Audio processing method, intelligent device and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, particularly provides an audio processing method, intelligent equipment and a storage medium, and aims to solve the problem of how to use audio to perceive an environment. The method provided by the invention comprises the steps of obtaining an environment audio of an environment where the intelligent equipment is located and a first problem text thereof, the first problem text being used for describing an environment perception problem, and the environment perception problem being a problem of perceiving the environment based on an acoustic phenomenon generated by the environment audio in the environment; the environment audio and the first question text are input into a voice large model for processing to obtain an answer text, the answer text is used for describing an answer of the question, and the answer is used for performing equipment control on the intelligent equipment; the large voice model comprises a large language model, the large voice model is obtained based on data sample training, the data sample comprises an audio sample of environment audio and annotation information of the audio sample, and the annotation information comprises a sample of a first question text and a sample of an answer text. Through the method, environment perception can be realized by using the environment audio.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an audio processing method, an intelligent device, and a storage medium. Background Art

[0002] Currently, some smart devices such as smart cars and robots are equipped with large language models (LLM). When users interact with smart devices through voice, the smart devices can recognize the voice to obtain voice text, and then input the voice text into the large language model for processing. The large language model can understand the user's intention based on the voice text and output response text. The response text is used to describe the response information to the voice, and the smart device is controlled according to the response information to meet the user's intention.

[0003] For example, if the smart device is a vehicle and the user's voice is to open the car window, the large language model can understand from the user's voice text that the user wants to open the car window and output a response text. The content of the response text can be to confirm to open the car window. Based on this response text, the vehicle can be controlled to open the window. However, in some cases, the voice may come from people outside the vehicle. If the car window is opened, it may cause safety risks. Currently, large language models are mainly used to process and generate text. Although some large language models have voice processing capabilities, they only recognize the semantics of speech and generate speech. They cannot simulate human auditory perception and perceive (or understand) the physical world through sound. Therefore, they cannot locate the source of the speech and cannot distinguish whether the speech comes from inside or outside the car. As long as the semantics of the speech is to open the car window, the vehicle will be controlled to open the window.

[0004] Accordingly, this field requires a new technical solution to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the following technical problem: how to use the audio generated in the physical world (or environment) to perceive (or understand) the physical world.

[0006] In a first aspect, an audio processing method is provided, wherein the method is applied to a smart device provided with an acoustic sensor, and the method comprises:

[0007] Acquiring ambient audio of the environment in which the smart device is located, collected by the acoustic sensor;

[0008] Obtaining a first question text corresponding to the ambient audio, where the first question text is used to describe an environmental perception problem, where the environmental perception problem is a problem of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment;

[0009] Inputting the environmental audio and the first question text into a large speech model for processing to obtain an answer text corresponding to the first question text, wherein the answer text is used to describe the answer to the environmental perception question, and the answer is used to control the smart device;

[0010] Among them, the large speech model includes a large language model, and the large speech model is trained based on data samples. The data samples include audio samples of environmental audio and their annotation information. The annotation information includes samples of the first question text and samples of the answer text.

[0011] In one technical solution of the above audio processing method, the smart device is further provided with a sound wave transmitting device, and the sound wave transmitting device is used to transmit a sound wave signal;

[0012] The problem type of the environmental perception problem includes distance measurement, and the distance measurement is measuring the distance between the target object in the environment and the smart device; when the problem type is the distance measurement, the environmental audio includes a first sound wave signal and a second sound wave signal, the first sound wave signal is a sound wave signal emitted to the target object by the sound wave emitting device, and the second sound wave signal is an echo signal formed by the first sound wave signal reflected by the target object.

[0013] In one technical solution of the above audio processing method, when the ambient audio is a voice control instruction of the smart device, the method further includes:

[0014] Obtaining a second question text corresponding to the ambient audio, where the second question text is used to instruct the smart device to execute an instruction operation of the voice control instruction;

[0015] Inputting the ambient audio and the second question text into the large speech model for processing to obtain an answer text corresponding to the second question text, the answer text being used to describe an environmental perception result and an execution decision of the instruction operation obtained based on the environmental perception result, the environmental perception result being a result of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment;

[0016] According to the execution decision, the smart device is controlled to execute the instruction operation.

[0017] In one technical solution of the above audio processing method, the large speech model further includes a text encoder and an audio encoder, and the audio encoder includes an audio phase extraction module and a speech recognition model;

[0018] The text encoder is configured to: encode the question text input into the large speech model to obtain text encoding information;

[0019] The audio phase extraction module is configured to: extract phase information of the ambient audio;

[0020] The speech recognition model is configured to: extract amplitude information of the ambient audio, and encode the phase information and the amplitude information to obtain audio encoding information;

[0021] The large language model is configured to process the text encoding information and the audio encoding information to obtain an answer text corresponding to the question text.

[0022] In one technical solution of the above audio processing method, the phase information includes a phase sine component and a phase cosine component, and the audio phase extraction module includes a sine component extraction unit and a cosine component extraction unit;

[0023] The sinusoidal component extraction unit is configured to: obtain a sine value of the phase angle of the ambient audio as the phase sinusoidal component;

[0024] The cosine component extraction unit is configured to obtain a cosine value of a phase angle of the ambient audio as the phase cosine component.

[0025] In one technical solution of the above audio processing method, the audio sample of the ambient audio is obtained by:

[0026] Using a channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end, wherein the first channel is represented in the form of a first channel impulse response;

[0027] performing convolution processing on the audio and the first channel, and using a result of the convolution processing as an audio sample of the ambient audio;

[0028] The channel simulator includes multiple components, each of which corresponds to a plurality of acoustic phenomena.

[0029] The component is configured to: simulate a second channel through which audio is transmitted from a transmitting end to a receiving end when a target acoustic phenomenon occurs, wherein the second channel is represented in the form of a second channel impulse response, and the target acoustic phenomenon is the acoustic phenomenon corresponding to the component;

[0030] The channel simulator is configured to obtain the first channel impulse response according to the second channel impulse response obtained by simulating all components.

[0031] In one technical solution of the above audio processing method, the multiple components include a LOS component, a reflection component, a reverberation component, a Doppler component, and an array component;

[0032] The acoustic phenomenon corresponding to the LOS component is the LOS path;

[0033] The acoustic phenomenon corresponding to the reflective component is early reflection;

[0034] The acoustic phenomenon corresponding to the reverberation component is reverberation;

[0035] The acoustic phenomenon corresponding to the Doppler component is the Doppler effect;

[0036] The acoustic phenomenon corresponding to the array assembly is the sensor array reception phenomenon;

[0037] The sensor array is a smart device equipped with a plurality of acoustic sensors, and the plurality of acoustic sensors are arranged in an array. The sensor array receiving phenomenon indicates that the ambient audio includes audio collected by each acoustic sensor in the sensor array from the same sound source.

[0038] In a technical solution of the above audio processing method, the second channel impulse response simulated by the LOS component is h1(τ), h1(τ)=A1δ(τ-τ los );

[0039] Where A1 represents the attenuation intensity of the LOS path, δ represents the impulse function, τ is the independent variable of the impulse function, τ represents the delay, τ los Indicates the propagation delay of the LOS path.

[0040] In one technical solution of the above audio processing method, the second channel impulse response simulated by the reflection component is h2(τ),

[0041] Wherein, A2 and A3 respectively represent the attenuation strength of the two reflection paths in the early reflection, and Respectively represent the propagation delays of the two reflection paths.

[0042] In one technical solution of the above audio processing method, the second channel impulse response simulated by the reverberation component is h3(τ), h3(τ)=R(τ);

[0043] in, s i (τ) represents the signal of the i-th frequency band in the preset Gaussian noise signal, and the Gaussian noise signal is decomposed into N band frequency band, e i(τ) represents the attenuation signal of the ith frequency band, e i (τ)=exp(-λ i τ), exp represents the exponential function, λ i Represents the attenuation rate of the i-th frequency band.

[0044] In one technical solution of the above audio processing method, the second channel impulse response simulated by the Doppler component is h4(τ),

[0045] Wherein, v represents the relative speed between the transmitting end and the receiving end, d0 represents the initial distance between the transmitting end and the receiving end, c represents the speed of sound, and t represents time.

[0046] In one technical solution of the above audio processing method, the second channel impulse response obtained by the array component is h5(τ),

[0047] Among them, h 5j (τ) represents the second channel impulse response when the receiving end is the j-th acoustic sensor in the sensor array, τ 51 represents the propagation delay of the transmission path when the audio is transmitted from the transmitter to the first acoustic sensor in the sensor array, and the transmission path at least includes the LOS path, τ θ(1j) represents the time difference between the audio transmission from the transmitter to the 1st and jth acoustic sensors respectively, j=1,…,n, and n is the total number of acoustic sensors in the sensor array.

[0048] In one technical solution of the above-mentioned audio processing method, the large speech model is trained based on multiple data sets, the multiple data sets corresponding one-to-one to problem types of various environmental perception problems, the data sets including multiple data samples, and the use of a channel simulator to simulate the first channel through which audio is transmitted from a transmitting end to a receiving end includes:

[0049] Obtaining a problem type corresponding to a data set to which the audio sample belongs, and determining a control state of each component in the channel simulator according to the problem type;

[0050] Based on the control states of the components, controlling the channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end;

[0051] Wherein, the control state includes target control, enabled, randomly enabled and disabled;

[0052] The target control is: enabling the component and adjusting the target parameter of the component;

[0053] The enabling includes: enabling the component and randomly adjusting the parameter values ​​of various parameters of the component;

[0054] The random activation is to randomly activate a component and randomly adjust the parameter values ​​of various parameters of the component when the component is activated.

[0055] In one technical solution of the above-mentioned audio processing method, the problem types include LOS path detection, Doppler effect estimation, multipath effect analysis, sound source direction estimation, and distance measurement, and determining the control state of each component in the channel simulator according to the problem type includes:

[0056] When the problem type is LOS path detection, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: target control, random enable, random enable, random enable and disable;

[0057] When the problem type is Doppler effect estimation, the control states of the LOS component, the reflection component, the reverberation component, the Doppler component and the array component are: random enable, random enable, random enable, target control and disable;

[0058] When the problem type is multipath effect analysis, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: random enable, random enable, target control, random enable and disable;

[0059] When the problem type is sound source direction estimation, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: enabled, randomly enabled, randomly enabled, randomly enabled and target controlled;

[0060] When the problem type is distance measurement, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: target control, random enable, random enable, random enable and disable respectively.

[0061] In a second aspect, a smart device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions provided in the first aspect is implemented.

[0062] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions provided in the first aspect.

[0063] Solution 1. An audio processing method, characterized in that the method is applied to a smart device, the smart device being provided with an acoustic sensor, the method comprising:

[0064] Acquiring ambient audio of the environment in which the smart device is located, collected by the acoustic sensor;

[0065] Obtaining a first question text corresponding to the ambient audio, where the first question text is used to describe an environmental perception problem, where the environmental perception problem is a problem of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment;

[0066] Inputting the environmental audio and the first question text into a large speech model for processing to obtain an answer text corresponding to the first question text, wherein the answer text is used to describe the answer to the environmental perception question, and the answer is used to control the smart device;

[0067] Among them, the large speech model includes a large language model, and the large speech model is trained based on data samples. The data samples include audio samples of environmental audio and their annotation information. The annotation information includes samples of the first question text and samples of the answer text.

[0068] Solution 2. The method according to Solution 1, characterized in that the smart device is further provided with a sound wave transmitting device, and the sound wave transmitting device is used to transmit a sound wave signal;

[0069] The problem type of the environmental perception problem includes distance measurement, and the distance measurement is measuring the distance between the target object in the environment and the smart device; when the problem type is the distance measurement, the environmental audio includes a first sound wave signal and a second sound wave signal, the first sound wave signal is a sound wave signal emitted to the target object by the sound wave emitting device, and the second sound wave signal is an echo signal formed by the first sound wave signal reflected by the target object.

[0070] Solution 3. The method according to Solution 1, characterized in that when the ambient audio is a voice control instruction of the smart device, the method further includes:

[0071] Obtaining a second question text corresponding to the ambient audio, where the second question text is used to instruct the smart device to execute an instruction operation of the voice control instruction;

[0072] Inputting the ambient audio and the second question text into the large speech model for processing to obtain an answer text corresponding to the second question text, the answer text being used to describe an environmental perception result and an execution decision of the instruction operation obtained based on the environmental perception result, the environmental perception result being a result of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment;

[0073] According to the execution decision, the smart device is controlled to execute the instruction operation.

[0074] Solution 4. The method according to any one of Solutions 1 to 3, wherein the large speech model further comprises a text encoder and an audio encoder, and the audio encoder comprises an audio phase extraction module and a speech recognition model;

[0075] The text encoder is configured to: encode the question text input into the large speech model to obtain text encoding information;

[0076] The audio phase extraction module is configured to: extract phase information of the ambient audio;

[0077] The speech recognition model is configured to: extract amplitude information of the ambient audio, and encode the phase information and the amplitude information to obtain audio encoding information;

[0078] The large language model is configured to process the text encoding information and the audio encoding information to obtain an answer text corresponding to the question text.

[0079] Solution 5. The method according to Solution 4, characterized in that the phase information includes a phase sine component and a phase cosine component, and the audio phase extraction module includes a sine component extraction unit and a cosine component extraction unit;

[0080] The sinusoidal component extraction unit is configured to: obtain a sine value of the phase angle of the ambient audio as the phase sinusoidal component;

[0081] The cosine component extraction unit is configured to obtain a cosine value of a phase angle of the ambient audio as the phase cosine component.

[0082] Solution 6. The method according to Solution 1, wherein the audio sample of the ambient audio is obtained by:

[0083] Using a channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end, wherein the first channel is represented in the form of a first channel impulse response;

[0084] performing convolution processing on the audio and the first channel, and using a result of the convolution processing as an audio sample of the ambient audio;

[0085] The channel simulator includes multiple components, each of which corresponds to a plurality of acoustic phenomena.

[0086] The component is configured to: simulate a second channel through which audio is transmitted from a transmitting end to a receiving end when a target acoustic phenomenon occurs, wherein the second channel is represented in the form of a second channel impulse response, and the target acoustic phenomenon is the acoustic phenomenon corresponding to the component;

[0087] The channel simulator is configured to obtain the first channel impulse response according to the second channel impulse response obtained by simulating all components.

[0088] Solution 7. The method according to solution 6, characterized in that the multiple components include a LOS component, a reflection component, a reverberation component, a Doppler component and an array component;

[0089] The acoustic phenomenon corresponding to the LOS component is the LOS path;

[0090] The acoustic phenomenon corresponding to the reflective component is early reflection;

[0091] The acoustic phenomenon corresponding to the reverberation component is reverberation;

[0092] The acoustic phenomenon corresponding to the Doppler component is the Doppler effect;

[0093] The acoustic phenomenon corresponding to the array assembly is the sensor array reception phenomenon;

[0094] The sensor array is a smart device equipped with a plurality of acoustic sensors, and the plurality of acoustic sensors are arranged in an array. The sensor array receiving phenomenon indicates that the ambient audio includes audio collected by each acoustic sensor in the sensor array from the same sound source.

[0095] Solution 8. The method according to Solution 7 is characterized in that the second channel impulse response simulated by the LOS component is h1(τ), h1(τ)=A1δ(τ-τ los );

[0096] Where A1 represents the attenuation intensity of the LOS path, δ represents the impulse function, τ is the independent variable of the impulse function, τ represents the delay, τ los Indicates the propagation delay of the LOS path.

[0097] Solution 9. The method according to Solution 8, characterized in that the second channel impulse response simulated by the reflection component is h2(τ),

[0098] Wherein, A2 and h3 represent the attenuation strength of the two reflection paths in the early reflection, respectively. and Respectively represent the propagation delays of the two reflection paths.

[0099] Solution 10. The method according to Solution 8, wherein the second channel impulse response simulated by the reverberation component is h3(τ), where h3(τ)=R(τ);

[0100] in, s i (τ) represents the signal of the i-th frequency band in the preset Gaussian noise signal, and the Gaussian noise signal is decomposed into N band frequency band, e i (τ) represents the attenuation signal of the ith frequency band, e i (τ)=exp(-λ i τ), exp represents the exponential function, λ i Represents the attenuation rate of the i-th frequency band.

[0101] Solution 11. The method according to Solution 8, wherein the second channel impulse response simulated by the Doppler component is h4(τ),

[0102] Wherein, v represents the relative speed between the transmitting end and the receiving end, d0 represents the initial distance between the transmitting end and the receiving end, c represents the speed of sound, and t represents time.

[0103] Solution 12. The method according to Solution 8, wherein the second channel impulse response obtained by the array component is h5(τ),

[0104] Among them, h 5j (τ) represents the second channel impulse response when the receiving end is the j-th acoustic sensor in the sensor array, τ 51 represents the propagation delay of the transmission path when the audio is transmitted from the transmitter to the first acoustic sensor in the sensor array, and the transmission path at least includes the LOS path, τ θ(1j) represents the time difference between the audio transmission from the transmitter to the 1st and jth acoustic sensors respectively, j=1,…,n, and n is the total number of acoustic sensors in the sensor array.

[0105] Solution 13. The method according to any one of Solutions 7 to 12, wherein the large speech model is trained based on multiple data sets, the multiple data sets corresponding one-to-one to problem types of various environmental perception problems, the data sets including multiple data samples, and the method of using a channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end comprises:

[0106] Obtaining a problem type corresponding to a data set to which the audio sample belongs, and determining a control state of each component in the channel simulator according to the problem type;

[0107] Based on the control states of the components, controlling the channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end;

[0108] Wherein, the control state includes target control, enabled, randomly enabled and disabled;

[0109] The target control is to enable the component and adjust the parameter value of the target parameter of the component; the enablement is to enable the component and randomly adjust the parameter value of each parameter of the component; the random enablement is to randomly enable the component and randomly adjust the parameter value of each parameter of the component when the component is enabled.

[0110] Solution 14. The method according to Solution 13, wherein the problem types include LOS path detection, Doppler effect estimation, multipath effect analysis, sound source direction estimation, and distance measurement, and determining the control state of each component in the channel simulator based on the problem type includes:

[0111] When the problem type is LOS path detection, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: target control, random enable, random enable, random enable and disable;

[0112] When the problem type is Doppler effect estimation, the control states of the LOS component, the reflection component, the reverberation component, the Doppler component and the array component are: random enable, random enable, random enable, target control and disable;

[0113] When the problem type is multipath effect analysis, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: random enable, random enable, target control, random enable and disable;

[0114] When the problem type is sound source direction estimation, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: enabled, randomly enabled, randomly enabled, randomly enabled and target controlled;

[0115] When the problem type is distance measurement, the control states of the LOS component, reflection component, reverberation component, Doppler component and array component are: target control, random enable, random enable, random enable and disable respectively.

[0116] Solution 15. A smart device, comprising:

[0117] at least one processor;

[0118] And, a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the audio processing method described in any one of Schemes 1 to 14 is implemented.

[0119] Solution 16. A computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the audio processing method according to any one of Solutions 1 to 14.

[0120] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0121] In a technical solution for implementing the audio processing method provided in this application, the method is applied to a smart device, which is provided with an acoustic sensor. The method includes: obtaining ambient audio of the environment in which the smart device is located, which is collected by the acoustic sensor; obtaining a first question text corresponding to the ambient audio, the first question text being used to describe an environmental perception problem, which is a problem of perceiving the environment based on the acoustic phenomena generated by the ambient audio in the environment; inputting the ambient audio and the first question text into a large speech model for processing to obtain an answer text corresponding to the first question text, the answer text being used to describe the answer to the environmental perception problem, the answer representing the result of environmental perception, and the answer being used to control the smart device. The large speech model includes a large language model, which is trained based on data samples, the data samples including audio samples of the ambient audio and their annotation information, the annotation information including examples of the first question text and examples of the answer text.

[0122] In the above implementation scheme, the trained speech model can perform environmental perception of the acoustic phenomena generated by the ambient audio in the environment, which is equivalent to giving the speech model an auditory perception ability similar to that of humans, and can understand the physical world through sound like humans. Based on this, the speech model is deployed on the smart device and the smart device executes the above method, which is equivalent to the smart device simulating the human auditory perception ability and using the ambient audio for environmental perception. For example, the ambient audio can be used to perceive the location and direction of the sound source, whether the environment is a closed space or an outdoor environment, and other information. In addition, the results of environmental perception can be used to control the smart device. Since the device control is obtained based on the result, the device control can be understood as a response (or interaction) to the environment based on the result of environmental perception, and the entire process from environmental perception to device control can be understood as the interaction between the smart device and the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily understand that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. in:

[0124] Figure 1 This is a flowchart of the main steps of an audio processing method according to an embodiment of the present application;

[0125] Figure 2 is a flowchart of the main steps of an audio processing method according to another embodiment of the present application;

[0126] Figure 3 is a schematic diagram of the main structure of a large speech model according to an embodiment of the present application;

[0127] Figure 4 This is a schematic diagram of a process in which a large speech model according to an embodiment of the present application processes ambient audio and question text to obtain an answer text;

[0128] Figure 5 is a schematic diagram of the main structure of an audio encoder according to an embodiment of the present application;

[0129] Figure 6 This is a flowchart of the main steps of the audio sample acquisition method according to an embodiment of the present application;

[0130] Figure 7 is a schematic diagram of two acoustic phenomena, LOS path and early reflection, existing in an environment according to an embodiment of the present application;

[0131] Figure 8 is Figure 7Schematic diagram of the second channel impulse response under the acoustic phenomenon shown;

[0132] Figure 9 is a schematic diagram of a channel impulse response in a real environment according to an embodiment of the present application;

[0133] Figure 10 yes Figure 9 A partial enlarged schematic diagram;

[0134] Figure 11 is a schematic diagram of a microphone array consisting of two microphones according to an embodiment of the present application;

[0135] Figure 12 Audio arrival Figure 11 A timeline diagram of two microphones in the microphone array shown

[0136] Figure 13 This is a flow chart of the main steps of transmitting analog audio from a transmitting end to a receiving end through a first channel according to an embodiment of the present application;

[0137] Figure 14 is a schematic diagram of a process for obtaining data samples according to an embodiment of the present application;

[0138] Figure 15 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0139] Reference numerals:

[0140] 11: Text encoder; 12: Audio encoder; 13: Large language model; 21: Memory; 22: Processor. DETAILED DESCRIPTION

[0141] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0142] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0143] The user personal information processed by this application will vary depending on the specific product / service scenario and must be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0144] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0145] The following describes an embodiment of the audio processing method provided by this application. The audio processing method provided by this application can be applied to smart devices, that is, the method can be performed by the smart device. Smart devices can include driving devices, smart cars, robots, smart home systems, etc. In some embodiments, the robot can be an onboard controller. The smart device is provided with an acoustic sensor that can collect audio. In some embodiments, the acoustic sensor can be a microphone.

[0146] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of the audio processing method according to an embodiment of the present application. Figure 1 As shown, the audio processing method in the embodiment of the present application mainly includes the following steps S101 to S103.

[0147] Step S101: Acquire ambient audio from the smart device's environment, as captured by an acoustic sensor. The range of the smart device's environment can be understood as the acoustic sensor's pickup range, and audio captured within this range is considered ambient audio. For example, if the smart device is a vehicle-mounted controller installed on the vehicle's center console, the acoustic sensor's pickup range is 5 meters, which covers the entire vehicle interior and also partially covers the exterior.

[0148] Step S102: Obtain a first question text corresponding to the ambient audio. The first question text is used to describe an environmental perception problem. The environmental perception problem is a problem of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment.

[0149] Acoustic phenomena can be understood as physical phenomena that occur when audio is transmitted from a transmitter (e.g., a sound source) to a receiver (e.g., an acoustic sensor). In embodiments of the present application, acoustic phenomena may include Line of Sight (LOS) paths, early reflections, reverberation, Doppler effects, and the like.

[0150] Environmental perception based on the acoustic phenomena generated by ambient audio in the environment can be based on the perception of the physical properties of the ambient audio in the environment. Physical properties can include the location, direction, and motion of the sound source (i.e., the source emitting the ambient audio), as well as the Doppler shift and multipath effects of the ambient audio.

[0151] The first question text is text information describing the question in natural language. For example, the first question text could be "Does the audio contain a LOS path?", "Can you evaluate the impact of multipath effects in a given audio clip?", "Please determine whether the sound source is approaching or moving away.", etc.

[0152] Step S103: Input the environmental audio and the first question text into the speech model for processing to obtain an answer text corresponding to the first question text. The answer text is used to describe the answer to the environmental perception question.

[0153] The answer text is text information that describes the answer in natural language. For example, for the question text "Does the audio contain the LOS path?", the answer text can be "Contains."

[0154] The answer described in the answer text represents the result of environmental perception, which can be used to control the smart device. For example: the smart device is an on-board controller in a vehicle. The on-board controller receives a voice control command to open the window. The environmental perception result can be whether there is a LOS path between the sound source and the on-board controller; if there is a LOS path, it indicates that the voice control command comes from inside the vehicle, and the window is opened; if there is no LOS path, it indicates that the voice control command comes from outside the vehicle, and in order to ensure vehicle safety, the window is not opened. For another example: the voice control command received by the on-board controller is a wake-up command. The environmental perception result includes the position of the sound source in the vehicle, and the on-board controller can be controlled to respond to the sound source in the direction of that position.

[0155] The large speech model includes a large language model (LLM). In this embodiment, a conventional large language model in the field of artificial intelligence technology can be used, and this embodiment does not specifically limit this. For example, an LLama model or a Qwen model can be used.

[0156] The speech large model is obtained through training based on data samples, and the data samples include audio samples of environmental audio and their annotation information, and the annotation information includes a sample of the first question text and a sample of the answer text. In this embodiment, a conventional training method can be used to supervise the speech large model using audio samples and their annotation information, and this embodiment does not specifically limit this. For example, the sample of the question text in the audio sample and its annotation information is input into the speech large model for processing to obtain the answer, and then the model parameters of the speech large model are optimized based on the difference between the sample of the answer text in the annotation information and the answer output by the speech large model. In some embodiments, when the speech large model is supervised, the next-token prediction loss can be used for supervised training. The goal of supervised training based on this loss is to maximize the probability of the next token predicted by the model so that the model can accurately predict the next token. The token refers to the token in the answer text output by the model, and the answer text consists of a sequence of tokens. The next token prediction loss is a conventional method used in large language model training, and its principle will not be elaborated here.

[0157] In the embodiment of the method described in the above steps S101 to S103, the trained large speech model is able to perform environmental perception of the acoustic phenomena generated by the ambient audio in the environment, which is equivalent to giving the large speech model auditory perception capabilities similar to those of humans, and being able to understand the physical world through sound like humans. Based on this, deploying the large speech model on a smart device and having the smart device execute the above method is equivalent to having the smart device simulate the auditory perception capabilities of humans and use the ambient audio for environmental perception. In addition, the results of environmental perception can be used to control the smart device. Since the device control is based on the environmental perception results, the device control can be understood as the response of the smart device to the environment based on the results, and the entire process from environmental perception to device control can be understood as the interaction between the smart device and the environment.

[0158] The following continues to describe the embodiments of the audio processing method provided by this application.

[0159] In some embodiments of the present application, the smart device is further provided with a sound wave transmitting device for transmitting a sound wave signal. Specifically, the smart device can control the sound wave transmitting device to transmit the sound wave signal toward a target object in the environment. In this embodiment, a conventional sound wave transmitting device can be used, and this embodiment is not specifically limited to this.

[0160] The problem type of environmental perception problems can include distance measurement, which is measuring the distance between the target object and the smart device in the environment. When the problem type is distance measurement, the environmental audio collected by the acoustic sensor in the aforementioned step S101 includes a first sound wave signal and a second sound wave signal. The first sound wave signal is a sound wave signal emitted to the target object by the sound wave emitting device, and the second sound wave signal is an echo signal formed by the first sound wave signal passing through the target object. In other words, in the distance measurement task, the smart device will actively emit a sound wave signal to the target object, and the sound source is equivalent to the smart device itself. At this time, it can be understood that the smart device is actively perceiving the environment.

[0161] Environmental perception problems can also include LOS path detection, Doppler effect estimation, multipath effect analysis, and sound source direction estimation. LOS path detection is to detect whether there is a LOS path. In these tasks, the sound source is an object in the environment, not the smart device itself. In this case, it can be understood that the smart device is passively perceiving the environment.

[0162] The following continues to describe the embodiments of the audio processing method provided by this application.

[0163] In some embodiments of the present application, ambient audio may be a voice control command of a smart device. A voice control command is used to instruct the smart device to perform an operation, and this operation can be understood as a command operation of the voice control command. For example, if the smart device is an in-vehicle controller and the semantic content of the ambient audio is "open the window," the ambient audio is a voice control command of the smart device, and the command operation is to open the window.

[0164] See attached Figure 2 In the case where the ambient audio is a voice control instruction, the audio processing method provided by this application may also include Figure 2 The following steps S201 to S203 are shown.

[0165] Step S201: Obtain a second question text corresponding to the ambient audio, where the second question text is used to instruct the smart device to execute a command operation of the voice control command.

[0166] For example, the semantic content of the ambient audio is "open the car window", and the second question text may be "follow the voice instructions in the input audio.".

[0167] Step S202: Input the environmental audio and the second question text into the speech model for processing to obtain the answer text corresponding to the second question text. The answer text is used to describe the environmental perception results and the execution decision of the instruction operation obtained based on the environmental perception results.

[0168] The environmental perception result is the result obtained by perceiving the environment based on the acoustic phenomena generated by the environmental audio in the environment. According to the method embodiment described in the aforementioned steps S101 to S103, the speech model has the ability to use environmental audio for environmental perception. This embodiment reuses the environmental perception ability of the speech model. In this embodiment, even if the first question text for describing the environmental perception problem is not additionally input to the speech model, when the input text is the second question text, the speech model can determine which physical properties of the environmental audio in the environment need to be perceived based on the specific content of the instruction operation represented by the environmental audio when processing the environmental audio, and obtain the perception result by processing the environmental audio. The answer text output by the speech model will describe the result.

[0169] Execution decisions can be understood as the strategy a smart device uses when executing a command. Commands originate from a sound source, and the smart device's execution of these commands can be understood as responding to that sound source. Environmental perception results reflect the sound source's physical properties within the environment (such as its position, direction, and state of motion). Therefore, determining execution decisions based on these environmental perception results can be understood as adopting different strategies to respond to the sound source based on its physical properties.

[0170] For example, if the smart device is an onboard controller in a vehicle and the ambient audio indicates a command to open a window, the environmental perception result could be whether there is a Loss of Sight (LOS) path between the sound source and the onboard controller. If there is a LOS path, the voice control command originates from inside the vehicle, and the decision is to open the window. If there is no LOS path, the voice control command originates from outside the vehicle, and to ensure vehicle safety, the decision is not to open the window.

[0171] For example, if the smart device is still an in-vehicle controller, the command operation represented by the ambient audio is to wake up the in-vehicle controller. The environmental perception result may be the location of the sound source in the vehicle. The execution decision may direct the in-vehicle controller to respond to the sound source in response to the location.

[0172] Step S203: Control the smart device to execute the instruction operation according to the execution decision.

[0173] Based on the method described in steps S201 to S203 above, after receiving the voice control command, the smart device can use the command to perceive the environment, determine the execution decision based on the perception result, and execute the command operation based on the decision, thereby realizing perception and interaction with the environment.

[0174] The following continues to describe the embodiment of the audio processing method provided in this application, specifically the structure of the large speech model.

[0175] See attached Figure 3 In some embodiments, the speech large model may include a text encoder 11, an audio encoder 12 and a large language model 13, which are described below.

[0176] 1. The text encoder 11 is described.

[0177] The text encoder 11 can be configured to encode the question text input into the large speech model to obtain text encoding information. The question text can be the first question text or the second question text in the aforementioned embodiment. In this embodiment, a conventional text encoding method can be used to encode the question text, which is not specifically limited in this embodiment.

[0178] 2. The audio encoder 12 is described.

[0179] The audio encoder may include an audio phase extraction module and a speech recognition model.

[0180] The audio phase extraction module can be configured to extract the phase information of the ambient audio. Specifically, the spectrum of the ambient audio can be obtained, and the phase information at multiple different frequencies can be obtained based on the spectrum. In some embodiments, the ambient audio can be converted into a spectrum by short-time Fourier transform (STFT). For example, the sampling rate of the ambient audio is 16 kHz, the window length in the configuration information of the short-time Fourier transform is 254, and the frame shift is 10 ms. Since the audio is a real number, the output of the short-time Fourier transform is conjugate symmetric. Therefore, only the first 254 / 2+1=128 unique frequency components are retained within a window length of the spectrum. In the prior art, when processing the spectrum, the speech recognition model usually converts the spectrum obtained by the short-time Fourier transform into a Mel spectrogram for processing. The Mel spectrogram emphasizes the human speech frequency segment. In this embodiment, the audio phase extraction module and the speech recognition model directly process the spectrum obtained by the short-time Fourier transform, without obtaining the Mel spectrum diagram. This is conducive to fine-grained analysis of the audio in the physical world and capturing fine audio features (or physical features of the physical world).

[0181] The speech recognition model can adopt a conventional speech recognition model in the field of artificial intelligence technology. The speech recognition model can extract the amplitude (Magnitude) information of the audio and encode the amplitude information to obtain audio encoding information. When extracting the amplitude information, the amplitude information at multiple different frequencies can also be obtained based on the spectrum of the ambient audio. The amplitude information represents the energy intensity of the audio. For example, the speech recognition model can be a Transformer-based speech recognition model. In this embodiment, the amplitude information can be a log-magnitude spectrum.

[0182] In this embodiment, the aforementioned capabilities of extracting audio amplitude and encoding amplitude information are reused. In this embodiment, the speech recognition model can be configured to extract the amplitude information of the ambient audio and encode the phase information and amplitude information to obtain audio encoding information. Specifically, the phase information and amplitude information are stacked or concatenated together for encoding.

[0183] In some embodiments, preset markers are added to the left and right sides of the audio coding information. The preset markers are used to mark the boundaries of the audio coding information. The audio coding information and the preset markers on both sides together constitute the final audio coding information. For example, the audio coding information is represented as Audio Tokens, and the preset marker added on the left is <soa>, the preset mark added on the right is <eoa>, the final audio coding information is expressed as <soa>Audio Tokens <eoa> 。 <soa>Indicates the start boundary of audio encoding information Audio Tokens (start of audio), <eoa>Indicates the end boundary of audio coding information AudioTokens (end of audio).

[0184] This embodiment simultaneously encodes both phase and amplitude information, extracting fine-grained features that go far beyond the sound content itself. This allows for more effective capture of the characteristics of the sound source and acoustic channel (the channel through which audio travels from the transmitter to the receiver), thereby facilitating accurate answers to environmental perception questions. This is particularly true for problems involving complex acoustic phenomena, such as Doppler effect estimation, multipath analysis, and sound source direction estimation.

[0185] 3. Explain the large language model 13.

[0186] The large language model 13 can be configured to process the text encoding information and the audio encoding information to obtain an answer text corresponding to the question text. Specifically, the text encoding information and the audio encoding information can be concatenated to form new encoding information, which is then input into the large language model 13 for processing to obtain the answer text. The text encoding information and the audio encoding information are represented in the form of vectors, and the vector dimensions of the two vectors are the same.

[0187] See attached Figure 4 , Figure 4 The following example illustrates the process in which a large speech model processes the ambient audio and question text to obtain the answer text in some embodiments. Figure 4 As shown, the semantics of the audio is "Open the window," and the question text can be "Follow the voice instructions in the input audio." After the audio and question text are input into the large speech model, the audio encoder encodes the audio to obtain audio encoding information, and the text encoder encodes the question text to obtain text encoding information. The audio and text encoding information are concatenated and input into the large language model. The large language model processes both and obtains the answer text, which is "The LOS path was not detected, and the sound source was blocked. Since the command comes from outside the vehicle, execution is rejected. Please issue the command from inside the vehicle."

[0188] The training method of the above-mentioned large speech model is described below.

[0189] When training the large speech model of the above structure, for the large language model, a pre-trained large language model can be first obtained as a base model, and then this base model is fine-tuned to obtain the final large language model 13, so as to reduce the training cost and give full play to the language understanding advantage of the large language model. In this embodiment, a conventional model fine-tuning method can be used, for example, LoRA (Low Rank Adaptation) technology can be used for model fine-tuning.

[0190] As for the speech recognition model in the audio encoder 12, since the speech recognition model adopts a conventional speech recognition model in the field of artificial intelligence technology, the model parameters of this conventional speech recognition model can be obtained before training, and then the model parameters of the speech recognition model in the audio encoder 12 are initialized to these model parameters, and training is performed after the initialization is completed.

[0191] The audio phase extraction modules in the text encoder 11 and the audio encoder 12 can be trained from scratch, especially the audio phase extraction module can be trained from scratch, which is beneficial for the audio phase extraction module to learn key audio phase features.

[0192] In addition, when training the large speech model of the above structure, the text encoder 11, the audio encoder 12 and the large language model 13 are trained in a joint training manner.

[0193] The following is an explanation of the large speech model.

[0194] In some embodiments of the present application, the audio phase extraction module may include a sine component extraction unit and a cosine component extraction unit. The sine component extraction unit may be configured to obtain the sine value of the phase angle of the ambient audio as the phase sine component, and the cosine component extraction unit may be configured to obtain the cosine value of the phase angle of the ambient audio as the phase cosine component. The phase sine component and the phase cosine component are combined to provide the phase information of the ambient audio.

[0195] The phase angle of ambient audio may jump between -π and π (for example, from -π to π, or from π to -π), resulting in numerical instability. Obtaining the phase sine and cosine components can avoid this. When the phase angles are -π and π, the sine value is 0, and when the phase angles are -π and π, the cosine value is -1. This numerical stability makes it easier to capture audio phase changes and helps the audio phase extraction module learn key audio phase features.

[0196] See attached Figure 5 , Figure 5 The main structure of the audio encoder in some embodiments of the present application is exemplified. Figure 5 As shown in the figure, the sine component extraction unit and the cosine component extraction unit form the audio phase extraction module, and the remaining structure is the structure of the speech recognition model. The number of channels of the audio amplitude, phase sine component, and phase cosine component is 128. These three features are processed separately through 3×3 one-dimensional convolution to expand the number of channels from 128 to 1280. They are then processed separately using activation functions, and the three processing results are stacked into a stacked feature with 3840 channels. After obtaining the stacked feature, it is convolved to reduce the number of channels to 1280. Then, sine position encoding is added and the features with this encoding are input into a 32-layer Transformer structure for processing, ultimately obtaining the audio encoding information. The activation function can be GELU (Gaussian Error Linear Unit).

[0197] The following continues to describe an embodiment of the audio processing method provided in this application, specifically a method for obtaining audio samples used in model training.

[0198] Model training requires the use of a large number of audio samples. If acoustic sensors are used to collect audio in a real physical environment, it is necessary to widely deploy acoustic sensors in different environments, which is costly. In addition, the acoustic phenomena (such as Doppler effect, multipath reflection, etc.) generated by audio in the environment cannot be recognized by the human eye alone, and often require the use of complex measuring instruments for measurement, which makes it very difficult to obtain the annotation information of audio samples. In order to solve these problems, the present application provides a method for obtaining audio samples. The method mainly decomposes the audio received by the receiving end (such as an acoustic sensor or a human ear) into two independent parts, one is the original audio emitted by the transmitting end (i.e., the sound source), and the other is the first channel through which the original audio is transmitted from the transmitting end to the receiving end, and then convolution processing is performed on these two parts. The result of the convolution processing represents the audio received by the receiving end after the original audio is transmitted to the receiving end through the first channel. Through the above method, it is only necessary to combine the existing audio (i.e., the original audio emitted by the sound source) with the channel simulated by the channel simulator to synthesize environmental audio with a high degree of simulation. Specifically, the audio sample acquisition method provided by the present application may include Figure 6 The following steps S301 to S302 are shown.

[0199] Step S301: using a channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end, wherein the first channel is represented in the form of a first channel impulse response (CIR).

[0200] The channel simulator consists of multiple components, each corresponding to a variety of acoustic phenomena. Each component can be configured to simulate the second channel through which audio is transmitted from the transmitter to the receiver when a target acoustic phenomenon occurs. The second channel is represented by a second channel impulse response (CIR), and the target acoustic phenomenon is the acoustic phenomenon corresponding to the component. In other words, each component within the channel simulator is independent of each other, and each component is responsible for simulating the channel when a specific acoustic phenomenon occurs.

[0201] The channel simulator may be configured to obtain a first channel impulse response based on the second channel impulse responses simulated by all components. In other words, the first channel impulse response integrates the channels simulated by all components.

[0202] Step S302: Convolve the audio with the first channel, and use the result of the convolution as an audio sample of the ambient audio. The audio sample can be represented as y, y=h*s, where h represents the first channel, s represents the audio, and * represents the convolution operation.

[0203] Based on the method described in steps S301 to S302 above, large-scale audio samples with rich varieties and convenient information annotation can be constructed efficiently and reliably.

[0204] In some embodiments according to the present application, the multiple components in the channel simulator include LOS components, reflection components, reverberation components, Doppler components and array components. The correspondence between these components and acoustic phenomena is shown in Table 1 below. Table 1 Components Acoustic phenomena corresponding to components LOS components LOS (Line of Sight) path Reflection Component Early Reflections Reverb Component Reverberation Doppler components Doppler Effects Array components Sensor array reception phenomenon

[0205] The sensor array is a smart device equipped with multiple acoustic sensors arranged in an array. The sensor array reception phenomenon indicates that the ambient audio includes the audio collected by each acoustic sensor in the sensor array from the same sound source.

[0206] The second channel impulse responses simulated by the above components are described below.

[0207] 1. Explain the LOS component and the reflector component.

[0208] See attached Figure 7 and attached Figure 8 , Figure 7 Schematic diagram showing the simultaneous occurrence of LOS path and early reflection in some embodiments. Figure 8 An example is shown in Figure 7 The second channel impulse response is displayed under the acoustic phenomenon shown in FIG. Figure 7 As shown in the figure, there is a direct path (LOS path) between the sound source and the microphone, and two reflection paths in the early reflection. One reflection path ECHO1 starts from the sound source and reflects from the upper wall to the microphone. The other reflection path ECHO2 starts from the sound source and reflects from the lower wall to the microphone. Figure 8 As shown, the second channel impulse response h(τ) is composed of the corresponding impulse responses of the three paths: the LOS path, the reflection path ECHO1, and the reflection path ECHO2. Figure 8 The amplitude in represents the attenuation strength of the transmission path.

[0209] The LOS component is used to simulate the second channel through which the audio is transmitted from the transmitter to the receiver in the event of a LOS path. The impulse response of the second channel h1(τ) is shown in the following equation (1):

[0210] h1(τ)=A1δ(τ-τ los ) (1)

[0211] Among them, A1 represents the attenuation intensity of the LOS path, δ represents the impulse function, τ is the independent variable of the impulse function, τ represents the delay, τ los Indicates the propagation delay of the LOS path.

[0212] The reflection component is used to simulate the second channel through which the audio is transmitted from the transmitter to the receiver in the case of early reflection. The impulse response of the second channel h2(τ) is shown in the following equation (2):

[0213]

[0214] Among them, A2 and A3 represent the attenuation intensity of the two reflection paths in the early reflection, and Represent the propagation delays of the two reflection paths respectively. It can be understood as the impulse response corresponding to a reflection path. It can be understood as the impulse response corresponding to another reflection path.

[0215] When the LOS path and early reflection occur simultaneously, the second channel impulse response is h(τ)=h1(τ)+h2(τ).

[0216] 2. Explain the reverb component.

[0217] See attached Figure 9 and attached Figure 10 , Figure 9 Schematic diagram of channel impulse response in a real environment in some embodiments is exemplified. Figure 10 yes Figure 9 A partial enlarged schematic diagram of the Figure 9 and Figure 10 In the figure, the initial peak is the LOS path, the following several small peaks are early reflections, and the diffuse tail further back represents reverberation. Reverberation describes the delayed reverberation effect caused by the continuous interaction between sound and the environment, that is, the tail sound caused by repeated reflections of sound in space.

[0218] The reverberation component is used to simulate the second channel that the audio passes through when it is transmitted from the transmitter to the receiver in the case of reverberation. The impulse response of the second channel h3(τ) is shown in the following equation (3):

[0219] h3(τ)=R(τ) (3)

[0220] Where R(τ) represents the reverberation function, s i (τ) represents the signal of the i-th frequency band in the preset Gaussian noise signal. The Gaussian noise signal is used to simulate random acoustic reflections. The Gaussian noise signal is decomposed into N band frequency bands. i (τ) represents the attenuation signal of the i-th frequency band, which is used to simulate the natural attenuation characteristics of sound in the environment. i (τ)=exp(-λ i τ), exp represents the exponential function, λ i represents the decay rate of the i-th frequency band. By controlling the decay rate in the reverberation function R(τ), we can simulate a variety of complex environments, from dry spaces (i.e., environments with minimal reverberation, such as a recording studio) to humid spaces (i.e., environments with strong echoes, such as an empty hall).

[0221] When the LOS path, early reflection, and reverberation occur simultaneously, the second channel impulse response is h(τ)=h1(τ)+h2(τ)+R(τ).

[0222] 3. Explain the Doppler component.

[0223] The Doppler effect refers to the frequency change caused by the relative motion between the transmitter and receiver. In this embodiment, a time delay that varies with time can be introduced into the channel impulse response to simulate the second channel through which audio is transmitted from the transmitter to the receiver when the Doppler effect occurs. The second channel impulse response h4(τ) is shown in the following equation (4):

[0224]

[0225] Where τ(t) represents the time delay that varies with time, b represents the relative speed between the transmitter and the receiver, d0 represents the initial distance between the transmitter and the receiver, c represents the speed of sound, and t represents time.

[0226] When LOS path, early reflection, reverberation and Doppler effect occur simultaneously, the second channel impulse response That is, the variable τ in h1(τ), h2(τ), and R(τ) is adjusted to

[0227] 4. Describe the array components.

[0228] The array component is used to simulate the second channel through which the audio is transmitted from the transmitter to the receiver when the sensor array receives the signal. The impulse response of the second channel h5(τ) is shown in the following equation (5):

[0229]

[0230] Among them, h 5j (τ) represents the second channel impulse response when the receiving end is the jth acoustic sensor in the sensor array, τ 51 represents the propagation delay of the transmission path when the audio is transmitted from the transmitter to the first acoustic sensor in the sensor array, τ θ(1j) represents the time difference between the audio transmission from the transmitter to the first and jth acoustic sensors, where j = 1, …, n, where n is the total number of acoustic sensors in the sensor array. Loss of sight (LOS) paths are inevitable during audio transmission; therefore, the aforementioned transmission paths include at least the LOS path.

[0231] See attached Figure 11 and attached Figure 12 , Figure 11 The embodiment exemplifies a microphone array composed of two microphones, which collect audio from the same sound source. Figure 12 A timeline diagram of the audio arriving at the two microphones.

[0232] like Figure 11 As shown in the figure, the distance between the two microphones is d. When the sound source emits sound from the target direction, a path difference Δd is generated in the target direction when the sound is transmitted along the target direction to the two microphones. The angle between the target direction and the line connecting the two microphones is θ (i.e., the direction angle of the sound source). Based on the path difference Δd, the time difference of arrival (TDoA) shown in the following formula (6) can be obtained.

[0233]

[0234] Where c represents the speed of sound, and dcos(θ) represents the projection of the path difference Δd onto the line between the two microphones.

[0235] like Figure 12 As shown, τ L and τ R Denote the delay time (i.e., delay τ) of the sound reaching the left microphone and the right microphone, respectively. Under simplified conditions without considering attenuation, the channel impulse responses of the two microphones are shown in the following equation (7):

[0236]

[0237] Among them, h L (τ) represents the channel impulse response of the left microphone, τ LOS h represents the propagation delay of the LOS path generated when the audio is transmitted from the sound source to the left microphone, R (τ) represents the channel impulse response of the right microphone, τ θ It is known from the above description that when the sound source direction angle θ is known, the sound source direction angle θ and the channel impulse response h of the left microphone can be used to calculate the time difference shown in formula (6). L (τ), derive the channel impulse response h of the right microphone R (τ), so that the channel impulse responses of both microphones can be represented by the channel impulse response of the same microphone (i.e., the left microphone). Based on this, when there are three or more microphones in the microphone array, the channel impulse responses of all microphones can still be represented by the channel impulse response of the same microphone, which is equivalent to expanding from a single channel impulse response (or single channel) to a multi-channel impulse response (or multi-channel).

[0238] The method of simulating channel impulse response in the prior art is mainly to model the environment and then simulate based on the environment model. When the environment is more complex, the modeling difficulty will be higher, and the simulation results are only applicable to the specific environment model obtained by modeling, and are not easy to expand to different environments. In this application, the combination of acoustic phenomena such as LOS path, early reflection, and reverberation can actually reflect the development process or life cycle of audio transmission. Therefore, this application constructs a channel simulator based on the above-mentioned LOS components, reflection components, reverberation components, Doppler components and array components. The channel simulator is used to simulate the channel passed through during audio transmission. It can be understood as a simulation from the perspective of the development process or life cycle of audio. There is no need to simulate the channel by modeling the environment, and the above-mentioned problem does not exist.

[0239] The following continues to describe the method for obtaining audio samples, specifically describing the method for obtaining the first channel in step S301 .

[0240] In some embodiments of the present application, the speech big model is trained based on multiple data sets, and the multiple data sets correspond one-to-one to the problem types (or tasks) of various environmental perception problems. The data sets include multiple data samples, so that the trained speech big model can understand various environmental perception problems.

[0241] Problem types can include LOS path detection, Doppler effect estimation, multipath effect analysis, sound source direction (DoA) estimation and distance measurement. LOS path detection mainly detects whether there is a LOS path, Doppler effect estimation mainly analyzes the Doppler effect, multipath effect analysis mainly analyzes the multipath effect, sound source direction estimation mainly determines the direction of the sound source, and distance measurement mainly measures the distance between the object and the sound source. Among them, sound source direction estimation is achieved by measuring the arrival time difference of the sound between multiple acoustic sensors (such as the time difference τ shown in the above formula (6) θ ). Since there is a spatial separation between acoustic sensors, sound waves arrive at each acoustic sensor at different times, and these time differences can be used to infer the direction of the sound source.

[0242] like Figure 13 As shown, when the analog audio is transmitted from the transmitting end to the receiving end through the first channel, simulation can be performed through the following steps S3011 to S3012.

[0243] Step S3011: Obtain the problem type corresponding to the data set to which the audio sample belongs, and determine the control state of each component in the channel simulator according to the problem type.

[0244] Control states include target control, enabled, random enabled, and disabled.

[0245] Target control is to enable the component and adjust the target parameter value of the component. This adjustment can be understood as adjusting the target parameter value according to the preset adjustment rules, that is, precise adjustment. Taking the reverberation component as an example, the target parameter is the decay rate λ i , the adjustment rule can be to gradually increase the attenuation rate λ according to the preset interval value i The purpose of precise adjustment is to generate different parameter values ​​so that the component simulation can obtain different second channels.

[0246] Enable: Enables the component and randomly adjusts the values ​​of its parameters. The difference between Enable and Target Control is that Target Control precisely adjusts the values ​​of the target parameters, while Enable randomly adjusts the values ​​of all parameters to improve channel diversity.

[0247] Random activation randomly activates components and randomly adjusts their parameters when they are activated. The difference between activation and random activation is that activation means a component is always enabled, while random activation means a component may be enabled or disabled at random. The purpose of random activation is also to improve channel diversity.

[0248] It should be noted that the random adjustment of parameter values ​​needs to be performed within the numerical range of each parameter and cannot exceed the upper and lower limits of each parameter.

[0249] Step S3012: Based on the control status of each component, the channel simulator is controlled to simulate the first channel through which the audio is transmitted from the transmitting end to the receiving end.

[0250] Different types of environmental perception problems will have different requirements for different acoustic phenomena in the simulated channel. Based on the method described in steps S3011 to S3012 above, the channel simulator can be dynamically adjusted according to the problem type (or task), so that the first channel obtained by the channel simulator can better meet the requirements of the problem type (or task) for acoustic phenomena.

[0251] In some implementations of step S3011, when determining the control state of each component in the channel simulator according to the problem type, it can be determined based on the correspondence between the preset problem type and the component control state, which can be shown in Table 2 below. Table 2 LOS components Reflection Component Reverb Component Doppler components Array components LOS path detection Target Control Randomly enabled Randomly enabled Randomly enabled Disable Doppler effect estimation Randomly enabled Randomly enabled Randomly enabled Target Control Disable Sound source direction estimation Enable Randomly enabled Randomly enabled Randomly enabled Target Control Multipath effect analysis Randomly enabled Randomly enabled Target Control Randomly enabled Randomly enabled Distance measurement Target Control Randomly enabled Randomly enabled Randomly enabled Disable

[0252] For the problem type of LOS path detection, when performing target control on the LOS component, the target parameters of the LOS component may include attenuation intensity A1, propagation delay τ los , adjusting the attenuation strength A1 can control whether there is a LOS path in the channel and adjust the propagation delay τ los Different LOS paths can exist in the control channel.

[0253] For the problem type of Doppler effect estimation, when performing target control on the Doppler component, the target parameter of the Doppler component may include the sound speed c. Adjusting the sound speed c can simulate different frequency shifts.

[0254] For the problem type of sound source direction estimation, when performing target control on the array component, the target parameters of the array component can include the distance between the sound source and the acoustic sensor. Adjusting this distance can control the time difference τ θ(1j) .

[0255] For the problem type of multipath effect analysis, when performing target control on the reverberation component, the target parameters of the reverberation component can include the decay rate λ i , adjust the attenuation rate λ i Can simulate different degrees of reflective environment.

[0256] For the problem type of distance measurement, when performing target control on the LOS component, the target parameters of the LOS component can include the propagation delay τ los , adjust the propagation delay τ los Different propagation distances can be simulated.

[0257] The following continues to describe an embodiment of the audio processing method provided in this application, specifically a method for obtaining annotation information of an audio sample.

[0258] In the embodiment of the present application, the data sample can be represented as a triplet <audio, question, answer>, where audio is an audio sample, the question and answer form a question-answer pair, which is also the annotation information of the audio, the question is an example of the first question text, and the answer is an example of the answer text. When the model is supervised for training, the audio and question in the triplet are used as input, and the answer is used as supervision information. Figure 14 As shown in the figure, when obtaining data samples, a channel simulator is used to simulate the channel, and then the channel is convolved with the audio of the sound source to obtain an audio sample. Finally, the audio sample and the question-answer pair are represented in the form of a triplet <audio, question, answer>.

[0259] In some implementations, the types of question-answer pairs may include closed question-answer pairs and open question-answer pairs, and these two types of question-answer pairs are described below.

[0260] 1. Explain the closed-ended question and answer pairs.

[0261] Closed-ended question-answer pairs focus on classification or quantitative analysis and have clear answers. They can include binary classification, multi-classification, and numerical estimation.

[0262] For a binary question-answer pair, for example, the question is “Does the audio contain a LOS path?”, the answer is one of yes and no.

[0263] For a multi-classification question-answer pair, for example, the question is "how severe is the multipath effect", and the answer is one of strong, medium, and weak.

[0264] For example, if the question is "What is the time delay?", the answer is the numerical value of the time delay; or if the question is "What is the Doppler shift?", the answer is the percentage of the Doppler shift.

[0265] Closed question-answer pairs can be generated using fixed question-answer pair templates.

[0266] 2. Explain the open-ended question and answer pairs.

[0267] Open-ended question-answer pairs ask questions in natural language, requiring the speech large model to give explanatory answers that demonstrate its understanding of physical phenomena. For example, in the problem type (or task) of distance measurement, the answer needs to include step-by-step calculations and a physical explanation of the time of flight (ToF). When obtaining open-ended question-answer pairs, a specific prompt can be input into the large language model, which is generated by the large language model based on the prompt. The large language model is not the large language model of the speech large model in this application, but a conventional large language model in the field of artificial intelligence technology that already has language understanding and generation capabilities. For example, the large language model can be a GPT-4 model.

[0268] See Table 3, which exemplifies closed question-answer pairs and open question-answer pairs. Table 3

[0269] In some embodiments of the present application, the audio dataset used when acquiring audio samples contains 2 million 10-second audio segments, covering more than 500 label categories (such as device audio, animal audio, singing audio, etc.). The dataset for each task includes 200,000 closed question-answer pairs and 10,000 open question-answer pairs. These datasets are used to train the speech model. The batch size (Batch Size) during training is 32, the number of training rounds (epoch) is 5, and the total training time is about 61 hours. The trained speech model is tested, and the test results are as follows:

[0270] The accuracy of the LOS path detection task is 0.924.

[0271] The mean absolute error (MAE) of the Doppler effect estimation task is 0.181.

[0272] The mean absolute error for the sound source direction estimation task is 0.907.

[0273] The accuracy of the multipath effect analysis task is 0.903.

[0274] The relative error of the distance measurement task is 1.599%.

[0275] The above test results show that the large speech model can accurately perceive the environment using audio.

[0276] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application, and therefore will also fall within the scope of protection of this application.

[0277] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0278] Another aspect of the present application provides a computer-readable storage medium.

[0279] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium can be configured to store a program for executing the audio processing method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned audio processing method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0280] Another aspect of the present application provides a smart device.

[0281] In an embodiment of a smart device according to the present application, the smart device may include at least one processor; and a memory in communication with the at least one processor; wherein the memory stores a computer program, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. Figure 15 , Figure 15 2 shows an exemplary embodiment of the memory 21 and the processor 22 being connected via a bus. In embodiments of the present application, smart devices may include driving devices, smart cars, robots, smart home systems, etc. In some embodiments, the robot may be an in-vehicle controller. The smart device is provided with an acoustic sensor capable of collecting audio. In some embodiments, the acoustic sensor may be a microphone.

[0282] In the description of this application, "processor" may include hardware, software, or a combination of both. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Computer-readable storage media include any suitable medium capable of storing program code, such as a magnetic disk, a hard disk, an optical disk, flash memory, read-only memory, random access memory, and the like.

[0283] Thus far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.< / eoa> < / soa> < / eoa> < / soa> < / eoa> < / soa>

Claims

1. An audio processing method, characterized in that: The method is applied to a smart device provided with an acoustic sensor, and the method includes: Acquiring ambient audio of the environment in which the smart device is located, collected by the acoustic sensor; Obtaining a first question text corresponding to the ambient audio, where the first question text is used to describe an environmental perception problem, where the environmental perception problem is a problem of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment; Inputting the environmental audio and the first question text into a large speech model for processing to obtain an answer text corresponding to the first question text, wherein the answer text is used to describe the answer to the environmental perception question, and the answer is used to control the smart device; in, The large speech model includes a large language model, which is trained based on data samples. The data samples include audio samples of environmental audio and their annotation information. The annotation information includes samples of the first question text and samples of the answer text.

2. The method according to claim 1, characterized in that The smart device is further provided with a sound wave transmitting device, which is used to transmit sound wave signals; The problem type of the environmental perception problem includes distance measurement, and the distance measurement is measuring the distance between the target object in the environment and the smart device; when the problem type is the distance measurement, the environmental audio includes a first sound wave signal and a second sound wave signal, the first sound wave signal is a sound wave signal emitted to the target object by the sound wave emitting device, and the second sound wave signal is an echo signal formed by the first sound wave signal reflected by the target object.

3. The method according to claim 1, characterized in that When the ambient audio is a voice control instruction of the smart device, the method further includes: Obtaining a second question text corresponding to the ambient audio, where the second question text is used to instruct the smart device to execute an instruction operation of the voice control instruction; Inputting the ambient audio and the second question text into the large speech model for processing to obtain an answer text corresponding to the second question text, the answer text being used to describe an environmental perception result and an execution decision of the instruction operation obtained based on the environmental perception result, the environmental perception result being a result of perceiving the environment based on acoustic phenomena generated by the ambient audio in the environment; According to the execution decision, the smart device is controlled to execute the instruction operation.

4. The method according to any one of claims 1 to 3, characterized in that The speech model also includes a text encoder and an audio encoder, and the audio encoder includes an audio phase extraction module and a speech recognition model; The text encoder is configured to: encode the question text input into the large speech model to obtain text encoding information; The audio phase extraction module is configured to: extract phase information of the ambient audio; The speech recognition model is configured to: extract amplitude information of the ambient audio, and encode the phase information and the amplitude information to obtain audio encoding information; The large language model is configured to process the text encoding information and the audio encoding information to obtain an answer text corresponding to the question text.

5. The method according to claim 4, characterized in that The phase information includes a phase sine component and a phase cosine component, and the audio phase extraction module includes a sine component extraction unit and a cosine component extraction unit; The sinusoidal component extraction unit is configured to: obtain a sine value of the phase angle of the ambient audio as the phase sinusoidal component; The cosine component extraction unit is configured to obtain a cosine value of a phase angle of the ambient audio as the phase cosine component.

6. The method according to claim 1, characterized in that The audio samples of the ambient audio are obtained in the following manner: Using a channel simulator to simulate a first channel through which audio is transmitted from a transmitting end to a receiving end, wherein the first channel is represented in the form of a first channel impulse response; performing convolution processing on the audio and the first channel, and using a result of the convolution processing as an audio sample of the ambient audio; in, The channel simulator includes multiple components, and the multiple components correspond one-to-one to multiple acoustic phenomena; The component is configured to: simulate a second channel through which audio is transmitted from a transmitting end to a receiving end when a target acoustic phenomenon occurs, wherein the second channel is represented in the form of a second channel impulse response, and the target acoustic phenomenon is the acoustic phenomenon corresponding to the component; The channel simulator is configured to obtain the first channel impulse response according to the second channel impulse response obtained by simulating all components.

7. The method according to claim 6, characterized in that The plurality of components include a LOS component, a reflection component, a reverberation component, a Doppler component, and an array component; The acoustic phenomenon corresponding to the LOS component is the LOS path; The acoustic phenomenon corresponding to the reflective component is early reflection; The acoustic phenomenon corresponding to the reverberation component is reverberation; The acoustic phenomenon corresponding to the Doppler component is the Doppler effect; The acoustic phenomenon corresponding to the array assembly is the sensor array reception phenomenon; The sensor array is a smart device equipped with a plurality of acoustic sensors, and the plurality of acoustic sensors are arranged in an array. The sensor array receiving phenomenon indicates that the ambient audio includes audio collected by each acoustic sensor in the sensor array from the same sound source.

8. The method according to claim 7, characterized in that The second channel impulse response simulated by the LOS component is h1(τ), h1(τ)=A1δ(τ-τ los ); Where A1 represents the attenuation intensity of the LOS path, δ represents the impulse function, τ is the independent variable of the impulse function, τ represents the delay, τ los Indicates the propagation delay of the LOS path.

9. The method according to claim 8, characterized in that The second channel impulse response simulated by the reflection component is h2(τ), Wherein, A2 and A3 respectively represent the attenuation strength of the two reflection paths in the early reflection, and Respectively represent the propagation delays of the two reflection paths.

10. The method according to claim 8, characterized in that The second channel impulse response simulated by the reverberation component is h3(τ), h3(τ)=R(τ); in, s i (τ) represents the signal of the i-th frequency band in the preset Gaussian noise signal, and the Gaussian noise signal is decomposed into N band frequency band, e i (τ) represents the attenuation signal of the ith frequency band, e i (τ)=exp(-λ i τ), exp represents the exponential function, λ i Represents the attenuation rate of the i-th frequency band.

Images