Audio control method and electronic equipment

Abstract

The invention provides an audio control method and electronic equipment, which are applied to the technical field of terminals, and can remove reverberation in a to-be-processed human voice signal, so that the human voice becomes clearer, and the user satisfaction is improved. The method comprises the following steps: acquiring a left surround sound channel signal, a right surround sound channel signal and a to-be-processed human voice signal; processing the left surround sound channel signal and the right surround sound channel signal by using an iterative crosstalk elimination algorithm to obtain a processed left surround sound channel signal and a processed right surround sound channel signal; determining a filtering compensator, wherein the filtering compensator is used for removing reverberation in the human voice signal to be processed; performing dereverberation processing on the to-be-processed human voice signal by using a filtering compensator to obtain a human voice calibration signal; and obtaining a stereo signal based on the human voice calibration signal, the processed left surround sound channel signal and the processed right surround sound channel signal.

Description

Technical Field

[0001] The present disclosure relates to the field of terminal technology, and in particular to an audio control method and an electronic device. Background Art

[0002] When using a terminal device to play stereo sound, a crosstalk cancellation algorithm or a crosstalk recursive iterative cancellation algorithm is generally used to improve the playback clarity of the human voice in the stereo sound. Before using the crosstalk cancellation algorithm or the crosstalk recursive iterative cancellation algorithm, the human voice signal in the stereo sound needs to be extracted, and then the background sound in the stereo sound is used to expand the sound field. Finally, the extracted human voice signal is superimposed on the expanded background sound to make the human voice clearer and achieve the effect of shortening the distance between the user and the human voice.

[0003] Since the human voice signal extracted by related technologies is not pure, and the crosstalk cancellation algorithm or crosstalk recursive iterative cancellation algorithm cannot completely cancel the crosstalk of stereo playback, it makes the human voice signal feel like reverberation is added when playing stereo, and the human voice becomes virtual and farther away from the user. Therefore, how to prevent the human voice from becoming virtual and farther away is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present disclosure provide an audio control method and an electronic device, which can remove reverberation in a human voice signal to be processed, so that the human voice becomes clearer and closer to the user, thereby improving user satisfaction.

[0005] To achieve the above objectives, the embodiments of the present disclosure adopt the following technical solutions:

[0006] In a first aspect, the present disclosure provides an audio control method, the method comprising: first, obtaining a left surround channel signal, a right surround channel signal and a human voice signal to be processed; then using an iterative crosstalk elimination algorithm to process the left surround channel signal and the right surround channel signal to obtain a processed left surround channel signal and a processed right surround channel signal; then determining a filter compensator, the filter compensator being used to remove reverberation in the human voice signal to be processed; then using the filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal; finally, based on the human voice calibration signal, the processed left surround channel signal and the processed right surround channel signal, obtaining a stereo signal.

[0007] Based on the audio control method of the first aspect, the present disclosure determines a filter compensator and then uses the filter compensator to remove reverberation in the human voice signal to be processed, thereby obtaining a human voice calibration signal. The human voice calibration signal has no reverberation or has very low reverberation. In this way, the stereo signal generated based on the human voice calibration signal will be clearer. Due to the reduction of reverberation, the human voice perceived by the user will also be closer, thereby improving user satisfaction.

[0008] In combination with the first aspect, in another possible implementation, before obtaining the left surround channel signal, the right surround channel signal and the human voice signal to be processed, the method further includes: obtaining the stereo signal to be processed; the stereo signal to be processed includes the left channel signal to be processed and the right channel signal to be processed; using an upmixing algorithm to process the left channel signal to be processed and the right channel signal to be processed to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal. Based on this implementation, if the stereo signal to be processed is obtained, it is also necessary to use an upmixing algorithm to obtain the human voice signal to be processed in the stereo signal to be processed, so as to facilitate the subsequent dereverberation processing of the human voice signal to be processed and improve the processing efficiency.

[0009] In combination with the first aspect, in another possible implementation, determining the filter compensator includes: determining the delay time corresponding to executing the iterative crosstalk elimination algorithm; determining the dereverberation time based on the delay time; and determining the filter compensator based on the delay time, the dereverberation time, and the iterative crosstalk elimination algorithm. The reverberation in the human voice signal to be processed is determined based on the dereverberation time. According to this implementation, the filter compensator can be generated based on the dereverberation time. Then, the dereverberation processing is performed using the filter compensator to achieve a better processing effect.

[0010] In combination with the first aspect, in another possible implementation, based on the delay time, the dereverberation time and the iterative crosstalk cancellation algorithm, determining the filter compensator includes: determining the number of samples corresponding to the delay time based on the iterative crosstalk cancellation algorithm; determining the number of reverberation samples based on the signal sampling rate and the dereverberation time; and determining the filter compensator based on the iterative crosstalk cancellation algorithm, the number of samples corresponding to the delay time and the number of reverberation samples. An implementation method for determining the filter compensator is provided.

[0011] In combination with the first aspect, in another possible implementation method, a filter compensator is used to perform dereverberation processing on the processed human voice signal to obtain a human voice calibration signal, including: performing Fourier transform on the processed human voice signal to obtain a frequency domain signal corresponding to the human voice signal to be processed; performing Fourier transform on the filter compensator to obtain a frequency domain signal corresponding to the filter compensator; performing inverse fast Fourier transform on the frequency domain signal corresponding to the processed human voice signal and the frequency domain signal corresponding to the filter compensator to obtain a human voice calibration signal.

[0012] In combination with the first aspect, in another possible implementation, the stereo signal includes a left channel signal and a right channel signal, and the stereo signal is obtained based on the vocal calibration signal, the processed left surround channel signal, and the processed right surround channel signal, including: obtaining the left channel signal based on the processed left surround channel signal and the vocal calibration signal; obtaining the right channel signal based on the processed right surround channel signal and the vocal calibration signal.

[0013] In combination with the first aspect, in another possible implementation, an iterative crosstalk elimination algorithm is used to process a left surround channel signal and a right surround channel signal to obtain a processed left surround channel signal and a processed right surround channel signal, including: determining a target number of iterations corresponding to the iterative crosstalk elimination algorithm; based on the target number of iterations, performing multiple preset processing on the right surround channel signal to obtain multiple first processing results; superimposing multiple first processing results on the left surround channel signal to obtain a processed left surround channel signal; based on the target number of iterations, performing multiple preset processing on the left surround channel signal to obtain multiple second processing results; superimposing multiple second processing results on the right surround channel signal to obtain a processed right surround channel signal. A specific implementation process of an iterative crosstalk elimination algorithm is provided.

[0014] In combination with the first aspect, in another possible implementation manner, the preset processing includes inversion processing, attenuation processing and delay processing.

[0015] In combination with the first aspect, in another possible implementation, determining the first and second iterations corresponding to the iterative crosstalk elimination algorithm includes: determining a first distance between a left speaker and a right speaker, a vertical distance between a user and the left and right speakers, and a second distance between a left ear of the user and a right ear of the user; determining a first ratio and a second ratio based on the first distance, the vertical distance, and the second distance; the first ratio is a ratio corresponding to a distance between a left path corresponding to the left speaker and the left path corresponding to the right ear of the user and the left speaker; the second ratio is a ratio corresponding to a distance between a right speaker and the left path and a distance between a right speaker and the right path; determining a first iteration based on the first ratio; determining a second iteration based on the second ratio; and determining a target iteration based on the first and second iterations. An implementation for determining the first and second iterations is provided.

[0016] In combination with the first aspect, in another possible implementation, determining the delay time corresponding to executing the iterative crosstalk elimination algorithm includes: determining the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path based on the first distance, the vertical distance, and the second distance; determining the first delay difference based on the distance between the left speaker and the left path and the distance between the left speaker and the right path; determining the second delay difference based on the distance between the right speaker and the left path and the distance between the right speaker and the right path; determining the first delay time based on the first delay difference and the first iteration number; determining the second delay time based on the second delay difference and the second iteration number; determining the delay time corresponding to the iterative crosstalk elimination algorithm according to the first delay time and the second delay time. Provided is an implementation method for determining the delay time.

[0017] In combination with the first aspect, in another possible implementation manner, the multi-channel signal to be processed includes a left surround channel signal, a right surround channel signal, and a human voice signal to be processed.

[0018] In a second aspect, an embodiment of the present disclosure provides an audio control device, which can be applied to an electronic device to implement the method in the first aspect. The functions of the audio control device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module, a processing module, a determination module, and a generation module.

[0019] Among them, the acquisition module is configured to acquire the left surround channel signal, the right surround channel signal and the human voice signal to be processed; the processing module is configured to use the iterative crosstalk elimination algorithm to process the left surround channel signal and the right surround channel signal to obtain the processed left surround channel signal and the processed right surround channel signal; the determination module is configured to determine the filter compensator, and the filter compensator is used to remove the reverberation in the human voice signal to be processed; the processing module is also configured to use the filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal; the generation module is configured to obtain a stereo signal based on the human voice calibration signal, the processed left surround channel signal and the processed right surround channel signal.

[0020] In combination with the second aspect, in a possible implementation, the acquisition module is further configured to acquire a stereo signal to be processed; the stereo signal to be processed includes a left channel signal to be processed and a right channel signal to be processed; the processing module is further configured to use an upmixing algorithm to process the left channel signal to be processed and the right channel signal to be processed to obtain a vocal signal to be processed, a left surround channel signal and a right surround channel signal.

[0021] In combination with the second aspect, in a possible implementation, the determination module is further configured to determine the delay time corresponding to executing the iterative crosstalk elimination algorithm; determine the de-reverberation time based on the delay time; and determine the filter compensator based on the delay time, the de-reverberation time and the iterative crosstalk elimination algorithm.

[0022] In combination with the second aspect, in a possible implementation, the determination module is also configured to determine the number of sample points corresponding to the delay time based on the iterative crosstalk elimination algorithm; determine the number of reverberation sample points based on the signal sampling rate and the de-reverberation time; and determine the filter compensator based on the iterative crosstalk elimination algorithm, the number of sample points corresponding to the delay time, and the number of reverberation sample points.

[0023] In combination with the second aspect, in a possible implementation method, the processing module is also configured to perform Fourier transform on the human voice signal to be processed to obtain a frequency domain signal corresponding to the human voice signal to be processed; perform Fourier transform on the filter compensator to obtain a frequency domain signal corresponding to the filter compensator; perform fast Fourier inverse transform on the frequency domain signal corresponding to the human voice signal to be processed and the frequency domain signal corresponding to the filter compensator to obtain a human voice calibration signal.

[0024] In combination with the second aspect, in a possible implementation, the generation module is further configured to obtain a left channel signal based on the processed left surround channel signal and the vocal calibration signal; and to obtain a right channel signal based on the processed right surround channel signal and the vocal calibration signal.

[0025] In combination with the second aspect, in a possible implementation, the processing module is further configured to determine a target number of iterations corresponding to the iterative crosstalk elimination algorithm; based on the target number of iterations, perform multiple preset processing on the right surround channel signal to obtain multiple first processing results; superimpose the multiple first processing results on the left surround channel signal to obtain a processed left surround channel signal; based on the target number of iterations, perform multiple preset processing on the left surround channel signal to obtain multiple second processing results; superimpose the multiple second processing results on the right surround channel signal to obtain a processed right surround channel signal.

[0026] In combination with the second aspect, in a possible implementation manner, the preset processing includes inversion processing, attenuation processing and delay processing.

[0027] In combination with the second aspect, in a possible implementation, the processing module is also configured to determine a first distance between the left speaker and the right speaker, a vertical distance between the user and the left speaker and the right speaker, and a second distance between the user's left ear and the user's right ear; based on the first distance, the vertical distance and the second distance, determine a first ratio and a second ratio; the first ratio is the ratio of the distance between the left speaker and the left path corresponding to the user's left ear and the distance between the left speaker and the right path corresponding to the user's right ear; the second ratio is the ratio of the distance between the right speaker and the left path and the distance between the right speaker and the right path; based on the first ratio, determine the first number of iterations; based on the second ratio, determine the second number of iterations; based on the first number of iterations and the second number of iterations, determine the target number of iterations.

[0028] In combination with the second aspect, in a possible implementation, the determination module is also configured to determine the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path based on the first distance, the vertical distance, and the second distance; determine the first delay difference based on the distance between the left speaker and the left path and the distance between the left speaker and the right path; determine the second delay difference based on the distance between the right speaker and the left path and the distance between the right speaker and the right path; determine the first delay time based on the first delay difference and the first number of iterations; determine the second delay time based on the second delay difference and the second number of iterations; determine the delay time corresponding to the iterative crosstalk elimination algorithm based on the first delay time and the second delay time.

[0029] In conjunction with the second aspect, in a possible implementation manner, the multi-channel signal to be processed includes a left surround channel signal, a right surround channel signal, and a human voice signal to be processed.

[0030] In a third aspect, the present disclosure provides an electronic device, comprising: a memory, a display screen, and one or more processors; the memory, the display screen, and the processor are coupled. The memory is used to store computer program codes, and the computer program codes include computer instructions; when the electronic device is running, the processor is used to execute one or more computer instructions stored in the memory, so that the electronic device executes the audio control method as described in any one of the first aspects above.

[0031] In a fourth aspect, the present disclosure provides a computer storage medium, comprising computer instructions, which, when executed on an electronic device, enable the electronic device to execute the audio control method as described in any one of the first aspects.

[0032] In a fifth aspect, the present disclosure provides a computer program product. When the computer program product is executed on an electronic device, the electronic device executes the audio control method as described in any one of the first aspects.

[0033] In a sixth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting a first device to implement the functions involved in the first aspect. In one possible design, the device also includes a memory for storing program instructions and data necessary for the first device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0034] It should be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1This is one of the scenario diagrams provided in the embodiment of the present disclosure.

[0036] Figure 2 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present disclosure.

[0037] Figure 3 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present disclosure.

[0038] Figure 4 This is one of the flow charts of an audio control method provided in an embodiment of the present disclosure.

[0039] Figure 5 A processing schematic diagram of an upmixing algorithm provided in an embodiment of the present disclosure.

[0040] Figure 6 The second flowchart of an audio control method provided in an embodiment of the present disclosure.

[0041] Figure 7 A processing schematic diagram of an iterative crosstalk cancellation algorithm provided in an embodiment of the present disclosure.

[0042] Figure 8 The present invention provides a second scenario diagram for an embodiment of the present invention.

[0043] Fig. 9 A schematic diagram of sound wave propagation provided in an embodiment of the present disclosure.

[0044] Fig.10 A schematic diagram of the relationship between sound amplitude and delay time provided in an embodiment of the present disclosure.

[0045] Fig.11 A schematic diagram of the relationship between distance and delay time provided in an embodiment of the present disclosure.

[0046] Fig.12 A schematic diagram of the relationship between distance and number of iterations provided in an embodiment of the present disclosure.

[0047] Fig.13 A schematic diagram of the relationship between the number of iterations and the delay time provided in an embodiment of the present disclosure.

[0048] Fig.14 A schematic diagram of the relationship between delay time and extended distance provided in an embodiment of the present disclosure.

[0049] Fig.15 A schematic diagram of the relationship between the number of sample points and the normalized amplitude provided in an embodiment of the present disclosure.

[0050] Fig.16 A schematic diagram of a reverberation filter and a filter compensator provided in an embodiment of the present disclosure.

[0051] Fig.17 A schematic diagram of a dereverberation effect provided by an embodiment of the present disclosure.

[0052] Fig.18 A schematic diagram of an effect without dereverberation provided in an embodiment of the present disclosure.

[0053] Fig.19 The third flowchart of an audio control method provided in an embodiment of the present disclosure.

[0054] Fig. 20 The fourth flowchart of an audio control method provided in an embodiment of the present disclosure.

[0055] Fig.21 The fifth flowchart of an audio control method provided in an embodiment of the present disclosure.

[0056] Fig. 22 A schematic diagram of the structure of an audio control device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The technical solution in the embodiment of the present disclosure will be described below in conjunction with the drawings in the embodiment of the present disclosure. Among them, in the description of the present disclosure, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present disclosure is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present disclosure, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present disclosure, in the embodiments of the present disclosure, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for easy understanding.

[0058] In addition, the network architecture and business scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0059] Some terms involved in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0060] 1. Stereo is an audio technology that uses two or more speakers (usually left and right speakers) to simulate the sound positioning and spatial sense in the real world. Stereo allows users to feel the horizontal position, distance and depth of the sound.

[0061] 2. Sound field expansion refers to increasing the width, depth and height of the sound field and enhancing the spatial separation between the left and right channels. Sound field expansion can improve the spatial expression of stereo sound, allowing users to feel a wider and more specific sound effect.

[0062] 3. Crosstalk refers to the interference phenomenon that occurs during the transmission or processing of stereo signals. It may come from external electromagnetic fields, transmission lines, internal circuits of equipment and other factors. Crosstalk will cause signal quality to deteriorate or generate additional noise. Crosstalk will also affect the clarity and accuracy of stereo signals, causing sound problems such as noise, distortion, intermittent or incoherent sound.

[0063] 4. Background sound can also be called environmental sound or ambient sound, which refers to the various noises and sounds produced in a specific environment, such as the sound of flowing water, wind, traffic, human voice, etc.

[0064] 5. Frequency points refer to the specific frequency positions that represent stereo signals in the frequency domain. In discrete Fourier transform (DFT) or fast Fourier transform (FFT), frequency points are sampling points in the spectrum. For an FFT transform result of length N, the number of frequency points is N / 2+1. In spectrum analysis or signal processing, effective frequency points refer to frequency components with significant energy or information.

[0065] 6. Sub-band refers to the division of the spectrum into sub-areas of different frequency ranges. In audio signal processing, a common method is to perform multi-stage filtering and downsampling operations on the spectrum through a filter bank to divide the original spectrum into multiple different sub-bands. Each sub-band contains the frequency components corresponding to the effective frequency points.

[0066] With the popularization and development of the Internet, people's functional requirements for terminal devices are becoming more and more diverse. For example, users have higher and higher requirements for the sound playback of terminal devices.

[0067] like Figure 1 As shown, when a user uses a speaker 101 in a terminal device 10 (e.g., a mobile phone or a tablet, etc.) to play stereo, in order to achieve a better playback effect, the related technology uses a human voice signal extraction algorithm to extract the human voice signal in the stereo, and then uses a crosstalk elimination algorithm or a crosstalk recursive iterative cancellation algorithm to cancel the crosstalk of the stereo during playback, so as to achieve the effect of widening the background sound field. Finally, the extracted human voice signal is superimposed on the widened background sound, so as to obtain a clearer human voice that is closer to the user.

[0068] Since the related technology cannot extract pure human voice signals and cannot completely offset the crosstalk of stereo sound during playback, the actual sound position 12 of the stereo sound heard by the user is further back than the target sound position 11 of the stereo sound, and the sound quality is more blurred, thus affecting the user's experience.

[0069] To this end, the embodiment of the present disclosure provides an audio control method, by which the reverberation in the human voice signal to be processed can be removed to obtain a relatively pure human voice. A stereo signal is generated based on the human voice after the reverberation is removed. The human voice in the stereo signal is clearer and closer to the user, thereby improving user satisfaction.

[0070] The audio control method provided in the embodiments of the present disclosure may be applied to a scenario in which a terminal device plays audio externally in any application. The embodiments of the present disclosure do not limit the application scenario of the audio control method.

[0071] It is understandable that the above terminal devices may also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The terminal device may be a mobile phone with at least two playback devices, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present disclosure do not limit the specific technology and specific device form adopted by the terminal device.

[0072] Figure 2 FIG. 1 shows a schematic diagram of the hardware structure of a terminal device. Figure 2As shown, the terminal device may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 230D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0073] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the terminal device. In other embodiments, the terminal device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0075] The charging management module 240 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0076] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 294, the camera 293, and the wireless communication module 260.

[0077] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem and baseband processor.

[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0079] The mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the terminal equipment.

[0080] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to terminal devices. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0081] The terminal device realizes the display function through the GPU, the display screen 294, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor.

[0082] The display screen 294 is used to display images, videos, etc. A series of graphical user interfaces (GUIs) can be displayed on the display screen 294 of the terminal device.

[0083] The terminal device can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor.

[0084] Camera 293 is used to capture still images or videos.

[0085] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device.

[0086] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes the instructions stored in the internal memory 221 to execute various functional applications and data processing of the terminal device.

[0087] The terminal device can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone interface 230D, and the application processor, etc. For example, music playback, recording, etc. The terminal device may also include a pressure sensor 280A, an air pressure sensor 280C, a gyroscope sensor 280B, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, an ambient light sensor 280L, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, a bone conduction sensor 280M, a button 290, a motor 291, an indicator 292, etc.

[0088] The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to or separated from the terminal device by inserting it into or removing it from the SIM card interface 295. The terminal device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The SIM card interface 295 is also compatible with external memory cards. The terminal device interacts with the network through the SIM card to implement functions such as calls and data communications.

[0089] In addition, an operating system, such as Hongmeng operating system, iOS operating system, Android operating system, Windows operating system, etc., is running on the above components. Applications can be installed and run on the operating system. In other embodiments, there may be multiple operating systems running in the terminal device.

[0090] It should be understood that Figure 2The hardware modules included in the terminal device shown are only illustrative descriptions and do not limit the specific structure of the terminal device. In fact, the terminal device provided by the embodiment of the present disclosure may also include other hardware modules that have an interactive relationship with the hardware modules illustrated in the figure, which are not specifically limited here. For example, the terminal device may also include a flashlight, a micro-projection device, etc. For another example, if the terminal device is a PC, then the terminal device may also include components such as a keyboard and a mouse.

[0091] The software system of the above terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the mobile phone is illustrated.

[0092] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0093] The application layer can include a series of application packages.

[0094] like Figure 3 As shown, the application package may include applications such as Bluetooth, camera, calendar, call, map, navigation, WLAN, music, video, short message, etc.

[0095] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0096] The application framework layer may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, etc., and the embodiments of the present disclosure do not impose any restrictions on this.

[0097] Activity Manager: used to manage the life cycle of each application. Applications usually run in the operating system in the form of Activities. For each Activity, there is a corresponding application record (ActivityRecord) in the Activity Manager. This ActivityRecord records the status of the Activity of the application. The Activity Manager can use this ActivityRecord as an identifier to schedule the Activity process of the application.

[0098] Window Manager Service: It is used to manage the graphical user interface (GUI) resources used on the screen. Specifically, it can be used to obtain the screen size, create and destroy windows, show and hide windows, layout windows, manage focus, and manage input methods and wallpapers.

[0099] The system library and kernel layer below the application framework layer can be called the underlying system. The underlying system includes an underlying display system for providing display services. For example, the underlying display system includes a display driver in the kernel layer and a surface manager in the system library.

[0100] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0101] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), algorithm library, etc.

[0102] The algorithm library includes multiple algorithms for processing audio signals, such as an upmix algorithm, an iterative crosstalk cancellation algorithm, a downmix algorithm, and the like.

[0103] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0104] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0105] OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0106] SGL is a graphics engine for 2D graphics.

[0107] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0108] The following is a description of the process of an audio control method provided by an embodiment of the present disclosure. Figure 4 As shown, specifically including:

[0109] Step 401: The terminal device obtains a stereo signal to be processed.

[0110] The stereo signal to be processed includes a left channel signal to be processed and a right channel signal to be processed.

[0111] The left channel signal to be processed refers to the sound signal played by the left speaker, and the right channel signal to be processed refers to the sound signal played by the right speaker.

[0112] In some examples, the stereo signal to be processed may be obtained by a receiver. For example, the receiver may obtain the stereo signal from another device or another system through a wired or wireless communication channel.

[0113] In some other examples, the stereo signal to be processed can be obtained according to a processor and at least two microphones. The at least two microphones are used to record information obtained from a sound source, and the processor is used to process the information recorded by the microphones to obtain the stereo signal to be processed.

[0114] In some examples, obtaining the stereo signal to be processed may include: first obtaining an initial audio signal; and then decomposing the initial audio signal to obtain the stereo signal to be processed. Decomposing the initial audio signal may be implemented by using one or more of an environmental phase estimation, a principal component analysis, or a least squares analysis method.

[0115] It should be noted that the initial audio signal may be audio recorded by a microphone; may also be sampled existing audio; or may be audio extracted from a music file. The present disclosure does not limit the method for obtaining the initial audio signal.

[0116] Step 402: The terminal device processes the stereo signal to be processed by using an upscaling algorithm to obtain a human voice signal to be processed, a left surround channel signal, and a right surround channel signal.

[0117] After the terminal device obtains the stereo signal to be processed, the left channel signal to be processed and the right channel signal to be processed in the stereo signal to be processed can be processed using an upscaling algorithm to obtain a human voice signal to be processed, a left surround channel signal and a right surround channel signal.

[0118] In some examples, such as Figure 5As shown, using the up-regulation algorithm to process the left channel signal to be processed and the right channel signal to be processed to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal may include: first, Fourier transform (FFT) is performed on the left channel signal to be processed and the right channel signal to be processed to obtain multiple effective frequency points. Then, the multiple effective frequency points are divided to obtain multiple sub-bands. Then, the sub-center extraction coefficient corresponding to each sub-band in the multiple sub-bands is determined. Based on the sub-center extraction coefficient of each sub-band, the sub-center signal corresponding to each sub-band is obtained. Then, according to the left channel signal to be processed, the right channel signal to be processed and the sub-center signal corresponding to each sub-band, the sub-left surround channel signal and the sub-right surround channel signal corresponding to each sub-band are determined. Then, the sub-left surround channel signal, the sub-right surround channel signal and the sub-center signal corresponding to each sub-band in the multiple sub-bands are synthesized to obtain the full-band signal. Finally, the full-band signal is inversely Fourier transformed to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal.

[0119] Exemplarily, Fourier transform (FFT) is performed on the left channel signal to be processed and the right channel signal to be processed in the stereo signal to be processed, and obtaining multiple valid frequency points may include: first, cutting the left channel signal to be processed in the stereo signal according to a certain frame length. For example, based on the time length at a signal sampling rate of 48kHz, the left channel signal to be processed is cut, and the frame length of each frame after cutting is 480 samples. The two adjacent frames after cutting are overlapped by 1 / 2 of the frame length, that is, the overlapping part between the two adjacent frames is 480 samples. Then, a Fourier transform (FFT) with a length of 1024 points is performed on each frame after cutting (including the overlapping part) to obtain 1024 complex frequency points. Since the complex frequency points include positive frequencies and negative frequencies, based on the symmetry of the spectrum, only 513 frequency points need to be retained as valid frequency points, and the valid frequency points can be used for processing or analysis of the stereo signal to be processed. Among them, Fourier transform is used to convert time domain signals into frequency domain signals.

[0120] It can be understood that the process of performing Fourier transform (FFT) on the right channel signal to be processed in the stereo signal to be processed to obtain multiple effective frequency points is similar to the process of performing Fourier transform on the left channel signal to be processed, which will not be repeated here.

[0121] After obtaining multiple valid frequency points, the multiple valid frequency points are divided to obtain multiple sub-bands. In some examples, the multiple valid frequency points may be divided evenly, or may be divided unevenly based on psychoacoustic sensitivity to frequency bands, for example, the multiple frequency points are divided based on a bark spectrum to obtain multiple sub-bands.

[0122] The number of subbands based on bark spectrum division can be determined by the following expression:

[0123]

[0124] Wherein, z(f) is the number of subbands; f is the frequency of Fourier transform.

[0125] After obtaining multiple sub-bands, the sub-midband extraction coefficient of each sub-band can be determined by the following expression.

[0126]

[0127] Among them, α is the sub-mid-place extraction coefficient corresponding to a sub-band; L r and L i is the real and imaginary parts of the left channel signal corresponding to a subband; R r and R i is the real and imaginary part of the right channel signal corresponding to a subband.

[0128] After obtaining the sub-center extraction coefficient corresponding to each sub-band, the sub-center signal corresponding to each sub-band can be determined according to the sub-center extraction coefficient corresponding to each sub-band, the left channel signal to be processed, and the right channel signal to be processed. Exemplarily, the sub-center signal corresponding to each sub-band can be determined by the following expression:

[0129] C'=α×(L1+R1)

[0130] Among them, C' is the sub-center signal; L is the left channel signal to be processed; R is the right channel signal to be processed.

[0131] In some examples, the sub-left surround channel signal and the sub-right surround channel signal corresponding to each sub-band satisfy the following expression:

[0132] L1'=L1-C'

[0133] R1'=R1-C'

[0134] Among them, L1' is the sub-left surround channel signal; R1' is the sub-right surround channel signal.

[0135] Then, the sub-vocal signals, sub-left surround channel signals and sub-right surround channel signals corresponding to all sub-bands are synthesized to obtain full-band signals. Finally, the full-band signals are inverse Fourier transformed to obtain the final vocal signals to be processed, left surround channel signals and right surround channel signals.

[0136] In some examples, Figure 6The processing flow of an audio control algorithm is shown in an exemplary manner. First, an upmixing algorithm is called to process the left channel signal to be processed and the right channel signal to be processed to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal. Then, the left surround channel signal and the right surround channel signal are iteratively crosstalk eliminated to obtain the processed left surround channel signal and the processed right surround channel signal. Then, the delay time and the number of samples corresponding to the delay time are determined in turn, and a filter compensator is generated based on the delay time and the number of samples corresponding to the delay time. Then, a human voice calibration signal is determined according to the filter compensator. Finally, a downmixing algorithm is used to process the human voice calibration signal, the processed left surround channel and the processed right surround channel to obtain a stereo signal. Among them, the delay time and the number of samples corresponding to the delay time are determined, and a filter compensator is generated based on the delay time and the number of samples corresponding to the delay time. Then, a human voice calibration signal is determined according to the filter compensator. Finally, the vocal calibration signal, the processed left surround channel and the processed right surround channel are processed by a downmixing algorithm to obtain a stereo signal. The process can be implemented by referring to steps 403 to 408 and will not be described in detail here.

[0137] Step 403: The terminal device processes the left surround channel signal and the right surround channel signal by using an iterative crosstalk cancellation algorithm to obtain a processed left surround channel signal and a processed right surround channel signal.

[0138] After the terminal device obtains the left surround channel signal and the right surround channel signal, it can process the left surround channel signal and the right surround channel signal using a related algorithm to obtain a processed left surround channel signal and a processed right surround channel signal.

[0139] In some examples, the above-mentioned related algorithm may be an iterative crosstalk cancellation algorithm, a crosstalk cancellation algorithm, or a crosstalk recursive iterative cancellation algorithm.

[0140] The following takes the iterative crosstalk cancellation algorithm as an example to process the left surround channel signal and the right surround channel signal.

[0141] For example, Figure 7 As shown, for the left surround channel signal, the iterative crosstalk cancellation algorithm is used to add a preset process to the right surround channel signal and iterate multiple times to obtain the processed left surround channel signal. The preset process includes inversion process, attenuation process and delay process. For the right surround channel signal, the iterative crosstalk cancellation algorithm is used to add a preset process to the left surround channel signal and iterate multiple times to obtain the processed right surround channel signal.

[0142] For example, the processed left surround channel signal and the processed right surround channel signal satisfy the following expression:

[0143]

[0144]

[0145] Where L2 is the processed left surround channel signal, R2 is the processed right surround channel signal; n is the target number of iterations; a n is the attenuation offset corresponding to the nth iteration; L1 is the left surround channel signal; t is the time (e.g., 1s, 2s, etc.), d n is the delayed sample point corresponding to the nth iteration; R1 is the right surround channel signal; d n It can be set based on experience, or can be determined by referring to the formula corresponding to the delayed sample point in step 404, and the present disclosure does not limit this.

[0146] When the left surround channel signal L1 = 0, simplifying the above formula, we can get:

[0147]

[0148] Let a n R1(t-2d n )=R1 nanb , we can get the updated L2, which satisfies the following expression:

[0149] L2=-R1 1a1b -R1 3a3b -R1 5a5b -R1 7a7b -R1 9a9b -R1 11a11b -……

[0150] Similarly, we can get R2, R2 satisfies the following expression:

[0151] R2=R1+R1 2a2b +R1 4a4b +R1 6a6b +R1 8a8b +R1 10a10b +……

[0152] In some examples, the total attenuation cancellation amount can be set based on user needs. The attenuation cancellation amount corresponding to each iteration can be determined based on the total attenuation cancellation amount; and then the target number of iterations of the iterative crosstalk cancellation algorithm (i.e., the number of iterations corresponding to the above preset processing) can be determined according to the attenuation cancellation amount corresponding to each iteration.

[0153] For example, the total attenuation offset may be -20 db. The present disclosure does not impose any limitation on the total attenuation offset, as long as it meets actual requirements.

[0154] The attenuation cancellation amount corresponding to each iteration can be determined based on the total attenuation cancellation amount. The attenuation cancellation amount corresponding to each iteration satisfies the following expression:

[0155] attenu_percent=10 (attenu_db / 20)

[0156] Among them, attenu_percent is the attenuation compensation amount corresponding to each iteration; attenu_db is the total attenuation compensation amount.

[0157] According to the attenuation compensation amount corresponding to each iteration, the target number of iterations can be determined. The target number of iterations satisfies the following expression:

[0158] interN=2*ceil((log10(attenu_percent) / log10(g)) / 2-1)

[0159] Wherein, interN is the target iteration number; ceil is used to represent the integer rounded downward; and g is the ratio of the distance between the left path corresponding to the speaker and the user's left ear to the distance between the right path corresponding to the speaker and the user's right ear.

[0160] For stereo signals, the playback of stereo signals usually requires two speakers, so the ratio g of the distance between the speaker and the left path corresponding to the user's left ear to the right path corresponding to the speaker and the right path corresponding to the user's right ear includes a first ratio (i.e., g1) corresponding to the distance between the left speaker and the left path and the distance between the left speaker and the right path and a second ratio (i.e., g2) corresponding to the distance between the right speaker and the right path and the distance between the right speaker and the left path.

[0161] Accordingly, since the target number of iterations is related to the ratio of the distance between the speaker and the left path corresponding to the user's left ear to the distance between the speaker and the right path corresponding to the user's right ear, the first number of iterations (i.e., interN1) can be determined based on the first ratio of the distance between the left speaker and the left path to the distance between the left speaker and the right path. The second number of iterations (i.e., interN2) can be determined based on the second ratio of the distance between the right speaker and the right path to the distance between the right speaker and the left path.

[0162] After the first number of iterations and the second number of iterations are obtained, the larger value of the first number of iterations and the second number of iterations may be used as the target number of iterations (ie, interN).

[0163] For example, Figure 8As shown, the distance between the left speaker and the right speaker is the first distance, which can be represented by ds. The vertical distance between the user and the left speaker and the right speaker can be represented by ys. The distance between the user's left ear and the user's right ear is the second distance. The second distance can be represented by 2r.

[0164] based on Figure 8 , the distance between the left speaker and the left path satisfies the following expression:

[0165]

[0166] Where ILL is the distance from the left speaker to the left path.

[0167] The distance between the left speaker and the right path satisfies the following expression:

[0168]

[0169] Where ILR is the distance from the left speaker to the right path.

[0170] The first ratio corresponding to the distance between the left speaker and the left path and the distance between the left speaker and the right path satisfies the following expression:

[0171]

[0172] The distance between the right speaker and the left path satisfies the following expression:

[0173]

[0174] Where IRL is the distance from the right speaker to the left path.

[0175] The distance from the right speaker to the right path satisfies the following expression:

[0176]

[0177] Where IRR is the distance from the right speaker to the right path.

[0178] The second ratio corresponding to the distance between the right speaker and the right path and the distance between the right speaker and the left path satisfies the following expression:

[0179]

[0180] After determining the first ratio and the second ratio, the first number of iterations and the second number of iterations can be obtained based on the first ratio, the second ratio and the expression corresponding to the number of iterations, and the larger value of the first number of iterations and the second number of iterations is used as the target number of iterations. Then, the left surround channel signal and the right surround channel signal can be iterated multiple times according to the target number of iterations to obtain a processed left surround channel signal and a processed right surround channel signal.

[0181] Step 404: The terminal device determines the delay time required to execute the iterative crosstalk cancellation algorithm.

[0182] Fig. 9 An example diagram of sound wave propagation is shown. According to the sound wave propagation diagram, when the sound point S emits a sound wave, the sound wave can be propagated to the user's ears through multiple propagation paths. For example, the sound point S can propagate the sound wave to the user's ears through path 1 (also called a direct path). The sound point S can also propagate the sound wave to the user's ears through path 2. Alternatively, the sound point S can propagate the sound wave to the user's ears through path 3.

[0183] Among them, path 2 and path 3 are the sound waves transmitted to the user's ears after the sound point S passes through a wall or other obstacles. When the sound point S passes through a wall or other obstacles, multiple reflections or diffractions will occur, so that the sound waves are transmitted to the user's ears in the form of synthetic waves.

[0184] When sound propagates in an indoor environment, it goes through different stages such as direct sound, early reflections, and late reverberation. Direct sound refers to the path of sound that propagates directly from the sound source to the user's ears. It is the first sound to reach the user's ears without any reflection or delay. Direct sound is very important for people to understand the content of language and sound because it provides clear and accurate information.

[0185] Early reflection refers to the reflection of sound from indoor walls, floors, ceilings and other objects within a short period of time after it is emitted, and re-reaching the user's ears through different paths. These early reflections are slightly later than direct sounds, but the time interval is shorter, which can provide users with a sense of space and positioning information.

[0186] Late reverberation refers to the sound gradually decaying after multiple reflections and spreading in the indoor space, forming the effect of echo and reverberation. Late reverberation adds a sense of space and lingering sound to the sound, but it may also reduce the clarity and intelligibility of the sound. For occasions such as speeches and music performances that require clear communication of information, proper control of late reverberation can improve the intelligibility of the sound.

[0187] It is understandable that Fig. 9For the sound point S in the sound wave, after multiple reflections or diffractions, it will enter the early reflection and late reverberation stages. In both the early reflection stage and the late reverberation stage, it can be considered that the propagation of sound is delayed.

[0188] Fig.10 The following is an exemplary diagram showing the relationship between the delay time and the sound amplitude. Fig.10 It can be seen that the sound amplitude will decay as the delay time increases. As the delay time increases, the sound amplitude of the direct sound will decay according to a certain attenuation relationship. When the delay time is less than 50ms, the sound is in the direct stage or the early reflection stage. When the delay time is greater than 50ms, reverberation will form and enter the late reverberation stage. Therefore, the early reflection stage and the late reverberation stage can be distinguished by the delay time. Generally, when the delay time is less than 50ms, it is considered that the current sound propagation is in the early reflection stage or the direct sound stage. If there is no reflection phenomenon in the sound, it is considered that the current sound propagation is in the direct sound stage. When the delay time is greater than 50ms, it is considered that the current sound propagation is in the late reverberation stage.

[0189] The iterative crosstalk elimination algorithm in step 403 mainly simulates the crosstalk path to offset the impact of crosstalk. If the sound is propagated in an ideal obstacle-free environment, theoretically, the effect of complete cancellation can be achieved, the sound field width can also reach the target sound field width, and the clarity of the human voice will not be affected. However, due to the existence of obstacles in the actual environment, the sound waves will inevitably have reflections, diffractions and reverberations, so the iterative crosstalk elimination algorithm in step 403 still cannot completely offset the impact of crosstalk.

[0190] Therefore, crosstalk signals of varying degrees will still remain in the processed left surround channel signal and the processed right surround channel signal obtained by the iterative crosstalk elimination algorithm. When the human voice signal to be processed is superimposed with these crosstalk signals, reverberation will occur in the human voice signal, which not only reduces the clarity of the human voice signal, but also increases the distance between the user and the human voice signal. Therefore, the present invention calculates the delay time required by the iterative crosstalk elimination algorithm, and generates a dereverberation filter based on the delay time. The dereverberation filter is used to remove the reverberation in the human voice signal, thereby improving the clarity of the human voice signal, and shortening the distance between the user and the human voice signal, thereby enhancing the user's auditory experience.

[0191] When the terminal device processes the left surround channel signal and the right surround channel signal using the iterative crosstalk cancellation algorithm, the terminal device may determine the delay time required to execute the iterative crosstalk cancellation algorithm.

[0192] In some examples, the delay time required for the iterative crosstalk elimination algorithm (i.e., refectT) can be determined based on the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path in step 403.

[0193] In some other examples, the first delay difference (i.e., delayTL1) can be determined based on the distance between the left speaker and the left path and the distance between the left speaker and the right path. The second delay difference (i.e., delayTL2) can be determined based on the distance between the right speaker and the left path and the distance between the right speaker and the right path. The delay difference (i.e., delayTL) includes the first delay difference and the second delay difference.

[0194] Then, based on the first delay difference and the first iteration number, the first delay time (i.e., refectT1) can be determined. Based on the second delay difference and the second iteration number, the second delay time (i.e., refectT2) can be determined. After determining the first delay time and the second delay time, the maximum value of the first delay time and the second delay time can be determined as the final delay time. (i.e., refectT). The delay time can also be called reverberation time.

[0195] Exemplarily, the first delay difference satisfies the following expression:

[0196]

[0197] Wherein, delayTL1 is the first delay difference; C is the speed of sound; C is 340m / s.

[0198] Based on the first delay difference and the first iteration number, a first delay time is determined. The first delay time satisfies the following expression:

[0199] refectT1=delayTL1*interN1

[0200] Among them, refectT1 is the first delay time.

[0201] In addition, the first delayed sample point can also be determined based on the first delay difference and the signal sampling rate. The first delayed sample point satisfies the following expression:

[0202] delayNL1=delayTL1*fs

[0203] Wherein, delayNL1 is the first delayed sample point; fs is the signal sampling rate.

[0204] When executing the iterative crosstalk cancellation algorithm in step 403, the crosstalk corresponding to the left speaker cannot be completely canceled due to the influence of the environment. This is equivalent to adding a reverberation of the first delay time to the human voice signal, which will cause the human voice signal to be pulled away. The first pull-away distance corresponding to the human voice signal can be determined by the following expression:

[0205] distC1=refectT1*C

[0206] Wherein, distC1 is the first distance to be pulled away; C is the speed of sound.

[0207] Then, the second delay difference corresponding to the right speaker is determined according to the distance between the right speaker and the left path and the distance between the right speaker and the right path. The second delay difference satisfies the following expression:

[0208]

[0209] Wherein, delayTL2 is the second delay difference; C is the speed of sound; C is 340m / s.

[0210] Then, based on the second delay difference and the second iteration number, a second delay time is determined. The second delay time satisfies the following expression:

[0211] refectT2=delayTL2*interN2

[0212] Among them, refectT2 is the second delay time.

[0213] In addition, a second delayed sample point may also be determined based on the second delay difference and the signal sampling rate. The second delayed sample point satisfies the following expression:

[0214] delayNL2=delayTL2*fs

[0215] Wherein, delayNL2 is the second delayed sample point; fs is the signal sampling rate.

[0216] When executing the iterative crosstalk cancellation algorithm in step 403, the crosstalk corresponding to the right speaker cannot be completely canceled due to the influence of the environment, which is equivalent to adding a reverberation of the second delay time to the human voice signal, thus causing the human voice signal to be pulled away. The second pull-away distance corresponding to the human voice signal can be determined by the following expression:

[0217] distC2=refectT2*C

[0218] Wherein, distC2 is the second distance to be pulled away; C is the speed of sound.

[0219] In combination with step 403, based on the total attenuation cancellation amount (ie, attenu_db), the first distance (ds), the vertical distance (ys) and the second distance (2r), the relationship between the delay time and the distance required when executing the iterative crosstalk cancellation algorithm can be calculated.

[0220] For example, attenu_db = -20dB, ds = 0.29, ys = 0.5, r = 0.08. Based on the formula corresponding to attenu_db, ds, ys, r and the delay difference (i.e., the first delay difference or the second delay difference mentioned above), it can be obtained Fig.11 .according to Fig.11 It can be seen that as the distance increases, the delay time required for one iteration becomes shorter. Based on the formula corresponding to attenu_db, ds, ys, r and the number of iterations (i.e. the first number of iterations or the second number of iterations mentioned above), we can get Fig.12 .according to Fig.12 It can be seen that as the distance increases, the number of iterations will also increase. Based on the formula corresponding to attenu_db, ds, ys, r and the delay time (i.e., the first delay time or the second delay time mentioned above), it can be obtained Fig.13 .according to Fig.13 It can be seen that as the number of iterations increases, the delay time will also increase. Based on the formula corresponding to attenu_db, ds, ys, r and the pull-out distance (i.e., the first pull-out distance or the second pull-out distance mentioned above), it can be obtained Fig.14 .according to Fig.14 It can be seen that as the delay time increases, the distance between the human voice signal to be processed and the user becomes longer.

[0221] Finally, after determining the first delay time and the second delay time, the maximum value of the first delay time and the second delay time can be determined as the final delay time. After the iterative crosstalk elimination algorithm, the reverberation corresponding to the delay time will be added to the human voice signal to be processed at most. The reverberation can then be eliminated by setting a de-reverberation filter, thereby shortening the distance between the user and the human voice signal to be processed.

[0222] Step 405: The terminal device determines the number of samples corresponding to the delay time.

[0223] Exemplarily, when the target number of iterations = 100, the first distance = 0.29, the vertical distance = 0.5, and the second distance = 0.08, using the formulas in the above steps 403 and 404, it can be concluded that the delay time is 12.9 ms.

[0224] Set iterN = 100; ds = 0.29; ys = 0.5; r = 0.08, and input as follows Figure 7 The iterative crosstalk cancellation algorithm shown in the figure can be obtained as follows Fig.15The output is shown. Fig.15 The vertical black line in the image represents the processed left surround channel. The dashed line represents the processed right surround channel. Fig.15 Using the output result as a reference line, it can be concluded that the number of samples corresponding to a delay time of 12.9ms is 619.

[0225] Step 406: The terminal device determines a filter compensator based on the delay time and the number of samples corresponding to the delay time.

[0226] After determining the delay time and the number of samples corresponding to the delay time, the dereverberation time can be determined based on the delay time. Then, the number of reverberation samples is determined based on the signal sampling rate and the dereverberation time. Then, based on the iterative crosstalk elimination algorithm, the number of samples corresponding to the delay time and the number of reverberation samples, the filter compensator is determined. Finally, the filter compensator is used to process the human voice signal to be processed to obtain a human voice calibration signal.

[0227] Exemplarily, in combination with step 405, the delay time is 12.9 ms, and the number of samples corresponding to the delay time is 619. Determining the dereverberation time based on the delay time can be determining the dereverberation time based on the delay time and user demand. The dereverberation time is less than or equal to the delay time. For example, the dereverberation time is 10 ms.

[0228] In some examples, based on the signal sampling rate and the dereverberation time, the process of determining the number of reverberation samples is as follows: first convert the dereverberation time into seconds, 10ms=0.01s. If the signal sampling rate is 48kHz (48kHz means 48,000 samples are collected per second), the number of samples that can be collected in 10ms is: 0.01×48000=480. That is, the number of reverberation samples is 480.

[0229] Combined with the above, it can be seen that the number of samples corresponding to the delay time of 12.9ms is 619. Based on the dereverberation time, it can be seen that the user only chooses to remove the reverberation of 10ms, that is, to remove 480 samples. Therefore, for the number of samples corresponding to the delay time, 139 samples are retained without performing the dereverberation operation. Among them, the dereverberation operation refers to using the filter compensator to process the human voice signal to be processed to obtain a human voice calibration signal.

[0230] In some examples, the processing corresponding to the iterative crosstalk elimination algorithm in step 403 can be converted into a linear processing system according to the convolution principle, and the linear processing system can be represented by the function h(t). After the unit pulse passes through the linear processing system h(t), the processing result ys(t) can be obtained (i.e., the reverberation filter ys(t) is obtained). In combination with step 401, the unit pulse can be the left channel signal to be processed and the right channel signal to be processed.

[0231] Assume that the unit pulse is idea_imp(t). Then the above process can be converted into the following expression:

[0232] ys(t)=convolution(h(t), idea_imp(t))

[0233] Introducing the above reverberation sample number and the sample number corresponding to the delay time into the processing result ys(t), we can get y(t)=ys(139:619). Combined with the above, we can know that the number of reverberation sample points is 480, so the 139:169 in ys(t) means counting from the 139th sample point to the 619th sample point. The effective length corresponding to the 139th sample point to the 619th sample point is 480 samples.

[0234] Then, perform Fourier frequency domain transform on y(t) to obtain the frequency domain representation result y(k). Then perform Fourier transform based inversion on y(k) to obtain the filter compensator g(t). Finally, perform Fourier frequency domain transformation on the filter compensator g(t) to obtain g(k).

[0235] Among them, y(k) satisfies the following expression:

[0236] y(k)=FFT(y(t))

[0237] Among them, y(k) is the frequency domain expression result of y(t).

[0238] g(k) satisfies the following expression:

[0239]

[0240] Among them, conj means conjugate negation; abs means averaging.

[0241] Fig.16 A schematic diagram of a reverberation filter and a filter compensator is shown as an example. The straight line corresponds to the reverberation filter, and the dashed line corresponds to the filter compensator. The reverberation filter is the filter corresponding to ys(t), and the filter compensator is the filter corresponding to g(k).

[0242] Step 407: The terminal device uses a filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal.

[0243] After the filter compensator is obtained, the Fourier frequency domain transform is performed on the human voice signal to be processed by using the filter compensator to achieve dereverberation processing, thereby obtaining a human voice calibration signal.

[0244] In some examples, before using a filter compensator to perform a Fourier frequency domain transform on the processed vocal signal, it is necessary to first perform a Fourier frequency domain transform on the processed vocal signal to obtain a frequency domain representation result C(k) corresponding to the processed vocal signal, and then use C(k) and g(k) to obtain a vocal calibration signal.

[0245] Exemplarily, the vocal calibration signal satisfies the following expression:

[0246] C'(t)=IFFT(C(K)*g(k))

[0247] Wherein, C'(t) is a human voice calibration signal; IFFT is an inverse fast Fourier transform (inverse fast Fourier transform).

[0248] IFFT is an algorithm that converts frequency domain signals into time domain signals. It is the inverse transform of Fourier transform (FFT) and can convert the frequency domain signal obtained by Fourier transform into a time domain signal.

[0249] The IFFT algorithm is mainly used in digital signal processing and digital communication. For example, in digital audio, IFFT synthesizes audio waveforms by transforming digital signals from the frequency domain to the time domain, thereby realizing filtering, equalization, encoding and decoding functions in the audio system. In digital communication, IFFT is usually used to transform digital signals into time domain signals for digital signal modulation and demodulation, as well as modeling and simulation of digital communication channels.

[0250] Step 408: The terminal device obtains a stereo signal based on the vocal calibration signal, the processed left surround channel, and the processed right surround channel.

[0251] After obtaining the vocal calibration signal, the vocal calibration signal can be superimposed on the processed left surround channel and the processed right surround channel using a downmix algorithm to obtain a stereo signal, wherein the stereo signal includes a left channel signal and a right channel signal.

[0252] In some examples, the left channel signal and the right channel signal satisfy the following expression:

[0253] L_out=g1*L2+g2*C'(t)

[0254] R_out=g3*R2+g2*C'(t)

[0255] Among them, L_out is the left channel signal; R_out is the right channel signal; g1 is the left channel gain control parameter; g2 is the vocal gain control parameter; g3 is the right channel gain control parameter; g1, g2, g3 are numbers between 0 and 1.

[0256] After the sound field expansion is performed on the processed stereo signal, if a filter compensator is set to perform dereverberation processing on the vocal signal, such as Fig.17 The sound waves of the vocal signal shown in FIG. 1 will become continuous, similar to the original stereo signal to be processed. If the filter compensator is not set to perform dereverberation processing on the vocal signal, Fig.18 The sound waves of the vocal signal shown become discontinuous.

[0257] Therefore, the present solution first determines the filter compensator, and then uses the filter compensator to remove the reverberation in the human voice signal to be processed, thereby obtaining a human voice calibration signal. The human voice calibration signal is obtained according to user needs. There is no reverberation or the reverberation is very low in the human voice calibration signal. In this way, the stereo signal generated based on the human voice calibration signal will be clearer. Due to the reduction of reverberation, the user will feel that the human voice will be closer, thereby improving user satisfaction.

[0258] The following is a description of another audio control method provided by an embodiment of the present disclosure. Fig.19 As shown, specifically including:

[0259] Step 1901: The terminal device obtains a multi-channel signal to be processed.

[0260] The multi-channel signal to be processed may be a 5.1-channel signal or a 7.1-channel signal.

[0261] The 5.1 channel signal is an audio signal with five independent channels. The 5.1 channel signal includes a left main channel signal to be processed, a right main channel signal to be processed, a vocal signal to be processed, a left surround channel signal to be processed, a right surround channel signal to be processed, and a bass channel signal.

[0262] The 7.1 channel signal has seven independent channels of audio signals. The 7.1 channel signal adds two additional surround channels on the basis of the 5.1 channel. The 7.1 channel signal includes four surround channels, namely the left front surround channel signal to be processed, the right front surround channel signal to be processed, the left rear surround channel signal to be processed, and the right rear surround channel signal to be processed. The multi-channel signal to be processed can provide a more realistic and three-dimensional sound effect, which is suitable for scenes with high requirements such as music and movies.

[0263] The following takes the 5.1 channel multi-channel signal to be processed as an example to illustrate the audio control method. It can be understood that the multi-channel signal to be processed includes a left main channel signal to be processed, a right main channel signal to be processed, a human voice signal to be processed, a left surround channel signal to be processed, and a right surround channel signal to be processed.

[0264] Step 1902: The terminal device processes the left main channel signal to be processed, the right main channel signal to be processed, the left surround channel signal to be processed, and the right surround channel signal to be processed using an iterative crosstalk cancellation algorithm to obtain a processed left main channel signal, a processed right main channel signal, a processed left surround channel signal, and a processed right surround channel signal.

[0265] In the embodiment of the present disclosure, an iterative crosstalk elimination algorithm is used to process the left main channel signal to be processed, the right main channel signal to be processed, the left surround channel signal to be processed, and the right surround channel signal to be processed. The processed left main channel signal, the processed right main channel signal, the processed left surround channel signal, and the processed right surround channel signal can be referred to the above Figure 4 Step 403 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0266] Among them, using the iterative crosstalk elimination algorithm to process the left main channel signal to be processed, the right main channel signal to be processed, the left surround channel signal to be processed and the right surround channel signal to be processed can be a group of signals, and the left main channel signal to be processed and the right main channel signal to be processed are input into the iterative crosstalk elimination algorithm to obtain a processed left main channel signal and a processed right main channel signal.

[0267] The left surround channel signal to be processed and the right surround channel signal to be processed are used as another group of signals, and the left surround channel signal to be processed and the right surround channel signal to be processed are input into an iterative crosstalk cancellation algorithm to obtain a processed left surround channel signal and a processed right surround channel signal.

[0268] In some examples, Fig. 20The processing flow of another audio control algorithm is shown in an exemplary manner. First, the iterative crosstalk elimination algorithm is called to process the left main channel signal to be processed and the right main channel signal to be processed to obtain the processed left main channel signal and the processed right main channel signal. The left surround channel signal to be processed and the right surround channel signal to be processed are processed to obtain the processed left surround channel signal and the processed right surround channel signal. Then, the delay time corresponding to the generated processed left main channel signal and the processed right main channel signal and the number of samples corresponding to the delay time are determined in sequence, and the first filter compensator is generated based on the delay time corresponding to the generated processed left main channel signal and the processed right main channel signal and the number of samples corresponding to the delay time. The delay time corresponding to the generated processed left surround channel signal and the processed right surround channel signal and the number of samples corresponding to the delay time are determined, and the second filter compensator is generated based on the delay time corresponding to the generated processed left surround channel signal and the processed right surround channel signal and the number of samples corresponding to the delay time. Finally, the human voice calibration signal is determined according to the first filter compensator and the second filter compensator. Finally, the vocal calibration signal, the processed left main channel signal, the processed right main channel signal, the processed left surround channel signal and the processed right surround channel signal are processed using a downmix algorithm to obtain a stereo signal. The above process can be implemented with reference to steps 1903 to 1906 and will not be repeated here.

[0269] Step 1903: The terminal device determines the delay time.

[0270] The terminal device determines the delay time 1 corresponding to the processed left main channel signal and the processed right main channel signal and the delay time 2 corresponding to the processed left surround channel signal and the processed right surround channel signal, and determines the delay time based on the delay time 1 and the delay time 2.

[0271] In some examples, based on delay time 1 and delay time 2, determining the delay time may include: taking a larger value of delay time 1 and delay time 2 as the final delay time.

[0272] In the embodiment of the present disclosure, the time delay 1 corresponding to the generated processed left main channel signal and the processed right main channel signal and the time delay 2 corresponding to the generated processed left surround channel signal and the processed right surround channel signal can be determined by referring to the above Figure 4 Step 404 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0273] Step 1904: The terminal device determines a filter compensator based on the delay time.

[0274] In some examples, in combination with step 1903, the terminal device can determine a first filter compensator based on delay time 1, determine a second filter compensator based on delay time 2, and finally use the filter compensator corresponding to the larger value of delay time 1 and delay time 2 as the final filter compensator.

[0275] In other examples, in combination with step 1903, the terminal device may determine the final delay time based on delay time 1 and delay time 2, determine the filter compensator corresponding to the delay time, and use the filter compensator as the final filter compensator.

[0276] In the embodiment of the present disclosure, the first filter compensator is determined based on the delay time 1, and the second filter compensator is determined based on the delay time 2. The filter compensator corresponding to the delay time can be determined by referring to the above Figure 4 Steps 405 and 406 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0277] Step 1905: The terminal device uses a filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal.

[0278] In the embodiment of the present disclosure, the filter compensator is used to perform dereverberation processing on the processed human voice signal to obtain the human voice calibration signal. Figure 4 Step 407 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0279] Step 1906: The terminal device obtains a stereo signal based on the vocal calibration signal, the processed left main channel signal, the processed right main channel signal, the processed left surround channel signal, and the processed right surround channel signal.

[0280] In the embodiment of the present disclosure, a stereo signal is obtained based on the human voice calibration signal, the processed left main channel signal, the processed right main channel signal, the processed left surround channel signal, and the processed right surround channel signal. Figure 4 Step 408 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0281] For ease of understanding, the following Fig.21 The audio control method provided by the embodiment of the present disclosure is described. Fig.21 As shown, the audio control method may include the following steps 2101-2105.

[0282] Step 2101: Obtain a left surround channel signal, a right surround channel signal, and a human voice signal to be processed.

[0283] In some examples, before obtaining the left surround channel signal, the right surround channel signal and the human voice signal to be processed, the method also includes: obtaining the stereo signal to be processed; the stereo signal to be processed includes the left channel signal to be processed and the right channel signal to be processed; using an upmixing algorithm to process the left channel signal to be processed and the right channel signal to be processed to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal.

[0284] In some examples, the multi-channel signal to be processed includes a left surround channel signal, a right surround channel signal, and a vocal signal to be processed.

[0285] In the embodiment of the present disclosure, the left surround channel signal, the right surround channel signal and the to-be-processed human voice signal can be obtained by referring to the above Figure 4 Steps 401 and 402 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0286] Step 2102: Use an iterative crosstalk cancellation algorithm to process the left surround channel signal and the right surround channel signal to obtain a processed left surround channel signal and a processed right surround channel signal.

[0287] In some examples, an iterative crosstalk elimination algorithm is used to process a left surround channel signal and a right surround channel signal to obtain a processed left surround channel signal and a processed right surround channel signal, including: determining a target number of iterations corresponding to the iterative crosstalk elimination algorithm; based on the target number of iterations, performing multiple preset processing on the right surround channel signal to obtain multiple first processing results; superimposing the multiple first processing results on the left surround channel signal to obtain a processed left surround channel signal; based on the target number of iterations, performing multiple preset processing on the left surround channel signal to obtain multiple second processing results; superimposing the multiple second processing results on the right surround channel signal to obtain a processed right surround channel signal.

[0288] In some examples, the preset processing includes an inversion process, an attenuation process, and a delay process.

[0289] In some examples, determining a target number of iterations corresponding to an iterative crosstalk elimination algorithm includes: determining a first distance between a left speaker and a right speaker, a vertical distance between a user and the left speaker and the right speaker, and a second distance between a left ear of the user and a right ear of the user; determining a first ratio and a second ratio based on the first distance, the vertical distance, and the second distance; the first ratio is a ratio corresponding to a distance between a left path corresponding to the left speaker and the left ear of the user and a distance between a right path corresponding to the left speaker and the right ear of the user; the second ratio is a ratio corresponding to a distance between the right speaker and the left path and a distance between the right speaker and the right path; based on the first ratio, determining a first number of iterations; based on the second ratio, determining a second number of iterations; based on the first number of iterations and the second number of iterations, determining a target number of iterations.

[0290] In the embodiment of the present disclosure, an iterative crosstalk cancellation algorithm is used to process the left surround channel signal and the right surround channel signal to obtain the processed left surround channel signal and the processed right surround channel signal. Figure 4 Step 403 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0291] Step 2103: determine a filter compensator, where the filter compensator is used to remove reverberation in the human voice signal to be processed.

[0292] In some examples, determining a filter compensator includes: determining a delay time corresponding to executing an iterative crosstalk cancellation algorithm; determining a dereverberation time based on the delay time; and determining a filter compensator based on the delay time, the dereverberation time, and the iterative crosstalk cancellation algorithm.

[0293] In some examples, a filter compensator is determined based on a delay time, a de-reverberation time, and an iterative crosstalk cancellation algorithm, including: determining the number of samples corresponding to the delay time based on the iterative crosstalk cancellation algorithm; determining the number of reverberation samples based on a signal sampling rate and the de-reverberation time; and determining the filter compensator based on the iterative crosstalk cancellation algorithm, the number of samples corresponding to the delay time, and the number of reverberation samples.

[0294] In some examples, determining the delay time corresponding to executing an iterative crosstalk elimination algorithm includes: determining the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path based on a first distance, a vertical distance, and a second distance; determining a first delay difference based on the distance between the left speaker and the left path and the distance between the left speaker and the right path; determining a second delay difference based on the distance between the right speaker and the left path and the distance between the right speaker and the right path; determining a first delay time based on the first delay difference and a first number of iterations; determining a second delay time based on the second delay difference and the second number of iterations; and determining the delay time corresponding to the iterative crosstalk elimination algorithm according to the first delay time and the second delay time.

[0295] In the embodiment of the present disclosure, the filter compensator can be determined by referring to the above Figure 4 Steps 404 to 406 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0296] Step 2104: Use a filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal.

[0297] In some examples, a filter compensator is used to perform dereverberation processing on a processed vocal signal to obtain a vocal calibration signal, including: performing Fourier transform on the processed vocal signal to obtain a frequency domain signal corresponding to the processed vocal signal; performing Fourier transform on the filter compensator to obtain a frequency domain signal corresponding to the filter compensator; performing inverse fast Fourier transform on the frequency domain signal corresponding to the processed vocal signal and the frequency domain signal corresponding to the filter compensator to obtain a vocal calibration signal.

[0298] In the embodiment of the present disclosure, a filter compensator is used to perform dereverberation processing on the processed human voice signal to obtain a human voice calibration signal. Figure 4 Step 407 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0299] Step 2105: Obtain a stereo signal based on the vocal calibration signal, the processed left surround channel signal, and the processed right surround channel signal.

[0300] In some examples, the stereo signal includes a left channel signal and a right channel signal, and the stereo signal is obtained based on the vocal calibration signal, the processed left surround channel signal, and the processed right surround channel signal, including: obtaining the left channel signal based on the processed left surround channel signal and the vocal calibration signal; obtaining the right channel signal based on the processed right surround channel signal and the vocal calibration signal.

[0301] In the embodiment of the present disclosure, a stereo signal is obtained based on the vocal calibration signal, the processed left surround channel signal and the processed right surround channel signal. Figure 4 Step 408 in the illustrated embodiment will not be described in detail in the embodiment of the present disclosure.

[0302] Corresponding to the method in the above-mentioned embodiment, the embodiment of the present disclosure further provides an audio control device. The audio control device can be applied to an electronic device to implement the method in the above-mentioned embodiment. The functions of the audio control device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

[0303] For example, Fig. 22 A schematic diagram of the structure of an audio control device 2200 is shown, such as Fig. 22 As shown, the audio control device 2200 may include: an acquisition module 2201, a processing module 2202, a determination module 2203, a generation module 2204, etc.

[0304] The acquisition module 2201 is configured as an acquisition module, configured to acquire a left surround channel signal, a right surround channel signal and a human voice signal to be processed;

[0305] The processing module 2202 is configured to process the left surround channel signal and the right surround channel signal by using an iterative crosstalk cancellation algorithm to obtain a processed left surround channel signal and a processed right surround channel signal;

[0306] The determination module 2203 is configured to determine a filter compensator, where the filter compensator is used to remove reverberation in the human voice signal to be processed;

[0307] The processing module 2202 is further configured to perform dereverberation processing on the human voice signal to be processed by using a filter compensator to obtain a human voice calibration signal;

[0308] The generating module 2204 is configured to obtain a stereo signal based on the vocal calibration signal, the processed left surround channel signal and the processed right surround channel signal.

[0309] In conjunction with the second aspect, in a possible implementation, the acquisition module 2201 is further configured to acquire a stereo signal to be processed; the stereo signal to be processed includes a left channel signal to be processed and a right channel signal to be processed;

[0310] The processing module is further configured to process the left channel signal to be processed and the right channel signal to be processed by using an upmixing algorithm to obtain a human voice signal to be processed, a left surround channel signal and a right surround channel signal.

[0311] In combination with the second aspect, in a possible implementation, the determination module 2203 is further configured to determine the delay time corresponding to executing the iterative crosstalk elimination algorithm; determine the de-reverberation time based on the delay time; and determine the filter compensator based on the delay time, the de-reverberation time and the iterative crosstalk elimination algorithm.

[0312] In combination with the second aspect, in a possible implementation, the determination module 2203 is also configured to determine the number of samples corresponding to the delay time based on the iterative crosstalk elimination algorithm; determine the number of reverberation samples based on the signal sampling rate and the de-reverberation time; and determine the filter compensator based on the iterative crosstalk elimination algorithm, the number of samples corresponding to the delay time, and the number of reverberation samples.

[0313] In one possible implementation, the processing module 2202 is also configured to perform Fourier transform on the human voice signal to be processed to obtain a frequency domain signal corresponding to the human voice signal to be processed; perform Fourier transform on the filter compensator to obtain a frequency domain signal corresponding to the filter compensator; perform inverse fast Fourier transform on the frequency domain signal corresponding to the human voice signal to be processed and the frequency domain signal corresponding to the filter compensator to obtain a human voice calibration signal.

[0314] In a possible implementation, the generating module 2204 is further configured to obtain a left channel signal based on the processed left surround channel signal and the vocal calibration signal; and to obtain a right channel signal based on the processed right surround channel signal and the vocal calibration signal.

[0315] In one possible implementation, the processing module 2202 is further configured to determine a target number of iterations corresponding to the iterative crosstalk elimination algorithm; based on the target number of iterations, perform multiple preset processing on the right surround channel signal to obtain multiple first processing results; superimpose the multiple first processing results on the left surround channel signal to obtain a processed left surround channel signal; based on the target number of iterations, perform multiple preset processing on the left surround channel signal to obtain multiple second processing results; superimpose the multiple second processing results on the right surround channel signal to obtain a processed right surround channel signal.

[0316] In combination with the second aspect, in a possible implementation manner, the preset processing includes inversion processing, attenuation processing and delay processing.

[0317] In one possible implementation, the processing module 2202 is further configured to determine a first distance between the left speaker and the right speaker, a vertical distance between the user and the left speaker and the right speaker, and a second distance between the user's left ear and the user's right ear; determine a first ratio and a second ratio based on the first distance, the vertical distance, and the second distance; the first ratio is the ratio of the distance between the left speaker and the left path corresponding to the user's left ear and the distance between the left speaker and the right path corresponding to the user's right ear; the second ratio is the ratio of the distance between the right speaker and the left path and the distance between the right speaker and the right path; determine a first number of iterations based on the first ratio; determine a second number of iterations based on the second ratio; determine a target number of iterations based on the first number of iterations and the second number of iterations.

[0318] In one possible implementation, the determination module 2203 is also configured to determine the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path based on the first distance, the vertical distance, and the second distance; determine the first delay difference based on the distance between the left speaker and the left path and the distance between the left speaker and the right path; determine the second delay difference based on the distance between the right speaker and the left path and the distance between the right speaker and the right path; determine the first delay time based on the first delay difference and the first number of iterations; determine the second delay time based on the second delay difference and the second number of iterations; and determine the delay time corresponding to the iterative crosstalk elimination algorithm based on the first delay time and the second delay time.

[0319] In a possible implementation manner, the multi-channel signal to be processed includes a left surround channel signal, a right surround channel signal, and a human voice signal to be processed.

[0320] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; or all in the form of hardware; or some units can be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware.

[0321] For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in a memory in the form of a program, and called and executed by a certain processing element of the device. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software called by a processing element.

[0322] It should be understood that each step in the above method embodiment provided by the present disclosure can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present disclosure can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0323] In one example, the unit in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0324] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system on a chip SOC.

[0325] In one implementation, the units of the above apparatus implementing the corresponding steps in the above method can be implemented in the form of a processing element scheduling program. For example, the apparatus may include a processing element and a storage element, and the processing element calls the program stored in the storage element to execute the method of the above method embodiment. The storage element may be a storage element on the same chip as the processing element, that is, an on-chip storage element.

[0326] In another implementation, the program for executing the above method may be in a storage element on a different chip from the processing element, i.e., an off-chip storage element. In this case, the processing element calls or loads the program from the off-chip storage element to the on-chip storage element to call and execute the method of the above method embodiment.

[0327] For example, the embodiments of the present disclosure may also provide a device, such as an electronic device, which may include a processor and a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the audio control method of the above embodiment. The memory may be located inside the electronic device or outside the electronic device. And the processor includes one or more.

[0328] In another implementation, the unit of the device implementing each step in the above method may be configured as one or more processing elements, which may be arranged on the corresponding electronic device, and the processing element here may be an integrated circuit, for example: one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of these integrated circuits. These integrated circuits may be integrated together to form a chip.

[0329] For example, the present disclosure also provides a chip, which can be applied to the above electronic device. The chip includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected through lines; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit to implement the method in the above method embodiment.

[0330] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device can implement the above-mentioned audio control method.

[0331] The disclosed embodiments also provide a computer program product, including computer instructions for the electronic device as described above to be executed. When the computer instructions are executed in the electronic device, the electronic device can implement the audio control method as described above. Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0332] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0333] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0334] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0335] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or optical disk and other media that can store program codes.

[0336] For example, the embodiments of the present disclosure may also provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the audio control method in the aforementioned method embodiment.

[0337] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An audio control method, characterized in that the method include: Obtaining a left surround channel signal, a right surround channel signal and a human voice signal to be processed; Using an iterative crosstalk cancellation algorithm, processing the left surround channel signal and the right surround channel signal to obtain a processed left surround channel signal and a processed right surround channel signal; Determining a filter compensator, wherein the filter compensator is used to remove reverberation in the human voice signal to be processed; Using the filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal; A stereo signal is obtained based on the vocal calibration signal, the processed left surround channel signal and the processed right surround channel signal.

2. The method according to claim 1, characterized in that Before obtaining the left surround channel signal, the right surround channel signal and the to-be-processed vocal signal, the method further comprises: Acquire a stereo signal to be processed; the stereo signal to be processed includes the left channel signal to be processed and the right channel signal to be processed; The left channel signal to be processed and the right channel signal to be processed are processed by using an upmixing algorithm to obtain the human voice signal to be processed, the left surround channel signal and the right surround channel signal.

3. The method according to claim 1, characterized in that The determining filter compensator comprises: Determine the delay time corresponding to executing the iterative crosstalk cancellation algorithm; Determining a dereverberation time based on the delay time; A filter compensator is determined based on the delay time, the dereverberation time and the iterative crosstalk cancellation algorithm.

4. The method according to claim 3, characterized in that The determining of the filter compensator based on the delay time, the de-reverberation time and the iterative crosstalk elimination algorithm comprises: Based on the iterative crosstalk cancellation algorithm, determining the number of samples corresponding to the delay time; Determining the number of reverberation samples based on the signal sampling rate and the dereverberation time; A filter compensator is determined based on an iterative crosstalk cancellation algorithm, the number of samples corresponding to the delay time, and the number of reverberation samples.

5. The method according to any one of claims 1 to 4, characterized in that The method of using the filter compensator to perform dereverberation processing on the human voice signal to be processed to obtain a human voice calibration signal includes: Performing Fourier transform on the human voice signal to be processed to obtain a frequency domain signal corresponding to the human voice signal to be processed; Performing Fourier transform on the filter compensator to obtain a frequency domain signal corresponding to the filter compensator; Perform inverse fast Fourier transform on the frequency domain signal corresponding to the human voice signal to be processed and the frequency domain signal corresponding to the filter compensator to obtain a human voice calibration signal.

6. The method according to any one of claims 1 to 5, characterized in that The stereo signal includes a left channel signal and a right channel signal, and the step of obtaining the stereo signal based on the human voice calibration signal, the processed left surround channel signal, and the processed right surround channel signal includes: Obtaining the left channel signal based on the processed left surround channel signal and the vocal calibration signal; The right channel signal is obtained based on the processed right surround channel signal and the vocal calibration signal.

7. The method according to any one of claims 1 to 6, characterized in that The left surround channel signal and the right surround channel signal are processed by an iterative crosstalk cancellation algorithm to obtain a processed left surround channel signal and a processed right surround channel signal, comprising: Determining a target number of iterations corresponding to the iterative crosstalk cancellation algorithm; Based on the target number of iterations, performing a plurality of preset processes on the right surround channel signal to obtain a plurality of first processing results; superimposing the plurality of first processing results on the left surround channel signal to obtain the processed left surround channel signal; Based on the target number of iterations, performing a plurality of preset processes on the left surround channel signal to obtain a plurality of second processing results; The plurality of second processing results are superimposed on the right surround channel signal to obtain the processed right surround channel signal.

8. The method according to claim 7, characterized in that The preset processing includes inversion processing, attenuation processing and delay processing.

9. The method according to claim 7, characterized in that: The determining a target number of iterations corresponding to the iterative crosstalk cancellation algorithm includes: Determine a first distance between a left speaker and a right speaker, a vertical distance between a user and the left speaker and the right speaker, and a second distance between a left ear of the user and a right ear of the user; Based on the first distance, the vertical distance and the second distance, a first ratio and a second ratio are determined; the first ratio is a ratio of a distance from the left speaker to a left path corresponding to the left ear of the user to a distance from the left speaker to a right path corresponding to the right ear of the user; the second ratio is a ratio of a distance from the right speaker to the left path to a distance from the right speaker to the right path; Based on the first ratio, determining the first number of iterations; Based on the second ratio, determining the second number of iterations; A target number of iterations is determined based on the first number of iterations and the second number of iterations.

10. The method according to claim 9, characterized in that The determining of the delay time corresponding to executing the iterative crosstalk cancellation algorithm includes: Based on the first distance, the vertical distance and the second distance, determine the distance between the left speaker and the left path, the distance between the left speaker and the right path, the distance between the right speaker and the left path, and the distance between the right speaker and the right path; Determine a first delay difference based on a distance between the left speaker and the left path and a distance between the left speaker and the right path; Determine a second delay difference based on a distance between the right speaker and the left path and a distance between the right speaker and the right path; Determine a first delay time based on the first delay difference and the first iteration number; Determine a second delay time based on the second delay difference and the second number of iterations; A delay time corresponding to the iterative crosstalk cancellation algorithm is determined according to the first delay time and the second delay time.

11. The method according to any one of claims 1 to 10, characterized in that The multi-channel signal to be processed includes the left surround channel signal, the right surround channel signal and the human voice signal to be processed.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method as claimed in any one of claims 1 to 11.

13. A computer-readable storage medium having computer program instructions stored thereon; characterized in that: When the computer program instructions are executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 11.

Images