Audio processing method, terminal equipment and computer readable medium

Abstract

The invention relates to the technical field of audio processing, in particular to an audio processing method, terminal equipment and a computer readable medium. The method comprises the following steps: determining that a terminal device carried by a user has a first spatial position relative to a plurality of external loudspeaker devices, wherein the plurality of external loudspeaker devices have a first stereo parameter; determining that the spatial position change of the terminal equipment carried by the user relative to the plurality of external loudspeaker equipment is a second spatial position; and adjusting the first stereo parameter to a second stereo parameter corresponding to the second spatial position, wherein the first stereo parameter and the second stereo parameter at least comprise sound channel control parameters played by each loudspeaker device. Therefore, the stereo parameters correspondingly played by each loudspeaker device arranged in the stereo playing scene can be adaptively adjusted according to the face orientation and position change of the user, so that the user can experience a stable and good stereo effect in real time, and the use experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio processing method, a terminal device and a computer-readable medium. Background Art

[0002] Stereo refers to sound with a three-dimensional sense. Stereo audio has a sense of direction and distribution of each sound source, which can improve the clarity of audio information. If the entire system can restore the spatial sense of the original sound to a certain extent from recording to playback, then this kind of playback sound with a certain degree of spatial distribution characteristics such as direction and level is called stereo in audio technology.

[0003] refer to Figure 1 As shown in Figure 1, in a relatively open space such as a smart home space, the ability of a single speaker device to achieve stereo surround sound is limited. Even if there are multiple speaker devices in the space, such as Figure 1 The speakers 101, 102, 103 and 104 shown in the figure can also achieve stereo surround effect by playing audio in the form of distributed speakers. However, since the positions of the speakers 101, 102, 103 and 104 are usually relatively fixed, and the relevant control parameters of the audio playing of each speaker device are also fixed, the user may Figure 1 The stereo effect experienced at position A shown is different from the stereo effect experienced by the user at position B. And the stereo effect experienced by the user facing direction a and facing direction b at position A is also quite different. Some of the above different stereo effects may give the user a better experience, while others may give the user a worse experience, that is, the stereo effect is unstable.

[0004] Current audio processing solutions are also unable to flexibly adjust the sound field of each speaker device in the corresponding space according to changes in the user's position and facial orientation to achieve a good and stable stereo effect. Summary of the invention

[0005] The present application provides an audio processing method, a terminal device and a computer-readable medium, which can adaptively adjust the stereo parameters of the corresponding playback of each speaker device arranged in a stereo playback scene according to the changes in the user's facial orientation and position, so that the user can experience a stable and good stereo effect at any position in the stereo playback scene, which is conducive to improving the user's experience of terminal products such as speaker devices.

[0006] Specifically, in the first aspect, the present application provides an audio processing method, applied to a terminal device, the method comprising: determining that a terminal device carried by a user has a first spatial position relative to multiple external speaker devices, wherein the multiple external speaker devices have first stereo parameters; determining that the spatial position of the terminal device carried by the user relative to the multiple external speaker devices changes to a second spatial position; adjusting the first stereo parameters to second stereo parameters corresponding to the second spatial position, wherein the first stereo parameters and the second stereo parameters at least include channel control parameters played by each speaker device.

[0007] For example, the terminal device may be a mobile phone carried by a user, or an electronic device such as smart glasses worn by a user. The speaker device may be, for example, an electronic device with a speaker such as a speaker, such as a distributed speaker arranged in a smart home space, and a distributed speaker or speaker device in a car audio system, etc. This is not limited here.

[0008] The audio processing method provided in the first aspect above is to detect whether the relative position of external speaker devices such as speakers arranged in the current stereo playback scene has changed through electronic devices such as mobile phones carried by users or smart glasses worn by users, that is, whether the spatial position of the above terminal device relative to multiple external speaker devices has changed from a first spatial position to a second spatial position. Among them, the change in the spatial position can be regarded as a change in the spatial position of each speaker device relative to the user. Furthermore, according to the change in the spatial position, each speaker device can adjust the stereo parameters accordingly to adapt to the changed spatial position to play stereo audio. In this way, users can experience stable and good stereo effects at any position in the stereo playback scene, which is conducive to improving the user experience of terminal products such as speaker devices.

[0009] In a possible implementation of the first aspect above, the multiple speaker devices include a first speaker device, and, when at least one of the following position parameters changes, it is determined that the spatial position of the terminal device carried by the user relative to the multiple speaker devices changes to a second spatial position: it is detected that the direction indicated by the first end of the terminal device changes from a first orientation to a second orientation, and it is detected that the distance of the first speaker device relative to the terminal device changes from a first distance to a second distance.

[0010] For example, the first end may be the front end of a mobile phone, and correspondingly, the direction indicated by the first end may be the direction indicated by the front end of the mobile phone, that is, the front end direction of the mobile phone. The possible implementation of the above is: by detecting whether the front end direction of the terminal device carried by the user changes, for example, from the first direction to the second direction; and detecting whether the distance of each speaker device relative to the terminal device changes, for example, from the first distance to the second distance, to detect whether the spatial position of the terminal device relative to the multiple speaker devices changes to the second spatial position. For another example, the first end may be the front end of the smart glasses in the direction of the user's face when they are worn, and correspondingly, the direction indicated by the first end may be the direction indicated by the front end of the smart glasses, which is not limited here.

[0011] In a possible implementation of the first aspect above, the first stereo parameter includes a first channel control parameter, and adjusting the first stereo parameter to a second stereo parameter corresponding to the second spatial position includes: determining the azimuth of the first speaker device relative to the first end of the terminal device based on the second distance and the second orientation; determining the second channel control parameter based on the determined azimuth and second orientation, and adjusting the first channel control parameter played by the first speaker device to the second channel control parameter.

[0012] That is, the channel control parameters in the stereo parameters adjusted for each speaker device can be determined according to the azimuth angle of each speaker device relative to the first end of the terminal device (e.g., the front end of the mobile phone) after the spatial position changes. For example, according to the second distance of each speaker device relative to the terminal device after the spatial position changes and the front end direction of the terminal device, the determined azimuth angle can determine that the first channel control parameter in the stereo parameters is adjusted to the second channel control parameter.

[0013] In a possible implementation of the first aspect above, the second channel control parameters include left channel control parameters corresponding to the user's left ear, and right channel control parameters corresponding to the user's right ear: and, based on the determined azimuth and second orientation, determining the second channel control parameters, including: regarding the direction indicated by the first end of the terminal device as the user's facial orientation, and, based on the second orientation and the azimuth, determining the proximity of the first speaker device to the user's left ear or right ear, respectively; corresponding to detecting that the first speaker device is close to the user's left ear, using the channel parameters that match the left channel audio in the stereo audio to be played as the adjusted second channel control parameters; corresponding to detecting that the first speaker device is close to the user's right ear, using the channel parameters that match the right channel audio in the stereo audio to be played as the adjusted second channel control parameters.

[0014] In a possible implementation of the first aspect, the first stereo parameter and the second stereo parameter further include: at least one of a sound intensity parameter and a phase parameter corresponding to the audio of each channel.

[0015] That is, after the channel adjustment is performed by determining the adjusted channel control parameters according to the azimuth angle of each speaker device relative to the terminal device in the spatial position, the sound intensity parameters, phase parameters and other audio parameters affecting the stereo effect of the stereo audio played by each speaker device in the corresponding channel can be further adjusted. In this way, it can be ensured that after adjusting the channel control parameters, the sound volume in the stereo effect experienced by the user is stable, and the audio playback can be smoother after the phase is aligned, and there will be no playback freeze problem.

[0016] In a possible implementation of the first aspect above, the first stereo parameter includes a first sound intensity parameter, and adjusting the first stereo parameter to a second stereo parameter corresponding to the second spatial position includes: determining the second sound intensity parameter based on the second distance, and adjusting the first sound intensity parameter in the first stereo parameter to the second sound intensity parameter, wherein the greater the second distance, the greater the sound intensity controlled by the second sound intensity parameter.

[0017] In a possible implementation of the first aspect above, the first end is the end of the terminal device away from the user, and whether the direction indicated by the first end of the terminal device changes is detected in the following manner: acquiring sensor data collected by an inertial sensor in the terminal device; determining that the motion state of the terminal device is a stationary state based on the sensor data; and determining whether the direction indicated by the first end of the terminal device changes based on the direction indicated by the first end of the terminal device, or the direction of the horizontal component of the vector corresponding to the direction indicated by the first end.

[0018] For example, when a user holds a mobile phone, the mobile phone can be placed horizontally. At this time, when the front end of the mobile phone is pointed in the same direction as the user's face, the front end of the mobile phone can be regarded as the user's face direction. For another example, when a user holds a mobile phone, the screen of the mobile phone can be pointed toward the face. At this time, the mobile phone is tilted, and the front end of the mobile phone points to the indicated direction projected in the horizontal direction, that is, the direction of the horizontal component of the vector corresponding to the direction indicated by the first end can be consistent with the user's face direction. At this time, the direction of the horizontal component of the vector corresponding to the direction indicated by the first end can be regarded as the user's face direction.

[0019] In a possible implementation of the first aspect above, whether the distance of the first speaker device relative to the user has changed is detected in the following way: based on a first time difference between a first moment when the first speaker device emits a sound wave signal and a second moment when the terminal device receives the sound wave signal, it is determined whether the distance of the first speaker device relative to the user has changed.

[0020] In a possible implementation of the first aspect above, the terminal device includes a first-end microphone and a second-end microphone, and the azimuth of the first speaker device relative to the first end of the terminal device is determined based on the second distance and the second orientation, including: determining a third distance of the first speaker device relative to the first-end microphone based on a second time difference between a first moment when the first speaker device emits a first sound wave signal and a third moment when the first-end microphone receives the first sound wave signal; determining a fourth distance of the first speaker device relative to the second-end microphone based on a third time difference between a first moment when the first speaker device emits the first sound wave signal and a fourth moment when the second-end microphone receives the first sound wave signal; determining the second distance according to the third distance and the fourth distance, and determining the azimuth based on the second distance and the second orientation.

[0021] In a possible implementation of the first aspect above, the second distance includes any one of the following: a third distance, a fourth distance, and an average of the third distance and the fourth distance.

[0022] In a possible implementation of the first aspect above, the first sound wave signal is an ultrasonic signal, and the method includes: the terminal device receives ultrasonic signals emitted by multiple external speaker devices, wherein the ultrasonic signals include the first sound wave signal emitted by the first speaker device; the terminal device determines that the spatial position relative to the multiple external speaker devices is changed to a second spatial position based on the ultrasonic signal and the direction indicated by the first end; the terminal device sends second stereo parameters adjusted to the second spatial position to each speaker device.

[0023] In a possible implementation of the first aspect above, the speaker device includes at least one of a speaker box, a smart TV, and a car audio device; the terminal device includes at least one of a mobile phone, a tablet computer, a head-mounted device, a watch, and a bracelet.

[0024] In a second aspect, the present application provides a terminal device, comprising: one or more processors; one or more memories; one or more memories storing one or more programs, when one or more programs are executed by one or more processors, the terminal device executes the audio processing method provided in the above-mentioned first aspect and various possible implementations of the first aspect.

[0025] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the audio processing method provided in the first aspect and various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Shown is a schematic diagram of a stereo playback scenario based on distributed speakers provided in an embodiment of the present application.

[0027] Figure 2 The figure is a schematic diagram of the implementation process of an audio processing method provided in Example 1 of the present application.

[0028] Figure 3 The figure is a schematic diagram of the principle of adjusting channel control parameters according to spatial position relationship provided in Example 1 of the present application.

[0029] Figure 4a A schematic diagram of the principle of correcting the sound intensity parameters of the corresponding channel audio played by each speaker according to the distance of each speaker relative to the user provided in Example 1 of the present application.

[0030] Figure 4b This is a schematic diagram of the principle of correcting sound intensity parameters according to distance equalization in a multi-user stereo playback scenario provided in Example 1 of the present application.

[0031] Figure 5 A schematic diagram of the principle of adjusting stereo parameters using an audio processing method provided in Example 1 of the present application.

[0032] Figure 6a A schematic diagram of an implementation process for detecting whether a user controls a mobile phone to be in a flat state provided in Example 1 of the present application.

[0033] Figure 6b A schematic diagram of the x- and y-axis directions used by an IMU sensor of a mobile phone provided in Example 1 of the present application.

[0034] Figure 7a A schematic diagram of an interactive process for measuring distance and azimuth provided in Example 1 of the present application.

[0035] Figure 7b A schematic diagram of the principle of measuring distance and azimuth provided in Example 1 of the present application.

[0036] Figure 8 A schematic diagram of a stereo playback scene in a car cabin provided in Example 2 of the present application.

[0037] Fig. 9 A schematic diagram of the implementation flow of another audio processing method provided in Example 2 of the present application.

[0038] Fig.10 Shown is a structural schematic diagram of a mobile phone 100 provided in an embodiment of the present application.

[0039] Fig.11 The figure shows a software structure block diagram of a mobile phone 100 provided in an embodiment of the present application.

[0040] Fig.12Shown is a schematic structural diagram of a speaker 200 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0042] In order to facilitate those skilled in the art to understand the solutions in the embodiments of the present application, some concepts and terms involved in the embodiments of the present application are explained below.

[0043] (1) Sound field: refers to the area in the medium where sound waves exist. A sound field where the sound source is in a uniform, isotropic medium and the sound can propagate without obstruction or interference is called a free sound field. However, there are often some obstructions and interferences in the actual sound propagation space, which causes the sound waves to be reflected, etc. Therefore, the sound received in the sound field usually includes direct sound and reverberation sound. When different degrees of direct sound and reverberation sound are combined and received by the human ear, different sound effects can be produced, such as the stereo effect of surround sound.

[0044] (2) Sound channel: refers to the independent audio signals collected or played back at different spatial locations when recording or playing sound. Correspondingly, the number of channels usually refers to the number of sound sources when recording sound, or the number of corresponding speakers when playing sound. It can be understood that mono sound gives the human ear a relatively simple auditory experience, while multi-channel sound can bring the human ear a three-dimensional sense of sound surround, that is, to achieve a stereo effect. Mono sound lacks the location of the sound source or speaker, but stereo technology can change this situation.

[0045] The left channel refers to the sound output produced by the electronic device that simulates the hearing range of the human left ear, as opposed to the right channel. The left and right channels of the stereo sound output of the electronic device can broadcast the same or different sounds respectively, producing a stereo sound change effect from left to right or from right to left.

[0046] In the actual process of controlling the speaker device to play the audio signal, since the hearing range of the human left ear tends to receive the low frequency range signal, and the hearing range of the human right ear tends to receive the middle and high frequency range signal. Therefore, it is generally designed to compress the relevant low frequency range signal and then input it into the human left ear after playing it through the audio track. The audio track that outputs the low frequency range signal is usually described as the left channel. Correspondingly, it is generally designed to compress the relevant high and middle frequency range signal and then input it into the human right ear after playing it through the audio track. The audio track that outputs the middle and high frequency range signal is usually described as the right channel.

[0047] Stereo: In addition to playing the audio tracks corresponding to the left and right channels, after deep analysis and stripping of the sound, the center audio and subwoofer audio processing methods are added and then input into the corresponding audio tracks for playback. In this way, the sound playback is clearer and more mellow, and the sound positioning can be accurately judged.

[0048] At present, stereo technology has developed into the widely used four-channel technology. In order to achieve four-channel surround, the four-channel technology stipulates four sound points: front left, front right, rear left, and rear right. The audience is surrounded by these four sound points and can feel the sound from multiple directions. They can get the auditory experience of being in various different environments, giving users a brand new experience.

[0049] (3) Phase: For a wave, phase refers to the position at a specific moment in the waveform cycle, that is, the scale of the peak, trough, or a point in between. Phase is used to describe the measurement of the change in the signal waveform, usually in degrees (°), also known as the phase angle. In the field of audio processing technology, it refers to the time relationship between multiple audio signals containing the same information. For a sine wave, 0° represents the starting point of the wave. The phase at the first peak is 90°. When the phase changes to 180°, the amplitude of the wave becomes negative, and when the phase changes to 360°, a complete waveform cycle is presented.

[0050] As mentioned earlier, in Figure 1 In some of the stereo playback scenarios shown, the current audio processing solutions cannot adapt to the user's position and facial orientation to the sound field of speaker devices such as speakers to achieve a better stereo effect.

[0051] In addition, in some audio systems that use a combination of multiple speakers, such as car audio systems, since the relevant control parameters of the sound field of each speaker are fixed after tuning, it is impossible to flexibly adjust the sound field of the speaker according to the user's position and facial orientation.

[0052] It is understandable that for the stereo sound effects produced by multiple speakers, the channel control parameters played by each speaker, as well as the stereo parameters such as the sound intensity parameters and phase parameters corresponding to the audio of each channel, will affect the stereo sound effects. Therefore, in order to solve the above problems, an embodiment of the present application provides an audio processing method. Specifically, in a stereo playback scenario, the method first determines the spatial position relationship of each speaker device relative to the user, and then adjusts the various stereo parameters that will affect the stereo sound effects according to the spatial position relationship.

[0053] For example, according to the above spatial position relationship, the channel control parameters for controlling the stereo channel audio played by each speaker device can be adjusted. The channel control parameters can control and adjust the corresponding channels played on each speaker device, so that the corresponding channels played by each speaker device can be adjusted accordingly following the changes in the user's facial orientation. In addition, according to the above spatial position relationship, the audio parameters affecting the stereo sense of orientation on each channel can be further adjusted according to the above spatial position relationship, such as the phase parameters affecting the phase difference of the sound heard by the left ear and the right ear and the sound intensity parameters affecting the strength of the sound heard by the left and right ears, so that the adjusted channels can still provide the user with a sense of orientation that is compatible with the above spatial position relationship. At this time, the user can feel that the loudness of the audio played corresponding to the spatial position of each sound source provided to the user can be adaptively adjusted following the changes in the user's position.

[0054] The above spatial position relationship can be comprehensively determined by detecting the user's facial orientation through a handheld terminal and detecting the distance, azimuth, etc. of each speaker device relative to the user through a sound wave signal emitted by a terminal device with a speaker such as a speaker device.

[0055] It can be understood that the above adjustment results of the stereo channels played by each speaker device must conform to the auditory characteristics of the human ear. Since the human right ear is more sensitive to the mid- and high-frequency signals played by the right channel, and the human left ear is more sensitive to the low-frequency signals played by the left channel, the above process of adjusting the channel control parameters according to the spatial position relationship can control the corresponding output of the left channel audio or right channel audio of each speaker device distributed relative to the current user's facial orientation. For example, adjust some speaker devices that originally output left channel audio close to the user's left ear, and switch to the right channel to output right channel audio when they are currently close to the user's right ear; or adjust some speaker devices that originally output right channel audio close to the user's right ear, and switch to the left channel to output left channel audio when they are currently close to the user's left ear.

[0056] It should be noted that the implementation process of adjusting the sound channel can also be described as a process of controlling the rotation of the sound channel in the following examples of the present application. The specific process of controlling the rotation of the sound channel according to the change of the user's facial orientation can be referred to the relevant drawings and descriptions below, which will not be repeated here.

[0057] After completing the channel adjustment, the handheld terminal may further adjust audio control parameters such as the sound intensity and phase of the audio played on each channel according to the above-mentioned spatial position relationship.

[0058] In this way, in some stereo playback scenarios, it is possible to flexibly adjust the sound field of each speaker device according to the user's position and facial orientation, bringing a stable and good stereo effect experience to the user.

[0059] It can be understood that based on the above audio processing method provided by this application, when the user Figure 1 When rotating or moving in the open space shown in the example, the user's handheld device can adjust the channels played by each speaker device and the phase, sound intensity and other audio control parameters of the corresponding audio played in the corresponding channel in real time, thereby flexibly adjusting the sound field of each speaker device.

[0060] In addition, when there are multiple users in a stereo playback scenario, the audio processing method provided by the present application can also determine multiple groups of channel control parameters and audio control parameters corresponding to each group of channels according to the spatial position relationship of each speaker device relative to each user. Furthermore, by performing weight processing on the above multiple groups of channel adjustment parameters and the corresponding audio adjustment parameters, the above multiple users can be balanced to obtain a good stereo effect. In this way, the audio processing method provided by the present application can meet the needs of multiple users to experience stereo effects in the same stereo playback scenario.

[0061] It can be understood that the audio processing method provided in the embodiments of the present application may be applicable to handheld terminals including but not limited to mobile phones, tablet computers, desktops, laptops, handheld computers, netbooks, as well as augmented reality (AR) and virtual reality (VR) devices, smart glasses, watches, bracelets and other wearable devices, smart TVs, and other electronic devices with one or more processors.

[0062] It can be understood that the audio processing method provided in the embodiments of the present application may be applicable to speaker devices or audio playback devices, including but not limited to speakers, car audio equipment, smart voice assistants, AR\VR devices, smart TVs, portable music players, reader devices, and electronic devices with one or more processors and one or more speakers.

[0063] The following describes in detail the specific implementation process of the audio processing method provided by the present application in various stereo playback scenarios in combination with the stereo playback scenarios provided in different embodiments.

[0064] Example 1

[0065] The present application embodiment is based on the above Figure 1 The stereo playback scenario based on the distributed speaker shown in the figure takes the user handheld terminal as the mobile phone 100 and the speaker device as the speaker 200 as an example to introduce in detail the specific implementation process of the audio processing method provided by the present application. Figure 1 In the scenario shown, the speaker 200 can be any device among the speakers 101 to 104. The speakers 101 to 104 and more or fewer speakers can be arranged as distributed devices in the above Figure 1 In the scene shown.

[0066] It is understood that the above-mentioned sound box 200 may include but is not limited to the above-mentioned Figure 1 One or more of the speakers 101 to 104 in the illustrated scene will not be described in detail here. Figure 1 In the scenario shown, when it is necessary to control the sound system formed by speakers 101 to 104 to play stereo sound, the spatial position relationship of each speaker device relative to the user can be determined by interacting with the mobile phone 100, wherein the front end direction of the mobile phone 100 can be determined as the detected facial orientation of the user, without limitation here.

[0067] Specifically, Figure 2 According to an embodiment of the present application, a schematic diagram of an implementation flow of an audio processing method is shown. Figure 2 The illustrated process may involve interaction between the mobile phone 100 and the speaker 200 .

[0068] It should also be stated that the steps in the methods and processes in the embodiments of the present application are numbered for ease of reference, rather than to limit the order of precedence. If there is an order between the steps, the text description shall prevail.

[0069] like Figure 2 As shown, the implementation process may include the following steps:

[0070] 201: The mobile phone 100 establishes a communication connection with the speaker 200.

[0071] Exemplarily, the mobile phone 100 and the speaker 200 can establish a connection through wireless communication methods such as wireless fidelity (WiFi), bluetooth low energy (BLE), and bluetooth basic rate / enhanced data rate (bluetooth BR / EDR), which are not limited here.

[0072] 202: The mobile phone 100 reminds the user to hold the mobile phone and control the front end of the mobile phone to point in the same direction as the face direction.

[0073] Exemplarily, when the user instructs to play stereo audio through the multiple speakers provided by the above-mentioned speaker 200, the mobile phone 100 can obtain the above-mentioned play request based on the communication connection established with the speaker 200, and then display a responsive prompt interface to remind the user to hold the mobile phone and control the front end of the mobile phone to point in the same direction as the direction of his or her face, so as to obtain a better stereo experience during movement. At this time, after viewing the responsive prompt interface, the user can hold the mobile phone 100 and control the direction indicated by the front end of the mobile phone 100 to be consistent with the current direction of his or her face. At this time, the mobile phone 100 can identify whether it is currently in a stationary state based on the built-in inertial measurement unit (IMU) such as accelerometer and gyroscope sensor, and then detect the direction indicated by the front end of the mobile phone 100 (for example Figure 7b As shown; hereinafter referred to as front-end pointing), to confirm whether the user performs the corresponding operation according to the content prompted on the prompt interface.

[0074] It can be understood that the mobile phone 100 being in a stationary state may correspond to a motion state in which the motion data collected by the IMU of the mobile phone 100 is lower than the corresponding preset value, for example, the acceleration data is lower than the corresponding threshold value, etc. At this time, the mobile phone 100 can obtain a more accurate detection result by detecting the front end direction again.

[0075] It is understood that when the user holds the mobile phone 100, the mobile phone 100 can be placed horizontally, or the front end of the mobile phone 100 can be placed at a certain angle to the horizontal plane, and the direction of the component of the front end pointing in the horizontal direction is kept consistent with the user's facial orientation. This is not limited here. At the same time, the mobile phone 100 can obtain touch screen data through a pressure sensor, a touch sensor, etc. to identify whether the user is currently holding the mobile phone.

[0076] Among them, for the example of the user controlling the mobile phone 100 to be placed horizontally, the process of the mobile phone 100 identifying whether it is currently in a horizontal state will be described in detail below in conjunction with the corresponding drawings and will not be repeated here.

[0077] It is understood that in other embodiments, the mobile phone 100 can also remind the user through voice or other means, which is not limited to the display prompt interface in the above example. In addition, the timing of the mobile phone 100 reminding the user is not limited to when the speaker 200 receives an instruction or request to play stereo audio. The mobile phone 100 can also remind the user when a communication connection is established with the speaker 200, or when the IMU sensor detects that the user moves more than a preset distance in the current space, which is not limited here.

[0078] In other embodiments, the mobile phone 100 may also identify the user's current facial orientation through other methods. For example, the mobile phone 100 may predict the user's current facial orientation through the direction of the picture taken by the rear camera, or recognize the user's face through the front camera and then analyze the user's current facial orientation, etc., without limitation here.

[0079] 203: The mobile phone 100 regards the detected front end direction of the mobile phone as the facial direction of the user.

[0080] Exemplarily, after executing the reminder in step 203, the mobile phone 100 may default its front end to point in the same direction as the user's facial orientation, so the front end orientation determined by the mobile phone 100 may be regarded as the user's facial orientation.

[0081] Referring to the relevant description in the above step 202, after reminding the user, the mobile phone 100 can detect whether it is in a stationary state or in a flat state through the IMU, and can detect whether the user is holding the mobile phone through the pressure sensor, touch sensor, etc., and thus infer whether the user controls the front end of the mobile phone 100 to point in the same direction as the face direction according to the reminder. No further details are given here.

[0082] It is understood that the determined facial orientation can be used to perform the following step 207 to determine the spatial position relationship of each speaker relative to the left ear and the right ear of the user. For details, please refer to the relevant description in the following step 207, which will not be repeated here.

[0083] 204: The speaker 200 detects a request to play stereo audio.

[0084] For example, referring to the above Figure 1 In the illustrated scenario 10, the speaker 200 can establish a communication connection with a terminal device such as a mobile phone 100, a large-screen device, a laptop computer, a tablet computer, and receive a playback request for playing stereo audio sent by the above-mentioned devices. It can be understood that the speaker 200 may include a main control unit for controlling each speaker to play the corresponding channel audio, and the main control unit may be, for example, a digital audio signal processor or other processor, which is not limited here.

[0085] In some embodiments, the above Figure 1 The stereo audio playback request of each device in the illustrated scenario 10 may also be sent to the speaker 200 via the mobile phone 100 by default, and no limitation is made here.

[0086] In other embodiments, the main control unit for controlling the speakers provided by the sound box 200 to play stereo audio may also be set on the mobile phone 100. At this time, the mobile phone 100 can also detect the playback request for playing stereo audio based on the main control unit, and then send a control instruction to the speakers provided by the sound box 200 to instruct them to emit sound wave signals, which is not limited in this application.

[0087] It can be understood that the process of the speaker 200 executing step 204 to detect the play request can occur before the mobile phone 100 executes the above steps 202 and 203, or it can occur during the mobile phone 100 executes the above steps 202 and 203, and there is no limitation here.

[0088] 205: The sound box 200 controls the loudspeaker to emit a sound wave signal for measuring distance and angle.

[0089] Exemplarily, the sound wave signal emitted by the above-mentioned speaker may be, for example, an audio signal that is not perceptible to the human body, such as an ultrasonic audio signal. It is understood that the sound wave signal such as ultrasonic audio emitted by the above-mentioned speaker may be used to measure the distance and azimuth of the speaker relative to the receiving device to determine the relative position of the two. The receiving device may be, for example, a microphone on the mobile phone 100, etc., which is not limited here.

[0090] In other embodiments, the speaker device such as the speaker 200 can also control the speaker to continuously emit a sound wave signal for distance and angle measurement within a certain period of time. Accordingly, the mobile phone 100 can synchronously detect whether the device posture meets the relevant conditions for being regarded as the user's facial orientation, and receive the sound wave signal when the conditions are met, and then execute the following step 206. That is, the speaker 200 can also execute this step 205 before or during the execution of the above steps 202 to 203 by the above mobile phone 100, which is not limited here.

[0091] 206: The mobile phone 100 determines the distance and azimuth angle of each speaker relative to the user according to the received sound wave signal.

[0092] For example, after receiving a sound wave signal such as ultrasound emitted by a speaker of the speaker box 200, the mobile phone 100 can calculate the distance of one or more speakers provided by the speaker box relative to the user based on the time difference between the time when the sound wave signal is received and the time when the sound wave signal is emitted. In the embodiment of the present application, the distance can be determined by calculating the relative distance between the front end of the mobile phone 100 and each speaker.

[0093] It can be understood that the distance can be the distance calculated by the time difference between the time when the sound wave signal is received by the microphone at the front end of the mobile phone 100 and the time when the sound wave signal is emitted. In other embodiments, the distance can also be the average of the distances calculated by the time difference when the sound wave signal is received by the microphones at the front end and the back end of the mobile phone 100, or can be the distance calculated by the time difference between the time when the sound wave signal is received by the microphone at the back end of the mobile phone 100 and the time when the sound wave signal is emitted. No limitation is made here.

[0094] Furthermore, the distance calculated based on the time difference between the sound wave signal emitted by the speaker and the time it reaches the microphones at the front and rear ends of the mobile phone 100, combined with the distance between the microphones at the front and rear ends of the mobile phone 100, can also be used to calculate the azimuth of each speaker relative to the front end of the mobile phone 100. In the embodiment of the present application, the azimuth is the azimuth of each speaker relative to the user's facial orientation.

[0095] It can be understood that the front end of the mobile phone 100 in the embodiment of the present application may refer to the end away from the user's body when the mobile phone 100 is held vertically by the user, or the end above the screen away from the user's body when the screen of the mobile phone 100 is opposite to the user's face, without limitation here. Correspondingly, the other end of the mobile phone 100 that keeps the user's face in the same direction and is close to the user's body is the rear end. In other embodiments, the front end of the mobile phone 100 may also be called the top end, and the rear end of the mobile phone 100 may also be called the bottom end, without limitation here.

[0096] Specifically, the process of the mobile phone 100 calculating and determining the distance and azimuth angle of each speaker relative to the user based on the received sound wave signal will be described in detail below in conjunction with the corresponding drawings and will not be repeated here.

[0097] 207: The mobile phone 100 determines the spatial position relationship of each speaker relative to the user according to the user's facial orientation, distance and azimuth.

[0098] Exemplarily, the mobile phone 100 can determine the spatial position relationship of each speaker relative to the user by combining the facial orientation determined in the above step 203 and the distance and azimuth of each speaker relative to the user determined in the above step 206. It can be understood that in the embodiment of the present application, in order to simplify the calculation, the above spatial position relationship is approximately the spatial position relationship of the speaker on the left side of the user relative to the left ear of the user and the spatial position relationship of the speaker on the right side of the user relative to the right ear of the user.

[0099] It can be understood that when the user's left ear and right ear receive the left channel audio and right channel audio and the corresponding reverberation audio respectively, they can experience stereo sound effects. The stereo sound effects obtained by adjusting the corresponding processing of the speakers according to the above-mentioned spatial position relationship can also meet the user's experience needs for stereo sound effects to a certain extent.

[0100] 208: The mobile phone 100 determines, according to the determined spatial position relationship, channel control parameters for adjusting the channel audio played by each speaker.

[0101] Exemplarily, the mobile phone 100 can adjust the channel audio currently played by each speaker to the channel audio suitable for the current facial orientation in the above-determined spatial position relationship based on the spatial position relationship of each speaker relative to the user determined in the above-mentioned step 207, for example, the above-detected rotation angle of the user's current facial orientation relative to the initial facial orientation.

[0102] As an example, Figure 3 According to an embodiment of the present application, a schematic diagram of the principle of adjusting channel control parameters according to spatial position relationship is shown.

[0103] For example, multiple speakers provided by multiple sound boxes 200 are distributed around the user, and the initial spatial position relationship of each speaker relative to the human ear can refer to Figure 3 At this time, refer to Figure 3 As shown in (a), speaker A plays the audio of channel ①, speaker B plays the audio of channel ②, speaker C plays the audio of channel ③, and speaker D plays the audio of channel ④. Among them, the audio played by the above-mentioned channels ① and ② can be the stereo audio obtained by mixing the treble and mid-range audio signals on the right channel audio track into the center audio and reverberating. In other embodiments, channel ① can also be called the right front channel, and channel ② can also be called the right rear channel. And the audio played by the above-mentioned channels ③ and ④ can be the stereo audio obtained by mixing the low audio signal on the left channel audio track into the heavy bass and reverberating. In other embodiments, channel ③ can also be called the left rear channel, and channel ④ can also be called the left front channel. This application is not limited here.

[0104] Continue to refer Figure 3 As shown in the change from (a) to (b) in FIG. 1 , when the user rotates from the facial orientation shown in (a) to the facial orientation shown in (b), the user's facial orientation rotates 90°. Figure 3The corresponding channels played by each speaker shown in (a) are no longer suitable for the user's left and right ears to hear stereo audio. At this time, based on the audio processing method provided by the present application, the mobile phone 100 can determine the channel control parameters for adjusting the channel audio played by each speaker according to the spatial position relationship of each speaker relative to the human ear after the user rotates. The channel control parameters can control the playback of each speaker. Figure 3 The channel audio shown in (c) is shown in Figure 1. Figure 3 As shown in (c), at this time, speaker D and speaker A can play the audio of channel ① and channel ② respectively, and speaker B and speaker D can play the audio of channel ③ and channel ④ respectively.

[0105] It can be understood that when the user is guided to hold the phone and the front end of the phone is controlled to be consistent with the facial orientation, the phone 100 can regard the rotation angle of the detected front end as the rotation angle of the user's facial orientation. In this case, when the user walks or turns around holding the phone 100, the phone 100 can regard the angle change caused by the change in the detected front end orientation, that is, the rotation angle corresponding to the front end orientation, as the rotation angle of the user's facial orientation.

[0106] In other embodiments, the mobile phone 100 may also identify the user's current facial orientation change by other means, such as detection and identification by the rear camera or front camera of the mobile phone 100, etc. For details, please refer to the relevant description in the above step 202, which will not be repeated here.

[0107] Since a slight turn of the user may also cause a large angle change in the facial orientation, when guiding the user to hold the phone and control the front end direction of the phone to be consistent with the facial orientation, the angle change corresponding to the front end direction detected by the phone 100 exceeding 45° or other preset angles can be preset as a trigger adjustment condition. When this condition is met, the phone 100 executes the above Figure 3 The illustrated process of adjusting channel control parameters according to spatial position relationship.

[0108] It can be understood that in a stereo playback scenario for multiple users, the above Figure 3 The channel control parameter adjustment process shown in the figure can be based on equalizing the channel control parameters determined for each user, for example, taking the mode value of the channel parameters determined corresponding to the rotation or movement of each user as the adjusted channel control parameters. For the specific equalization process, please refer to the following Figure 4b The exemplified equalization process of the sound intensity parameters will not be described in detail here.

[0109] 209: The mobile phone 100 determines, according to the distance between the speakers, audio parameters for adjusting the corresponding channel audio played by each speaker.

[0110] Exemplarily, the above audio parameters may be related parameters that can affect the sense of orientation of the stereo sound effects heard by the user, such as sound intensity parameters, phase parameters, etc. Among them, the sound intensity parameters can be used to control the loudness of the left and right channel audio played by each speaker. The phase parameter can be used to control the phase difference between the left and right channel audio played by each speaker and reaching the left and right ears of the user respectively. The size of the phase difference can affect the sense of orientation of the stereo sound effects heard by the user, so that the user can clearly feel the position of each speaker as the sound source.

[0111] It can be understood that after adjusting the channels played corresponding to each speaker, in order not to affect the continuity of the stereo audio playback, it is necessary to adjust the waveform phase of the stereo audio played corresponding to the adjusted channel forward or backward by a certain amount. Therefore, in some embodiments, it is also necessary to adjust the phase of the audio played corresponding to the adjusted channel, that is, to determine the corresponding phase parameters to adjust the corresponding channel to play the stereo audio.

[0112] Since the human ear's experience of sound intensity and left-right channel audio phase difference is related to distance, the mobile phone 100 can adjust the above-mentioned audio parameters such as sound intensity parameters and phase parameters according to the distance of each speaker relative to the user.

[0113] Take the sound intensity parameter as an example, refer to Figure 4a As shown, after adjusting the channel audio played by each speaker, the sound intensity L1 of the corresponding channel audio played by speaker A can be preset as a reference according to the distance of each speaker relative to the user, and the sound intensity of the corresponding channel audio played by other speakers can be corrected using the following formula (1):

[0114] Ln=L1*(d1 / dn) 2 (1)

[0115] Wherein, L1 is the sound intensity of the corresponding channel audio played by speaker A, in dB; d1 is the distance of speaker A relative to the user, dn is the distance of other speakers relative to the user, and n=2, 3 or 4. For example, refer to Figure 4a As shown, d2 is the distance between the speaker B and the user, d3 is the distance between the speaker C and the user, and d4 is the distance between the speaker D and the user.

[0116] In some embodiments, if there are multiple users in a stereo playback scenario, the sound intensity parameters of the corresponding channel audio played by each speaker can be determined in a balanced manner for the multiple users.

[0117] refer to Figure 4b As shown, for example, for user 1 and user 2, the sound intensity correction values ​​corresponding to each speaker can be calculated using the following formulas (2) and (3) respectively.

[0118] User 1: Ln = L1*(d1 / dn) 2 (2)

[0119] User 2: Ln' = L1*(d1' / dn') 2 (3)

[0120] Among them, L1 can be the sound intensity of the corresponding channel audio played by speaker A at a distance relative to user 1; Ln is the sound intensity correction value of the corresponding channel audio played by any one of speakers B to D relative to user 1; Ln' is the sound intensity correction value of the corresponding channel audio played by any one of speakers A to D relative to user 2.

[0121] The sound intensity correction value calculated based on the above formulas (2) and (3) can be further substituted into the following formula (4) to calculate the average value, which is used as the final equalized sound intensity parameter:

[0122] Lx=L1 / 2*(d1' / dx') 2 +L1 / 2*(d1 / dn) 2 (4)

[0123] In some other embodiments, L1 / 2 in the above equalization calculation formula (4) may also be L1 multiplied by a corresponding weight coefficient, such as L1*w1, L1*w2, etc., where w1+w2=1. This is not limited here.

[0124] It can be understood that the channel control parameters determined in the above step 208 and the audio parameters such as the sound intensity parameters and phase parameters determined in the above step 209 can be uniformly described as stereo parameters for each speaker to play the stereo audio of the corresponding channel in some embodiments. In other embodiments, the stereo parameters can also include other parameters that can affect the stereo sound effects experienced by the user's two ears, and the parameters can be parameters different from the above channel control parameters and audio parameters such as the sound intensity parameters and phase parameters, and the present application does not limit this.

[0125] 210 : The mobile phone 100 sends the adjusted channel control parameters and audio parameters to the speaker 200 .

[0126] Exemplarily, after executing the above step 208 to determine the channel control parameters, and executing the above step 209 to determine the audio parameters such as sound intensity parameters and phase parameters, the mobile phone 100 can package the adjusted stereo parameters into data and send them to the speaker 200, so that the speaker 200 can adjust the effect of the stereo audio played by the corresponding speaker.

[0127] It is understandable that the mobile phone 100 may simultaneously execute the above step 208 and the above step 209 to determine the above-mentioned adjusted stereo parameters, or may execute the above step 208 and the above step 209 successively, and the present application does not make any limitation thereto.

[0128] 211: The speaker 200 controls the speaker to play the corresponding channel audio according to the received parameter adjustment result.

[0129] Exemplarily, the main control unit of the speaker 200 can control each speaker to play the audio of the corresponding channel using the adjusted stereo parameters based on the received adjustment results of various stereo parameters, such as the adjusted channel control parameters and audio parameters such as sound intensity parameters and phase parameters.

[0130] It can be understood that the audio processing method for stereo audio implemented in the above steps 201 to 211 can adjust the parameters related to stereo sound effects to adapt to changes in user position and facial orientation, and can also be referred to in Figure 5 As shown, speakers A to D play stereo audio of channels ① to ④ respectively when the user is in a certain position and facing the user.

[0131] When a change in the user's position or facial orientation is detected, the main control unit controls each speaker to emit an ultrasonic audio signal, such as Figure 5 The ultrasonic waves A to D are emitted by the speakers A to D shown in the figure. At this time, the distance of each speaker relative to the user can be calculated based on the ultrasonic ranging principle. Further, the azimuth of each speaker relative to the user's position and facial orientation can be calculated by combining the ranging results and the position distribution of microphones on terminal devices such as mobile phones that receive ultrasonic audio signals.

[0132] In this way, the channel control parameters and audio parameters can be adjusted according to the distance and angle measurement results and the front direction of the handheld device. Figure 5 In the execution process of "adjusting parameters I" shown in the figure, the main control unit can first control and adjust the channel control parameters corresponding to speakers A to D. For example, adjust the parameters corresponding to channel ① originally played by speaker A to the parameters corresponding to channel ②, adjust the parameters corresponding to channel ② originally played by speaker B to the parameters corresponding to channel ③, adjust the parameters corresponding to channel ③ originally played by speaker C to the parameters corresponding to channel ④, adjust the parameters corresponding to channel ④ originally played by speaker D to the parameters corresponding to channel ①, etc. The above is completed. Figure 3 Then, refer to the channel control parameter adjustment process shown in (a) to (c) in FIG. Figure 5In the execution process of "adjusting parameters II" shown in the figure, the main control unit can then control each speaker to adjust the audio parameters of the corresponding channel playing stereo audio, such as sound intensity parameters, phase parameters, etc.

[0133] Finally, based on the audio processing method provided in this application, in some stereo playback scenarios in open spaces, the user's ears can also receive the stereo audio played by each speaker controlled based on the parameter adjustment results, allowing the user to experience a good stereo effect.

[0134] It can be understood that the above-mentioned main control unit and each speaker can be set in the same or different speaker devices, or the main control unit can be set in a handheld terminal such as a mobile phone, and each speaker can be set in a different speaker device, and there is no limitation here. The channel parameters ① to ② corresponding to the playback of each speaker can be synchronously transmitted based on the communication connection between the devices, and the channel control parameters and audio parameter decision mechanism for controlling each speaker to play the corresponding channel stereo audio can be set in the main control unit. For example, the relevant instructions of the above-mentioned decision mechanism can be set in the corresponding memory or register for the main control unit to read and control the execution, etc., which will not be repeated here.

[0135] Combine the following Figure 6a The process shown in detail introduces the specific implementation process of the mobile phone 100 detecting whether it is in a horizontal state in an example in which the user controls the mobile phone 100 to be in a horizontal state according to a prompt involved in the above step 202.

[0136] Figure 6a According to an embodiment of the present application, a schematic diagram of an implementation process of detecting whether a mobile phone is in a flat state is shown.

[0137] It can be understood that in a possible example involved in the above step 202, the user can hold the mobile phone and control the mobile phone to be in a horizontal state according to the prompt interface or voice prompt displayed by the mobile phone 100. At this time, the mobile phone 100 can execute the process shown in Figure 6 to detect whether it is in a horizontal state. Then, the mobile phone 100 can continue to execute the above step 203 based on the detection result, and regard the detected front end direction of the mobile phone as the user's facial orientation.

[0138] In some other embodiments, after the mobile phone 100 executes the above 202 to remind the user, it may not execute Figure 6a The detection process shown in the figure determines whether it is in a stationary state based on the IMU, and then the direction of the horizontal component of the vector corresponding to the front end pointing to the direction of the user's face is regarded as the direction of the user's face, etc., without limitation here.

[0139] Understandably, Figure 6aThe execution entity of each step of the illustrated process can be the mobile phone 100 or other terminal devices held by the user. The execution entity of each step will not be repeatedly described below when introducing each step.

[0140] like Figure 6a As shown, the detection process may include the following steps:

[0141] 601: Obtain inertial sensor data within a preset time period.

[0142] Exemplarily, the preset duration may be, for example, 2s, 1s, or other reasonable preset values. The inertial sensor data (IMU data for short) may be, for example, relevant data collected by an acceleration sensor and / or a gyroscope sensor provided on the mobile phone 100. The IMU data may be used to determine whether the mobile phone 100 is in a horizontal state within the current spatial range.

[0143] In other embodiments, the mobile phone 100 may also detect touch screen data generated when the user touches the edge of the screen while holding the mobile phone, as one of the reference data for detecting whether the user is currently holding the mobile phone. This application does not impose any restrictions on this.

[0144] 602: Determine whether the device is currently in a horizontal state.

[0145] If the result of the judgment is yes, that is, when the mobile phone 100 detects that it is in a horizontal state, the mobile phone 100 determines that the front end is pointing in the same direction as the user's face orientation, then the following step 603 may be executed to directly regard the detected front end orientation of the mobile phone as the user's face orientation;

[0146] If the judgment result is no, that is, the mobile phone 100 is not in a flat state, the mobile phone 100 can return to the above step 601 to continue to obtain IMU data.

[0147] For example, the mobile phone 100 can obtain IMU data, such as the reading or detection value of the acceleration sensor, and then calculate the components of the IMU data corresponding to each coordinate axis, and compare them with the corresponding threshold value, so as to determine whether the mobile phone 100 is currently in a horizontal state. For example, the mobile phone 100 can perform a judgment based on the component data of the obtained IMU data corresponding to the x, y, and z axes, and substitute them into the following relationship (5):

[0148]

[0149] Where abs(acc_x) represents the absolute value of the acceleration component in the x-axis direction, which can be referred to as Figure 6b abs(acc_y) represents the absolute value of the acceleration component in the y-axis direction, which can be referred to as Figure 6bThe y-axis direction is shown; abs(acc_z) represents the absolute value of the acceleration component in the z-axis direction; acc_z represents the acceleration component in the z-axis direction; std(acc_x) represents the standard deviation of the acceleration component in the x-axis direction.

[0150] It can be understood that when the mobile phone 100 is in a flat state, the acceleration data collected by its acceleration sensor has a negative component on the z-axis, while the components on the x-axis and y-axis are usually zero or close to zero. Therefore, the first threshold in the above relationship (5) can be a preset value close to zero. No limitation is made here.

[0151] At the same time, when the mobile phone 100 is in a stable state or in a stationary state, the acceleration data collected by its acceleration sensor has a smaller component change on the x-axis or y-axis, and the corresponding standard deviation or variance is also smaller. Therefore, the second threshold in the above relationship (5) can also be a preset value close to zero. No limitation is made here.

[0152] 603: The detected front direction of the mobile phone is regarded as the user's facial direction.

[0153] The specific execution process can refer to the relevant description in the above step 203, which will not be repeated here.

[0154] Combine the following Figure 7a to Figure 7b , the specific implementation process of the speaker 200 and the mobile phone 100 interacting to determine the distance and azimuth of each speaker provided by the speaker relative to the mobile phone in the above steps 205 to 206 is described in detail. It can be understood that the distance and azimuth can also be considered as the distance of each speaker relative to the user and the azimuth relative to the user's facial orientation.

[0155] As an example, Figure 7a According to an embodiment of the present application, a schematic diagram of an interactive process for measuring distance and azimuth is shown. The interactive process involves interaction between the mobile phone 100 and the speaker 200. In other embodiments, the mobile phone 100 and the speaker 200 can also achieve the purpose of measuring the distance and azimuth between the two through other interactive methods and processes, which are not limited here.

[0156] Specifically, Figure 7a As shown, the process may include the following steps:

[0157] 701: The mobile phone 100 and the speaker 200 perform time synchronization.

[0158] For example, the main control unit for controlling the speaker to play audio on the mobile phone 100 and the speaker 200 can synchronize the system time after establishing a communication connection. The synchronization process, for example, can be that the mobile phone 100 transfers its own system time to the system of the speaker 200, or the mobile phone 100 and the speaker 200 are both synchronized to the universal coordinated time (UTC), such as Beijing time, etc., which is not limited here.

[0159] It can be understood that after time synchronization, the mobile phone 100 and the speaker 200, or the mobile phone 100 and the main control unit, can have the same system time, or the system time error between the two is within a preset range, for example, less than 10ms. At this time, the mobile phone 100 and the speaker 200 recording the same timestamp at any time can be considered to be the same, and the timestamp can be, for example, the time record corresponding to the moment when the speaker 200 controls each speaker to emit an ultrasonic audio signal.

[0160] 702: The sound box 200 controls the speaker to emit a sound wave signal.

[0161] The specific process of controlling the speaker to emit a sound wave signal may refer to the relevant description in the above step 205, which will not be described in detail here.

[0162] 703: The speaker 200 synchronously sends the first time information of the sound wave signal to the mobile phone 100.

[0163] Exemplarily, the first time information may be a timestamp recorded by the system when the sound box 200 controls the speaker to emit an ultrasonic audio signal, and the corresponding indication may be the moment when the speaker emits the ultrasonic audio signal.

[0164] 704: The mobile phone 100 receives the sound wave signal and the first time information, and records the second time information when the front-end and rear-end microphones respectively receive the sound wave signal.

[0165] Exemplarily, the mobile phone 100 may respectively record the second time information corresponding to when the front-end and rear-end microphones receive the sound wave signal. Figure 7b As shown, the distance between the microphone (mic) 101 at the front end of the mobile phone 100 and the microphone (mic) 102 at the rear end may be c, and the distance c is usually a fixed value.

[0166] 705: The mobile phone 100 calculates the distances a and b of each speaker relative to the front and rear microphones of the mobile phone according to the time difference between the first time information and the second time information.

[0167] For example, continue to refer to the above Figure 7bAs shown, the first time difference between the second time information recorded when the sound wave signal emitted by the speaker reaches the microphone 101 and the aforementioned first time information can be used to calculate the corresponding transmission distance a; the second time difference between the second time information recorded when the sound wave signal emitted by the speaker reaches the microphone 102 and the aforementioned first time information can be used to calculate the corresponding transmission distance b. Among them, the mobile phone 100 can regard the calculated distance a or distance b as the distance of the speaker relative to the user, or the average of distance a and distance b as the distance of the speaker relative to the user, which is not limited here.

[0168] 706: The mobile phone 100 calculates and determines the azimuth angle of each speaker relative to the user according to the calculated distances a and b and the orientation of the front end of the mobile phone.

[0169] For example, continue to refer to the above Figure 7b As shown in FIG. 1 , since the front end of the mobile phone 100 is in the same direction as the user's face and the mobile phone 100 is in a flat position, the mobile phone 100 can use its front end direction as a reference, and in combination with the distances a and b calculated in step 705 above, the azimuth of the speaker relative to the microphone 101 at the front end of the mobile phone 100 and the front end direction can be calculated. Figure 7b The angle β shown can be considered as the azimuth of the speaker relative to the orientation of the user's face.

[0170] In other embodiments, the mobile phone 100 may also calculate the azimuth angle of the speaker relative to the microphone 102 at the rear end of the mobile phone 100 based on the distances a and b and the front direction of the mobile phone, referring to Figure 7b The γ angle shown is regarded as the azimuth angle of the speaker relative to the user's face direction. No limitation is made here.

[0171] In other embodiments, the mobile phone 100 can also calculate the azimuth of the speaker relative to the front microphone or the rear microphone of the mobile phone based on the above distances a, b, and the distance c between the front microphone and the rear microphone of the mobile phone, and regard the azimuth as the azimuth of the speaker relative to the user. This is not limited here. It can be understood that in the stereo playback scenario of the smart home space illustrated in Example 1 of the present application, the above is implemented through the interaction between the mobile phone 100 and the speaker 200 or the home audio system in the home space. Figures 2 to 7b The corresponding audio processing method can realize that when the user turns his body or moves, the mobile phone will also rotate or move accordingly. The stereo parameters such as the channels played by each speaker can be flexibly adjusted according to the result of the user's rotation of the body or movement, so that the user can feel a better stereo effect at any position in the home space.

[0172] In addition, for multiple users, the audio processing method provided in the embodiment of the present application can also perform equalization processing on the corresponding stereo parameters to meet the needs of multiple users to experience the stereo effect at the same time. In this way, the user experience of the speaker device or system such as a speaker box can be improved.

[0173] The following introduces the specific implementation process of the audio processing method provided by the present application in another application scenario in combination with the application scenario provided by another embodiment 2.

[0174] Example 2

[0175] Figure 8 According to an embodiment of the present application, another schematic diagram of a stereo playback scenario is shown.

[0176] like Figure 8 As shown, the scene 20 may include a plurality of speakers and a central control device 800 disposed in a car cabin. The central control device 800 may control some or all of the plurality of speakers to play stereo audio. It is understood that in some embodiments, the number of the plurality of speakers disposed in the car cabin may be any number, such as 4, 6, 8, 9, etc.

[0177] As an example, Figure 8 The speakers provided in the scene 20 shown may include, for example, speakers A, B, C, D or more. Figure 8 The corresponding locations of the speakers in the example are described based on the facial orientation of the user driving or riding in the car facing the front of the car. For example, speakers A and B can be set on the door on the right ear side of the user to play low-frequency signals. Correspondingly, speakers C and D can be set on the door on the left ear side of the user to play mid- and high-frequency signals. For example, in other embodiments, the locations of the speakers in the car cabin may be different from Figure 8 The positions shown are not limiting.

[0178] It is understood that the central control device 800 may be, for example, an electronic device such as a vehicle-mounted computer. The user's facial orientation and the distance and azimuth of each speaker relative to the user may be detected by a head-mounted device worn by the user. The head-mounted device may include, but is not limited to, smart glasses, smart headphones, and VR / AR devices, etc., which are not limited here.

[0179] Based on the above Figure 8 The scene 20 shown, Fig. 9 According to an embodiment of the present application, another implementation flow diagram of an audio processing method is shown. Figure 8The process shown may involve the interaction between the head mounted device and the central control device 800. Figure 8 When describing each step of the process shown, the smart glasses 900 will be taken as an example of a head-mounted device.

[0180] It can be understood that in the embodiments of the present application Fig. 9 The implementation process of the audio processing method applied to the car cockpit scene is as follows: Figure 2 The audio processing method used in the smart home scenario can more accurately detect the user's facial orientation based on the head-mounted device, and it is also more accurate to regard the distance of each speaker detected relative to the head-mounted device worn by the user as the distance of each speaker relative to the user. Fig. 9 The example implementation process of the audio processing method applied to the car cockpit scene can also more accurately adjust the channel control parameters corresponding to each speaker and stereo parameters such as sound intensity parameters and phase parameters.

[0181] Specifically, Fig. 9 As shown, the interaction process includes the following steps:

[0182] 901: The smart glasses 900 establish a communication connection with the central control device 800.

[0183] Exemplarily, the smart glasses 900 may establish a communication connection such as Bluetooth with the vehicle-mounted central control device 800. The specific process of establishing the communication connection may refer to the relevant description in step 201 in the above embodiment 1, and will not be described in detail here.

[0184] 902: The smart glasses 900 obtain the collected inertial sensor data and / or image data.

[0185] For example, while the user is wearing the smart glasses 900, the smart glasses 900 can collect and obtain IMU data based on its own acceleration sensor, gyroscope sensor, etc., and can also collect and obtain image data in front of the glasses based on its own camera.

[0186] 903: The smart glasses 900 determine the current user's facial orientation based on the inertial sensor data and / or the image data.

[0187] For example, while the user is wearing the smart glasses 900, the smart glasses 900 may identify the user's facial orientation based on the IMU data and / or image data collected and acquired. The process of the smart glasses 900 determining the user's facial orientation based on the IMU data may refer to steps 202 to 203 and Figure 6aThe descriptions of the steps in the process shown, and the process of determining the user's facial orientation based on the collected image data, can be implemented with reference to current image recognition related technologies, and will not be elaborated in this application.

[0188] 904: The central control device 800 detects a demand for playing stereo audio.

[0189] It can be understood that the process of the central control device 800 executing step 904 to detect the playback request can occur before the smart glasses 900 execute the above steps 902 and 903, or it can occur during the smart glasses 900 execute the above steps 902 and 903, and there is no limitation here.

[0190] For the specific process of the central control device 800 executing step 904 to detect the need to play stereo audio, reference may be made to the relevant description in step 204 in the above embodiment 1, which will not be described in detail here.

[0191] 905: The central control device 800 controls each speaker to emit a sound wave signal for measuring distance and angle.

[0192] It is understandable that the central control device 800 can also control each speaker to continuously emit a sound wave signal for distance and angle measurement within a certain period of time. Accordingly, the smart glasses 900 can synchronously detect whether the device posture meets the relevant conditions for being regarded as the user's facial orientation, and receive the sound wave signal when the conditions are met, and then execute the following step 206. That is, the central control device 800 can also execute this step 905 before or during the execution of the above-mentioned smart glasses 900 by the above-mentioned steps 902 to 903, and there is no limitation here.

[0193] The specific process of the central control device 800 executing this step 905 to control each speaker to emit a sound wave signal can refer to the relevant description in step 205 in the above embodiment 1, and will not be repeated here.

[0194] 906: The smart glasses 900 determine the distance and azimuth angle of each speaker relative to the user based on the received sound wave signal.

[0195] 907: The smart glasses 900 determine the spatial position relationship of each speaker relative to the user according to the user's facial orientation, distance and azimuth.

[0196] 908: The smart glasses 900 determine channel control parameters for adjusting the channel audio played by each speaker according to the determined spatial position relationship.

[0197] 909: The smart glasses 900 determine audio parameters for adjusting the audio of the corresponding channel played by each speaker according to the distance between the speakers.

[0198] 910: The smart glasses 900 send the adjusted channel control parameters and audio parameters to the central control device 800.

[0199] 911: The central control device 800 controls the speaker to play the corresponding channel audio according to the received parameter adjustment result.

[0200] The specific execution process of the above steps 906 to 911 is the same as the specific execution process of steps 206 to 211 in the above embodiment 1. For details, please refer to the relevant description in the above embodiment 1, and no further details will be given here.

[0201] It can be understood that in the stereo playback scenario of the car cabin space illustrated in Example 2 of the present application, the above-mentioned audio system is implemented through the interaction between the head-mounted device such as the smart glasses 900 worn by the user and the cabin audio system composed of the speakers controlled by the central control device 800 in the car cabin. Figures 8 to 9 The corresponding audio processing method can flexibly adjust the stereo parameters affecting the stereo sound effect, such as the sound channels played by each speaker, when the user turns his body or changes seats in the car cabin, so that the user can feel a better stereo effect in any seat in the car cabin.

[0202] If there are multiple users in the car cabin, the audio processing method provided in the embodiment of the present application can also perform equalization processing on the channel control parameters and the like corresponding to the detection and determination of the head-mounted devices worn by each user. The equalization processing process can refer to the above Figure 4b The above and related descriptions will not be repeated here.

[0203] Furthermore, the audio processing method provided in the embodiment of the present application can adjust the stereo parameters of the corresponding channel audio played by each speaker according to the channel control parameters obtained by equalization processing, etc. In this way, the needs of multiple users in the same car cabin to experience the stereo effect at the same time can be met, which is conducive to improving the user experience of the car audio system constructed based on multiple speakers in the car cabin.

[0204] It can be understood that in the stereo playback scenarios respectively illustrated in the above-mentioned Embodiment 1 or Embodiment 2, if the user is wearing a head-mounted device and holding a terminal device such as a mobile phone, the audio processing method provided in this application can decide the electronic device that performs the detection task first from the head-mounted device and the mobile phone. For example, at this time, the head-mounted device can be used to detect and determine the spatial position relationship of each speaker device relative to the user, and then determine the corresponding channel control parameters, etc., as the stereo parameters for adjusting each speaker device to play the corresponding channel stereo audio. When it is detected that the head-mounted device is not in a wearing state, the mobile phone can be used to detect and determine the spatial position relationship of each speaker device relative to the user, and then determine the corresponding channel control parameters, etc., as the stereo parameters for adjusting each speaker device to play the corresponding channel stereo audio. There is no limitation here.

[0205] Fig.10 FIG. 1 is a schematic structural diagram of a mobile phone 100 according to an embodiment.

[0206] like Fig.10 As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0207] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0208] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0209] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0210] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0211] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0212] Among them, the GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0213] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the mobile phone 100, and can also be used to transfer data between the mobile phone 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0214] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0215] The charging management module 140 is used to receive charging input from a charger. The charging management module 140 is used to connect the power management module 141 and the battery 142 , and the power management module 141 is connected to the processor 110 .

[0216] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0217] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0218] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the mobile phone 100 .

[0219] The wireless communication module 160 can provide wireless communication solutions for application in the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0220] The mobile phone 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0221] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the mobile phone 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0222] The mobile phone 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0223] The ISP is used to process data fed back by the camera 193 .

[0224] The camera 193 is used to capture still images or videos. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0225] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, digital audio signal processors can process digital audio signals and adjust audio parameters related to the output stereo sound effects, such as sound intensity parameters, phase parameters, etc.

[0226] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, audio and video files are stored in the external memory card.

[0227] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0228] The mobile phone 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0229] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0230] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The mobile phone 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0231] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the mobile phone 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0232] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The mobile phone 100 can be provided with at least one microphone 170C. In other embodiments, the mobile phone 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the mobile phone 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0233] The earphone jack 170D is used to connect a wired earphone.

[0234] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The mobile phone 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the mobile phone 100 detects the corresponding touch screen information based on the pressure sensor 180A to determine whether the user is holding the mobile phone. The mobile phone 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0235] The gyroscope sensor 180B can be used to determine the motion posture of the mobile phone 100. In some embodiments, the angular velocity of the mobile phone 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. In the embodiment of the present application, it can be determined whether the mobile phone is in a horizontal state based on the angular velocity data collected by the gyroscope sensor 180B.

[0236] The acceleration sensor 180E can detect the magnitude of the acceleration of the mobile phone 100 in various directions (generally three axes). When the mobile phone 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device. In the embodiment of the present application, based on the acceleration data collected by the acceleration sensor 180E, it can be determined whether the mobile phone is in a horizontal state.

[0237] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations on or near it. In the embodiment of the present application, the mobile phone 100 can also determine whether the user is holding the mobile phone based on the touch screen information detected by the touch sensor 180K at the edge of the screen.

[0238] The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key or a touch key. The mobile phone 100 can receive key input and generate key signal input related to the user settings and function control of the mobile phone 100.

[0239] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Motor 191 can also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194.

[0240] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0241] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 100 by inserting the SIM card interface 195 or pulling the SIM card out of the SIM card interface 195.

[0242] Fig.11 It is a software structure block diagram of the mobile phone 100 according to an embodiment of the present invention.

[0243] The software system of the mobile phone 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the mobile phone 100.

[0244] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0245] like Fig.11 As shown, the application layer may include a series of application packages, which may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0246] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0247] In some embodiments, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0248] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0249] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0250] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0251] The phone manager is used to provide communication functions of the mobile phone 100, such as management of call status (including answering, hanging up, etc.).

[0252] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0253] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0254] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0255] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0256] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0257] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.

[0258] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0259] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0260] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0261] A 2D graphics engine is a drawing engine for 2D drawings.

[0262] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0263] Fig.12According to an embodiment of the present application, a structural schematic diagram of a speaker 200 is shown. In other embodiments, Fig.12 The structure of the speaker box 200 device shown may also be other speaker devices capable of controlling the playback of stereo audio, such as the central control device 800 in the car cabin shown in the above-mentioned embodiment 2.

[0264] like Fig.12 As shown, the speaker 200 may include a processor 1211, a memory 1212, a display screen 1213, a communication module 1214, a power switch 1215, an input-output interface 1216, and an audio module 1217. The audio module 1217 may include a microphone 171 and a speaker 172. The speaker 172 may be, for example, the speaker whose related channel control parameters and audio parameters are adjusted in the above-mentioned embodiment 1. For the specific functions of the above-mentioned structures, please refer to the above-mentioned Fig.10 The functional description of the corresponding structures in the mobile phone 100 is not repeated here.

[0265] It can be understood that the number of speakers 172 provided in the speaker 200 can be one or more, and the speakers used to implement the audio processing method provided in the present application can come from the same or different speakers 200. In addition, when the user is too close or too far away from a certain speaker, in order to achieve a better stereo effect, the speaker can be turned off. For example, based on the audio processing method provided in the present application, the channel control parameter played by the speaker can be adjusted to a null value or the corresponding audio parameter can be adjusted to 0, etc. In other words, in some embodiments, in order to achieve a better stereo effect, in some stereo playback scenarios, based on the spatial position relationship of each speaker relative to the user's ear, some speakers 200 or some speakers in the speaker 200 can also be turned off, without limitation here.

[0266] It is understandable that the above Fig.12 The illustrated structure does not constitute a specific limitation on the speaker 200. In other embodiments of the present application, the speaker 200 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0267] In addition, an embodiment of the present application also provides a computer program product for implementing the audio processing methods provided in the above embodiments.

[0268] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program module or module code executed on a programmable system, and the programmable system includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0269] A computer program module or module code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0270] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0271] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0272] References to "one embodiment" or "an embodiment" in the specification mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one exemplary implementation or technology disclosed according to the embodiment of the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily all refer to the same embodiment.

[0273] The disclosure of the embodiment of the present application also relates to an operating device for executing the text. The device can be specially constructed for the required purpose or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC) or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.

[0274] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present application embodiment disclosure is intended to illustrate rather than limit the scope of the concepts discussed herein.

Claims

1. An audio processing method, applied to a terminal device, characterized in that: include: Determining that the terminal device carried by the user has a first spatial position relative to a plurality of external speaker devices, wherein the plurality of external speaker devices have first stereo parameters; Determining that the spatial position of the terminal device carried by the user relative to the plurality of external speaker devices changes to a second spatial position; The first stereo parameter is adjusted to a second stereo parameter corresponding to the second spatial position, wherein: The first stereo parameters and the second stereo parameters at least include channel control parameters played by each speaker device.

2. The method according to claim 1, characterized in that The plurality of speaker devices include a first speaker device, and, In the case where at least one of the following position parameters changes, determining that the spatial position of the terminal device carried by the user relative to the multiple speaker devices changes to a second spatial position: detecting that the direction indicated by the first end of the terminal device changes from a first orientation to a second orientation, and, It is detected that the distance of the first speaker device relative to the terminal device changes from a first distance to a second distance.

3. The method according to claim 2, characterized in that The first stereo parameters include first channel control parameters, and, The adjusting the first stereo parameter to a second stereo parameter corresponding to the second spatial position includes: Determine an azimuth angle of the first speaker device relative to a first end of the terminal device based on the second distance and the second orientation; Based on the determined azimuth angle and the second orientation, a second channel control parameter is determined, and the first channel control parameter played by the first speaker device is adjusted to the second channel control parameter.

4. The method according to claim 3, characterized in that The second channel control parameters include left channel control parameters corresponding to the left ear of the user and right channel control parameters corresponding to the right ear of the user: and, The determining of the second channel control parameter based on the determined azimuth angle and the second orientation includes: The direction indicated by the first end of the terminal device is regarded as the facial orientation of the user, and Based on the second orientation and the azimuth angle, determining the proximity of the first speaker device to the left ear side or the right ear side of the user respectively; In response to detecting that the first speaker device is close to the left ear of the user, using a channel parameter that matches the left channel audio in the stereo audio to be played as the adjusted second channel control parameter; In response to detecting that the first speaker device is close to the right ear of the user, a channel parameter matching the right channel audio in the stereo audio to be played is used as the adjusted second channel control parameter.

5. The method according to claim 2, characterized in that: The first stereo parameter and the second stereo parameter further include: At least one of the sound intensity parameter and the phase parameter corresponding to each channel audio.

6. The method according to claim 5, characterized in that The first stereo parameter includes a first sound intensity parameter, and the adjusting the first stereo parameter to a second stereo parameter corresponding to the second spatial position includes: A second sound intensity parameter is determined based on the second distance, and a first sound intensity parameter in the first stereo parameter is adjusted to the second sound intensity parameter, wherein: The larger the second distance is, the larger the sound intensity controlled by the second sound intensity parameter is.

7. The method according to claim 2, characterized in that The first end is an end of the terminal device that is away from the user, and whether the direction indicated by the first end of the terminal device changes is detected in the following manner: Acquiring sensor data collected by an inertial sensor in the terminal device; Determining, based on the sensor data, that the motion state of the terminal device is a stationary state; Based on the direction indicated by the first end of the terminal device or the direction of the component of the vector corresponding to the direction indicated by the first end in the horizontal direction, it is determined whether the direction indicated by the first end of the terminal device changes.

8. The method according to claim 2, characterized in that: Detecting whether the distance of the first speaker device relative to the user changes by: Based on a first time difference between a first moment when the first speaker device sends a sound wave signal and a second moment when the terminal device receives the sound wave signal, it is determined whether the distance of the first speaker device relative to the user changes.

9. The method according to claim 3, characterized in that: The terminal device comprises a first end microphone and a second end microphone, and The determining, based on the second distance and the second orientation, an azimuth angle of the first speaker device relative to the first end of the terminal device comprises: determining a third distance of the first speaker device relative to the first end microphone based on a second time difference between a first moment when the first speaker device sends out a first sound wave signal and a third moment when the first end microphone receives the first sound wave signal; determining a fourth distance of the first speaker device relative to the second end microphone based on a third time difference between a first moment when the first speaker device sends out a first sound wave signal and a fourth moment when the second end microphone receives the first sound wave signal; The second distance is determined according to the third distance and the fourth distance, and the azimuth is determined based on the second distance and the second orientation.

10. The method according to claim 9, characterized in that The second distance includes any one of the following: The third distance, the fourth distance, and an average of the third distance and the fourth distance.

11. The method according to claim 9, characterized in that The first sound wave signal is an ultrasonic signal, and the method comprises: The terminal device receives the ultrasonic signals emitted by the plurality of external speaker devices, wherein the ultrasonic signals include the first sound wave signal emitted by the first speaker device; The terminal device determines, according to the ultrasonic signal and the direction indicated by the first end, that the spatial position relative to the plurality of external speaker devices is changed to a second spatial position; The terminal device sends the second stereo parameters adjusted corresponding to the second spatial position to each speaker device.

12. The method according to any one of claims 1 to 9, characterized in that The speaker device includes at least one of a sound box, a smart TV, and a car audio device; The terminal device includes at least one of a mobile phone, a tablet computer, a head-mounted device, a watch, and a bracelet.

13. A terminal device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the terminal device executes the audio processing method described in any one of claims 1 to 12.

14. A computer readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, enable the computer to execute the audio processing method according to any one of claims 1 to 12.

Images