Calibration-based audio playing method and system, head-mounted device and storage medium

Abstract

The invention provides an audio playing method and system based on calibration, head-mounted equipment and a storage medium, and relates to the field of audio processing. The calibration-based audio playing method is applied to a processor of the head-mounted device. The head-mounted device is connected with at least one wireless device through wireless communication, and the calibration-based audio playing method comprises the following steps: determining relative azimuth information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device; calibrating a null drift parameter of the IMU based on the relative azimuth information; the zero drift parameter is a parameter influenced by zero drift; acquiring a current calibration pose of the head-mounted device from the IMU after the zero drift parameter is calibrated; and according to the current calibration pose, adjusting orientation features in audio data to be played and sending the orientation features to the loudspeaker for playing. The calibration-based audio playing method is beneficial to improving the three-dimensional effect of audio data playing.

Description

Technical Field

[0001] The present application relates to the field of wearable devices, and in particular, to a calibration-based audio playback method, system, head-mounted device, and storage medium. Background Art

[0002] Audio data has directional characteristics. When music is played through head-mounted devices such as headphones, the head-mounted devices will use IMU (Inertial Measurement Unit) to detect posture. The IMU will feed back the posture information to the processor of the head-mounted device so that the processor of the head-mounted device can adjust the directional characteristics of the audio data based on the posture information, thereby making the playback of the audio data more three-dimensional.

[0003] However, as the usage time increases, the IMU's zero drift (abbreviated as "zero drift") parameters will change due to noise, errors, etc., thereby affecting the accuracy of posture detection and the playback effect of audio data. Summary of the invention

[0004] In view of this, the present application aims to provide a calibration-based audio playback method, system, head-mounted device and storage medium to improve the three-dimensional effect of audio data playback.

[0005] In the first aspect, an embodiment of the present application provides a calibration-based audio playback method, which is applied to a processor of a head-mounted device; the head-mounted device is connected to at least one wireless device through wireless communication, and the head-mounted device also includes an inertial measurement unit IMU and a speaker; the IMU is used to determine the posture of the head-mounted device; the calibration-based audio playback method includes: determining the relative orientation information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device; wherein, in the process of determining the relative orientation information, the wireless device is fixed; based on the relative orientation information, the zero drift parameter of the IMU is calibrated; the zero drift parameter is a parameter affected by zero point drift; the current calibration posture of the head-mounted device is obtained from the IMU after the zero drift parameter is calibrated; the orientation features in the audio data to be played are adjusted according to the current calibration posture and sent to the speaker for playback.

[0006] In an embodiment of the present application, a wireless device is set in the scene to be wirelessly connected to the head-mounted device. As a result, the head-mounted device transmits signals with the wireless device through the wireless communication connection to determine its relative orientation information compared to the wireless device. Based on the relative orientation information, the head-mounted device can calibrate the zero drift parameters of the IMU, thereby reducing the impact of zero drift on the posture detection of the IMU, so that the IMU can determine a more accurate posture. When adjusting the orientation features in the audio data to be played based on the current calibration posture determined by the calibrated IMU, the orientation features can be adjusted more accurately, thereby improving the playback effect of the audio data, that is, the three-dimensional effect is more accurate.

[0007] In one embodiment, the head-mounted device is a headset, the processor of the headset is set in either a left earphone or a right earphone, and the left earphone and the right earphone of the headset are communicatively connected via an audio line; or, the head-mounted device is either a left earphone or a right earphone of a true wireless headset.

[0008] The calibration-based audio playback method provided in the embodiment of the present application is applicable to different types of headphones. For example, the processor of the headset is one, and the processor can process the audio data to be played by the left and right headphones, and transmit the audio data from the headset where the processor is located to the other headset through the audio line in the headband to achieve the adjustment of the audio data to be played by the left and right headphones. The method can also be applied to true wireless headphones, and the left and right headphones of the true wireless headphones are each provided with a processor to process the audio data to be played by the respective headphones.

[0009] In one embodiment, the calibrating the zero drift parameters of the IMU based on the relative orientation information includes: obtaining a Kalman filter, wherein the Kalman filter is used to estimate the zero drift parameters; constraining the Kalman filter based on the relative orientation information; estimating based on the constrained Kalman filter to obtain calibrated zero drift parameters; and updating the zero drift parameters in the IMU according to the calibrated zero drift parameters.

[0010] The Kalman filter makes an optimal estimate of the system state through the system input and output observation data. The observation data includes the influence of noise and interference, and the zero drift parameter is also affected by error and noise. Therefore, in the embodiment of the present application, the Kalman filter can be used to estimate the zero drift parameter to estimate the optimal zero drift parameter. Among them, constraining the Kalman filter based on the relative orientation information can make the zero drift parameter estimated by the Kalman filter match the actual state of the head-mounted device, so that the zero drift parameter of the calibrated IMU is more accurate. On the other hand, the Kalman filter uses the estimated value of the previous moment and the observed value of the current moment to update the estimate of the state variable, which can make the calibration of the zero drift parameter more real-time, which helps to improve the accuracy of calibrating the IMU.

[0011] In one embodiment, the head-mounted device includes at least two receiving antennas arranged at different positions; determining the relative orientation information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device includes: obtaining the same radio frequency signal from the same wireless device received by different receiving antennas respectively; calculating the phase difference between the radio frequency signals received by different receiving antennas; and determining the relative orientation information of the head-mounted device relative to the wireless device based on the phase difference.

[0012] In the embodiment of the present application, the relative position information between the head-mounted device and the wireless device can be determined by using the phase difference of the same radio frequency signal received by different receiving antennas. This method can be applied to various types of wireless communication methods that need to receive radio frequency signals through receiving antennas, so that the head-mounted device can use more types of wireless devices to determine relative position information, thereby calibrating the zero drift parameters and improving the scope of application of the calibration-based audio playback method.

[0013] In one embodiment, the head-mounted device is wirelessly connected to the wireless device via Bluetooth; determining the relative position information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device includes: determining the relative position information between the head-mounted device and the wireless device based on Bluetooth channel detection.

[0014] Bluetooth channel detection is a high-precision positioning and ranging method. In the embodiment of the present application, the relative position information is determined by using Bluetooth channel detection, which can make the relative position information have a higher accuracy, thereby making the zero drift calibration more accurate and the determined posture more accurate. In addition, compared with other structures, the cost of the Bluetooth module is lower and does not require more structures, which can make the calibration-based audio playback method using Bluetooth channel detection more versatile. There is no need to set up additional structures in the head-mounted device, and the existing Bluetooth module can be reused to save costs.

[0015] In one embodiment, the Bluetooth channel detection is also used to determine the relative distance between the head-mounted device and the wireless device; constraining the Kalman filter based on the relative orientation information includes: constraining the Kalman filter based on the relative orientation information and the relative distance.

[0016] In the embodiment of the present application, Bluetooth channel detection is a positioning technology that can not only determine the phase position between the head-mounted device and the wireless device, but also determine the distance between the head-mounted device and the wireless device. When performing Kalman filtering, compared with constraining a single relative position information, constraining the Kalman filter based on both relative position information and distance data can make the zero drift parameter estimated by the Kalman filter more accurate, thereby making the position determined by the IMU more accurate and improving the playback effect of the audio data.

[0017] In one embodiment, the orientation features in the audio data to be played are adjusted according to the current calibration posture and sent to the speaker for playback, including: obtaining the audio data and a reference posture recorded at a reference time; the reference time is before the current time, and the current time is the time for obtaining the current calibration posture; determining the posture change based on the reference posture and the current calibration posture; adjusting the orientation features in the audio data to be played based on the posture change and a preset head-related transformation function module; the head-related transformation function module includes a conversion relationship between posture and the orientation features in the audio data.

[0018] HRTF (Head Related Transfer Function) can simulate the processing of sound signals by the human ear to restore the spatial orientation features of audio signals based on dual-channel sound signals. The orientation features in the audio data themselves match the posture at the reference moment. Therefore, the processing of the orientation features in the audio data by HRTF also matches the position at the reference moment. Therefore, in the embodiment of the present application, the relative change of the posture can be determined based on the reference posture and the current calibration posture, and then the HRTF can be adjusted based on the relative change, so that the HRTF can adjust the orientation features in the audio data to be played based on the current calibration posture.

[0019] In a second aspect, an embodiment of the present application provides a calibration-based audio playback device, which is applied to a processor of a head-mounted device; the head-mounted device is connected to at least one wireless device through wireless communication, and the head-mounted device also includes an inertial measurement unit IMU and a speaker; the IMU is used to determine the posture of the head-mounted device; the calibration-based audio playback device includes: a detection module, which is used to determine the relative orientation information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device; wherein the wireless device is fixed in the process of determining the relative orientation information; a calibration module, which is used to calibrate the zero drift parameter of the IMU based on the relative orientation information; the zero drift parameter is a parameter affected by zero point drift; a playback unit, which obtains the current calibration posture of the head-mounted device from the IMU after calibrating the zero drift parameter, and adjusts the orientation features in the audio data to be played according to the current calibration posture and sends it to the speaker for playback.

[0020] In a third aspect, an embodiment of the present application provides a head-mounted device, comprising: an IMU, a speaker; a wireless communication module for connecting to at least one wireless device; and a processor, respectively connected to the IMU, the speaker and the wireless communication module, for executing the calibration-based audio playback method as described in any one of the first aspects.

[0021] In a fourth aspect, an embodiment of the present application further provides an audio playback system, comprising a head-mounted device as described in the third aspect; and at least one wireless device, communicatively connected to the head-mounted device.

[0022] In a fifth aspect, an embodiment of the present application provides a readable storage medium, wherein the readable storage medium stores a program. When the program is executed by a processor, the processor implements the calibration-based audio playback method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A structural block diagram of a head mounted device provided in one embodiment of the present application;

[0025] Figure 2 A flowchart of a calibration-based audio playback method provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a calibration-based audio playback device provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of an audio playback system provided in accordance with an embodiment of the present application.

[0028] Icons: head mounted device 100; processor 110; IMU 120; wireless transmission module 130; speaker 140; audio digital-to-analog converter 150; memory 160; audio playback system 400; wireless device 410; smart device 420. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application more clear, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0030] First, an embodiment of the present application provides a head-mounted device. In the embodiment of the present application, the head-mounted device can be headphones, smart glasses, etc.

[0031] In an embodiment of the present application, if the head-mounted device is headphones, the head-mounted device includes but is not limited to headphones, true wireless headphones, etc., among which the headphones and true wireless headphones can be in-ear headphones, semi-in-ear headphones, and open-ear headphones, which are not limited here.

[0032] The headset includes a left earphone and a right earphone, the left earphone and the right earphone are connected to each other via an audio line in the headband, and the processor 110 is disposed in one of the left earphone and the right earphone. The audio data to be played includes the audio data to be played by the left earphone and the right earphone respectively. In an embodiment of the present application, the processor 110 can simultaneously adjust the audio data to be played by the left earphone and the right earphone respectively, and send the audio data to be played after adjusting the azimuth characteristics to the speaker 140 of the corresponding earphone for playing.

[0033] The true wireless headset includes a left earphone and a right earphone. The difference from the headphone is that the left earphone and the right earphone of the true wireless headset each include a processor 110. In an embodiment of the present application, the processors 110 of the left earphone and the right earphone of the true wireless headset can be used to execute the calibration-based audio playback method provided in the present application, or, in some embodiments, either the left earphone or the right earphone of the true wireless headset can be used as a head-mounted device in an embodiment of the present application, including the structure of the head-mounted device described later.

[0034] See also Figure 1 , Figure 1A structural block diagram of a head mounted device is provided for an embodiment of the present application. The head mounted device 100 includes: a processor 110 , an inertial measurement unit IMU 120 , a wireless transmission module 130 , a speaker 140 , a memory 160 and an audio digital-to-analog converter 150 .

[0035] The processor 110 is used to execute the calibration-based audio playback method provided by the present application, and the calibration-based audio playback method will be expanded later. In the embodiment of the present application, the processor 110 can be a SOC (System on Chips) chip, a CPU (Central Processing Unit), an MPU (Microprocessor), etc., which is not limited here.

[0036] The inertial measurement unit IMU 120 is used to determine the posture of the head mounted device 100. The posture may include position and attitude, and the attitude includes orientation and / or azimuth.

[0037] In an embodiment of the present application, the IMU 120 may include sensor elements such as a gyroscope and / or an accelerometer, which may be used to detect acceleration, velocity, angular velocity, and thus determine attitude, position, etc. In some embodiments, if the processor 110 is a SOC chip, the IMU 120 may be a chip integrated on the SOC.

[0038] The measurements of the IMU 120 may be affected by systematic errors and random noise; for example, the accelerometer may be affected by noise, zero bias, scale factor, and installation error, and the gyroscope may be affected by g-sensitivity.

[0039] Due to the influence of system errors and random noise, IMU 120 is prone to zero drift (abbreviated as "zero drift"), which refers to the error in the measured posture and / or position when IMU 120 does not receive any external measurement signal (i.e., IMU is in a stationary state). Zero drift is caused by the drift and noise of the IMU sensor itself. For example, due to the mechanical structure or circuit noise inside the device, the signal output by the IMU sensor is offset. Zero drift will cause the sensor accuracy of the IMU to decrease, which in turn affects the accuracy and reliability of posture and / or position measurement. And the zero drift of the inertial sensor in the IMU will make the cumulative error of the IMU attitude estimation result and the position estimation result become larger and larger as time goes by.

[0040] Audio data has directional characteristics, and users can obtain better three-dimensional effects through directional characteristics. For example, they can know that the sound source is in front of, behind, on the left or on the right of the user, which makes the surround and stereo sound playback effects better. At the same time, when applied to hearing aid and transparent transmission functions, it is convenient for users to determine the source of the sound.

[0041] The azimuth feature of the audio data is determined based on the current posture and position of the head mounted device 100. For example, assuming that the position of the sound source is fixed, the head mounted device 100 is facing the first direction at the first moment, and facing the second direction at the second moment. The first direction and the second direction are completely opposite directions. For example, if the user turns 180 degrees (the head mounted device 100 also turns accordingly), assuming that at the first moment, when the head mounted device 100 faces the first direction, the sound comes from the left side of the user, then at the second moment, when the head mounted device 100 faces the second direction, the sound should come from the right side of the user. If the IMU has zero drift between the first moment and the second moment, such as the azimuth deviation of 20 degrees, 30 degrees, 45 degrees, etc., the sound heard at the second moment may come from the oblique front, oblique rear, etc., rather than the right side. The position of the sound source is fixed, which will affect the three-dimensional effect of the user listening to the sound.

[0042] Therefore, in the calibration-based audio playback method provided in the embodiment of the present application, the processor 110 can be used to calibrate the zero drift of the IMU 120 so that when the audio data is played, the influence of the zero drift on the three-dimensional effect of the audio data playback is reduced.

[0043] The wireless communication module is used to connect with at least one wireless device through wireless communication.

[0044] In the embodiment of the present application, the wireless communication module may be a Bluetooth module, a WIFI (Wireless Fidelity) module, a cellular module, a UWB (Ultra Wide Band) module, etc. Through the wireless communication module, the wearable device can be wirelessly connected with a smart device such as a mobile phone or a tablet computer or other audio source device to receive the audio data to be played sent by the smart device / audio source device and play it through the speaker 140.

[0045] The memory 160 is connected to the wireless transmission module 130 , the processor 110 and the IMU 120 , and is used to store the audio data to be played, the posture data measured by the IMU 120 , and the like.

[0046] The speaker 140 is used to play audio data. In some embodiments, the audio data is a digital signal. Therefore, in the embodiment of the present application, the head mounted device 100 may further include an audio digital-to-analog converter 150, which is used to perform digital-to-analog conversion on the audio to be played, so as to play the converted analog signal through the speaker 140.

[0047] The above-mentioned structures and circuits can refer to the prior art and will not be elaborated here. Next, the calibration-based audio playback method executed by the processor 110 in the present application will be described.

[0048] See also Figure 2 , Figure 2 A flowchart of a calibration-based audio playback method provided in an embodiment of the present application, the calibration-based audio playback method comprising:

[0049] S110: Determine relative position information between the head mounted device and the wireless device based on the wireless communication connection between the head mounted device and the wireless device.

[0050] In an embodiment of the present application, the head mounted device 100 includes a wireless communication module, through which the head mounted device 100 is connected to at least one wireless device through wireless communication, and the wireless device is fixed in the process of determining the relative position information. The wireless device can be a different fixed device in the scene, for example, a smart home device in the scene, such as an air conditioner, a television, a lamp, a curtain, and other devices that are usually fixed, or a movable device, such as a speaker, a humidifier, etc., which only needs to remain still in the process of determining the relative position information.

[0051] In the embodiment of the present application, the process of determining the relative position information includes from the reference time to the current time, that is, the wireless device is fixed from the reference time to the current time. For example, the user specifies the reference time through an interactive method such as a key, gesture, or voice, and collects the posture at this time as the reference posture. At least during the period from the user specifying the reference time to determining the relative position information at the current time, the wireless device remains fixed.

[0052] In some embodiments, the head mounted device 100 may obtain the position of each wireless device to improve the accuracy of the relative position information obtained based on the obtained position. For fixed wireless devices, the position of each wireless device may be pre-configured, and for mobile / movable wireless devices, each wireless device may determine its own position through positioning and send it to the head mounted device 100.

[0053] In some embodiments of the present application, the wireless device may also be a mobile device such as a mobile phone or a tablet computer, and similarly, it also needs to remain motionless during the process of determining the relative position.

[0054] In an embodiment of the present application, the number of wireless devices may include multiple ones, and the relative orientation information of the head-mounted device relative to different wireless devices is determined based on different wireless devices, so that when the zero drift parameters of IMU120 are calibrated based on multiple relative orientation information, more accurate zero drift calibration parameters can be obtained.

[0055] In an embodiment of the present application, if the head mounted device 100 is a true wireless headset, the processors 110 in the left headset and the right headset may respectively determine the relative position information between each of them and the wireless device.

[0056] In one embodiment of the present application, the head-mounted device 100 may include at least two receiving antennas arranged at different positions, and S110 may include: obtaining the same radio frequency signal from the same wireless device received by different receiving antennas respectively; calculating the phase difference between the radio frequency signals received by different receiving antennas; and determining the relative orientation information of the head-mounted device 100 relative to the wireless device based on the phase difference.

[0057] In this embodiment, more than two receiving antennas are set at different positions on the head-mounted device 100. Different receiving antennas can receive the same radio frequency signal emitted by the same wireless device. The positions of the receiving antennas are different, which makes different receiving antennas receive the same radio frequency signal at different times, so that there is a phase difference in the radio frequency signals received and stored by each. The positions of different receiving antennas are fixed and known. Therefore, positioning can be performed based on the phase difference and the distance between different receiving antennas.

[0058] The specific process of determining the relative orientation information of the head mounted device 100 relative to the wireless device based on the phase difference can refer to the wireless signal detection method in the prior art, such as dual antenna positioning technology, arrival angle positioning technology, etc., which will not be elaborated here.

[0059] In the prior art, the positioning between two devices, including the direction and distance, can be achieved based on the wireless signal detection method. In the embodiments of the present application, the phase difference can be used to detect only the relative direction, that is, the distance does not need to be detected. Compared with the simultaneous detection of the direction and the distance, the detection result of only detecting the relative direction using the phase difference has higher accuracy, which helps to improve the effect of zero drift calibration.

[0060] In this embodiment, providing a receiving antenna having a number greater than 2 helps to further improve the accuracy of the relative position information. Similarly, providing multiple wireless devices to respectively detect and determine the relative position information can also help to improve the accuracy of the obtained relative position information.

[0061] In some other embodiments of the present application, the wireless communication module may include a Bluetooth module, and the head-mounted device 100 may be wirelessly connected to the wireless device via Bluetooth. In this embodiment, S110 may include: determining the relative position information between the head-mounted device 100 and the wireless device based on Bluetooth channel detection.

[0062] Bluetooth channel detection technology is a high-precision ranging technology in the Bluetooth standard. It combines channel detection, phase measurement and round-trip time measurement to estimate distance and correct each other. It has higher accuracy and a safety protection mechanism, and can achieve a ranging accuracy of 10cm.

[0063] In the embodiments of the present application, determining the relative position information based on the Bluetooth channel detection technology helps to improve the accuracy of the determined relative position information.

[0064] In addition, the cost of Bluetooth modules is low and they are widely used. They are installed in many devices. Therefore, the relative position information determined based on Bluetooth channel detection can be used in various devices, expanding the scope of application. When the existing head mounted device 100 implements the method provided by the present application, there is no need to configure various structures additionally. The existing Bluetooth modules can be reused, which reduces the cost.

[0065] Bluetooth channel detection technology has a large ranging accuracy, such as a ranging accuracy of 10 cm. Therefore, in an embodiment of the present application, when using Bluetooth channel detection technology to determine relative orientation information, the relative distance between the head-mounted device 100 and the wireless device can also be determined at the same time, so as to calibrate the IMU 120 based on the relative orientation information and the relative distance.

[0066] In addition, there are other positioning methods in the prior art, and the methods provided in the above embodiments should not be considered as limitations to the present application.

[0067] S120, calibrating the zero drift parameters of the IMU based on the relative orientation information.

[0068] In this embodiment, the zero drift parameter is a parameter affected by zero drift. The zero drift parameter may include acceleration included in IMU 120, and the zero drift parameter includes the zero drift parameter of the accelerometer, for example, the DC or low-frequency offset of the acceleration in each direction, and / or, the DC or low-frequency offset of the integral of the acceleration (i.e., velocity), and / or, the DC or low-frequency offset of the quadratic integral of the acceleration (i.e., displacement, or position). IMU 120 also includes a gyroscope, and the zero drift parameter may also include the zero drift parameter of the gyroscope, for example, the DC or low-frequency offset of the angular velocity in each rotation direction, and / or, the DC or low-frequency offset of the integral of the angular velocity (i.e., azimuth, also referred to as attitude angle, heading angle).

[0069] In one embodiment of the present application, calibrating the zero drift parameters of the IMU 120 based on the relative orientation information may include: obtaining a Kalman filter, constraining the Kalman filter based on the relative orientation information; estimating based on the constrained Kalman filter to obtain a calibrated zero drift parameter; and updating the zero drift parameter in the IMU 120 according to the calibrated zero drift parameter.

[0070] Kalman filtering is an algorithm that utilizes linear system state equations, through system input and output observation data, to optimally estimate the system state. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process. In an embodiment of the present application, IMU 120 is a system that needs filtering, and the zero drift parameter of IMU 120 is the observation data that needs to be filtered, and the observation data is affected by systematic error and random noise. Therefore, in an embodiment of the present application, the zero drift parameter can be estimated based on the kalman filter.

[0071] In order to make the estimated zero drift parameters match the zero drift parameters corresponding to the current actual posture of IMU 120, it is necessary to constrain the Kalman filter. In an embodiment of the present application, relative orientation information can be obtained to constrain the Kalman filter so that the constrained Kalman filter can estimate the calibration zero drift parameters that match the current posture of IMU 120.

[0072] The kalman filter can refer to the prior art, and the principle is not expanded here. The kalman filter is a state space module, which includes a state quantity. In the embodiment of the present application, the state quantity of the kalman filter includes the position, azimuth, etc. of the IMU (wherein, the IMU is connected to the head-mounted device, and the IMU is adjacent to the head-mounted device or arranged in the head-mounted device, and the position of the IMU can be used to characterize the position of the head-mounted device), and the wireless device is fixed. Therefore, based on the kalman filter, the IMU and the wireless device, the difference between the azimuth of the IMU relative to the wireless device and the corresponding azimuth at the reference time can be obtained, wherein the azimuth or azimuth difference determined based on the kalman filter will be affected by zero drift. On the other hand, the azimuth of the head-mounted device and the wireless device can also be measured by the wireless signal of the wireless communication connection, and the azimuth measured in this way is not affected by zero drift. Therefore, the azimuth measured by the wireless signal or the difference in the azimuth corresponding to the parameter time can be used as a constraint or measurement equation to correct the state quantity of the kalman filter, thereby calibrating the zero drift.

[0073] In one embodiment of the present application, the method based on Bluetooth channel detection can not only determine the relative orientation information, but also determine the relative distance between the head-mounted device 100 and the wireless device. Therefore, in an embodiment of the present application, the Kalman filter can also be constrained based on both the relative distance and the relative orientation information, so that the Kalman filter can estimate a more accurate calibration zero drift parameter.

[0074] In an embodiment of the present application, the zero drift parameters in the IMU 120 are updated according to the calibration zero drift parameters. The calibration zero drift parameters may be used to replace the corresponding zero drift parameters in the IMU 120, or the parameters may be adjusted multiple times in stages until the corresponding zero drift parameters in the IMU 120 are adjusted to be the same as the calibration zero drift parameters.

[0075] In other embodiments of the present application, the zero drift parameters can also be estimated using methods such as the Euler angle method, the quaternion method, the equivalent rotation vector method, the Rodriguez parameter method, the direction cosine matrix method, etc., and the relative orientation information (or relative distance and relative orientation information) can be used to constrain the above methods to obtain the calibrated zero drift parameters. The above are only examples and are not limited here.

[0076] S130, obtaining a current calibration posture of the head mounted device from the IMU after calibrating the zero drift parameters.

[0077] IMU 120 can perform posture detection, and the detection method and process can refer to the prior art, which will not be expanded here. In the embodiment of the present application, the current calibration posture refers to the posture detected by IMU 120 after calibrating the zero drift parameters.

[0078] In an embodiment of the present application, the posture detected by the IMU 120 also includes a reference posture, which is the posture at the initial moment / the posture collected at the preset reference moment, and the reference posture is stored in the memory 160. The user can specify the reference moment and collect the posture at this moment as the reference posture through interactive methods such as buttons, gestures, and voice. For example, when the user starts to enjoy music or audio, or when the audio player is turned on, the moment can be determined as the reference moment; the user can also specify a predetermined moment at any time through interactions such as buttons and voice.

[0079] S140, adjusting the orientation features in the audio data to be played according to the current calibration posture and sending it to the speaker 140 for playback.

[0080] The audio data to be played may be a PCM audio signal or a compressed audio signal, such as AAC (Advanced Audio Coding) audio data, MP3 (Moving Picture Experts Group Audio Layer III, an audio compression technology) audio data, SBC (Sub-Band Coding, sub-band coding in Bluetooth audio codec) audio data, FLAC (Free Lossless Audio Codec, lossless audio compression coding) audio data, etc. For compressed audio signals, for example, AAC audio data and MP3 audio data are compressed audio data, and the processor 110 may perform audio decompression on the compressed audio data and then make adjustments.

[0081] In some embodiments of the present application, the audio data to be played may be received from a smart device / audio source device (such as), for example, the audio data to be played may be call data of a mobile phone or music data to be played, which is transmitted to the head-mounted device 100 for playback via wireless communication.

[0082] In some embodiments of the present application, the audio data to be played can also be collected by the head mounted device 100 itself or pre-stored in the memory 160. For example, the head mounted device 100 is used to implement at least one of the functions of transparent transmission, hearing aid or side tone, and the head mounted device 100 also includes a microphone, and the audio data collected by the microphone will be played through the in-ear speaker 140, and the audio data collected by the microphone is the audio to be played. For another example, the audio to be played can also be some prompt sounds or other audio data pre-stored in the memory 160. The above is only an example and should not be regarded as a limitation of the present application.

[0083] In an embodiment of the present application, the audio data to be played is N-channel audio data, where N is greater than or equal to 2. For example, the audio data to be played may be 5.1-channel audio data, dual-channel audio data, etc. The number of channels of the audio data may characterize the azimuth characteristics in the audio data. For example, the 5.1-channel audio data may include 2 front channels, 1 center channel, 2 surround channels, and 1 bass channel. The dual-channel audio data may include a left channel and a right channel. The contents related to the audio data and the channels may refer to the prior art and will not be expanded here.

[0084] In one embodiment of the present application, adjusting the orientation features in the audio data to be played according to the current calibration posture may include: obtaining the audio data to be played and a reference posture recorded at a reference time; determining a posture change based on the reference posture and the current calibration posture; and adjusting the orientation features in the audio data to be played based on the posture change and a preset head-related transformation function module.

[0085] The reference pose is the pose determined at a preset reference time, and the current calibration pose is the pose determined by the IMU 120 after the IMU 120 is calibrated, that is, the reference time is before the current time. The orientation features in the audio data are based on the reference pose, or it can be understood that the orientation features in the audio data match the reference pose in all directions. For example, the left side of the orientation feature in the audio data matches the left side of the reference pose, the right side of the orientation feature in the audio data matches the right side of the reference pose, and so on.

[0086] In this embodiment, based on the current calibration posture and the reference posture, the posture change can be determined, for example, compared with the reference moment, the current moment is rotated 90 degrees to the left, rotated 150 degrees, rotated up or down 30 degrees, etc.

[0087] HRTF (Head Related Transfer Function) is used to simulate the human ear's positioning of sounds in three-dimensional space. People have two ears and can locate sounds from three-dimensional space. This is due to the human ear's analysis system for sound signals. The signal transmitted from any point in space to the human ear (in front of the eardrum) can be described by a filter system. The sound source obtained by the filter system is the sound signal in front of the eardrums of both ears. If the filter system can be converted into multiple filters (or transfer functions) that describe spatial information, namely HRTF, then the sound signal from this direction in space can be restored based on HRTF. In an embodiment of the present application, the principle of HRTF can be used to construct a head-related transfer function module, which includes a filter group that simulates all directions of space to both ears. Based on the head-related transfer function module, the directional characteristics of the sound can be restored, so that the user can listen to the sound with a three-dimensional effect. The principle and implementation method of HRTF can refer to the prior art and will not be expanded here. The filter in the head-related transfer function module may be a FIR (Finite Impulse Response, finite unit impulse response filter) filter or an IIR (Infinite Impulse Response, recursive filter) filter, which is not limited here.

[0088] In the embodiment of the present application, the azimuth characteristics of the sound can be restored based on the head-related transformation function module. The audio data finally played by the speaker 140 of the head-mounted device 100 can be dual-channel audio data, and the audio data sent by the smart device, the sound source device, etc. can be audio data of more channels. Therefore, in the embodiment of the present application, the processor 110 can also convert the audio data of other channels into dual-channel audio data, and adjust the azimuth characteristics in the dual-channel audio data. If the audio data to be played received by the head-mounted device 100 is dual-channel audio data, only the azimuth characteristics in the dual-channel audio data can be adjusted. Among them, the process is based on the reference posture and the current calibration posture, and the head-related transformation function module is used to adjust the audio data, and the audio data of multiple channels is converted into dual-channel audio data to be played, so that although the posture of the head-mounted device 100 worn by the user may be changing and rotating, the dual-channel audio data to be played by the user after being played through the speaker maintains the original spatial audio characteristics and sound azimuth characteristics, so the dual-channel audio data to be played also includes azimuth characteristics.

[0089] Therefore, in an embodiment of the present application, the head-related transformation function module includes a conversion relationship between posture and orientation features in audio data.

[0090] For example, based on the reference posture, the azimuth feature of a sound in the audio data indicates that the sound comes from the first direction. After the posture is changed, the azimuth feature of the sound should be adjusted to match the change in posture, so that the sound heard by the user still feels that the sound comes from the first direction, and the sound originally played by the left earphone will not still be emitted by the left earphone after the user rotates 180 degrees. For example, at the reference moment, the sound of the door closing is heard from behind the user, and when the user turns to the direction of the door, the sound heard by the user should come from in front of the user, thereby making the sound playback more three-dimensional.

[0091] Taking the audio data to be played as 5.1-channel audio data as an example, the audio data to be played and the reference posture determined at the reference moment are obtained, and the dual-channel audio data to be played is obtained by using the head-related transformation function module. The audio played at the reference moment is based on the 5.1-channel audio data and the reference posture. For example, the left and right front channels in the 5.1 channel are the posture of the head-mounted device relative to the reference moment. At the current moment, based on the reference posture and the current calibrated posture, the preset head-related transformation function module is used to adjust the audio data to be played to obtain the dual-channel audio data to be played, and because the dual-channel audio data to be played contains azimuth features, the final played audio can be unaffected by the rotation of the head-mounted device, thereby maintaining the spatial characteristics of the audio listened to by the user.

[0092] In other embodiments of the present application, after determining the current calibration posture, the conversion relationship between the reference posture included in the head-related transformation function module and the orientation features in the audio data can also be updated to the conversion relationship between the current calibration posture and the orientation features in the audio data, so as to facilitate the subsequent rapid processing of audio data.

[0093] In an embodiment of the present application, a wireless device is arranged in the scene so as to be wirelessly connected to the head mounted device 100. Thus, the head mounted device 100 transmits signals with the wireless device through the wireless communication connection, thereby determining its relative orientation information relative to the wireless device. Then, based on the relative orientation information, the head mounted device 100 can calibrate the zero drift parameters of the IMU 120, thereby reducing the influence of the zero drift on the posture detection of the IMU 120, so that the IMU 120 can determine a more accurate posture. Then, when adjusting the orientation features in the audio data to be played based on the current calibration posture determined by the calibrated IMU 120, the orientation features can be adjusted more accurately, thereby improving the playback effect of the audio data, that is, the three-dimensional effect is more accurate.

[0094] Based on the same inventive concept, the embodiment of the present application also provides a calibration-based audio playback device, which can be applied to the processor 110 of the head-mounted device 100 provided in the aforementioned embodiment. The head-mounted device 100 is connected to at least one wireless device via wireless communication, and the head-mounted device 100 further includes an inertial measurement unit IMU 120 and a speaker 140; the IMU 120 is used to determine the position and posture of the head-mounted device 100, and the details can be referred to the aforementioned embodiment, which will not be expanded here.

[0095] See also Figure 3 , Figure 3 This is a schematic diagram of a calibration-based audio playback device provided in an embodiment of the present application. The calibration-based audio playback device includes: a detection module 410 , a calibration module 420 and a playback unit 430 .

[0096] The detection module 410 is used to determine the relative position information between the head-mounted device and the wireless device based on the wireless communication connection between the head-mounted device and the wireless device; wherein the wireless device is fixed during the process of determining the relative position information.

[0097] The calibration module 420 is used to calibrate the zero drift parameters of the IMU based on the relative orientation information; the zero drift parameters are parameters affected by the zero point drift.

[0098] The playback unit is used to obtain the current calibration posture of the head-mounted device from the IMU after calibrating the zero drift parameters; adjust the orientation features in the audio data to be played according to the current calibration posture and send it to the speaker for playback.

[0099] The calibration module 420 is used to obtain a Kalman filter, which is used to estimate zero drift parameters; constrain the Kalman filter based on relative orientation information; estimate based on the constrained Kalman filter to obtain calibration zero drift parameters; and update the zero drift parameters in the IMU according to the calibration zero drift parameters.

[0100] In other embodiments, the calibration module 420 may also be based on a quaternion method, an equivalent rotation vector method, a Rodrigues parameter method, or the like.

[0101] The head-mounted device includes at least two receiving antennas arranged at different positions. The detection module 410 is used to obtain the same radio frequency signal from the same wireless device received by different receiving antennas; calculate the phase difference between the radio frequency signals received by different receiving antennas; and determine the relative orientation information of the head-mounted device relative to the wireless device based on the phase difference.

[0102] The head mounted device is wirelessly connected to the wireless device via Bluetooth, and the detection module 410 is used to determine the relative position information between the head mounted device and the wireless device based on Bluetooth channel detection.

[0103] The Bluetooth channel detection is also used to determine the relative distance between the head mounted device and the wireless device, and the calibration module 420 is also used to constrain the Kalman filter based on the relative orientation information and the relative distance.

[0104] The playback unit 430 includes a head-related transformation function module, which is used to obtain the audio data to be played and the reference posture recorded at a reference time; the reference time is before the current time, and the current time is the time for obtaining the current calibration posture; the posture change amount is determined based on the reference posture and the current calibration posture; the orientation features in the audio data to be played are adjusted based on the posture change amount and a preset head-related transformation function module; the head-related transformation function module includes a conversion relationship between the posture and the orientation features in the audio data.

[0105] The calibration-based audio playback device provided in the embodiment of the present application is similar to the aforementioned calibration-based audio playback method. For details, please refer to the aforementioned content and will not be elaborated here.

[0106] Based on the same inventive concept, the present application embodiment also provides an audio playback system based on calibration, see Figure 4 , Figure 4 A schematic diagram of an audio playback system provided in one embodiment of the present application, the audio playback system includes: a head-mounted device and at least one wireless device.

[0107] The structure of the head mounted device may refer to the aforementioned embodiment and will not be elaborated here.

[0108] The wireless device is connected to the head-mounted device through wireless communication, and the wireless communication method can be WIFI, Bluetooth, UWB, etc. The wireless device can be a smart home device such as a desk lamp, air conditioner, etc., or a device with a wireless communication module such as a Bluetooth speaker.

[0109] In addition, the calibrated audio playback system may also include a smart device, such as a mobile phone, a tablet computer, a smart TV, etc., which is connected to the head-mounted device through wireless communication. The audio data of the head-mounted device can be obtained from the smart device through wireless communication.

[0110] In an embodiment of the present application, the head-mounted device can execute the calibration-based audio playback method provided in the embodiment of the present application based on the wireless device. The calibration-based audio playback method can be specifically referred to the aforementioned embodiment and will not be expanded here.

[0111] Based on the same inventive concept, an embodiment of the present application further provides a readable storage medium, in which instructions are stored. The instructions can be executed by one or more processing units to implement the calibration-based audio playback method provided in the above embodiment.

[0112] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD (digital video disc)), or a semiconductor medium (e.g., an SSD (Solid State Disk)).

[0113] If the calibration-based audio playback method is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0114] In the embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0115] The above embodiments can be freely combined without conflict, and the embodiments obtained by the combination are included in the protection scope of this application.

[0116] The above detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0117] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0118] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0119] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A calibration-based audio playback method, characterized in that: A processor applied to a head-mounted device; the head-mounted device is connected to at least one wireless device via wireless communication, and the head-mounted device also includes an inertial measurement unit IMU and a speaker; The IMU is used to determine the position and posture of the head mounted device; The calibration-based audio playback method comprises: Determining relative position information between the head mounted device and the wireless device based on the wireless communication connection between the head mounted device and the wireless device; wherein the wireless device is fixed during the process of determining the relative position information; Calibrate the zero drift parameter of the IMU based on the relative orientation information; the zero drift parameter is a parameter affected by zero drift; Obtaining the current calibration posture of the head mounted device from the IMU after calibrating the zero drift parameters; The orientation features in the audio data to be played are adjusted according to the current calibration posture and sent to the speaker for playing.

2. The calibration-based audio playback method according to claim 1, characterized in that: The head-mounted device is a headset, the processor of the headset is arranged in any one of the left headset or the right headset, and the left headset and the right headset of the headset are communicatively connected via an audio line; Or, the head-mounted device is any one of the left earphone or the right earphone of a true wireless headset.

3. The calibration-based audio playback method according to claim 1, characterized in that: The calibrating the zero drift parameter of the IMU based on the relative orientation information includes: Acquire a Kalman filter, wherein the Kalman filter is used to estimate a zero drift parameter; constraining the Kalman filter based on the relative orientation information; Estimation is performed based on the constrained Kalman filter to obtain calibration zero drift parameters; The zero drift parameters in the IMU are updated according to the calibration zero drift parameters.

4. The calibration-based audio playback method according to claim 3, characterized in that: The head mounted device comprises at least two receiving antennas arranged at different positions; The determining the relative position information between the head mounted device and the wireless device based on the wireless communication connection between the head mounted device and the wireless device comprises: Acquire the same radio frequency signal from the wireless device received by different receiving antennas respectively; Calculating the phase difference between the radio frequency signals received by different receiving antennas; The relative position information of the head mounted device with respect to the wireless device is determined based on the phase difference.

5. The calibration-based audio playback method according to claim 3, characterized in that: The head mounted device is wirelessly connected to the wireless device via Bluetooth; The determining the relative position information between the head mounted device and the wireless device based on the wireless communication connection between the head mounted device and the wireless device comprises: The relative position information between the head mounted device and the wireless device is determined based on Bluetooth channel detection.

6. The calibration-based audio playback method according to claim 5, characterized in that: The Bluetooth channel detection is also used to determine the relative distance between the head mounted device and the wireless device; The constraining the Kalman filter based on the relative orientation information comprises: The Kalman filter is constrained based on the relative orientation information and the relative distance.

7. The calibration-based audio playback method according to any one of claims 1 to 6, characterized in that: Adjusting the orientation features in the audio data to be played according to the current calibration posture includes: Acquire the audio data to be played and a reference posture recorded at a reference time; the reference time is before the current time, and the current time is the time when the current calibration posture is acquired; Determine a posture change amount based on the reference posture and the current calibration posture; The orientation features in the audio data to be played are adjusted based on the posture change amount and a preset head-related transformation function module; the head-related transformation function module includes a conversion relationship between the posture and the orientation features in the audio data.

8. A calibration-based audio playback device, characterized in that: A processor applied to a head-mounted device; the head-mounted device is connected to at least one wireless device via wireless communication, and the head-mounted device also includes an inertial measurement unit (IMU) and a speaker; The IMU is used to determine the position and posture of the head mounted device; The calibration-based audio playback device comprises: A detection module, configured to determine relative position information between the head mounted device and the wireless device based on a wireless communication connection between the head mounted device and the wireless device; wherein the wireless device is stationary during the process of determining the relative position information; A calibration module, used for calibrating a zero drift parameter of the IMU based on the relative orientation information; the zero drift parameter is a parameter affected by zero drift; The playing unit obtains the current calibration posture of the head mounted device from the IMU after calibrating the zero drift parameters, and adjusts the orientation features in the audio data to be played according to the current calibration posture and sends it to the speaker for playing.

9. A head mounted device, characterized in that: include: IMU, speaker; A wireless communication module, used for connecting with at least one wireless device; A processor is respectively connected to the IMU, the speaker and the wireless communication module, and is used to execute the calibration-based audio playback method as described in any one of claims 1 to 7.

10. An audio playback system, characterized in that: include: The head mounted device as claimed in claim 9; At least one wireless device is communicatively connected to the head mounted device.

11. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by a processor, the processor implements the calibration-based audio playback method according to any one of claims 1 to 7.

Images