Smart home device positioning method based on dual-microphone mobile terminal

Abstract

The present invention discloses a smart home device positioning method based on a dual-microphone mobile terminal. The method first uses the dual microphones of the mobile terminal as sensors to collect sound signals and simultaneously collects the orientation of the mobile terminal. During the positioning process, information is collected at multiple locations to obtain the sound signal of the sound source device to be located. Then, by calculating the TDoA of the two microphones, the relative orientation of the sound source device to be located is analyzed, and the positioning space is modeled. For each collected position, corresponding four-tuple positioning information is constructed. Finally, based on the positioning information four-tuple, the perpendicular midline of the dual microphones is calculated for each four-tuple to achieve spatial segmentation. The subspace where the sound source device is located is determined through multiple sets of spatial segmentation. The subspace mean coordinates are mapped to the indoor environment to obtain the position of the sound source device. The present invention does not rely on a large number of sensors or high-precision microphone arrays, and is more economical and more robust than existing indoor sound source positioning technologies.

Description

Technical Field

[0001] The present invention belongs to the technical field of sound source localization and indoor positioning, and particularly relates to a smart home device positioning method based on a dual-microphone mobile terminal. Background Art

[0002] With the continuous development of IoT technology, various IoT devices are being widely used in scenarios such as smart cities, smart healthcare, and smart homes. Smart home technology primarily addresses people's demand for comfort and convenience in their homes. Currently, the location of smart devices in smart home systems often relies on manual user input. If the device's location is lost, the user must manually search for it in a relatively complex indoor environment. Smart home devices often include sound-emitting components, making smart home positioning solutions based on sound source localization valuable. Current sound source localization solutions fall into two main categories: one that relies on the collaboration of three or more sensors deployed in the environment to achieve sound source localization, and the other that relies on reflection analysis of acoustic signals. The former, since most smart home devices lack microphones, requires additional sensors, increasing positioning costs and making it unsuitable for smart home scenarios. The latter, however, requires the device to be equipped with a high-precision microphone array, also making it unsuitable for smart home scenarios. Therefore, a more convenient sound source localization solution is needed to help users locate smart home devices without increasing equipment costs. Summary of the Invention

[0003] Purpose of the Invention: To address the lack of economical and efficient solutions for locating sound source devices in smart home scenarios, the present invention proposes a smart home device positioning method based on a dual-microphone mobile terminal. This method requires only one smart mobile terminal, such as a mobile phone, to achieve spatial modeling by combining information such as the sound source device, the location of the smart mobile terminal, and the orientation of the smart mobile terminal. Furthermore, a spatial segmentation and positioning solution is used to locate sound source devices in smart furniture.

[0004] Technical Solution: This invention proposes a smart home device positioning method based on a dual-microphone mobile terminal. The solution is divided into three parts: raw data acquisition and processing, mobile terminal positioning information modeling, and sound source device positioning based on spatial segmentation. The details are as follows:

[0005] (1) Raw data acquisition and processing: The dual microphones of the mobile terminal are used as sensors to collect sound signals and the orientation of the mobile terminal. Information is collected at multiple locations during the positioning process. The sound signal of the sound source device to be located is obtained through filtering, denoising, and effective sound extraction.

[0006] (2) Mobile terminal positioning information modeling: By calculating the TDoA of the two microphones, analyzing the relative position of the sound source device to be located to the microphones, and combining the indoor coordinates of the mobile terminal with the orientation information of the mobile terminal, the positioning space is modeled and the corresponding four-tuple positioning information is constructed for each acquisition position.

[0007] (3) Sound source device positioning based on spatial segmentation: Based on the positioning information quadruple, the median perpendicular line of the two microphones is calculated for each quadruple to achieve spatial segmentation; the subspace where the sound source device is located is determined through multiple groups of spatial segmentation; the subspace mean coordinates are mapped to the indoor environment to obtain the position of the sound source device.

[0008] Furthermore, the raw data collection and processing in step (1) specifically includes:

[0009] (11) Original audio data acquisition: The dual microphones of the mobile terminal include a top microphone and a bottom microphone. Dual-channel acquisition is used to obtain sound information with the same sampling frequency and a synchronized clock. Therefore, the time difference between the sound reaching the top and bottom microphones is used to analyze the position of the sound source relative to the mobile terminal.

[0010] (12) Low-pass filtering to process high-frequency noise: Since the noise distribution is often high-frequency relative to the sound generated by the sound source equipment, a low-pass filter is used to process the microphone data to remove some of the noise;

[0011] (13) Extraction of effective sound information: Compared with the ambient noise, the effective sound amplitude generated by the sound source device has a waveform feature of sudden increase and decrease. Therefore, a sliding window is used to calculate the variance of the sound amplitude, and the moment when the variance is greater than the threshold is identified as the start or end moment of the effective sound, thereby extracting the effective sound information.

[0012] Furthermore, the mobile terminal positioning information modeling in step (2) specifically includes:

[0013] (21) Dual-microphone TDoA calculation: The effective sound is divided into a fixed-length sampling sample set, and the GCC-PHAT algorithm is used to calculate the TDoA of the top-bottom microphone for each sample set;

[0014] (22) Determination of the direction of the sound source device: The microphones at both ends of the mobile terminal use the same sampling frequency and the distance between the microphones at both ends is fixed. TDoA is expressed by the number of sampling samples within a fixed range. The positive and negative values ​​of TDoA are used to determine which half of the mobile terminal the sound source device is located in.

[0015] (23) Positioning information modeling: By combining the mobile terminal position coordinates (x, y), the mobile terminal orientation ο, and the sound source device orientation information f, the positioning information quadruple (x, y, ο, f) is constructed. Under this modeling, the positive direction of the y-axis of the position coordinate system is the geographic north direction, and the mobile terminal orientation ο represents the clockwise rotation angle of the mobile terminal relative to the north direction, with a value range of 0 to 360. The sound source device orientation information f takes 1 to indicate that the sound source is closer to the bottom microphone, and takes -1 to indicate that it is closer to the top.

[0016] Furthermore, the sound source device positioning based on space segmentation in step (3) specifically includes:

[0017] (31) For each positioning information quadruple (x, y, o, f), the dual-microphone midpoint perpendicular line is calculated, and its analytical expression is calculated by the following formula:

[0018]

[0019]

[0020] c=yk ver *x (3)

[0021] Among them, k is the slope of the line where the two microphones are located in the modeling coordinate system, k ver is the slope of the median line, and the analytical expression of the median line of the dual microphone is y=k ver x+c, c is a constant;

[0022] (32) Calculate which side of the perpendicular bisector the sound source device is located on based on the mobile terminal's orientation ο and the sound source device's orientation information f. Use the parameter flag to indicate this. 1 indicates it is located below the perpendicular bisector, and -1 indicates it is located above. The parameter flag is calculated using the following formula:

[0023]

[0024] (33) For m positioning information measurement points, we get m triplets (k ver ,c,flag), for the coordinate points (x i ,y i ), where the subspace coordinates of the sound source device satisfy constraints (5) or (6)

[0025] y i <k ver *x i +c,flag=1 (5)

[0026] y i >k ver *x i +c,flag=-1 (6)

[0027] (34) Find the subspace consisting of n coordinate points that meet the constraints, calculate the mean of the coordinate points in the subspace, and predict the position coordinates of the sound source device. The coordinate calculation formula is as follows:

[0028]

[0029]

[0030] Among them, (x i ,y i ) is the coordinate in the subspace, (x pred ,y pred ) is the predicted sound source position coordinate.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0032] 1. The present invention does not rely on a large number of sensors or high-precision microphone arrays. It uses a mobile terminal, such as a single smartphone, to achieve sound source device positioning in smart home scenarios without the need for additional equipment. Compared with existing indoor sound source positioning technologies, it is more economical.

[0033] 2. The present invention uses dual-microphone TDoA to determine the direction of the sound source device. Since TDoA is only used for binary judgment and does not rely too much on its accuracy, it has stronger robustness than the existing technology.

[0034] 3. By combining different types of signals, the present invention allows users to locate indoor sound source devices by simply moving their smartphones to collect information. This method is convenient and efficient, requiring no additional operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a workflow diagram of the positioning system of the present invention.

[0036] Figure 2 It is a schematic diagram of the sound source localization process of the present invention. DETAILED DESCRIPTION

[0037] This invention designs and implements a smart home device positioning method based on a dual-microphone mobile terminal to locate sound source devices in a smart home environment. The main processes include raw data acquisition and processing, smart mobile terminal positioning information modeling, and sound source device positioning based on spatial segmentation. While this embodiment uses a smartphone as an example of a smart mobile terminal, actual use is not limited to smartphones; other smart mobile terminals capable of implementing the present invention's solution can also be used. Figure 1 The workflow of the positioning system of the present invention is shown, which specifically includes:

[0038] 1. Raw data collection and processing

[0039] The original data collection and processing part of the present invention includes four parts: original sound source data collection, original sound source data denoising, effective sound information extraction, and smartphone orientation angle data collection.

[0040] Original sound source data acquisition: The present invention uses a top microphone and a bottom microphone in each smartphone to acquire sound information with the same sampling frequency and a synchronous clock using dual-channel acquisition.

[0041] Denoising of original sound source data: Analyze the original sound information distribution and use a low-pass filter to process the microphone data to remove obvious high-frequency noise.

[0042] Extraction of effective sound information: Analyze the waveform characteristics of the effective sound amplitude with steep increases and decreases, divide the entire audio data into m segments with 400 samples in each segment, and calculate the variance of each segment; then use the mean of these m variances as the threshold, start from both ends of the overall data, and use a sliding window of size 400, sliding 10 samples each time to traverse towards the middle, and calculate the variance within the window; if it is greater than the threshold, it is considered that the start or end moment of the effective sound is found, and then the audio of the effective sound T is extracted to extract the effective sound information.

[0043] Smartphone orientation data collection: The smartphone's accelerometer and magnetic sensor data are collected through the smartphone's sensor module to calculate the smartphone's deflection angle relative to true north. Alternatively, the smartphone's orientation angle is obtained using the deflection angle calculation interface packaged within the sensor of some smartphone models. The smartphone's coordinate position is recorded at each sampling time.

[0044] 2. Smartphone Positioning Information Modeling

[0045] The smartphone positioning information modeling in the present invention includes three parts: dual-microphone TDoA calculation, sound source device orientation determination, and sound source information modeling.

[0046] Dual-microphone TDoA calculation: The TDoA of the two microphones is calculated based on the extracted effective sound T. Because the number of samples in the effective sound portion can be large, calculating the TDoA for the entire effective sound segment will result in significant error. Therefore, this article divides the effective sound into m segments, each with 1500 samples. The TDoA calculation is performed on the same segment of samples from both microphones, resulting in m TDoA results. For example, in a Xiaomi phone, the distance between its two microphones is 15.5 cm. Based on the speed of sound of 340 m / s and a sampling frequency of 44100 Hz, it can be calculated that the sound reaches the two microphones at most 20 samples apart. Therefore, the TDoA expressed in terms of the number of samples should be an integer between -20 and 20.

[0047] Sound source device orientation determination: If one or more of the m TDoA results are outside this range, the sound data collected at that location is deemed invalid and will no longer be used in subsequent calculations. If all m results are within this range, the number of positive and negative numbers in these m TDoAs is compared. If there are more positive numbers, it means that the sound source is closer to the bottom microphone; if there are more negative numbers, it means that the sound source is closer to the top microphone. If there are the same number of positive and negative numbers, the measurement is considered inaccurate and the data is discarded.

[0048] Sound source information modeling: Combine the smartphone's position coordinates (x, y), the smartphone's orientation ο, and the sound source device orientation information f to construct a positioning information quadruple (x, y, ο, f). In this modeling, the positive direction of the y-axis of the position coordinate system is the geographic north direction. The smartphone orientation ο represents the clockwise rotation angle of the phone relative to the north direction, with a value range of 0 to 360. The sound source device orientation information f indicates which half of the phone the sound source is in. 1 indicates the lower half, that is, the sound source is closer to the bottom microphone, and -1 indicates the lower half, that is, closer to the top microphone.

[0049] 3. Sound source device positioning based on spatial segmentation

[0050] The dual-microphone perpendicular midline is calculated for each positioning information quadruple, and its analytical expression can be calculated by the following formula:

[0051]

[0052]

[0053] c=yk ver *x (3)

[0054] Among them, k is the slope of the line where the two microphones are located in the modeling coordinate system, k ver is the slope of the median line, and the analytical expression of the median line of the dual microphone is y=k ver x+c;

[0055] (32) Calculate which side of the perpendicular bisector the sound source device is located on based on the mobile phone orientation ο and the sound source device orientation information f. Use the parameter flag to indicate this. 1 indicates that the sound source device is located on the lower side of the perpendicular bisector, and -1 indicates that the sound source device is located on the upper side. The parameter flag can be calculated using the following formula:

[0056]

[0057] (33) For m positioning information measurement points, we get m triplets (k ver ,c,flag), for the coordinate points (x i ,y i), which may contain subspace coordinates of the sound source device satisfying constraints (5) or (6)

[0058] y i <k ver *x i +c,flag=1 (5)

[0059] y i >k ver *x i +c,flag=-1 (6)

[0060] (34) Find the subspace consisting of n coordinate points that meet the constraints, calculate the mean of the coordinate points in the subspace, and predict the position coordinates of the sound source device. The coordinate calculation formula is as follows:

[0061]

[0062]

[0063] Among them, (x i ,y i ) is the coordinate in the subspace, (x pred ,y pred ) is the predicted sound source position coordinate.

[0064] Figure 2 The sound source localization process of the present invention is demonstrated.

[0065] The English abbreviations used in the present invention are TDoA, which is interpreted as Time Difference of Arrival (TDoA); GCC-PHAT, which is interpreted as Generalized Cross Correlation-Phase Transform (GCC-PHAT).

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit its scope of protection. Although the present application has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading this application, those skilled in the art may still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the application.

Claims

1. A smart home device positioning method based on a dual-microphone mobile terminal, characterized in that: The method includes: (1) Raw data acquisition and processing: The dual microphones of the mobile terminal are used as sensors to collect sound signals and the orientation of the mobile terminal. Information is collected at multiple locations during the positioning process. The sound signal of the sound source device to be located is obtained through filtering, denoising, and effective sound extraction. (2) Mobile terminal positioning information modeling: By calculating the TDoA of the two microphones, analyzing the relative position of the sound source device to be located relative to the microphones, and combining the indoor coordinates of the mobile terminal and the mobile terminal orientation information, the positioning space is modeled and the corresponding four-tuple positioning information is constructed for each acquisition position; The details are as follows: (21) Dual-microphone TDoA calculation: The effective sound is divided into a fixed-length sampling sample set, and the GCC-PHAT algorithm is used to calculate the TDoA of the top-bottom microphone for each sample set; (22) Determination of the direction of the sound source device: The microphones at both ends of the mobile terminal use the same sampling frequency and the distance between the microphones at both ends is fixed. TDoA is expressed by the number of sampling samples within a fixed range. The positive and negative values ​​of TDoA are used to determine which half of the mobile terminal the sound source device is located in. (23) Positioning information modeling: By combining the mobile terminal position coordinates (x, y), the mobile terminal orientation o, and the sound source device orientation information f, a positioning information quadruple (x, y, o, f) is constructed. In this modeling, the positive direction of the y-axis of the position coordinate system is the geographic north direction. The mobile terminal orientation o represents the clockwise rotation angle of the mobile terminal relative to the north direction, with a value range of 0 to 360. The sound source device orientation information f takes 1 to indicate that the sound source is closer to the bottom microphone, and takes -1 to indicate that the sound source is closer to the top. (3) Sound source device positioning based on spatial segmentation: Based on the positioning information quadruple, the median perpendicular line of the two microphones is calculated for each quadruple to achieve spatial segmentation; the subspace where the sound source device is located is determined through multiple groups of spatial segmentation; the subspace mean coordinates are mapped to the indoor environment to obtain the position of the sound source device.

2. The smart home device positioning method based on a dual-microphone mobile terminal according to claim 1, characterized in that: The raw data collection and processing in step (1) specifically includes: (11) Original audio data acquisition: The dual microphones of the mobile terminal include a top microphone and a bottom microphone. Dual-channel acquisition is used to obtain sound information with the same sampling frequency and a synchronized clock. The position of the sound source relative to the mobile terminal is analyzed by the time difference between the sound reaching the top and bottom microphones. (12) Low-pass filtering to process high-frequency noise: Since the noise distribution is often high-frequency relative to the sound generated by the sound source equipment, a low-pass filter is used to process the microphone data to remove some of the noise; (13) Extraction of effective sound information: Compared with the ambient noise, the effective sound amplitude generated by the sound source device has a waveform feature of sudden increase and decrease. Therefore, a sliding window is used to calculate the variance of the sound amplitude, and the moment when the variance is greater than the threshold is identified as the start or end moment of the effective sound, thereby extracting the effective sound information.

3. The smart home device positioning method based on a dual-microphone mobile terminal according to claim 1, characterized in that: The sound source device positioning based on space segmentation in step (3) specifically includes: (31) For each positioning information quadruple (x, y, o, f), the dual-microphone midpoint perpendicular line is calculated, and its analytical expression is calculated by the following formula: c=y-k ver *x (3) Among them, k is the slope of the line where the two microphones are located in the modeling coordinate system, k ver is the slope of the median line, and the analytical expression of the median line of the dual microphone is y=k ver x+c, c is a constant; (32) Calculate which side of the perpendicular bisector the sound source device is located on based on the mobile terminal's orientation ο and the sound source device's orientation information f. Use the parameter flag to indicate this. 1 indicates it is located below the perpendicular bisector, and -1 indicates it is located above. The parameter flag is calculated using the following formula: (33) For m positioning information measurement points, we get m triplets (k ver , c, flag), for coordinate points (x i ,y i ), where the subspace coordinates of the sound source device satisfy constraints (5) or (6) y i <k ver *x i +c,flag=1 (5) y i >k ver *x i +c,flag=-1 (6) (34) Find the subspace consisting of n coordinate points that meet the constraints, calculate the mean of the coordinate points in the subspace, and predict the position coordinates of the sound source device. The coordinate calculation formula is as follows: Among them, (x i ,y i ) is the coordinate in the subspace, (x pred ,y pred ) is the predicted sound source position coordinate.

Images