A device selection system and method

Abstract

A device selection system and method are provided. The system includes a head-mounted device with a wireless sensor array disposed thereon. The wireless sensor array includes at least two wireless sensors for receiving wireless signals emitted by devices in the environment. The head-mounted device is configured to: calculate the direction of arrival (AOI) of a corresponding device based on the wireless signals received by the wireless sensor array from the devices in the environment; calculate the direction of the user's selection action relative to the head-mounted device based on the user's selection action; and select a target device based on the AOI of the device and the direction of the user's selection action.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and more particularly to a device selection system and method based on a head-mounted device. Background Technology

[0002] Existing smart device selection technologies mainly include infrared signal-based selection technology, audio signal-based selection technology, image signal-based selection technology, and wireless signal synchronization-based selection technology.

[0003] Infrared signal-based selection technology is often used in remote controls paired with smart devices. The remote control is equipped with an infrared transmitter, and the smart device has an infrared receiver. Users select the device by pointing the remote control at the receiver. Audio signal-based selection technology is mainly used in smart devices with microphones and speakers. Users can wave a smart device with a speaker (such as a mobile phone) towards the target device. The target device performs Doppler analysis on the audio signal received by its microphone to confirm the selection. In image signal-based selection technology, users can take a picture of the target smart device and identify and recognize it in the photo for pairing. In wireless signal synchronization-based selection technology, users observe an LED on the target smart device and tap the wireless device in sync with the LED's flashing frequency. This causes a periodic change in the wireless broadcast signal strength. The target smart device analyzes the correlation between the wireless broadcast signal strength change and its own flashing frequency to determine whether to pair.

[0004] The aforementioned technologies typically require users to hold objects and make gestures for selection, such as remote controls or mobile phones, which reduces the interactive experience. Existing smart device selection methods usually require users to learn, such as shaking or simultaneous tapping, which is not natural enough, and usually requires the addition of hardware on the target smart device, such as infrared receivers, microphones, and LED lights, increasing system costs. Summary of the Invention

[0005] In view of the above-mentioned problems of the prior art, the present invention proposes a device selection system, the system including a head-mounted device, on which a wireless sensor array is arranged, the wireless sensor array including at least two wireless sensors for receiving wireless signals emitted by devices in the environment;

[0006] The head-mounted device is configured as follows:

[0007] The direction angle of arrival of the corresponding device is calculated based on the wireless signals emitted by the devices in the environment received by the wireless sensor array.

[0008] Calculate the user's orientation relative to the head-mounted device based on the user's selection action; and

[0009] The target device is selected based on the incoming wave direction angle of the device and the direction of the user's selection action.

[0010] In one embodiment, the at least two wireless sensors are arranged on a first side and a second side of the frame of the head-mounted device.

[0011] In one embodiment, the wireless sensor array includes five wireless sensors arranged along the frame of the head-mounted device, wherein the second, third, and fourth wireless sensors are evenly arranged along a first side of the frame, and the first and fifth wireless sensors are arranged along a second side of the frame and are staggered with the second, third, and fourth wireless sensors, wherein the first and second sides are opposite each other.

[0012] In one embodiment, the wireless sensor array includes five wireless sensors arranged along the frame of the head-mounted device, wherein the second, third, and fourth wireless sensors are evenly arranged along a first side of the frame, the first and fifth wireless sensors are arranged along a second side of the frame, and the distance between the first and fifth wireless sensors is as far as possible, wherein the first and second sides are opposite each other.

[0013] In one embodiment, the head-mounted device calculates the direction angle of arrival of the corresponding device based on the MUSIC algorithm.

[0014] In one embodiment, the user's selection action is either a head movement or a hand movement.

[0015] In one embodiment, the difference between the incoming wave direction angle of the device and the direction of the user's selection action is calculated using the following formula, and the device with the smallest difference is selected as the target device:

[0016] ΔE(D h D dev )=-(a1 sin(θ h sin(θ) dev )+a2 cos(θ h cos(θ) dev cos(φ) h -φ dev ))

[0017] Where ΔE is the difference between the incoming wave direction angle of the device and the direction of the user's selected action, D h D is the direction of the user's selection action. devθ is the direction angle of arrival of the device. h θ is the pitch angle of the user's selected action. dev It is the pitch angle of the equipment, φ h It is the azimuth angle of the user's selection action, φ dev A is the azimuth angle of the device, and a1 and a2 are weights.

[0018] In one embodiment, when the user's selection action is a head movement, θ h and φ h All are 0.

[0019] In one embodiment, a target device is selected based on the direction angle of arrival of the device and the direction of the user's selection action using a machine learning classifier, wherein the machine learning classifier is obtained by training on the direction angle of arrival of each device and the direction of the user's selection action.

[0020] In one embodiment, the user's selected action is a dynamic nodding action, a static pointing action, or a dynamic circling action, and a corresponding machine learning classifier is trained for each of the dynamic nodding action, static pointing action, and dynamic circling action.

[0021] In one embodiment, the system further includes a wrist-worn device for transmitting wireless signals, the head-mounted device determining the direction of the user's selected action based on the direction angle of arrival of the wrist-worn device and a human body model.

[0022] In one embodiment, in the human body model, the origin O is set at the chest of the human body model, and positions G, S, D, and F are the positions of the head-mounted device, the shoulders of the human body model, the wrist-mounted device, and the fingers of the human body model, respectively.

[0023] In one embodiment, the position D of the wrist-worn device is determined based on the incoming wave direction angle of the wrist-worn device and the human body model, and the direction of the user's selection action is determined based on the position D of the wrist-worn device and the position G of the head-mounted device.

[0024] The present invention also provides a device selection method for the above-mentioned device selection system, the method comprising:

[0025] Receives wireless signals emitted by devices in the environment through a wireless sensor array in a head-mounted device;

[0026] The direction angle of arrival of the corresponding device is calculated based on the wireless signal received by the wireless sensor array;

[0027] Calculate the user's orientation relative to the head-mounted device based on the user's selection action; and

[0028] The target device is selected based on the incoming wave direction angle of the device and the direction of the user's selection action.

[0029] The device selection system and method based on a head-mounted device of the present invention utilize a wireless sensor array arranged on the head-mounted device to accurately detect the azimuth and pitch angles of wireless signals emitted by external devices. The head-motion-based device selection method allows users to perform head movements such as nodding towards the target device, similar to communicating with a person, achieving a natural device selection. The hand-motion-based device selection method allows the head-mounted device's wireless sensor array to simultaneously track the hand and the azimuth angles of the target device in the environment when the user is wearing a wrist-worn device such as a watch or ring, and matches these azimuth angles to achieve a natural device selection. Attached Figure Description

[0030] Figure 1 A device selection system based on a head-mounted device according to an embodiment of the present invention is shown.

[0031] Figure 2 A schematic diagram of a head-mounted device according to an embodiment of the present invention is shown.

[0032] Figure 3A A schematic diagram of a wireless sensor array 1 according to an embodiment of the present invention is shown.

[0033] Figure 3B A schematic diagram of a wireless sensor array 2 according to an embodiment of the present invention is shown.

[0034] Figure 4A The cross-coupling between antennas ANT1, ANT2 and ANT3 of wireless sensor array 1 and wireless sensor array 2 is shown.

[0035] Figure 4B The azimuth directionality of the wireless sensor array 1 is shown.

[0036] Figure 4C The pitch directionality of the wireless sensor array 1 is shown.

[0037] Figure 4D The azimuth directionality of the wireless sensor array 2 is shown.

[0038] Figure 4E The pitch directionality of the wireless sensor array 2 is shown.

[0039] Figure 5 A schematic diagram of the incoming wave direction angle measurement is shown.

[0040] Figure 6 A schematic diagram of wave direction angle measurement based on the MUSIC algorithm is shown.

[0041] Figure 7 A schematic diagram of gesture direction calculation is shown.

[0042] Figure 8 The effect of different body sizes on gesture orientation estimation is shown.

[0043] Figure 9 The selection action according to an embodiment of the present invention is illustrated.

[0044] Figure 10 A flowchart of a device selection method based on a head-mounted device according to an embodiment of the present invention is shown. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments given in this invention are for illustrative purposes only and do not limit the scope of protection of this invention.

[0046] Figure 1 A device selection system based on a head-mounted device according to an embodiment of the present invention is illustrated. The head-mounted device 100 has a wireless sensor array disposed thereon for receiving wireless signals emitted by devices W1-W4 in the environment. The wireless sensor array includes at least two wireless sensors. According to an embodiment of the present invention, the head-mounted device 100 is configured to: calculate the direction of arrival (AoA) of the corresponding device based on the wireless signals emitted by the devices in the environment received by the wireless sensor array; calculate the direction of the user's selection action relative to the head-mounted device based on the user's selection action; and select a target device based on the AoA of the devices in the environment and the direction of the selection action.

[0047] In one embodiment, the head-mounted device 100 further includes a transceiver for receiving and decoding signals from the wireless sensor array. In another embodiment, the head-mounted device 100 further includes an RF switch for controlling the sequential activation of the at least two wireless sensors.

[0048] Head-mounted devices can be, for example, head-mounted displays, smart glasses, etc. Wireless sensors can be, for example, antennas. Devices W1-W4 in the environment can transmit wireless signals, and these can include mobile phones, tablets, personal computers, smartwatches, etc.

[0049] In one embodiment, devices W1-W4 may include wireless signal transmitters, and the wireless signals may include Bluetooth, Wi-Fi, etc. The wireless signal transmitters on devices W1-W4 periodically transmit wireless signals, and the wireless sensors synchronize with the periodically transmitted wireless signals from multiple devices through time-division multiplexing. The wireless signals typically include data for identifying different devices; after receiving this data, the head-mounted device can identify the data transmitted by different devices.

[0050] Wireless sensor array design

[0051] Figure 2 A schematic diagram of a head-mounted device (e.g., glasses) according to an embodiment of the present invention is shown, wherein five wireless sensors R1, R2, R3, R4, and R5 are disposed on the head-mounted device. The five wireless sensors R1, R2, R3, R4, and R5 are located on the outer frame of the head-mounted device. The horizontal (x-axis direction) wireless sensors R1, R2, and R3, and the horizontal wireless sensors R4 and R5, can be used for azimuth angle measurement, while the vertical (y-axis direction) wireless sensors, such as R1 and R4, R3 and R5, R2 and R5, etc., can be used for pitch angle measurement. Figure 2 As shown, the x-direction is along the length of the head-mounted device frame, the y-direction is along the height of the head-mounted device frame, and the z-direction is perpendicular to the plane containing the head-mounted device frame. In this invention, the plane containing the azimuth angle, i.e., the xz plane, is defined as the azimuth angle plane, and the plane containing the pitch angle, i.e., the yz plane, is defined as the pitch angle plane.

[0052] Figure 2 The arrangement of the wireless sensor array in the example is only for illustration. In practical applications, if the devices in the environment are basically on the same horizontal plane, such as being on the same azimuth plane or the same pitch plane, only two wireless sensors may be used. If the devices in the environment are on both the azimuth plane and the pitch plane, at least three wireless sensors are required to detect the azimuth and pitch angles.

[0053] The resolvable distance d1 of the head-mounted device on any plane (e.g., the azimuth plane and the pitch plane) can be roughly estimated by equation (1):

[0054] d1≈λR / P (1)

[0055] Where λ is the wavelength of the wireless signal, R is the distance between the receiver of the head-mounted device and the target device emitting the wireless signal, and P is the size (e.g., length or area) of the wireless sensor array. As can be seen from equation (1), the larger the size of the wireless sensor array, the smaller the resolvable distance, and the higher the corresponding resolution.

[0056] To avoid ambiguity in the estimated direction of arrival (DOA), the distance Dw between adjacent wireless sensors should be less than half the shortest wavelength in the communication band. For example, in a 2480MHz Bluetooth channel, the distance Dw is less than 60.5mm. Simultaneously, the distance Dw should be as large as possible to allow for a larger wireless sensor array size, thereby minimizing cross-coupling between wireless sensor elements. For instance, in an embodiment using a 2480MHz Bluetooth channel, the distance Dw could be set to 60mm.

[0057] In theory, the size of the wireless sensor array should be as large as possible (i.e., the number of wireless sensors should be as large as possible) to achieve high resolution in both the azimuth and pitch planes. However, the length and height of the wireless sensor array are limited by the head-mounted device frame (e.g., a smart glasses frame). In one embodiment, the length of the prototype device frame can be set according to the width of a human face, for example, the average of the 95th percentile of the width of the bifacial region of men and women, which is 145 mm. The height of the device frame is set to Dw to provide the largest possible area of ​​the wireless sensor array in the pitch plane while maintaining the typical appearance of a head-mounted device (e.g., glasses). In one embodiment, the width W of the wireless sensor array is 145 mm, and the height H is 60 mm.

[0058] The size, shape, directivity, and efficiency of the wireless sensor are also key factors to consider. Preferably, the length of the wireless sensor along the x-axis should be less than Dw, the height along the y-axis should be comparable to the width of the head-mounted device frame, the grounding area should be small or nonexistent, and the directivity should have a zero point on the plane of the head-mounted device frame (i.e., the xy-plane) to minimize cross-coupling in the azimuth direction. The wireless sensor should be small and efficient to save power in the head-mounted device and avoid heat generation, thus preventing user discomfort. In one embodiment, the wireless sensor can be a dipole antenna, which is small in size and highly efficient. In another embodiment, the wireless sensor can be a meander dipole, which is smaller than a dipole antenna and only slightly less efficient.

[0059] Figure 3A A schematic diagram of a wireless sensor array 1 according to an embodiment of the present invention is shown; Figure 3B A schematic diagram of a wireless sensor array 2 according to an embodiment of the present invention is shown. The wireless sensors are antennas, and the wireless sensor array includes antennas ANT1-ANT5 arranged on a head-mounted device frame 20. Figure 3A In the wireless sensor array 1, three antennas ANT2-ANT4 are evenly arranged along the upper length of the frame 20, and two antennas ANT1 and ANT5 are arranged along the lower length of the frame 20, with the distance between antennas ANT1 and ANT5 being as far as possible. Figure 3B In the wireless sensor array 2, three antennas ANT2-ANT4 are evenly arranged along the upper part of the frame 20, and two antennas ANT1 and ANT5 are arranged along the lower part of the frame 20, and are staggered with the three antennas ANT2-ANT4 above, that is, antenna ANT1 is located between antenna ANT2 and antenna ANT3, and antenna ANT5 is located between antenna ANT3 and antenna ANT4.

[0060] Figures 4A-4E The simulation results for wireless sensor array 1 and wireless sensor array 2 are shown. The simulation uses a default zigzag antenna generated by MATLAB at a wireless frequency of 2450 MHz. Figure 4A The diagram illustrates the cross-coupling between antennas ANT1, ANT2, and ANT3 of wireless sensor arrays 1 and 2. Curve L1 represents cross-coupling S13 between antennas ANT1 and ANT3 of wireless sensor array 1; curve L2 represents cross-coupling S13 between antennas ANT1 and ANT3 of wireless sensor array 2; curve L3 represents cross-coupling S12 between antennas ANT1 and ANT2 of wireless sensor array 1; curve L4 represents cross-coupling S12 between antennas ANT1 and ANT2 of wireless sensor array 2; curve L5 represents cross-coupling S23 between antennas ANT2 and ANT3 of wireless sensor array 1; and curve L6 represents cross-coupling S23 between antennas ANT2 and ANT3 of wireless sensor array 2. Figure 4A It can be seen that the cross-coupling of wireless sensor array 2 is smaller than that of wireless sensor array 1. Taking the cross-coupling S12 between antennas ANT1 and ANT2 in the elevation direction as an example, the cross-coupling S12 of wireless sensor array 2 is only 63% of that of wireless sensor array 1. The null point of the bent-line antenna makes the cross-coupling S23 between antennas ANT2 and ANT3 in the azimuth direction around -6.6dB.

[0061] Figure 4B The azimuth directionality of the wireless sensor array 1 is shown. Figure 4C The pitch directionality of wireless sensor array 1 is shown. Figure 4D The azimuth directionality of the wireless sensor array 2 is shown. Figure 4EThe elevation directivity of wireless sensor array 2 is shown. In the azimuth plane, the directivity of array 1 (8.17 dBi) is higher than that of array 2 (7.66 dBi). However, the sidelobes of array 1 (-2.66 dBi) are three times greater than those of array 2 (-7.4 dBi). Higher sidelobes introduce stronger interference from directions not faced by the user, thus reducing the estimated direction of arrival (AOI). Furthermore, the directivity differences in the elevation plane are mainly in the 90° and 270° directions, aligned with the user's body, and have little impact on AIO estimation. Based on these simulation results, the arrangement of array 2 is a better option, and the following explanation uses the arrangement of array 2 as an example.

[0062] Calculation of incoming wave direction angle

[0063] Figure 5 A schematic diagram of the arrival direction angle measurement is shown. In the basic arrival direction angle estimation mechanism, it is assumed that the wireless signal is a monochromatic plane wave. For example... Figure 5 As shown, wireless signals from devices in the environment are incident on wireless sensors R1 and R2, and arrive at wireless sensors R1 and R2 at different times. The phase difference Δφ between the wireless signals received by wireless sensors R1 and R2 is:

[0064] Δφ=2πdcosθ / λ (2)

[0065] Where λ is the wavelength of the wireless signal, δ is the distance between wireless sensors R1 and R2, and θ is the direction angle of arrival of the wireless signal. The phase difference Δφ can usually be calculated. For example, wireless sensors R1 and R2 can read the IQ values ​​of the wireless signal (I is the same-direction component, Q is the quadrature component) through an analog-to-digital converter (ADC), calculate the phases φ1 and φ2 of the wireless signal respectively, and subtract them to obtain the phase difference Δφ. Therefore, the direction angle of arrival θ can be calculated as:

[0066] θ=arcos(λΔφ / 2πd) (3)

[0067] The direct calculation of the incoming wave direction angle based on the phase difference φ is easily affected by multipath effects, polarization mismatch, etc.

[0068] The MUSIC algorithm is one of the most popular radio direction finding algorithms. Figure 6A schematic diagram of the arrival direction angle measurement based on the MUSIC algorithm is shown. It decomposes the wireless signal from device W1 into two orthogonal spaces: a signal space and a noise space. It should be noted that two wireless signals are unlikely to arrive at the antenna array simultaneously because the duration of the wireless signal is shorter (e.g., a maximum of 160 µs) compared to the wireless signal broadcast interval (e.g., a minimum of 20 ms), and the broadcast channel is randomly selected. Therefore, it can be assumed that only one wireless signal arrives at the antenna at any given time. The signal x(t) received by the antenna can be modeled as follows:

[0069] x(t)=As(t)+n(t), t=1, 2,...,N, (4)

[0070] Where s(t) is the wireless signal vector from the device, and n(t) is a vector with variance σ. N 2 The Gaussian noise term, A is the ideal steering matrix, and t represents the device in the environment.

[0071] The phase of x(t) generated by the IQ value of the wireless signal is α. i The array output definition for (t) (i = 1, 2, 3, 4, 5, representing antennas ANT1-ANT5):

[0072]

[0073] The guidance matrix A is:

[0074]

[0075] Among them, (d) xi d yi β1 is the position of each antenna i on the head-mounted device in the XY plane, β2 is the projection direction of the target device in the XY plane, and β3 is the azimuth angle of the target device relative to the Z-axis. Figure 6 As shown.

[0076] In practical applications, the covariance matrix R of the received signal xx A rough estimate is made by averaging the time of the signal x(t) received by the antenna:

[0077]

[0078] The MUSIC algorithm then uses the eigenvector decomposition and eigenvalues ​​of the covariance matrix to separate the signal space and noise space, as shown in the formula:

[0079] R xx =VΛV -1 (8)

[0080] Here, Λ and V are composed of eigenvalues ​​and corresponding eigenvectors.

[0081] The eigenvectors are sorted according to the values ​​of their respective eigenvalues. The first eigenvector with the largest eigenvalue is used as the signal space, and the remaining four eigenvalues ​​and eigenvectors are used as the noise space. This yields the noise vector V. n = [v2, v3, v4, v5]. Finally, the pseudo-spectrum can be obtained according to the following formula:

[0082]

[0083] By selecting the peak values ​​of the spectrum, the directions of the transmitted signal in three-dimensional space, β1 and β2, can be obtained. Then, the direction vector is projected onto the YZ plane, and the true azimuth φ and elevation θ of the device are obtained according to the following equations:

[0084]

[0085] Gesture direction calculation

[0086] In this invention, the appropriate device can be selected by the user's hand and head movements. When selecting a device by hand movements, the device selection system based on a head-mounted device of this invention may further include a wrist-worn device capable of emitting wireless signals. The head-mounted device calculates the direction angle of arrival of the wrist-worn device based on the above method, and determines the user's gesture direction based on the direction angle of arrival of the wrist-worn device and the human body model. The wrist-worn device may be, for example, a watch, a bracelet, or a wristband.

[0087] Figure 7 A schematic diagram illustrating gesture direction calculation is shown. It depicts a human body model, based on which the orientation of the wrist-worn device relative to the head-mounted device can be determined. This is converted into the pointing direction of a gesture. In one embodiment, the origin O is set at the user's chest. Positions G, S, D, and F are the positions of the head-mounted device, shoulder, wrist-mounted device, and finger, respectively. In one embodiment of the invention, a calibration model is built based on the body dimensions of a 21-year-old male with a height of 178 cm to estimate the pointing direction for all users. Data includes... (Half the width of your shoulders) (arm length) (the length of fingers and hand), and (Eye height). Assumption The estimated direction D of the wrist-worn device f for The direction vector. The estimated direction D based on the wrist-worn device. f ,as well as The position D of the wrist-worn device can then be determined. From this, we can obtain... Since the position F of the finger is known and the position G of the head-mounted device is known, the direction D of the user's finger relative to the head-mounted device can be obtained. h ,Right now The direction vector.

[0088] Figure 8 The effect of different body sizes on gesture orientation estimation is illustrated in the figure. Six additional participants with heights between 155 cm and 185 cm were shown, and they were asked to point to a device with both azimuth and pitch angles of 0. The results show that the maximum error in gesture orientation estimation was only 0.48°, occurring in a participant who was 184 cm tall. These results indicate that the correction model does not introduce significant estimation errors for individuals of different heights.

[0089] When selecting a device via head movement, no additional wrist-worn device is required; only the direction angle of arrival of the device needs to be detected. In this case, the pitch and azimuth angles of the gesture direction can be considered to be 0.

[0090] Action selection and recognition

[0091] In this invention, three selection actions that users are already familiar with are designed, such as... Figure 9 As shown, from left to right:

[0092] 1) Static pointing action: The user points to the target device and holds the position for a moment;

[0093] 2) Dynamic circling motion: The user draws a circle in the air around the target device;

[0094] 3) Dynamic nodding action: The user nods at the target device, just like people nodding to each other.

[0095] Users can improve selection accuracy by nodding multiple times, holding a pointing gesture for a few seconds, or drawing multiple circles. In practice, users can also use other gestures to make selections.

[0096] The following explains how to identify dynamic nodding, static pointing, and dynamic circling movements.

[0097] First, it is determined whether the user's head is stationary by analyzing changes in the direction of arrival (AOI) of nearby devices. If the pitch change of the AOI is greater than a predetermined threshold (e.g., 28°) and the azimuth change is less than a predetermined threshold (e.g., 4°), then the nodding motion is considered to have begun. These predetermined thresholds are determined empirically. When the pitch change decreases again and the azimuth change is similar to the value before the nodding motion began, then the nodding motion is considered to have ended.

[0098] The device can be the one closest to the user or the one the user is facing.

[0099] If the head remains stationary, the head-mounted device detects the wireless signal strength of the wrist-worn device to determine whether the hand has been raised. The transceiver in the head-mounted device typically has a signal strength indicator; raising the hand causes a significant increase in signal strength, while lowering the hand causes a significant decrease in signal strength, marking the start and end of the gesture selection.

[0100] During hand motion detection, if the average value of the incoming wave direction angle of the wrist-worn device satisfies a predetermined threshold (e.g., 25°) for both the azimuth and pitch angle changes, the system will mark the hand motion as a random hand movement. If the azimuth change is less than the predetermined threshold (e.g., 25°) and the pitch angle change is less than the predetermined threshold (e.g., 25°), the system will proceed to the next step of hand motion detection.

[0101] For hand gesture recognition, if the standard deviation of the incoming direction angle of the wrist-worn device is less than a predetermined threshold (e.g., 3°) over a period of time, it is determined to be a static pointing motion. Dynamic circling motion produces sinusoidal changes in azimuth and pitch angles. In one embodiment, a 64-point Fast Fourier Transform (FFT) is applied to the changes in pitch and azimuth angles. If the amplitude of frequency components less than 0.5 Hz in the azimuth or pitch plane is greater than a predetermined threshold (e.g., 4.5), it is determined to be a dynamic circling motion.

[0102] Target device selection algorithm

[0103] This invention provides two methods for selecting target devices: an angular distance method and a method based on a machine learning classifier.

[0104] Angular distance method: The angular distance method selects based on the angular difference between the device's incoming wave direction angle and the gesture direction. It quantifies the 3D angular distance on the azimuth and elevation planes separately. Figure 3B The antenna array shown estimates the azimuth angle more accurately. The range metric has a greater weight in the azimuth angle than the elevation angle, as shown below:

[0105] ΔE(D h D dev )=-(a1 sin(θ h sin(θ) dev )+a2 cos(θ h cos(θ) dev cos(φ) h -φ dev (11)

[0106] Where ΔE represents the difference between the incoming wave direction angle and the gesture direction angle of the device, D h Indicates the direction of the gesture, D dev θ represents the direction angle of incoming waves from the device. h The pitch angle, θ, represents the gesture. dev φ represents the pitch angle of the equipment. h The azimuth angle of a gesture, φ dev This represents the azimuth angle of the device. a1 and a2 are weights, which can be set to 1 and 1.5 respectively based on experience. Then, the target device i is selected by solving the following equation. * :

[0107]

[0108] Among them, D h (j) represents the j-th sample of the gesture direction during the selection period. It is the average estimated incoming wave direction angle of the i-th device between the start and end of the gesture. For a nodding gesture, the gesture direction D... h Both the azimuth and elevation angles are set to 0°. In this method, the device with the smallest angle difference ΔE is selected as the target device by calculating the difference between the incoming wave direction angle and the gesture direction.

[0109] Machine learning classifier-based methods: These methods take the direction angle of arrival of the device as input and output whether the device is the target device.

[0110] The device selection task can be viewed as a classification problem. Each device is classified using a model generated from the selection action data to determine whether it is a target device. When the azimuth and pitch angles are close to zero, triangulation is first applied to improve the model's sensitivity. Then, a machine learning classifier is trained for each selection action instance, and these models are used to determine whether a device is a selection target.

[0111] In this invention, the user typically looks directly at the target device during selection, which adjusts the orientation of the antenna array so that the azimuth and elevation angles of the target device are close to zero. Therefore, a trigonometric mapping (a,e) = (sinφ,sinθ) is applied to all incoming wave direction angle data to improve sensitivity around such areas.

[0112] 1) For nodding motions, a classifier can be pre-trained. The sample region of this classifier is -10° < pitch angle < 30°, -5° < azimuth angle < 5°, and filled with 0.5° density. This sample region is obtained in advance through experiments. Using this region as samples, a pose model of the nodding motion is obtained by training based on the One-class SVM algorithm. The corresponding device can then be classified by this pose model.

[0113] When a user's nodding action is detected, for each device, its corresponding incoming wave direction angle is input into the nodding action posture model. The model outputs the probability of each device in a specific category (i.e. the category of the selected device). Finally, the average selection probability of each device is calculated, and the device with the highest selection probability is selected as the target device.

[0114] 2) For pointing gestures, a machine learning model is trained in real-time for each selection gesture immediately after a drop event is detected (i.e., the pointing gesture is completed). Gesture direction data is extracted during the pointing gesture (i.e., the time from raising to lowering the hand), which approximates the direction of arrival angle of the target device. Using the gesture direction data as samples, a pose model for the pointing gesture is trained based on the One-class SVM algorithm, allowing the corresponding device to be classified by this pose model.

[0115] When a user's action is detected as a pointing action, for each device, its corresponding incoming wave direction angle is input into the gesture model of the pointing action. The model outputs the probability of each device in a specific category (i.e. the category of the selected device). Finally, the average selection probability of each device is calculated, and the device with the highest selection probability is selected as the target device.

[0116] 3) For circling gestures, similar to pointing gestures, a new model is trained in real time for each gesture. First, a Kalman filter (Q=10) is applied. -4 R = 4 × 10 -5 The system smooths the movement trajectory of the gesture direction during the circling motion. Then, it fills the area enclosed by the smoothed movement trajectory in 0.5° increments on the azimuth and elevation planes. Using this area as samples, it trains a pose model of the circling motion based on the One-class SVM algorithm, allowing the corresponding device to be classified by this pose model.

[0117] When a user's action is detected as a circling motion, for each device, its corresponding incoming wave direction angle is input into the circling motion pose model. The model outputs the probability of each device in a specific category (i.e. the category of the selected device). Finally, the average selection probability of each device is calculated, and the device with the highest selection probability is selected as the target device.

[0118] The present invention also provides a device selection method based on a head-mounted device; Figure 10 A flowchart of the method is shown, which includes:

[0119] Step S1: Receive wireless signals emitted by devices in the environment through the wireless sensor array of the head-mounted device;

[0120] Step S2: Calculate the direction angle of arrival of the corresponding device based on the wireless signal received by the wireless sensor array;

[0121] Step S3: Calculate the user's orientation relative to the head-mounted device based on the user's selection action;

[0122] Step S4: Select the target device based on the incoming wave direction angle of the device and the direction of the selection action.

[0123] This invention utilizes a wireless sensor array mounted on a head-mounted device to receive wireless signals emitted by devices in the environment and calculate their three-dimensional spatial direction angles (including azimuth and pitch angles) to obtain the relative angle between the head-mounted device and the devices in the environment. When the user moves their head or hand, the relative angle between the head-mounted device, their hand, and the devices in the environment will change. By monitoring and analyzing the angle change signals, the device selected by the user can be determined.

[0124] The device selection system and method based on a head-mounted device of the present invention utilize a wireless sensor array arranged on the head-mounted device to accurately detect the azimuth and pitch angles of incoming waves from external devices. The head-motion-based device selection method allows users to perform head movements such as nodding towards the target device, similar to communicating with a person, achieving a natural device selection. The hand-motion-based device selection method allows the head-mounted device's wireless sensor array to simultaneously track the hand's azimuth angle and the incoming wave direction angle of the target device in the environment when the user is wearing a wireless wrist-worn device such as a watch or ring, and matches the azimuth angles of the two to achieve a natural device selection.

[0125] This invention eliminates the need for additional hardware on the target device, saving costs. Simultaneously, users can select devices using natural gestures such as nodding or pointing, achieving a natural and immediate interactive experience.

[0126] While the present invention has been described through preferred embodiments, it is not limited to the embodiments described herein, and various changes and modifications are made without departing from the scope of the invention.

Claims

1. A device selection system, the system comprising a head-mounted device having a wireless sensor array disposed thereon, the wireless sensor array comprising at least two wireless sensors for receiving wireless signals emitted by devices in the environment; The head-mounted device is configured as follows: The direction angle of arrival of the corresponding device is calculated based on the wireless signals emitted by the devices in the environment received by the wireless sensor array. Calculate the user's orientation relative to the head-mounted device based on the user's selection action; and The target device is selected based on the incoming wave direction angle of the device and the direction of the user's selection action, wherein the user's selection action is the user's head movement or hand movement.

2. The equipment selection system according to claim 1, wherein, The at least two wireless sensors are arranged on the first and second sides of the frame of the head-mounted device.

3. The equipment selection system according to claim 2, wherein, The wireless sensor array includes five wireless sensors arranged along the frame of the head-mounted device, wherein the second, third, and fourth wireless sensors are evenly arranged along a first side of the frame, and the first and fifth wireless sensors are arranged along a second side of the frame and are staggered with the second, third, and fourth wireless sensors, wherein the first and second sides are opposite each other.

4. The equipment selection system according to claim 2, wherein, The wireless sensor array includes five wireless sensors arranged along the frame of the head-mounted device, wherein the second, third, and fourth wireless sensors are evenly arranged along a first side of the frame, and the first and fifth wireless sensors are arranged along a second side of the frame, with the first wireless sensor located away from the fifth wireless sensor and the distance between them being less than the length of the second side of the frame, wherein the first and second sides are opposite each other.

5. The equipment selection system according to claim 1, wherein, The head-mounted device calculates the incoming wave direction angle of the corresponding device based on the MUSIC algorithm.

6. The equipment selection system according to claim 1, wherein, The difference between the incoming wave direction angle of the device and the direction of the user's selected action is calculated using the following formula, and the device with the smallest difference is selected as the target device: Where ΔE is the difference between the incoming wave direction angle of the device and the direction of the user's selected action. It is the direction of the user's selection action. It is the direction angle of arrival of the device. It is the pitch angle of the user's selected action. It is the pitch angle of the equipment. It is the azimuth angle of the user's selection action. α1 and α2 are the azimuth angles of the device.

7. The equipment selection system according to claim 6, wherein, When the user's selection action is a head gesture. and All are 0.

8. The equipment selection system according to claim 1, wherein, The target device is selected by a machine learning classifier based on the incoming wave direction angle of the device and the direction of the user's selection action. The machine learning classifier is obtained by training on the incoming wave direction angle of each device and the direction of the user's selection action.

9. The equipment selection system according to claim 8, wherein, The user's selected action is either a dynamic nodding action, a static pointing action, or a dynamic circling action. A corresponding machine learning classifier is trained for each of the dynamic nodding action, static pointing action, and dynamic circling action.

10. The equipment selection system according to any one of claims 1-8, wherein, The system also includes a wrist-worn device for transmitting wireless signals, the head-mounted device determining the direction of the user's selected action based on the incoming wave direction angle of the wrist-worn device and the human body model.

11. The device selection system according to claim 10, wherein, In the human body model, the origin O is set at the chest, and positions G, S, D, and F are the positions of the head-mounted device, the shoulders, the wrist-mounted device, and the fingers, respectively. ⊥ , ⊥ , ∥ .

12. The device selection system according to claim 11, wherein, The position D of the wrist-worn device is determined based on the incoming wave direction angle of the wrist-worn device and the human body model. The direction of the user's selection action is determined based on the position D of the wrist-worn device and the position G of the head-mounted device.

13. A device selection method for a device selection system according to any one of claims 1-12, the method comprising: Receives wireless signals emitted by devices in the environment through a wireless sensor array in a head-mounted device; The direction angle of arrival of the device is calculated based on the wireless signal received by the wireless sensor array. Calculate the user's orientation relative to the head-mounted device based on the user's selection action; as well as The target device is selected based on the incoming wave direction angle of the device and the direction of the user's selection action.

Images