Wearable device control method and wearable device

Abstract

The invention discloses a control method of wearable equipment and the wearable equipment. The method comprises the following steps: responding to equipment entering an interactive interface, and displaying a plurality of selectable items in a plurality of display areas of a screen; detecting the noise of the environment where the equipment is located; in response to the fact that the noise is larger than the first noise threshold value and smaller than the second noise threshold value, staring coordinates of the wearer on the screen and a voice recognition result are determined; at least inputting a display area of a first selectable item corresponding to the noise and the voice recognition result, a first distance of a second selectable item closest to the gazing coordinate and a display area of the second selectable item into a trained machine learning model to obtain a selected display area; and controlling the interactive interface based on the selected options of the selected display area. According to the method, the camera arranged on the wearable device is used for collecting the eye image of the user to obtain the staring coordinate, and the recognized voice instruction is combined to be jointly used for determining the user instruction based on machine learning network reasoning in the noisy environment.

Description

Technical Field

[0001] The present disclosure relates to the field of interactive control, and in particular to a control method for a wearable device and the wearable device. Background Art

[0002] Wearable devices such as virtual reality devices, augmented reality devices, and mixed reality devices can provide a rich immersive experience, allowing wearers to interact with real objects, real environments, virtual objects, and / or virtual environments with the help of wearable devices.

[0003] Common operating methods for wearable devices include keystrokes, gestures, head movements, and / or voice control. Keystrokes, performed through physical buttons or touchscreens, are simple and reliable, while head movements offer limited functionality. In situations where hands-free operation is inconvenient, such as running, voice control, which uses voice commands to control functions, is more convenient.

[0004] However, voice control has certain requirements for the usage environment. When the surrounding noise is large, the signal-to-noise ratio of the voice signal is low, which may lead to a decrease in the voice command recognition rate, thereby reducing the accuracy and success rate of voice interaction. Summary of the Invention

[0005] According to a first aspect of the present disclosure, a control method for a wearable device is provided, comprising: in response to the wearable device entering an interactive interface, displaying multiple optional items in multiple display areas of a screen of the wearable device; detecting noise in an environment in which the wearable device is located; in response to the noise being greater than a first noise threshold and less than a second noise threshold, determining the gaze coordinates of a wearer of the wearable device on the screen, and determining a speech recognition result corresponding to the wearer's speech information; inputting at least the noise, the display area of ​​a first optional item corresponding to the speech recognition result, a first distance of a second optional item closest to the gaze coordinate, and the display area of ​​the second optional item into a trained machine learning model to obtain a selected display area, wherein the selected display area is one of the display area of ​​the first optional item and the display area of ​​the second optional item; and controlling the interactive interface based on the selected optional item of the selected display area.

[0006] Optionally, the first optional item corresponding to the speech recognition result is the same as or different from the second optional item closest to the gaze coordinates.

[0007] Optionally, the method further includes: inputting the first distance of the third optional option that is second closest to the gaze coordinate and the display area of ​​the third optional option into the trained machine learning model to obtain the selected display area.

[0008] Optionally, the method also includes: in response to the noise being less than the first noise threshold, determining a voice recognition result corresponding to the wearer's voice information, and using the first optional option corresponding to the voice recognition result to control the interactive interface; and / or in response to the noise being greater than the second noise threshold, determining the gaze coordinates of the wearer of the wearable device on the screen, and using the second optional option closest to the gaze coordinates to control the interactive interface.

[0009] Optionally, determining the gaze coordinates of the wearer of the wearable device on the screen includes: using a camera on the wearable device to capture an eye image of the wearer; extracting a pupil position from the eye image; and determining the gaze coordinates corresponding to the pupil position based on a mapping relationship.

[0010] Optionally, the mapping relationship is determined according to a calibration operation, and the calibration operation includes: displaying multiple calibration points on the screen; obtaining the pupil coordinates of the wearer when looking at each calibration point; and determining the mapping relationship based on the pupil coordinates and the coordinates of the multiple calibration points on the screen.

[0011] Optionally, using the camera on the wearable device to capture the wearer's eye image includes: after displaying the multiple optional items on the screen, detecting the wearer's voice input in response to voice activity detection, continuously capturing the eye image, and determining the gaze coordinates corresponding to the pupil position based on the mapping relationship includes: determining the gaze coordinates in response to the pupil position extracted from the continuously captured eye images indicating that the gaze at the same area of ​​the multiple display areas reaches a predetermined threshold within a predetermined time period.

[0012] Optionally, determining the voice recognition result corresponding to the wearer's voice information includes: using the microphone of the wearable device to collect the voice information input by the wearer and sending the voice into a voice recognition model; obtaining the voice recognition result output by the voice recognition model, the voice recognition result including a character string and a confidence level, wherein the display area of ​​the first optional item is determined based on the character string; and the method also includes: sending the confidence level as input into the machine learning model to obtain the selected display area.

[0013] Optionally, the machine learning model includes output categories corresponding to the multiple display areas and a detection failure category respectively; and when the output of the machine learning model corresponds to the output category of a display area and the probability is higher than a predetermined threshold value, the display area is determined to be the selected display area.

[0014] According to a second aspect of the present disclosure, a wearable device is provided, comprising: a screen for displaying an interactive interface and displaying multiple optional items in multiple display areas; a microphone for collecting noise and voice input by the wearer; a camera for collecting eye images of the wearer to determine gaze coordinates; a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method based on the first aspect.

[0015] According to a third aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to the first aspect.

[0016] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, implement the method according to the first aspect.

[0017] Therefore, for the current noise level that falls within the predetermined range, the present disclosure comprehensively considers the speech recognition and eye tracking results and noise based on machine learning model reasoning to determine the user command, thereby improving the accuracy of user interaction in non-button control scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0019] Figure 1 An example of the structure of the smart glasses provided by the present disclosure is shown.

[0020] Figure 2 A schematic flowchart of a control method for a wearable device according to an embodiment of the present disclosure is shown.

[0021] Figure 3 An example of a smart glasses control method provided by the present disclosure including operation branches under different noise levels is shown.

[0022] Figure 4A -B shows examples of optional options for different forms of display provided by the present disclosure.

[0023] Figure 5A -D shows an example of displaying multiple options for eye tracking provided by the present disclosure.

[0024] Figure 6A -C shows an example of human eye and pupil extraction provided by the present disclosure.

[0025] Figure 7 An example of the correspondence between pupil coordinates and smart glasses screen coordinates provided by the present disclosure is shown.

[0026] Figure 8A -B shows two examples of calibration operations provided by the present disclosure.

[0027] Figures 9A-9D An example is shown in which the screen provided by the present disclosure includes four quadrants and gaze coordinates are located at different positions. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art. In addition, it should be understood that the "first", "second" and "third" or similar expressions in the present disclosure are only used for descriptive and distinguishing purposes, and should not be understood as indicating or implying the order of appearance or importance of the corresponding objects.

[0029] Wearable devices such as virtual reality devices, augmented reality devices, and mixed reality devices can provide a rich immersive experience, where the wearer can interact with real objects, real environments, virtual objects, and / or virtual environments with the help of the wearable device. Artificial reality content (including augmented reality content, virtual reality content, and mixed reality content) can include completely computer-generated content (such as virtual reality content), or computer-generated content combined with collected content (such as real-world augmented reality content, mixed reality content). Artificial reality content can include video, audio, tactile feedback, or some combination thereof, any of which can be presented in a single channel or in multiple channels (for example, a video that produces a two-dimensional or three-dimensional effect to the wearer). In addition, in some embodiments, artificial reality can also be associated with applications, products, accessories, services, or some combination thereof, which are used to, for example, create content in artificial reality and / or otherwise be used in artificial reality (for example, to perform activities in artificial reality).

[0030] Artificial reality systems can be implemented in a variety of different form factors and configurations. Some artificial reality systems can be designed to work without a near-eye display (e.g., audio / AI glasses, head-mounted patches, headsets, wrist-mounted devices, etc.). Other artificial reality systems can include near-eye displays that also provide visibility to the real world, or near-eye displays that visually immerse the user in an artificial reality. Some artificial reality devices can be stand-alone systems, while others can communicate and / or coordinate with external devices to provide an artificial reality experience to the user. Examples of external devices include handheld controllers (e.g., handheld mobile stations that provide various social, video, game, etc. resources), mobile devices, desktop computers, devices worn by the wearer, devices worn by one or more other wearers, and / or any other suitable external systems.

[0031] Artificial reality systems are often used for office work, social interaction, gaming, and other purposes. For example, artificial reality systems can be used for sports training, for practicing surgery, as visualization aids, and to facilitate interaction and collaboration between people around the world.

[0032] In the present disclosure, artificial reality can be constituted to include a form of virtual reality to be presented to the wearer. Such artificial reality can include virtual reality, augmented reality, mixed reality, or some combination and / or variation of one or more of them.

[0033] Artificial reality can be provided by a display system such as glasses worn by the user. In some examples, the artificial reality display can look similar to and / or the same as vision correction glasses and / or vision improvement glasses. By bringing the display screen close to the user's eyes in the form of lenses and adjusting the focal length of the optical path, the head-mounted device can generate a wide-angle image at a close distance in a much smaller volume than an ordinary display, and can be combined with headphones or speakers close to the ears to provide the wearer with a more immersive audio and video experience. Head-mounted devices are divided into two types: light-transmissive and light-impermeable. In addition to displaying images, light-transmissive head-mounted devices can also allow users to see images behind the display screen. This feature is often used in the development of augmented reality (AR) applications.

[0034] Smart glasses are a type of light-transmitting head-mounted device. Figure 1An example of the structure of the smart glasses provided by the present disclosure is shown. As shown in the figure, the smart glasses 100 are first of all a type of glasses, and therefore can have a structure similar to that of conventional glasses, including lenses 110, frames 120, and temples 130. Furthermore, the smart glasses 100 can provide various "intelligent" functions that conventional glasses cannot provide, such as voice interaction, real-time translation, and audio and video playback. Therefore, the smart glasses 100 also need to integrate relevant components required for intelligent computing devices. Specifically, the processing unit of the smart glasses 100 can be arranged in the temples 130, for example, at the position shown in Figure 131. The processing unit can perform various operations according to the acquired instructions, such as using the left and right display units to project the left and right images onto the left and right lenses 110 for display. Figure 1 In the example of , the display can be performed only on a part of the left and right lenses 110 (for example, the gray rectangular area shown in the figure). In this case, the gray area shown in the figure corresponds to the screen range of the smart glasses 100. In other implementations, the left and right display units can be more than Figure 1 The gray area shown can be displayed in a larger or smaller range, for example, the entire lens area can be used for display.

[0035] To receive voice commands from the wearer, the smart glasses 100 may also be equipped with a microphone, such as the illustrated microphone 132. The microphone 132 may preferably be located on the temple 130 so that it can receive the wearer's spoken commands from a position where the wearer's voice can be clearly heard. Accordingly, the smart glasses 100 may also include a speaker 133 for playing voice feedback or wearer-specified audio content, such as music, to the wearer. To provide the wearer with better sound effects, speakers 133 may be provided on both temples near the ears, as shown.

[0036] In a preferred embodiment, the smart glasses 100 may be provided with multiple microphones, which may be arranged at different positions of the smart glasses to pick up sounds from different directions. Figure 1 In the example shown, in addition to the microphone 132 provided on the temple 130, a microphone 121 is also provided on the frame 120. In this case, the microphone 132 can be regarded as a microphone primarily used to obtain the wearer's voice commands, while the microphone 121 can be regarded as a microphone primarily used to obtain the voice input of the wearer's conversation partner. In this case, the microphones 132 and 121 can form a microphone array. A specific algorithm can be executed on the voice signal collected by the microphone array to extract a signal containing valid voice information for voice wake-up and / or voice recognition. In addition to collecting specific human voices, the microphones 132 and 121 can also collect ambient noise for use in the operating method of the present disclosure described below.

[0037] Furthermore, the smart glasses 100 may also be equipped with cameras, such as cameras 122 and 123 shown in the figure. Camera 122 faces the wearer's visual field and captures the wearer's surroundings to provide corresponding functions. For example, in a translation scenario, it can capture images of text to be recognized. Camera 123 can be positioned inward and used to capture images of the wearer's eyes. These eye images can be used to perform eye tracking in the operating method of the present disclosure described below.

[0038] As a computing device, in addition to the processing unit 131 shown in the figure, the smart glasses 100 should also include a storage unit and a communication unit (not shown in the figure).

[0039] The processing unit 131 may be a single processor or may include multiple processors. In some embodiments, the processing unit 131 may include a general-purpose main processor and one or more specialized coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, at least a portion of the processing unit 131 may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0040] The storage unit may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processing unit or other modules of the computing device. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computing device is powered off. The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store instructions and data required by some or all processing units during operation. Executable code is stored on the storage unit, and when the executable code is processed by the processing unit, the processing unit can execute various methods required to implement the relevant functions of the smart glasses.

[0041] The communication unit can be used to communicate wirelessly with the outside world. The full functionality of smart glasses usually requires the installation of a smart glasses app on other smart devices (e.g., smartphones). For example, smart glasses can communicate with the smart glasses app via the communication unit for short-range wireless communication, and the latter then interacts with the voice model deployed on the server based on long-range wireless communication to obtain the results of voice recognition.

[0042] It should be understood that although Figure 1While a specific framed eyeglass design is shown, in other embodiments, the smart glasses may also have a frameless design. In this case, components originally placed in the frame can be relocated elsewhere. In other embodiments, the smart glasses may also include other components beyond the main structure of the glasses, such as an image projection device attached to the outside of the lenses, removable additional lenses with myopia / hyperopia prescription located inside the lenses, or removable in-ear earbuds. This disclosure does not limit the specific implementation of the smart glasses.

[0043] As above combined Figure 1 As can be seen from the description, head-mounted devices such as smart glasses are typically equipped with microphones and speakers and can be used to provide various voice interaction services. Voice control is particularly suitable for scenarios such as cycling and running, where manual keystrokes are inconvenient. However, if there is high background noise in the voice control scenario, the voice command recognition rate may decrease due to the low signal-to-noise ratio, reducing the accuracy and success rate of voice interaction.

[0044] To this end, the present disclosure provides a control method for a wearable device and the wearable device. Figure 2 A schematic flow chart of a control method for a wearable device according to one embodiment of the present disclosure is shown. This control method utilizes a camera on the wearable device to capture eye images of a user (the "user" of a wearable device may also be referred to as the "wearer," and the two are used interchangeably hereinafter) to obtain gaze coordinates. These images, combined with recognized voice commands, are used to determine user commands in a noisy environment based on machine learning network inference.

[0045] The control method can be performed by any kind of wearable device equipped with a camera and capable of photographing the eyes, such as a smartwatch that can capture eye images by raising the wrist to direct the camera toward the wearer's eyes. Figure 1 The smart glasses equipped with a camera 123 arranged toward the human eye can capture eye images without the wearer's additional operation (in other words, the wearer's eyes are already within the image capture range of the camera 123). Therefore, this method is also applicable to head-mounted devices equipped with cameras arranged toward the human eye, such as Figure 1 In the following description, the principles of the present disclosure will be described mainly with reference to an example of smart glasses, which does not limit the scope of use or application of the present disclosure.

[0046] In step S210, in response to the wearable device entering the interactive interface, multiple optional items are displayed in multiple display areas of the screen of the wearable device. When the application of the wearable device enters the interactive scene and the user is required to perform the next interaction, multiple optional items can be displayed on the screen of the wearable device at the same time, and each optional item corresponds to a display area. For example, in the navigation application of smart glasses, if the wearer says "navigate to People's Square", the navigation application will show the user multiple routes, and the user is required to select one of them. In normal operation, the user can use the buttons to scroll up and down or slide to one of them on the touchpad, and then click to confirm to make a selection, or use voice input, such as "Route 1" to select one of the paths. When the user is required to perform the next interaction, in addition to visual prompts (i.e., displaying multiple optional items), the application can also provide voice prompts to the user.

[0047] In step S220, the noise of the environment in which the wearable device is located is detected. The wearable device can use the microphone equipped therein, for example Figure 1 The smart glasses can use microphones 132 and 121 to pick up sound and detect the noise contained therein.

[0048] After the noise is detected, the noise level needs to be determined. Only when the noise is greater than a certain threshold will it be considered to have an impact on the accuracy of speech recognition. This threshold can be called the "first noise threshold" (also called "T1", for example, 30dB). When the noise is too loud, it is completely unsuitable for speech recognition. The threshold for determining that the noise is too loud can be called the "second noise threshold" (also called "T2", for example, 65dB). When the detected noise is in a predetermined interval between the first and second noise thresholds, it means that the noise will have a certain impact on the speech recognition rate, but the speech recognition results can be combined with the results of eye tracking to make a comprehensive judgment on the user's intention. At this time, the following operations for steps S230 to S250 can be performed.

[0049] In step S230, in response to the noise being greater than a first noise threshold and less than a second noise threshold, the gaze coordinates of the wearer of the wearable device on the screen are determined, as well as the speech recognition result corresponding to the wearer's speech information. When the noise is within a predetermined interval between T1 and T2, the wearer's eyes can be photographed using a camera built into the wearable device, and the gaze coordinates of the wearer's eyes on the screen can be determined from the photographed image. Simultaneously or sequentially in time, the wearer's input speech information is collected and speech recognition is performed to obtain a speech recognition result.

[0050] In one embodiment, when it is necessary to obtain the wearer's gaze coordinates on the screen, for example, when the noise level is determined to be greater than T1, the display format of the multiple optional items in step S210 can be changed as needed to facilitate eye tracking. For example, the multiple optional items can be displayed as icons with large spacing between them, as shown in Figure 5. In this case, only the multiple optional item icons can be displayed on the screen, and the screen can be considered to be divided into multiple display areas, with each icon corresponding to a display area on the screen. This facilitates the determination of gaze coordinates and the determination of user intent.

[0051] After obtaining the results of voice recognition and eye tracking, they can be input into a machine learning model for classification to infer user intent. To this end, in step S240, at least the noise, the display area of ​​the first optional item corresponding to the voice recognition result, the first distance to the second optional item closest to the gaze coordinates, and the display area of ​​the second optional item are input into a trained machine learning model to obtain a selected display area. In the aforementioned steps, multiple optional items are displayed to the user, each of which corresponds to a display area on the screen. Here, "first optional item" refers to the optional item among the multiple optional items that corresponds to the recognition result of the user's input voice information; "second optional item" refers to the optional item among the multiple optional items that is closest to the user's eye tracking result (i.e., gaze coordinates). Information related to these two optional items is fed as input into the trained machine learning model, which performs inference and outputs a classification result. The classification result typically corresponds to one of the multiple display areas, which can be referred to as the "selected display area." The selected display area is typically one of the display area of ​​the first optional item and the display area of ​​the second optional item. In practice, the first option corresponding to the speech recognition result and the second option closest to the gaze coordinates may or may not be the same. Therefore, when the first and second options are the same, the selected display area corresponds to the display area where the first option is located. However, when the first and second options are different, a machine learning model is required to perform inference and make a comprehensive judgment.

[0052] As mentioned above, the information related to the eye tracking result of the input model includes the first distance of the second optional item closest to the gaze coordinate and the display area of ​​the second optional item. Here, the first distance refers to the distance between the gaze coordinate and the second optional item. For example, if the gaze coordinate is within the corresponding icon of the second optional item, the first distance can be considered to be zero, but if there is a deviation from the icon position, the distance is not zero and the value of the first distance can be obtained based on the icon outline or center position. The display area of ​​the second optional item corresponds to the area where the second optional item is located, for example, in the following Figure 9A In the example of quadrant-based division shown in -D, the display area corresponds to the quadrant where the icon of the second optional item is located.

[0053] After determining which area is the selected display area, the interactive interface can be controlled in step S250 based on the selected option in the selected display area. For example, in the example above where the interactive interface of the smart glasses displays multiple navigation routes, if the voice recognition result corresponds to "Route 1" and the gaze coordinates fall into the display area corresponding to "Route 2", but the output result of the machine learning model (i.e., the selected display area) corresponds to the display area where "Route 1" is located, then it is considered that the user has selected the option "Route 1", and navigation content based on "Route 1" can then be displayed on the screen.

[0054] Therefore, for the current noise level that falls within the predetermined range, the control method disclosed herein can comprehensively consider the speech recognition and eye tracking results and noise based on machine learning model reasoning to determine the user instructions, thereby improving the accuracy of user interaction in non-button control scenarios.

[0055] When the noise level falls outside the predetermined range, there is no need to perform machine learning model inference for both speech recognition and eye tracking results. This is because when the noise level is low, speech recognition is usually very accurate, and the speech recognition result can be directly used for control. When the noise level is too high, speech recognition is meaningless, and control can be based solely on eye tracking. To this end, the control method of the wearable device disclosed in the present invention also includes: in response to the noise being less than the first noise threshold, determining the speech recognition result corresponding to the wearer's voice information, and using the first optional item corresponding to the speech recognition result to control the interactive interface; and / or in response to the noise being greater than the second noise threshold, determining the wearer's gaze coordinates on the screen, and using the second optional item closest to the gaze coordinates to control the interactive interface.

[0056] The following will be combined Figure 3 The principles of the present disclosure are described in more detail with reference to examples of controlling smart glasses. Figure 3 An example of a smart glasses control method provided by the present disclosure, including operation branches under different noise levels, is shown. For ease of distinction, the three branches corresponding to voice recognition only, eye tracking only, and simultaneous voice recognition and eye tracking are shown in orange, blue, and red, respectively. The common operations of the two branches of voice recognition only and simultaneous voice recognition and eye tracking are marked in orange (which can be seen as a superposition of orange and red), and the common operations of the two branches of eye tracking only and simultaneous voice recognition and eye tracking are marked in purple (which can be seen as a superposition of blue and red).

[0057] In step S301, the smart glasses enter an interactive scene. Figure 3The process shown can be started when the application of smart glasses enters the interactive scene and the user is required to perform the next interaction. The next interaction usually requires the user to select from multiple options, such as requiring the user to "confirm" or "cancel", or to select from more options. When entering the interactive scene, multiple options will be displayed in multiple display areas of the screen of the smart glasses. In different implementations, the display areas and options may have different corresponding forms. For ease of understanding, Figure 4A -B shows examples of optional options for different forms of display provided by the present disclosure. Figure 4A The figure shows three routes displayed by the navigation app on the smart glasses screen after the user says "Navigate to People's Square." Because the map displays a portion of the screen, the display area for the options corresponds only to the lower half of the screen. It can be assumed that the three areas 1-3 where the three routes are located correspond to the three options, with each display area representing one option. Figure 4B The two options corresponding to the "Confirm" and "Cancel" icons are displayed. In this case, each icon area can be considered to correspond to a display area; alternatively, a larger area (e.g., one side of the screen) containing the icons can be considered to correspond to a display area, in which case the screen includes both left and right display areas. When icon selection is required via gaze coordinates, the screen is typically divided into multiple sections based on the number of icons displayed, with each section containing one icon and corresponding to one display area.

[0058] In interactive scenarios, in addition to displaying interactive content, voice prompts can also be used to prompt users to interact. For example, the speaker can play "Please select confirmation or area." In subsequent scenarios where eye tracking is required, the speaker can also prompt users to look, such as "Please look at the screen to confirm or cancel."

[0059] In step S302, the noise level in the environment of the smart glasses is detected. The built-in microphone of the smart glasses can be used to pick up ambient sound and use a supporting algorithm to detect the noise level (e.g., decibel level). In some embodiments, specific noise scenarios can be classified and identified based on the picked-up sound. In this case, noise scenarios can be identified based on a neural network classification model, such as a deep neural network (DNN). The collected raw audio signal can be converted into a numerical form more suitable for processing by a machine learning model, such as MFCC (Mel-Frequency Cepstral Coefficients) or FBANK (Filter Bank Features) as input to the neural network. For example, in one example, a 128x128 Mel-frequency spectrum can be input to the neural network, where the first 128 represents the number of Mel-frequency points and the second 128 represents the number of Mel-values, i.e., the number of samples. The neural network classification model can then infer the input and output a specific noise scenario category. For example, noise scenario categories can include surrounding vehicle noise when walking on the road, noise when inside a car, noise inside a subway car, and wind noise when moving quickly outdoors (e.g., cycling or running).

[0060] Then, in step S303, the noise threshold can be determined. In the present disclosure, there are two noise thresholds, namely, the first noise threshold T1 and the second noise threshold T2. The values ​​of T1 and T2 can be set based on experience or can vary based on the noise scenario. In one embodiment, T1 can be set to 30dB and T2 to 65dB. When the noise level is greater than T2, it is considered that the noise is too loud and is no longer suitable for voice interaction. At this time, the process enters Figure 3 On the right side, steps S304 to S309 are executed, using only eye tracking to collect the user's selections, without voice recognition processing to reduce processing overhead. When the noise level is less than T1, it can be considered that the user's voice input can be correctly detected. At this time, eye tracking is not required and the process directly enters Figure 3 Steps S310 and S311 are executed on the left. When the noise level is within the predetermined interval between T1 and T2, it is considered that the speech recognition result has a certain reference value. At this time, the speech device and eye tracking are performed simultaneously. Figure 3 Part of the left and right branches (step S310 and steps S304 to S308) are shown, and after obtaining the speech recognition results and eye tracking results, the arbitration process of the middle path is entered, that is, steps S312 to S314 are executed.

[0061] The eye tracking branch will be described below. As mentioned above, when it is determined in step S303 that the noise level is greater than T1, eye tracking needs to be performed.

[0062] In step S304, multiple options for eye tracking are displayed on the smart glasses. Since the screen size of smart glasses is relatively small, the resolution of the pupil range is usually low, and large changes in pupil position are easier to distinguish. Therefore, when displaying the multiple options for eye tracking, it is necessary to display them in a format that is easy to use for eye tracking, such as displaying them as large icons with as much distance between them as possible.

[0063] Figure 5A -D shows an example of displaying multiple options for eye tracking provided by the present disclosure. As shown in the figure, when two, three, and four options need to be displayed, two, three, and four icons can be displayed on the screen, and the icons can be separated as much as possible and as far apart as possible while maintaining the appearance of the display. If there are more than four options, the following can be used: Figure 5D The display is divided into multiple pages, and page turning icons are displayed at the top and bottom.

[0064] In some embodiments, the display of multiple options for eye tracking in step S304 can be consistent with the display of multiple options when entering the interaction step in step S301. In other embodiments, the display of multiple options in step S304 can be changed to make it more suitable for eye tracking. For example, after the user says "navigate to People's Square", if the noise level is detected to be greater than T1 (i.e., eye tracking is required), the display of multiple options in step S304 can be changed to make it more suitable for eye tracking. Figure 4A The partition options are converted to Figure 5B The icon optional display shown improves eye tracking accuracy by not displaying non-optional content (or displaying non-optional content as the background) and arranging different optional items as far as possible on the screen.

[0065] Subsequently, in step S305, eye movement detection is performed, and in step S306, gaze coordinates are determined. In one embodiment of the present disclosure, determining the gaze coordinates of the wearer of the wearable device on the screen may include: capturing an eye image of the wearer using a camera on the wearable device; extracting a pupil position from the eye image; and determining the gaze coordinates corresponding to the pupil position based on a mapping relationship.

[0066] Figure 6A -C shows an example of human eye and pupil extraction provided by the present disclosure. Figure 6A The image of the left eye is shown. Figure 6AAs shown in Figure 1, the human eye consists of the cornea, iris, and pupil. The pupil is similar to the pinhole in pinhole imaging. By adjusting the size of the pupil, the amount of light entering the eye can be controlled, thereby adapting to environments of varying brightness. Furthermore, the eye adjusts the curvature of the lens (adjusting focal length) to see objects at different distances. Furthermore, people change their line of sight by rotating their eyes; the direction in which the pupil is aligned is the direction of gaze.

[0067] A camera deployed in the smart glasses and facing the wearer can be used to capture the wearer's eye image in real time and detect the pupil direction through image technology to obtain the user's gaze direction. Figure 6B The position of the pupil when the left eye is gazing straight ahead, to the right, above, and to the left is shown. Since the camera is arranged inward on the glasses, when the user wears the glasses, it can be assumed that the position of the camera relative to the wearer's eyes remains unchanged, and the distance of the icons displayed on the lenses relative to the eyes remains unchanged. Therefore, eye tracking can be equivalent to pupil position detection, and the gaze direction can be known by the change in the position of the wearer's pupil. After the camera captures the user's eye image, the pupil can be extracted from the image using various known methods (for example, image processing methods) to obtain a circular pupil, and the position of the circular pupil in the image represents the gaze direction. Figure 6C The relative positions of the pupils when the human eye looks at 9 angles are shown. In some embodiments, a neural network trained based on large-scale sample data can also be used for inference to obtain the coordinate position of the current pupil (e.g., the horizontal coordinate x and the vertical coordinate y of the current pupil).

[0068] After obtaining the position of the pupil in the eye (ie, the pupil position coordinates), the position of the pupil gaze direction on the glasses screen (ie, the gaze coordinates) can be determined according to the mapping relationship. Figure 7 An example of the correspondence between pupil coordinates and smart glasses screen coordinates provided by the present disclosure is shown. The left side of the figure shows the pupil coordinates, and the right side shows the corresponding glasses screen coordinates. The center of the eye can be used as the pupil coordinate origin, and the center of the screen as the screen coordinate origin, and a one-to-one correspondence can be established between the pupil coordinates and the screen coordinates. Based on this mapping relationship, the pupil coordinates extracted from the image are converted into the user's gaze coordinates when looking at the smart glasses screen, thereby obtaining the gaze result used to determine the user's selection.

[0069] Due to individual differences (e.g., eye size, nose bridge position, and ear height), the mapping relationship between pupil coordinates and screen coordinates varies for each wearer. Therefore, in certain embodiments, for accuracy reasons, the mapping relationship can be determined based on a calibration operation. The calibration operation can include: displaying multiple calibration points on the screen; obtaining the pupil coordinates of the wearer when looking at each calibration point; and determining the mapping relationship based on the pupil coordinates and the coordinates of the multiple calibration points on the screen. Figure 8A -B shows two examples of calibration operations provided by the present disclosure. Specifically, Figure 8A 9 calibration points and 4 quadrants calibration are shown. The 9 calibration points shown are located in the center, top, bottom, left, right, top left, bottom left, top right, and bottom right, respectively, corresponding to the top corners of the four quadrants. At this time, the coordinates can be obtained based on the linear mapping within the quadrant. For example, the coordinates of the pupil in the upper left corner are (m, n), and the corresponding coordinates of the upper left corner of the screen are (M, N). If the coordinates of the pupil at a point in the upper left quadrant are (x, y), then the coordinates of the corresponding screen at this time (X', Y') are obtained according to the mapping relationship as follows:

[0070] X'=x*M / m

[0071] Y'=y*N / n

[0072] Since pupil coordinates and glasses screen coordinates are not always linearly corresponding in actual situations, more calibration points and quadrant coordinates tend to make the mapping more accurate. Figure 8B The 13 calibration points and 16 quadrants calibration are shown. At this time, the 13 calibration points divide the entire mapping space into 16 sub-quadrants. From the pupil coordinates on the left, we can see that the coordinate change is nonlinear, and the change trend decreases towards the edge of the eye. Figure 8B In practice, however, the required number of calibration points and the corresponding mapping relationship are usually determined in combination with mapping resolution and user-friendliness.

[0073] After obtaining the gaze coordinates, gaze determination can be performed in step S307. Specifically, it can be determined whether the coordinates are within a certain interactive icon (or within a predetermined range from the icon). If they are within the icon, it indicates that the user has interaction intention, and the subsequent step S308 can be performed. If they are not within the interactive icon coordinate range, it is assumed that the user has no intention to use eyes for interaction.

[0074] To use eye contact interaction, the eyes must remain fixed on a specific icon, rather than simply looking past it. To this end, gaze duration determination is performed in step S308. To this end, capturing the wearer's eye images using a camera on the wearable device may include: after displaying the multiple selectable items on the screen, continuously capturing the eye images in response to voice activity detection detecting voice input from the wearer. Determining the gaze coordinates corresponding to the pupil position based on the mapping relationship may include: determining the gaze coordinates in response to pupil positions extracted from the continuously captured eye images indicating gaze on the same area of ​​the multiple display areas for a predetermined duration exceeding a predetermined threshold. The "same area" here can be the icon area or a specific size area outside of the icons in the display area. For example, all coordinates within a period of time can be counted. If the majority of the coordinates are on a particular icon, the user is deemed to have gazed on that icon. For example, if the gaze detection duration is 2 seconds and 80% of the time within the 2 seconds is spent on the same target, the gaze is considered detected successfully. If the duration is less than 1.6 seconds, the user is deemed to have no interaction intent.

[0075] In actual operation, after entering the interactive session, the user needs to browse all the content on the screen before determining the gaze target. The browsing time depends on the amount of content and the user's reading speed. This makes it difficult to accurately determine the start time of gaze, which in turn affects the stability of subsequent judgments. To this end, in one embodiment, the user's voice signal can be introduced as a trigger mechanism for the start of gaze. For example, when the VAD (Voice Activity Detection) system detects that the user has uttered a voice of any content, it is determined that the gaze has started. In this way, gaze detection can be synchronized with the user's actual gaze action, thereby improving detection accuracy.

[0076] When the noise level is greater than T2, user intent is determined solely based on eye tracking. Therefore, in step S309, the smart glasses application can be controlled based on the gaze results obtained through eye tracking. Specifically, if the user's gaze duration on an icon exceeds a threshold, this is recognized as the user selecting that icon. For example, if "Confirm" is selected, the application enters the confirmation process.

[0077] As mentioned above, when it is determined in step S303 that the noise level is less than T2, speech recognition needs to be performed.

[0078] Therefore, in step S310, voice command recognition detection is performed. Voice recognition usually requires the help of a voice recognition model. At this time, determining the voice recognition result corresponding to the wearer's voice information may include: using the microphone of the wearable device to collect the voice information input by the wearer and sending the voice to the voice recognition model; and obtaining the voice recognition result output by the voice recognition model. Since the voice recognition model is usually located on a smart device (for example, a smart phone) or a server paired with the smart glasses, the audio signal collected by the smart glasses needs to be processed and transmitted to the mobile phone or the cloud for recognition, and then mapped into the corresponding command based on the recognition result.

[0079] If the noise level is less than T1, application control can be performed directly based on the voice device output in step S311. For example, if the user's voice command is recognized as "Route 1," navigation based on Route 1 will be displayed on the smart glasses. If the noise level is between T1 and T2, arbitration based on the machine learning model is performed in step S312.

[0080] It can be considered that Figure 3 The arbitration step S312 shown corresponds to Figure 2 In step S240, at this time, at least the noise (for example, the noise level, such as decibels), the display area of ​​the first optional item corresponding to the speech recognition result, the first distance of the second optional item closest to the gaze coordinate, and the display area of ​​the second optional item can be input into the trained machine learning model to obtain the selected display area.

[0081] In one embodiment, only the noise, the display area of ​​the first optional item corresponding to the speech recognition result, the first distance to the second optional item closest to the gaze coordinates, and the display area of ​​the second optional item as described above can be used as input to the machine learning model for inference. In other embodiments, more information can be used as model input. Specifically, the speech recognition result output by the speech recognition model typically includes a string and a confidence level. The string corresponds to the recognized speech content of the user input and can therefore be used to determine the display area of ​​the first optional item. The confidence level represents the confidence level of the speech recognition model in outputting the string, and therefore, in some embodiments, this confidence level can also be input into the machine learning model. In some embodiments, the eye tracking results used as input to the machine learning model can include not only information about the optional item closest to the gaze coordinates (i.e., the "second optional item" mentioned above), but also information about the optional item with the second closest gaze coordinates (herein referred to as the "third optional item"). In this case, the control method of the present disclosure can also include: inputting the first distance to the third optional item with the second closest gaze coordinates and the display area of ​​the third optional item into the trained machine learning model to obtain the selected display area.

[0082] In the present disclosure, a neural network model or a decision tree model can be used as a machine learning model for inferring the selected display area. The model can be a classification model. In one embodiment, the classification model can only include an output classification corresponding to each display area. In another embodiment, the classification model can also additionally include a detection failure classification. In this case, the machine learning model includes output classifications corresponding to the multiple display areas and a detection failure classification. And when the output of the machine learning model corresponds to the output classification of a display area and the probability is higher than a predetermined threshold value, the display area can be determined to be the selected display area.

[0083] In one embodiment, the trained machine learning model used for arbitration can be a neural network model. The neural network model can be a convolutional neural network, including multiple stacked convolutional layers and activation layers, followed by a fully connected layer and a softmax layer. The input features of the model can include seven dimensions, namely: noise value (decibel), speech recognition confidence, the quadrant corresponding to the speech recognition result string, the quadrant closest to the gaze (i.e., top1) command word, the distance between the gaze coordinates and the closest command word, the quadrant second closest to the gaze (i.e., top2) command word, and the distance between the gaze coordinates and the second closest command word. The following Table 1 shows the name, dimension, format and description of the input features.

[0084]

[0085] Table 1: Neural network input features

[0086] Here, the screen area can be divided into a plurality of quadrants according to the distribution of the optional items, wherein each optional item corresponds to a command word, and the display area where each optional item is located corresponds to a quadrant. Figures 9A-9D An example is shown in which the screen provided by the present disclosure includes four quadrants and gaze coordinates are located at different positions.

[0087] like Figure 9A As shown, in a certain interactive scenario, four command words are displayed on the screen, corresponding to four icons "louder volume", "lower volume", "brighter screen" and "darker screen". These four icons correspond to four optional items, and each optional item corresponds to a display area. These four display areas can be called "first quadrant", "second quadrant", "third quadrant" and "fourth quadrant" respectively.

[0088] exist Figures 9B-9D In the figure, the red dots correspond to the gaze coordinates, and examples of gaze coordinates at different positions are shown. In an ideal situation, the gaze coordinates are within the command word frame, that is, corresponding to Figure 9BAt this point, the distance between the gaze coordinates and the closest command word is 0, and the distance between the gaze coordinates and the next closest command word is maximum (because the gaze coordinates are located within the "Voice Louder" icon and are at their maximum distance from the "Brighter Screen" icon). If the speech recognition result is also "Voice Louder," the neural network model can determine with a high probability that the selected display area corresponds to the first quadrant.

[0089] When the gaze coordinates deviate from the icon area, such as Figure 9C As shown, the gaze coordinates are still in the first quadrant, but outside the icon area. The distance between the gaze coordinates and the closest command word, "Louder," is a, and the distance between the gaze coordinates and the next closest command word, "Brighter screen," is b, with a < b. If the speech recognition result is also "Louder," the neural network model can still determine with a high probability that the selected display area corresponds to the first quadrant.

[0090] When the gaze coordinates deviate from the icon area and are not located in the first quadrant, such as Figure 9D The distance between the gaze coordinates and the closest command word "brighten the screen" is b, and the distance between the gaze coordinates and the second closest command word "louder voice" is a, and a>b. At this time, if the speech recognition result is also "louder voice", since the gaze coordinates deviate from the first quadrant by a small distance, it is usually still tended to be believed that the user expects to choose "louder voice". Therefore, by introducing the relevant information of the second closest command word as the input feature of the machine learning model, combined with the speech recognition result to comprehensively judge the user's intention, it is possible to Figure 9D Slight deviations as shown are tolerated to ensure smooth and accurate interaction.

[0091] In one embodiment, the output dimension of the neural network model can be the number of quadrants + 1. For example, if the number of quadrants is 4, the output dimension is 5, and the extra dimension corresponds to the detection command failure classification. Each output value can represent the probability of selecting a command word in a quadrant. For example, when there are 4 quadrants, the output result of the model is [0.9, 0.02, 0.02, 0.03, 0.04] (at this time, the 5 dimensions correspond to the first, second, third, and fourth quadrant classifications and the detection and recognition classification failure respectively), which means that the selected display area of ​​this reasoning corresponds to the first quadrant where "louder voice" is located. A threshold THESH (for example, equal to 0.6) can be set. When the highest probability in the output dimension exceeds the threshold, the command recognition is considered to be successful. If it is less than the threshold or the value of the last dimension is the largest, it means that no valid command was detected and the detection failed.

[0092] In other implementations, the screen can be divided into more quadrants. With fewer quadrants, there are fewer options, but the display area for each optional command is larger, making the commands more distinguishable and easier to track. With more quadrants, there are more options, but the display area for each command word is smaller, and the distance between command words is also smaller. This makes it difficult for gaze judgment to locate the desired command range, resulting in eye tracking failure. Therefore, the number of options displayed and the corresponding number of quadrants can be flexibly determined based on actual conditions.

[0093] After the selected display area is determined in step S312, command mapping is performed in step S313 to map the selected display area, such as the quadrant index inferred by the neural network model, to the corresponding command. Figures 9B-9D In the example, if the model infers that the classification is in the first quadrant, the mapped command is "louder".

[0094] Then, in step S314 , the application is controlled based on the result of the model arbitration, for example, the output volume of the speaker of the smart glasses is increased.

[0095] As above combined Figures 3 to 9D In one embodiment, the present disclosure can also be implemented as a wearable device, comprising: a screen for displaying an interactive interface and displaying multiple options in multiple display areas; a microphone for collecting noise and voice input by the wearer; a camera for collecting eye images of the wearer to determine gaze coordinates; a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to execute the above combined Figure 2 and Figure 3 In one embodiment, the wearable device may be smart glasses, such as Figure 1 Smart glasses shown with a built-in camera.

[0096] The control method for a wearable device according to the present disclosure and the wearable device using the method have been described in detail above with reference to the accompanying drawings. The present disclosure determines an interaction path based on noise levels, simultaneously performs voice recognition and eye tracking when the noise level falls within a predetermined range (T1-T2), and uses machine learning model reasoning to comprehensively consider the voice recognition and eye tracking results and noise to determine user commands, thereby improving the accuracy of user interactions in non-button control scenarios.

[0097] In addition, the method according to the present disclosure may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present disclosure.

[0098] Alternatively, the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to perform the various steps of the above-mentioned method according to the present disclosure.

[0099] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0100] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A control method for a wearable device, comprising: In response to the wearable device entering an interactive interface, displaying a plurality of optional items in a plurality of display areas of a screen of the wearable device; detecting noise in an environment in which the wearable device is located; In response to the noise being greater than a first noise threshold and less than a second noise threshold, determining a gaze coordinate of a wearer of the wearable device on the screen, and determining a speech recognition result corresponding to the voice information of the wearer; Inputting at least the noise, the display area of ​​the first optional item corresponding to the speech recognition result, the first distance of the second optional item closest to the gaze coordinate, and the display area of ​​the second optional item into a trained machine learning model to obtain a selected display area, wherein the selected display area is one of the display area of ​​the first optional item and the display area of ​​the second optional item; The interactive interface is controlled based on the selected option of the selected display area.

2. The method according to claim 1, further comprising: The first distance of the third optional item that is second closest to the gaze coordinate and the display area of ​​the third optional item are also input into the trained machine learning model to obtain the selected display area.

3. The method according to claim 1, further comprising: In response to the noise being less than the first noise threshold, determining a speech recognition result corresponding to the wearer's speech information, and controlling the interactive interface using the first optional item corresponding to the speech recognition result; and / or In response to the noise being greater than the second noise threshold, determining the gaze coordinates of the wearer of the wearable device on the screen, and controlling the interactive interface using the second optional item closest to the gaze coordinates.

4. The method according to claim 1, wherein Determining the gaze coordinates of the wearer of the wearable device on the screen includes: Using a camera on the wearable device to capture an eye image of the wearer; extracting a pupil position from the eye image; The gaze coordinates corresponding to the pupil position are determined according to the mapping relationship.

5. The method according to claim 1, wherein The mapping relationship is determined according to a calibration operation, and the calibration operation includes: displaying a plurality of calibration points on the screen; Obtaining pupil coordinates of the wearer when looking at each calibration point; The mapping relationship is determined according to the pupil coordinates and the coordinates of the multiple calibration points on the screen.

6. The method of claim 4, wherein: Capturing the eye image of the wearer using a camera on the wearable device includes: After the plurality of optional items are displayed on the screen, in response to voice activity detection detecting the voice input of the wearer, the eye images are continuously collected, and Determining the gaze coordinates corresponding to the pupil position according to the mapping relationship includes: The gaze coordinates are determined in response to pupil positions extracted from the continuously acquired eye images indicating gaze at the same area of ​​the plurality of display areas for a predetermined threshold within a predetermined time period.

7. The method according to claim 1, wherein Determining the voice recognition result corresponding to the wearer's voice information includes: Using the microphone of the wearable device to collect voice information input by the wearer and feeding the voice into a speech recognition model; Obtaining the speech recognition result output by the speech recognition model, the speech recognition result including a character string and a confidence level, wherein a display area of ​​the first optional item is determined based on the character string; and The method further comprises: The confidence level is also fed as input into the machine learning model to obtain the selected display area.

8. The method according to claim 1, wherein The machine learning model includes output categories corresponding to the plurality of display regions and a detection failure category; And when the output of the machine learning model corresponds to an output classification of a display area and the probability is higher than a predetermined threshold value, the display area is determined to be the selected display area.

9. The method according to claim 1, wherein The first optional item corresponding to the speech recognition result is the same as or different from the second optional item closest to the gaze coordinate.

10. A wearable device comprising: A screen, used to display an interactive interface and display multiple options in multiple display areas; Microphone, used to collect noise and voice input by the wearer; A camera for capturing images of the wearer's eyes to determine gaze coordinates; processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 9. 11 . A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to claim 1 .

12. A computer program product comprising computer program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images