Device interaction method and head-mounted device

Abstract

The invention relates to the technical field of communication interaction, in particular to an equipment interaction method and head-mounted equipment. The device interaction method is applied to a first head-mounted device, the first head-mounted device comprises an image acquisition module, and the method comprises the following steps: respectively acquiring a first image and a second image through the image acquisition module; determining a second head-mounted device for interacting with the first head-mounted device based on the first image and the second image; interacting with the second head-mounted device through the talkback mode; or the first user wearing the first head-mounted device interacts with the second user wearing the second head-mounted device through the transparent transmission mode. According to the embodiment of the invention, the dependence on user awareness can be effectively reduced, the operation difficulty is simplified, the autonomous switching of the operation mode is realized, the intelligence of the first head-mounted device is improved, and the use experience of the user is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of communication and interaction technology, specifically to a device interaction method and a head-mounted device. Background Technology

[0002] With technological advancements, head-mounted devices such as headphones, smart glasses, and AI glasses are becoming increasingly widespread. These devices often incorporate image acquisition modules, allowing them to determine whether to operate in active noise cancellation or pass-through mode based on captured images, enabling automatic mode switching. However, when the head-mounted device needs to operate in intercom mode, manual selection is required, impacting the user experience. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, an exemplary embodiment of this disclosure provides a device interaction method applied to a first head-mounted device. The first head-mounted device includes an image acquisition module. The method includes: acquiring a first image and a second image through the image acquisition module; determining a second head-mounted device for interacting with the first head-mounted device based on the first image and the second image; interacting with the second head-mounted device through an intercom mode; or a first user wearing the first head-mounted device interacting with a second user wearing the second head-mounted device through a pass-through mode.

[0004] In some embodiments, determining a second head-mounted device for interacting with a first head-mounted device based on a first image and a second image includes: identifying, based on the first image, whether there is a head-mounted device to be interacted with at a position relative to the location of a first user; determining, based on the second image, the gaze direction of the first user; and, in response to the existence of a head-mounted device to be interacted with, and the head-mounted device to be interacted with being located in the gaze direction of the first user, determining that the head-mounted device to be interacted with is a second head-mounted device for interacting with the first head-mounted device.

[0005] In some embodiments, the method further includes: detecting the current environment and obtaining a detection result; and based on the detection result, determining the operating mode for interacting with the second headset as either a talkback mode or a pass-through mode.

[0006] In some embodiments, based on the detection results, determining the operating mode for interacting with the second headset as an intercom mode or a pass-through mode includes: determining the operating mode for interacting with the second headset as an intercom mode in response to the detection results indicating that the noise of the current environment is greater than or equal to a first threshold; and determining the operating mode for interacting with the second headset as a pass-through mode in response to the detection results indicating that the noise of the current environment is less than the first threshold.

[0007] In some embodiments, based on the detection results, determining the operating mode for interacting with the second headset as an intercom mode or a pass-through mode further includes: in response to the detection results indicating that the noise of the current environment is greater than or equal to a second threshold, determining the operating mode for interacting with the second headset as a mode in which intercom mode and active noise cancellation mode run in parallel, wherein the second threshold is greater than the first threshold.

[0008] In some embodiments, the method further includes: sending a mode notification based on the communication connection and operating mode with the second head-mounted device, so that the second head-mounted device adjusts its operating mode to match the operating mode of the first head-mounted device in response to the mode notification, thereby enabling device interaction.

[0009] In some embodiments, the first head-mounted device includes at least one microphone, a speaker, and a self-speaking detection module, and interacts with the second head-mounted device through an intercom mode, including: in the intercom mode: acquiring a first voice signal through at least one microphone, detecting whether the first voice signal is voice emitted by a first user through the self-speaking detection module, and in response to detecting that the first voice signal is voice emitted by the first user, sending the first voice signal based on the communication connection with the second head-mounted device, so that the second head-mounted device plays audio based on the first voice signal through the speaker of the second head-mounted device; or receiving a second voice signal based on the communication connection with the second head-mounted device, and playing audio based on the second voice signal through the speaker of the first head-mounted device, wherein the second voice signal is acquired by the second head-mounted device.

[0010] In some embodiments, a first head-mounted device includes at least one microphone and a speaker, and a first user wearing the first head-mounted device interacts with a second user wearing a second head-mounted device through a pass-through mode, including: in the pass-through mode: acquiring a third voice signal through at least one microphone, wherein the third voice signal includes the second user's speech content; and playing audio through the speaker based on the third voice signal so that the first user acquires the second user's speech content.

[0011] Secondly, this disclosure also provides a head-mounted device, including: a first camera for capturing a first image; a second camera for capturing a second image; a processing module connected to the first camera and the second camera respectively, for determining a second head-mounted device for interacting with the head-mounted device based on the first image and the second image; a communication module connected to the processing module, for interacting with the second head-mounted device through an intercom mode; and an interaction module connected to the processing module, for a first user wearing the head-mounted device to interact with a second user wearing the second head-mounted device through a pass-through mode.

[0012] In some embodiments, the interaction module includes: at least one microphone connected to the processing module for acquiring voice signals; and a speaker connected to the processing module for audio playback.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0014] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: According to the device interaction method provided by this disclosure, by acquiring the content of the first image and the content of the second image through the image acquisition module, it can be determined whether the first head-mounted device needs to interact with other head-mounted devices. If it is determined that interaction is required, the device can interact with the second head-mounted device through intercom mode, or the first user wearing the first head-mounted device can interact with the second user wearing the second head-mounted device through transparent transmission mode. This can effectively reduce the reliance on user consciousness, ensure the smoothness of user communication, thereby helping to simplify the operation difficulty, realize the autonomous switching of the operating mode, help improve the intelligence of the first head-mounted device, and help enhance the user experience. Attached Figure Description

[0015] This disclosure can be better understood by describing exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, in which:

[0016] Figure 1 This is a schematic diagram of signal processing transmission according to an exemplary embodiment disclosed in a publication;

[0017] Figure 2 This is a schematic diagram of another signal processing transmission according to an exemplary embodiment of a disclosed document;

[0018] Figure 3 This is a flowchart illustrating a device interaction method according to an exemplary embodiment disclosed in a book.

[0019] Figure 4 This is a flowchart illustrating another device interaction method according to an exemplary embodiment of a published document;

[0020] Figure 5 This is a flowchart illustrating yet another device interaction method according to an exemplary embodiment disclosed in a publication;

[0021] Figure 6 This is a schematic diagram of the frame of a head-mounted device according to an exemplary embodiment disclosed in a book;

[0022] Figure 7 This is a schematic diagram of the frame of another head-mounted device according to an exemplary embodiment disclosed in a publication. Detailed Implementation

[0023] The following describes specific embodiments of this disclosure. It should be noted that, in order to provide a concise description, this specification cannot exhaustively describe all features of the actual embodiments. It should be understood that, in the actual implementation of any embodiment, just as in any engineering or design project, various specific decisions are often made to achieve the developer's specific goals and to meet system-related or business-related constraints, and this can change from one embodiment to another. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this disclosure, changes in design, manufacturing, or production based on the technical content disclosed in this disclosure are merely conventional technical means and should not be construed as insufficient content of this disclosure.

[0024] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The terms “a” or “one,” etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” etc., mean that the element or object preceding “comprising” or “including” encompasses the elements or objects listed following “comprising” or “including” and their equivalents, and do not exclude other elements or objects. The terms “connected,” “linked,” etc., are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0025] In related technologies, the operating modes of head-mounted devices can at least include active noise cancellation mode, pass-through mode, and intercom mode. Head-mounted devices can include, but are not limited to, headphones, smart glasses, and AI glasses. Headphones can include wireless headphones or true wireless headphones.

[0026] Taking headphones as an example, the operating principle block diagram in active noise cancellation mode can be as follows: Figure 1As shown. In active noise cancellation mode, the headset's microphone (MIC) collects sound signals and converts them into analog electrical signals (the sound collected by the MIC includes ambient noise and audio played from the speaker). These signals pass through an analog gain module and are then converted into digital signals by an analog-to-digital converter (ADC). Specifically, the MIC includes a feed-forward (FF) channel (the topmost path in the diagram) and a feedback (FB) channel (the bottommost path in the diagram). The FF and FB channels can work together or independently. The FF channel MIC is placed on the outside of the earpiece, while the FB channel MIC is placed on the inside, close to the ear canal when worn. In active noise cancellation mode, the FF MIC collects sound signals, which pass through an analog gain module, an ADC module, a low-pass filter and a downsampling filter module, and then through an FF active noise cancellation (ANC) filter, serving as the input to the digital-to-analog converter (DAC). The FB MIC collects the audio signal, which passes through an analog gain module, an ADC module, a low-pass and downsampling filter module, and then through an FB ANC filter before being used as the input to the DAC. When playing audio, the audio signal is sent to the DAC for playback. Simultaneously, the audio signal passes through an audio echo module to cancel out the audio component in the FB noise reduction path, ensuring that the sound collected in the FB noise reduction path is unaffected by the audio component. In other words, the audio signal is added to or subtracted from the low-pass and downsampling filter modules in the FB noise reduction channel to eliminate the influence of the audio component on the noise reduction path, and then the remaining ambient noise is sent to the FB ANC filter. The signal filtered by the FB ANC filter can then pass through a limiter 1 (limiter 1 is not mandatory) before being sent to the DAC for playback, thus completing the ANC noise reduction function and the audio signal playback function.

[0027] In some examples, if the MIC is a digital MIC, then Figure 1 The analog gain module and ADC may be omitted. In other examples, the FF ANC filter and FB ANC filter can be adaptive or fixed filters. They can be any of the following structures: Infinite Impulse Response (IIR) structure, Finite Impulse Response (FIR) structure, or a hybrid IIR and FIR filter structure. In still other examples, the active noise cancellation mode can have only FF channels, only FB channels, or both FF and FB channels.

[0028] The pass-through mode refers to a system where, when a user wears headphones, the various external sounds they hear are as close as possible to what they would experience without headphones. This allows users to perceive external sounds in the most natural way, thus solving the problem of users having difficulty hearing external sounds easily due to headphone obstruction, which affects the user experience. It also allows headphone wearers to better receive external voice messages, environmental noise, and various alarm sounds.

[0029] The operating principle block diagram in transparent transmission mode can be as follows: Figure 2 As shown, the head-mounted device may include a feedforward (FF) talk-through channel and a feedback (FB) active noise cancellation (ANC) channel. For the FF talk-through channel (the topmost path in the diagram), the FF microphone collects the audio signal, which is then processed sequentially through an analog gain module, an ADC module, a low-pass and downsampling filter module, an FF talk-through filter, and a limiter 2. The processed result is used as the input to the DAC. For the FB active noise cancellation channel (the bottommost path in the diagram), the FB microphone collects the audio signal, which is then processed sequentially through an analog gain module, an ADC module, a low-pass and downsampling filter module, an FB ANC filter, and a limiter 3. The processed result is used as the input to the DAC. When playing audio, the audio signal to be played is sent to the DAC for playback. Simultaneously, when the FB active noise cancellation channel is activated, the audio signal to be played passes through an audio echo module to cancel out the audio component in the FB noise cancellation path, ensuring that the sound collected in the FB noise cancellation path is unaffected by the audio component being played. Specifically, the audio signal to be played is added to or subtracted from the low-pass and downsampling filter modules in the FB noise reduction channel after the output of the audio echo module is passed through the audio echo module. After eliminating the influence of the audio signal to be played on the noise reduction path, the remaining environmental noise is sent to the FB active noise reduction filter and limiter 3 as the input of the DAC.

[0030] In some embodiments, when the FB active noise cancellation channel is enabled, the output of the FF pass-through channel can be used by the audio echo module to cancel out the FF pass-through signal component in the FB noise cancellation path, so that the sound collected in the FB noise cancellation path is not affected by the FF pass-through signal. In some examples, the output of the FF pass-through channel is added to the played audio and can be used as the input of the audio echo module.

[0031] In some embodiments, limiter 2 and limiter 3 are optional and can be used as needed. The analog gain module, ADC module, low-pass and downsampling filter module can be different.

[0032] In some embodiments, an upsampling and filtering module may be included before the DAC. After the aforementioned signals are added together, they can first pass through an upsampling and filtering module before being used as the input to the DAC, thereby improving the output quality of the DAC.

[0033] In some embodiments, the MIC can be a digital MIC, in which case, the upper Figure 2 The analog gain and ADC may be omitted. In other examples, the FF ANC filter and FB ANC filter can be adaptive or fixed filters. They can be any of the following structures: Infinite Impulse Response (IIR) structure, Finite Impulse Response (FIR) structure, or a hybrid IIR and FIR filter structure. In still other examples, the pass-through mode can have only an FF pass-through channel, only an FB active noise cancellation channel, or both FF pass-through and FB active noise cancellation channels.

[0034] Two-way mode refers to the microphone of a head-mounted device collecting voice signals and wirelessly transmitting them to another head-mounted device, as well as receiving wireless audio signals sent by another head-mounted device and playing them through a speaker.

[0035] In related technologies, head-mounted devices are equipped with an image acquisition module, which can then capture images. During use, the head-mounted device can autonomously determine its operating mode—whether it's in active noise cancellation mode or pass-through mode—based on the captured images, thus automatically switching modes to provide users with a more comfortable and efficient user experience.

[0036] However, when the headset needs to be in intercom mode, manual selection is required, increasing operational complexity. Manually switching intercom modes in emergencies or busy situations can be inconvenient. Furthermore, because intercom mode selection is manual, it becomes overly reliant on user awareness, potentially leading to brief communication interruptions, affecting the smoothness of communication, and ultimately impacting the user experience.

[0037] To address the aforementioned issues, this disclosure provides a device interaction method applied to a first head-mounted device. The first head-mounted device refers to the head-mounted device currently used to perform device interaction. This first head-mounted device includes an image acquisition module, which can be used to acquire images of the application environment of the first head-mounted device to determine whether the first head-mounted device needs to interact with a second head-mounted device, thereby facilitating the timely execution of device interaction functions by the first head-mounted device. The second head-mounted device refers to the head-mounted device used to interact with the first head-mounted device. The first and second head-mounted devices can be devices of the same specifications but with different roles.

[0038] That is, the device interaction method provided in this disclosure is applicable to application scenarios involving at least two head-mounted devices in the same space, and each head-mounted device is equipped with an image acquisition module. During application, each head-mounted device can determine whether it needs to interact with other head-mounted devices based on the images acquired by its image acquisition module, and then achieve interaction through intercom mode or transparent transmission mode. When a head-mounted device needs to interact with other head-mounted devices, if there are two head-mounted devices in the same space, then the first head-mounted device is the first head-mounted device, and the other head-mounted device is the second head-mounted device; if there are more than two head-mounted devices in the same space, then the second head-mounted device is one or more of the other head-mounted devices.

[0039] The following will explain in detail the process by which the first head-mounted device identifies the second head-mounted device and the interaction between the devices. For example... Figure 3 As shown, the device interaction method may include the following steps:

[0040] Step S110: The first image and the second image are acquired through the image acquisition module.

[0041] To determine whether the first head-mounted device needs to interact with other head-mounted devices, an image acquisition module is used to acquire images of the scene in which the first head-mounted device is located, thereby obtaining a first image and a second image with different perspectives. This allows for subsequent determination of whether a second head-mounted device exists for interacting with the first head-mounted device based on the content of the first image and the content of the second image.

[0042] The image acquisition module may include multiple cameras, each capturing different image content. The placement of these cameras on the first head-mounted device can be customized as needed and is not limited here. For example, multiple cameras can be positioned at the same location on the first head-mounted device, but in different poses, to capture images from different angles relative to the head-mounted device, thereby improving image quality and scene understanding. Alternatively, multiple cameras can be positioned at different locations on the first head-mounted device to capture images from corresponding angles, significantly enhancing the device's sensing capabilities, interactivity, and application range.

[0043] Step S120: Based on the first image and the second image, determine a second head-mounted device for interacting with the first head-mounted device.

[0044] By analyzing the content of the first image and the content of the second image, the environment in which the first head-mounted device is located and the context information of the environment can be determined. Then, during the operation of the first head-mounted device, the content of the first image and the content of the second image can be used to determine whether it is necessary to interact with other head-mounted devices. If it is determined that interaction is necessary, the head-mounted device used to interact with the first head-mounted device will be designated as the second head-mounted device to ensure the timeliness of device interaction.

[0045] In some examples, the content of the first image and the content of the second image can be analyzed using image algorithms or neural network models to determine whether the scene in which the first head-mounted device is located is a device interaction scene. Then, if it is determined that the scene in which the first head-mounted device is located is a device interaction scene, another head-mounted device in the device interaction scene can be used as the second head-mounted device.

[0046] In other examples, features can be extracted from the first image and the second image separately to identify the content of the first image and the content of the second image. Then, based on the intersection of the content of the first image and the content of the second image, it can be determined whether the scene in which the first head-mounted device is located is a device interaction scene. If it is determined that the scene in which the first head-mounted device is located is a device interaction scene, another head-mounted device in the device interaction scene can be used as the second head-mounted device.

[0047] Step S130: Interact with the second headset via intercom mode.

[0048] In active noise cancellation mode, the user of the first headset will have difficulty hearing ambient sounds or voices. If the distance between the first and second headsets is too great, they will also have difficulty hearing each other's voices. In this case, when it is determined that the first headset needs to interact with the second headset, it can be done through intercom mode.

[0049] In intercom mode, the first headset can send the acquired voice signal to the second headset, or receive the voice signal sent by the second headset and play the audio, thus enabling device interaction.

[0050] In step S140, the first user wearing the first head-mounted device interacts with the second user wearing the second head-mounted device through a pass-through mode.

[0051] In transparent transmission mode, the first head-mounted device can autonomously collect the speech content of the second user wearing the second head-mounted device and play it back to the first user wearing the first head-mounted device, so that the first user can clearly understand the speech content and facilitate communication with the second user, thereby achieving interaction.

[0052] According to the device interaction method provided in this disclosure, by acquiring the content of the first image and the content of the second image through the image acquisition module, it can be determined whether the first head-mounted device needs to interact with other head-mounted devices. If interaction is required, the device can interact with the second head-mounted device through intercom mode, or the first user wearing the first head-mounted device can interact with the second user wearing the second head-mounted device through transparent transmission mode. This can effectively reduce reliance on user awareness, ensure smooth user communication, simplify operation, enable autonomous switching of operating modes, improve the intelligence of the first head-mounted device, and enhance the user experience.

[0053] In some embodiments, such as Figure 4 As shown, step S110 above may include the following steps:

[0054] Step S111: Based on the first image, identify whether there is a head-mounted device to be interacted with at a position relative to the location of the first user.

[0055] Determining whether the scenario in which the first head-mounted device is located is a device interaction scenario mainly involves analyzing whether the first user needs to communicate with another user using the head-mounted device by examining the content of the first image and the content of the second image.

[0056] The first image can be understood as an image captured from the environment in which the first head-mounted device is located. Image analysis of the content of the first image can determine whether a head-mounted device to be interacted with exists at a location relative to the first user's location. For example, by analyzing the content of the first image, it can be identified whether a head-mounted device exists within the image. If a head-mounted device exists, it can be considered that its location is relative to the first user's location, and that it is the head-mounted device to be interacted with. If no head-mounted device exists, it can be considered that there is no head-mounted device available for interaction at a location relative to the first user's location.

[0057] In some examples, the content of the first image may be obtained through inference by an AI model or through image processing by a neural network model. The specific model or algorithm used to identify the image content can be determined according to actual needs and is not limited here.

[0058] In other examples, the image acquisition direction of the first camera is the same as the field of view of the first user, thus ensuring that the position of the head-mounted device appearing in the first image is relative to the position of the first user, thereby improving the positioning efficiency of the head-mounted device to be interacted with. Here, the first camera is the camera in the image acquisition module used to acquire the first image, and there can be at least one such first camera.

[0059] In some other examples, the image acquisition direction of the first camera is a specified direction, which overlaps with the field of view of the first user. This effectively avoids misidentification and ensures that the position of the head-mounted device appearing in the first image is relative to the position of the first user, thereby improving the accuracy and efficiency of determining the head-mounted device to be interacted with. The number of first cameras can be at least one.

[0060] Step S112: Determine the gaze direction of the first user based on the second image.

[0061] The second image can be understood as an image captured of the first user's face, including an image of the first user's eyes. Image analysis of the second image can determine the first user's gaze direction. Combined with the content of the first image, it can be determined whether the first user has a need for communication. This allows for timely determination of the operating mode of the head-mounted device to meet device interaction requirements, enhance the intelligence of the head-mounted device, and improve the user experience.

[0062] In some examples, the content of the second image can be obtained through AI model inference or through image processing using a neural network model. The specific model or algorithm used to identify the image content can be determined according to actual needs and is not limited here.

[0063] In other examples, the second image can be a series of multiple frames. Based on the eye images in each frame, the eye movements of the first user can be tracked to determine the direction of gaze, ensuring the reliability and accuracy of the determination. The second camera in the image acquisition module, used to acquire the second image, can be configured in the first head-mounted device to capture the position of the first user's face.

[0064] In some other examples, there can be multiple second cameras, which are configured at different positions on the first head-mounted device. However, the image acquisition direction of each second camera includes the position of the first user's face. By combining the second images acquired by each second camera, image information from multiple angles can be integrated to determine the direction of movement of the first user's eyeballs, thereby determining the direction of the first user's gaze and ensuring determination efficiency.

[0065] In step S113, in response to the existence of a head-mounted device to be interacted with, and the head-mounted device to be interacted with is in the line of sight of the first user, the head-mounted device to be interacted with is determined to be a second head-mounted device for interacting with the first head-mounted device.

[0066] In response to the presence of a head-mounted device to be interacted with, and the head-mounted device to be interacted with is in the line of sight of the first user, indicating that the first user is looking directly at or focusing their attention on the head-mounted device to be interacted with, it can be assumed that the first user needs to interact with the head-mounted device to be interacted with through the first head-mounted device. Thus, the head-mounted device to be interacted with can be determined as a second head-mounted device for interacting with the first head-mounted device, so that the first head-mounted device can interact with the second head-mounted device in a timely manner, meet the user's needs, and thereby improve the user experience.

[0067] In some examples, the first and second images can be acquired simultaneously or asynchronously. For instance, with simultaneous acquisition, the first and second images can be acquired separately, and then the second head-mounted device can be determined jointly based on the content of the first and second images. This ensures that the correlation between the content of the first and second images is not affected by time, thereby improving the reliability of determining the second head-mounted device. With asynchronous acquisition, the first image can be acquired first, and if it is determined based on the content of the first image that a head-mounted device to be interacted with exists, the second image can then be acquired to determine whether the first user needs to interact with the head-mounted device through the first head-mounted device, thereby improving the efficiency of determining the second head-mounted device. If it is determined based on the content of the first image that no head-mounted device to be interacted with does not exist, the second image is not acquired to save image analysis costs.

[0068] In other examples, determining the head-mounted device to be interacted with may further include: determining whether a head-mounted device located opposite the first user's location has a wireless connection to the first head-mounted device. If the head-mounted device located opposite the first user's location has a wireless connection to the first head-mounted device, it can be determined that the head-mounted device can interact with the first head-mounted device, and thus the head-mounted device can be used as the head-mounted device to interact with the first head-mounted device, providing a reliable connection guarantee for subsequent device interaction. In response to the existence of a head-mounted device wirelessly connected to the first head-mounted device, and the head-mounted device to be interacted with being located in the first user's line of sight, the head-mounted device to be interacted with is determined as a second head-mounted device for interacting with the first head-mounted device. Subsequently, whether through intercom mode or pass-through mode, device interaction can be achieved without interference, thereby helping to ensure the convenience and effectiveness of device interaction.

[0069] In some embodiments, such as Figure 5 As shown, the device interaction method may also include the following steps:

[0070] Step S150: Detect the current environment and obtain the detection results.

[0071] Because head-mounted devices are used in a wide range of scenarios, and the impact of environmental noise on the transmitted audio signal varies in different environments. For example, in busy streets, construction sites, or concert venues, environmental noise can degrade the quality of the received audio signal, resulting in unclear speech and affecting communication. However, in relatively quiet places such as libraries, open squares, or dormitories, environmental noise will not affect the quality of the transmitted audio signal.

[0072] Therefore, to ensure a good user experience, the current environment is detected to determine the noise level and obtain a detection result. For example, the ambient sound can be captured by the microphone in the first head-mounted device, and then the noise intensity, power, and signal amplitude can be evaluated using a noise analysis algorithm to obtain a detection result that assesses the noise level of the current environment.

[0073] Step S160: Based on the detection results, determine the operating mode for interacting with the second headset as either intercom mode or pass-through mode.

[0074] Based on the detection results, the ambient noise level can be determined, allowing selection of either intercom mode or pass-through mode as the operating mode for interacting with the second headset. For example, if the ambient noise is relatively low, indicating a quiet environment that won't affect the quality of the first headset's microphone's voice signal acquisition, then pass-through mode can be chosen for interaction with the second headset. This minimizes latency during interaction, allowing the first user to quickly receive the voice from the second headset's user, resulting in a better conversational experience. Conversely, if the ambient noise is relatively high, indicating a noisy environment that will affect the quality of the first headset's microphone's voice signal acquisition, then intercom mode can be chosen for interaction with the second headset. This ensures high-quality wireless voice signal transmission, improving playback quality and allowing the first user to hear the voice clearly, minimizing the impact of ambient noise. Wireless transmission may include, but is not limited to, Bluetooth, Bluetooth Low Energy, Wi-Fi, or other radio wave transmission methods, without any restrictions.

[0075] In some embodiments, step S160 above may include the following steps:

[0076] Step a1: In response to the detection result indicating that the noise of the current environment is greater than or equal to the first threshold, the operating mode for interacting with the second headset is determined to be the intercom mode;

[0077] Step a2: In response to the detection result indicating that the noise of the current environment is less than the first threshold, the operating mode for interacting with the second head-mounted device is determined to be the pass-through mode.

[0078] Specifically, the first threshold can be understood as the maximum critical value used to measure whether the noise in the current environment can be ignored. If the noise in the current environment is greater than or equal to the first threshold, it indicates that the environmental noise cannot be ignored, and the ambient sound of the current environment is relatively noisy. If the noise in the current environment is less than the first threshold, it indicates that the environmental noise can be ignored, and the ambient sound of the current environment is relatively quiet.

[0079] Therefore, in response to the detection result indicating that the noise of the current environment is greater than or equal to a first threshold, and considering that the ambient sound of the current environment affects the quality of the voice signal collected by the microphone of the first head-mounted device, the operating mode for interacting with the second head-mounted device is determined to be the intercom mode, so as to ensure the transmission quality of the voice signal through wireless transmission, thereby improving the playback quality of the voice signal, enabling the first user to hear the content of the voice signal as clearly as possible, and reducing the impact of ambient noise.

[0080] In response to the detection result indicating that the noise in the current environment is less than a first threshold, and that the ambient sound in the current environment will not affect the quality of the voice signal collected by the microphone of the first headset, the operating mode for interacting with the second headset is determined to be the pass-through mode. This is to minimize latency when interacting with the second headset, so that the first user can quickly obtain the voice emitted by the user of the second headset, thereby bringing a better dialogue experience to the first user.

[0081] In some embodiments, step S160 may further include the following steps:

[0082] Step a3: In response to the detection result indicating that the noise of the current environment is greater than or equal to the second threshold, determine the operating mode for interacting with the second head-mounted device as a mode that combines intercom mode and active noise cancellation mode.

[0083] The second threshold is greater than the first threshold. The second threshold can be understood as the maximum critical value used to measure whether the noise in the current environment needs active noise reduction. Since the detection result indicates that the noise in the current environment is greater than or equal to the second threshold, indicating that the ambient sound is too noisy, the transmission of voice signals wirelessly will inevitably be affected by noise. Therefore, the operating mode for interacting with the second headset is determined to be a parallel mode of intercom mode and active noise reduction mode. In intercom mode, when interacting with the device, the active noise reduction function provided by the active noise reduction mode can be used to perform environmental noise reduction processing on the voice signal collected by the microphone of the first headset, minimizing interference from environmental noise and thus improving the quality of voice signal acquisition, thereby ensuring the quality of voice signal transmission to the second headset.

[0084] In other examples, based on the content of the first image and the content of the second image, it can be determined that although there is a head-mounted device to be interacted with, the first user is not looking directly at or paying attention to the head-mounted device to be interacted with. It can be assumed that the first user is talking to himself or communicating with others. Therefore, in order to reduce interference with the first user, the operating mode for interacting with the second head-mounted device can be determined to be the pass-through mode.

[0085] In some embodiments, the device interaction method may further include: sending a mode notification based on the communication connection and operating mode with the second head-mounted device, so that the second head-mounted device adjusts its operating mode to match that of the first head-mounted device in response to the mode notification, thereby achieving device interaction. By sending mode notifications, it can be ensured that the first and second head-mounted devices can synchronize their operating modes according to current usage needs and environmental conditions, thereby achieving seamless device interaction and collaborative work, which helps to enhance the collaborative functions between devices and the user experience.

[0086] In some examples, the first headset and the second headset may be pre-paired to enable quick intercom functionality. There may be at least one second headset. If there is only one second headset, the first headset can quickly establish intercom functionality with it based on pre-pairing. If there are multiple second headsets, the first headset can quickly establish intercom functionality with multiple second headsets based on pre-pairing.

[0087] In other examples, the first headset can be paired with the second headset after the second headset has been identified and it is determined that device interaction with the second headset is required via intercom mode, in order to save power consumption during device connection and thus extend the usage time of the first headset.

[0088] In some embodiments, the first head-mounted device may further include at least one microphone and a speaker to acquire speech signals via the microphone and play audio via the speaker. In some examples, there may be multiple microphones. Therefore, in intercom mode, because the first head-mounted device is close to the first user's mouth and the microphones on the first head-mounted device are relatively fixed in position to the first user's mouth, configuring multiple microphones can enhance the speech in the direction of the user's mouth, improving speech quality. In other examples, in a mode where intercom mode and active noise cancellation mode run in parallel, speech noise cancellation processing can be performed after acquiring the first user's speech from each microphone, further improving speech quality. In still other examples, in pass-through mode, configuring multiple microphones can achieve lower latency (compared to intercom mode) for receiving the second user's speech, providing a better conversational experience for the first user.

[0089] In some application scenarios, the first head-mounted device includes at least one microphone, a speaker, and a self-speech detection module. In intercom mode, the first head-mounted device can acquire a first voice signal through at least one microphone and detect whether the first voice signal is spoken by a first user through the self-speech detection module. In response to detecting that the first voice signal is spoken by the first user, indicating that the first voice needs to be sent from the first head-mounted device to a second head-mounted device, the first voice signal can be sent based on the communication connection with the second head-mounted device. The second head-mounted device then plays the audio based on the first voice signal through its speaker, allowing the user of the second head-mounted device (the second user) to receive the content of the first voice signal, facilitating real-time voice communication with the first user and making voice communication unrestricted by distance. The self-speech detection module can use a neural network model to detect whether the first voice signal is spoken by the first user. The first voice signal input to the self-speech detection module can be acquired by an FF microphone or jointly acquired by an FF microphone and an FB microphone. If the first voice signal input to the self-speaking detection module is jointly acquired by the FF microphone and the FB microphone, then by combining the first voice signal acquired by the FB microphone, it can be ensured that the vibration sound in the first voice signal is generated when the first user speaks, and it is impossible to include the vibration sound generated when others speak; moreover, since the FB microphone is placed inside the first user's ear, it is even less likely to collect the voices of other people, which can ensure the accuracy and reliability of the first voice signal.

[0090] In some application scenarios, in intercom mode, the first headset can receive a second voice signal based on the communication connection with the second headset, and play the audio based on the second voice signal through the speaker of the first headset. This allows the first user to receive the content of the second voice signal, facilitating real-time voice communication with the user of the second headset (the second user), making voice communication unrestricted by distance. The second voice signal is acquired through the second headset.

[0091] In some application scenarios, under pass-through mode, the first head-mounted device can acquire a third voice signal through at least one microphone and play audio through a speaker based on the third voice signal. This allows the first user to access the second user's speech, providing an instant, latency-free voice transmission experience, enhancing communication efficiency between the first and second users, and improving the user experience. The third voice signal includes the second user's speech.

[0092] In some optional application scenarios, during the first user's use of the first head-mounted device, the first head-mounted device can acquire a first image and a second image through an image acquisition module. The first image is used to determine whether the head-mounted device to be interacted with exists at a position relative to the first user's location. The second image is used to determine the first user's line of sight.

[0093] By analyzing the content of the first and second images, if there is a headset to be interacted with, and the headset to be interacted with is in the line of sight of the first user, then it can be assumed that the first user needs to interact with the headset to be interacted with through the first headset. Therefore, the headset to be interacted with is determined to be a second headset used to interact with the first headset. Thus, to ensure the quality of device interaction, interaction with the second headset can be achieved through intercom mode.

[0094] If the ambient noise of the current environment is less than the first threshold, it indicates that the ambient sound of the current environment will not affect the quality of the voice signal collected by the microphone of the first head-mounted device. Therefore, the first user can interact with the second user wearing the second head-mounted device through the pass-through mode of the first head-mounted device, so as to reduce latency, enhance the communication efficiency between the first user and the second user, and improve the user experience.

[0095] If the ambient noise detected is greater than or equal to the first threshold, it indicates that the ambient sound will affect the quality of the voice signal collected by the microphone of the first headset. Therefore, the second headset can be interacted with in intercom mode to reduce the impact of ambient noise, ensure the transmission quality of the voice signal, and thus improve the playback quality of the speech signal.

[0096] If the ambient noise detected is greater than or equal to the second threshold, it indicates that the ambient sound is too noisy. When transmitting voice signals wirelessly, the noise will inevitably affect the signal. Therefore, the second headset can be interacted with in a parallel mode of intercom and active noise cancellation to minimize the interference caused by ambient noise, improve the quality of voice signal acquisition, and thus ensure the quality of voice signal transmission to the second headset.

[0097] Based on the same inventive concept, this disclosure also provides a head-mounted device. For example... Figure 6 As shown, the head-mounted device 200 may include:

[0098] The first camera 210 can be used to capture the first image;

[0099] The second camera 220 can be used to capture a second image;

[0100] The processing module 230 is connected to the first camera 210 and the second camera 220 respectively, and can be used to determine a second head-mounted device for interacting with the head-mounted device 200 based on the first image and the second image.

[0101] The communication module 240, connected to the processing module 230, can be used to interact with the second headset in a two-way communication mode;

[0102] The interaction module 250, connected to the processing module 230, is used for the first user wearing the head-mounted device to interact with the second user wearing the second head-mounted device through a pass-through mode.

[0103] In some embodiments, the processing module 230 may be configured to: identify, based on a first image, whether there is a head-mounted device to be interacted with at a position relative to the location of the first user; determine, based on a second image, the direction of the first user's gaze; and, in response to the existence of a head-mounted device to be interacted with, and the head-mounted device to be interacted with being located in the direction of the first user's gaze, determine that the head-mounted device to be interacted with is a second head-mounted device for interacting with the head-mounted device.

[0104] In some embodiments, the processing module 230 can also be used to detect the current environment and obtain detection results; and to determine, based on the detection results, the operating mode for interacting with the second head-mounted device as intercom mode or pass-through mode.

[0105] In some embodiments, the processing module 230 may be configured to determine, in response to a detection result indicating that the noise of the current environment is greater than or equal to a first threshold, an operating mode for interacting with the second head-mounted device is an intercom mode; and to determine, in response to a detection result indicating that the noise of the current environment is less than the first threshold, an operating mode for interacting with the second head-mounted device is a pass-through mode.

[0106] In some embodiments, the processing module 230 may also be configured to determine, in response to a detection result indicating that the noise of the current environment is greater than or equal to a second threshold, an operating mode for interacting with the second head-mounted device as a mode in which intercom mode and active noise cancellation mode run in parallel, wherein the second threshold is greater than the first threshold.

[0107] In some embodiments, the communication module 240 can also be used to send a mode notification based on the communication connection and operating mode with the second head-mounted device, so that the second head-mounted device adjusts its operating mode to match the operating mode of the head-mounted device in response to the mode notification, thereby realizing device interaction.

[0108] In some embodiments, such as Figure 7 As shown, the interaction module 250 may include: at least one microphone 251, connected to the processing module 230, for acquiring voice signals; and a speaker 252, connected to the processing module 230, for audio playback.

[0109] In some embodiments, at least one microphone 251 may be used to acquire a first voice signal, and the communication module 240 may be used to send the first voice signal based on the communication connection with the second head-mounted device, so that the second head-mounted device can play audio through the speaker of the second head-mounted device based on the first voice signal.

[0110] In some embodiments, the communication module 240 can be used to receive a second voice signal based on a communication connection with the second head-mounted device; the speaker 252 can be used to play audio based on the second voice signal, wherein the second voice signal is acquired through the second head-mounted device.

[0111] In some embodiments, at least one microphone 251 may be used to acquire a third voice signal, wherein the third voice signal includes the speech of a second user using a second head-mounted device; and a speaker 260 may be used to play audio based on the third voice signal so that a first user using a head-mounted device can acquire the speech of the second user.

[0112] Regarding the head-mounted device in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0113] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a program for performing the device interaction method of any of the foregoing embodiments.

[0114] This disclosure uses specific terms to describe embodiments of the present disclosure. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics in one or more embodiments of the present disclosure can be appropriately combined.

[0115] In the context of this disclosure, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0116] Similarly, it should be noted that, in order to simplify the description of this disclosure and thus aid in the understanding of one or more embodiments, the foregoing description of embodiments of this disclosure may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of this disclosure requires more features than the features claimed. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0117] The basic concepts have been described above. It is obvious that the above disclosure is merely illustrative and does not constitute a limitation of this disclosure. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this disclosure by those skilled in the art. Such modifications, improvements, and corrections are suggested in this disclosure and therefore remain within the spirit and scope of the embodiments of this disclosure.

Claims

1. A device interaction method applied to a first head-mounted device, the first head-mounted device comprising an image acquisition module, the method comprising: acquiring, by the image acquisition module, a first image and a second image respectively; determining, based on the first image and the second image, a second head-mounted device for interacting with the first head-mounted device; interacting with the second head-mounted device in a talk-through mode; or interacting, by a first user wearing the first head-mounted device, with a second user wearing the second head-mounted device in a pass-through mode. The determining, based on the first image and the second image, a second head-mounted device for interacting with the first head-mounted device, comprises:

2. The device interaction method of claim 1, wherein, identifying, based on the first image, whether there is a head-mounted device to be interacted with in a position opposite to a position where the first user is located; determining, based on the second image, a line-of-sight direction of the first user; in response to the presence of the head-mounted device to be interacted with and the head-mounted device to be interacted with being in the line-of-sight direction of the first user, determining the head-mounted device to be interacted with as the second head-mounted device for interacting with the first head-mounted device. The method further comprises:

3. The device interaction method according to claim 1 or 2, wherein, detecting a current environment to obtain a detection result; determining, based on the detection result, an operation mode for interacting with the second head-mounted device to be the talk-through mode or the pass-through mode. The determining, based on the detection result, an operation mode for interacting with the second head-mounted device to be the talk-through mode or the pass-through mode, comprises:

4. The device interaction method of claim 3, wherein, in response to the detection result representing that noise of the current environment is greater than or equal to a first threshold value, determining the operation mode for interacting with the second head-mounted device to be the talk-through mode; in response to the detection result representing that noise of the current environment is less than the first threshold value, determining the operation mode for interacting with the second head-mounted device to be the pass-through mode. The determining, based on the detection result, an operation mode for interacting with the second head-mounted device to be the talk-through mode or the pass-through mode, further comprises:

5. The device interaction method of claim 4, wherein, in response to the detection result representing that noise of the current environment is greater than or equal to a second threshold value, determining the operation mode for interacting with the second head-mounted device to be a mode in parallel with the talk-through mode and an active noise reduction mode, wherein the second threshold value is greater than the first threshold value. The method further comprises:

6. The device interaction method of claim 5, wherein, based on a communication connection between the first head-mounted device and the second head-mounted device and the operation mode, sending a mode notification to cause the second head-mounted device to adjust, in response to the mode notification, an operation mode of the second head-mounted device to be matched with the operation mode of the first head-mounted device, to realize device interaction. The first head-mounted device comprises at least one microphone, a speaker and a self-speech detection module, and the interacting with the second head-mounted device in the talk-through mode comprises:

7. The device interaction method of claim 1, wherein, in the talk-through mode: ​ acquire a first voice signal through at least one microphone, and detect, through the self-speech detection module, whether the first voice signal is speech uttered by the first user, and in response to detecting that the first voice signal is speech uttered by the first user, send the first voice signal based on a communication connection between the first head-mounted device and the second head-mounted device, to cause the second head-mounted device to perform audio playback through a loudspeaker of the second head-mounted device based on the first voice signal; or receive a second voice signal based on a communication connection between the first head-mounted device and the second head-mounted device, and perform audio playback through the loudspeaker of the first head-mounted device based on the second voice signal, wherein the second voice signal is acquired through the second head-mounted device.

8. The device interaction method of claim 1, wherein, The first head-mounted device includes at least one microphone and a loudspeaker, and a first user wearing the first head-mounted device interacts with a second user wearing a second head-mounted device in a pass-through mode, including: In the pass-through mode: acquire a third voice signal through at least one microphone, wherein the third voice signal includes speech content of the second user; and perform audio playback through the loudspeaker based on the third voice signal, to cause the first user to acquire the speech content of the second user.

9. A head-mounted device, comprising: a first camera configured to capture a first image; a second camera configured to capture a second image; a processing module connected to the first camera and the second camera, respectively, and configured to determine, based on the first image and the second image, a second head-mounted device for interacting with the head-mounted device; a communication module connected to the processing module and configured to interact with the second head-mounted device in a talk-through mode; an interaction module connected to the processing module and configured to cause a first user wearing the head-mounted device to interact with a second user wearing the second head-mounted device in a pass-through mode.

10. The head-mounted device of claim 9, wherein, The interaction module includes: at least one microphone connected to the processing module and configured to acquire a voice signal; a loudspeaker connected to the processing module and configured to perform audio playback.

Images