Method, apparatus and device for voice interaction and storage medium

Abstract

Embodiments of the present disclosure relate to an interaction method, device, equipment and storage medium. The method proposed herein is applied to an electronic device, and the method comprises: when the electronic device is in a first mode, continuously capturing a plurality of images by using an image capturing unit carried by the electronic device; and in response to a visual interaction request of a user, outputting response content to the user, wherein the response content is generated by a pre-trained model based on at least one image of the plurality of images. In this way, embodiments of the present disclosure can meet the needs of users to interact with electronic devices.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatuses, devices, and computer-readable storage media for voice interaction. Background Technology

[0002] With the increasing maturity of science and technology, the use of electronic devices has gradually become a daily habit. For example, users can use electronic devices to play audio content, check the time, and measure physiological data. In addition, people can use electronic devices for interactive operations. However, the interactive operations supported by existing electronic devices are limited to simple operations such as playing and switching audio, which cannot meet users' deeper interactive needs. Summary of the Invention

[0003] In a first aspect of this disclosure, an interaction method is provided for use in an electronic device. The method includes: continuously capturing multiple images using an image acquisition unit mounted on the electronic device while the electronic device is in a first mode; and outputting response content to the user in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the multiple images.

[0004] In a second aspect of this disclosure, an interaction method is provided for use in a terminal device. The method includes: acquiring a plurality of images from an electronic device wirelessly connected to the terminal device, the plurality of images being continuously captured using an image acquisition unit mounted on the electronic device while the electronic device is in a first mode; acquiring response content in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; and sending the response content to the electronic device to output the response content to the user using the electronic device.

[0005] In a third aspect of this disclosure, an interaction method is provided for use in an interaction system. The interaction system includes a wirelessly communicating electronic device and a terminal device. The method includes: when the electronic device is in a first mode, continuously capturing multiple images using an image acquisition unit on the electronic device; transmitting the multiple images from the electronic device to the terminal device; in response to a user's visual interaction request, obtaining response content from the terminal device, wherein the response content is generated by a pre-trained model based on at least one of the multiple images; and transmitting the response content from the terminal device to the electronic device to output the response content to the user using the electronic device.

[0006] In a fourth aspect of this disclosure, an apparatus for interaction is provided, applied to an electronic device. The apparatus includes: an image acquisition module configured to continuously capture multiple images using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; and a playback module configured to output response content to a user in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the multiple images.

[0007] In a fifth aspect of this disclosure, an apparatus for interaction is provided, applied to a terminal device. The apparatus includes: an image acquisition module configured to acquire a plurality of images from an electronic device wirelessly connected to the terminal device, the plurality of images being continuously captured using an image acquisition unit mounted on the electronic device while the electronic device is in a first mode; a second receiving module configured to acquire response content in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; and a sending module configured to send the response content to the electronic device to output the response content to the user using the electronic device.

[0008] In a sixth aspect of this disclosure, an apparatus for interaction is provided, applied to an interactive system. The interactive system includes a wireless communication electronic device and a terminal device. The apparatus includes: an image capturing module configured to continuously capture multiple images using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; and an image transmitting module configured to transmit the multiple images from the electronic device to the terminal device.

[0009] The voice acquisition module is configured to acquire response content from the terminal device in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of a plurality of images; and the voice playback module is configured to send the response content from the terminal device to the electronic device to output the response content to the user using the electronic device.

[0010] In a seventh aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the methods of the first, second, or third aspects.

[0011] In an eighth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the methods of the first, second, or third aspects.

[0012] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0014] Figure 1 A schematic diagram is shown in which an example interactive system according to embodiments of the present disclosure may be implemented;

[0015] Figure 2 A flowchart illustrating an example process for voice interaction according to some embodiments of this disclosure is shown;

[0016] Figure 3A and Figure 3B Example interfaces according to some embodiments of this disclosure are shown;

[0017] Figure 4 A flowchart illustrating an example process of interaction according to some embodiments of this disclosure is shown;

[0018] Figure 5 A flowchart illustrating an example process of interaction according to yet another embodiment of this disclosure is shown;

[0019] Figure 6 A flowchart illustrating an example process of interaction according to yet another embodiment of this disclosure is shown;

[0020] Figure 7 A schematic structural block diagram of an example device for interaction according to some embodiments of the present disclosure is shown;

[0021] Figure 8 A schematic structural block diagram of an example device for interaction according to further embodiments of the present disclosure is shown;

[0022] Figure 9 A schematic structural block diagram of an example device for interaction according to further embodiments of the present disclosure is shown; and

[0023] Figure 10 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0026] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0028] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0029] As mentioned above, with the increasing maturity of science and technology, the use of electronic devices has gradually become a daily habit for people. For example, users can use electronic devices to play audio content, check the time, and measure physiological data. In addition, people can also use electronic devices for interactive operations. However, the interactive operations supported by existing electronic devices are limited to simple operations such as playing and switching audio, which cannot meet users' deeper interactive needs.

[0030] Embodiments of this disclosure propose a scheme for voice interaction in electronic devices. According to this scheme, when the electronic device is in a first mode, multiple images can be continuously captured using an image acquisition unit mounted on the electronic device; and in response to a user's visual interaction request, response content can be output to the user, wherein the response content is generated by a pre-trained model based on at least one of the multiple images.

[0031] In this manner, embodiments of the present disclosure can continuously capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in the first mode. Furthermore, embodiments of the present disclosure can, after receiving a user's visual interaction request, acquire and output response voice content generated by a model based on multiple images. Thus, the electronic device provided by embodiments of the present disclosure can generate and output corresponding response content based on image content and visual interaction requests, thereby improving the efficiency of information acquisition for users, expanding the interaction capabilities between the electronic device and the user, and meeting the user's interaction needs. In addition, since the electronic device provided in embodiments of the present disclosure captures images continuously, rather than starting to capture images only after acquiring the user's voice content, this improves the efficiency of voice content processing and reduces user waiting time.

[0032] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0033] Example interactive system

[0034] Figure 1 A schematic diagram of an example interactive system 100 according to an embodiment of this disclosure is shown. (As follows) Figure 1As shown, the interactive system 100 may include an electronic device 110 and a terminal device 120. A communication connection can be established between the electronic device 110 and the terminal device 120. As an example, the electronic device 110 may connect to the terminal device 120 based on an appropriate wireless connection method; or, for example, the electronic device 110 may also establish a wired connection with the terminal device 120. For example, the communication connection may include, but is not limited to, Bluetooth connection, mobile network connection, Universal Serial Bus (USB) connection, Wireless Fidelity (WiFi) connection, etc., and the embodiments of this disclosure are not limited in this respect.

[0035] like Figure 1 As shown, the electronic device 110 may be equipped with one or more image acquisition units 115. For example, the image acquisition unit 115 may be implemented as a camera assembly (e.g., a camera).

[0036] As an example, electronic device 110 may include wearable devices or other devices (e.g., handheld devices). For example, wearable devices may include headphone devices of suitable form, such as in-ear headphones, over-ear headphones, and headphones. This disclosure is not intended to limit the form of headphone devices.

[0037] Additionally, the headphone device may be equipped with an image acquisition unit 115 (e.g., a camera) located in an appropriate position to enable the use of the camera to capture environmental images of the user 150 when the user 150 is wearing the headphone device.

[0038] In some examples, the headphone device may include one or more cameras. For example, the left and right earbuds may each have one camera. As another example, the headphone device may also include only a single camera. For example, only the right earbud may have a camera, while the left earbud may not.

[0039] In other examples, the electronic device 110 may also be implemented in other suitable forms. For example, the electronic device 110 may include an eyeglass device equipped with one or more cameras. For example, such cameras may be located in the frame or rim of the eyeglass device.

[0040] In some embodiments, the electronic device 110 may be provided with an audio acquisition unit (e.g., a microphone) and an audio playback unit (e.g., a speaker). Specifically, the electronic device 110 may use the audio acquisition unit to acquire audio data and send the audio data to the terminal device 120. Alternatively, the electronic device 110 may also use the audio playback unit to play audio data from the terminal device 120.

[0041] In some embodiments, electronic device 110 may also be associated with a voice assistant application installed on terminal device 120. For example, user 150 may use electronic device 110 to interact with the voice assistant application via voice. Additionally, as will be described below, terminal device 120 may also provide a graphical interface for the voice assistant application to provide visual interactive information associated with the voice assistant application.

[0042] In some embodiments, the terminal device 120 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 120 may also support any type of user-facing interface (such as "wearable" circuitry).

[0043] like Figure 1 As shown, the voice assistant application utilizes a pre-trained model 140 to interact with the user 150 in multiple appropriate modalities. As an example, the pre-trained model 140 can be a language model or a generative model that supports multimodal data processing.

[0044] In some embodiments, the pre-trained model 140 can be deployed on the terminal device 120. Alternatively, the pre-trained model 140 can also be deployed on the server 130. When the pre-trained model 140 is deployed on the server 130, the terminal device 120 can invoke the model 140 via a communication connection with the server 130 to achieve interaction within the voice assistant application. For example, the server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The server 130 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. The server 130 can provide background services for the voice assistant application in the terminal device 120.

[0045] In some embodiments, the electronic device 110 may have multiple operating modes. Different operating modes may be related to, for example, the operating mode of the image acquisition unit 115.

[0046] Taking a headset device as an example, when the headset device is in normal conversation mode, the image acquisition unit 115 will not be activated by default. The image acquisition unit 115 will only be activated when it receives an explicit instruction from the user (e.g., to take a photo).

[0047] As will be detailed below, the electronic device 110 can also support operation in a first mode. The first mode, for example, allows the image acquisition unit 115 to be in a normally open state, and in this first mode, the image acquisition unit 115 can periodically capture images 125.

[0048] For example, when a user wears electronic device 110, the user can activate the first mode of electronic device 110. The specific process for activating the first mode will be described in detail below. For example, the user can trigger electronic device 110 to enter the first mode via voice command (e.g., “Activate first mode”).

[0049] Accordingly, during the first mode, the image acquisition unit 115 of the electronic device 110 can periodically capture images. For example, when the headphone device worn by the user is in the first mode, the camera of the headphone device can periodically capture images of the user's environment.

[0050] like Figure 1 As shown, the captured image 125 can be sent to the terminal device 120 in response to a visual interaction request initiated by the user 150.

[0051] As an example, the audio acquisition unit of the electronic device 110 can acquire first voice content 135 from the user 150. As an example, the first voice content 135 could be "What is this?". This first voice content 135 can express the user 150's interactive request regarding visual content.

[0052] Furthermore, model 140 can respond to user 150's visual interaction request using at least one of the plurality of images 125 captured in the first mode. Specifically, when the first voice content 135 is determined to be associated with the visual interaction request, the plurality of images 125 or related images from the plurality of images 125 can be provided to model 140 to generate response content for the visual interaction request, such as second voice content 145.

[0053] As will be detailed below, electronic device 110, terminal device 120, server 130 and / or model 140 can determine whether the first voice content 135 is associated with the visual interaction request based on the first voice content 135 or its corresponding text content.

[0054] Furthermore, the response content generated by model 140 based on the image captured by image acquisition device 115 and / or the first voice content 135 can be the second voice content 145, or text content corresponding to the second voice content 145. The generated text content can be converted into the second voice content 145 by model 140, server 130, and / or terminal device 120.

[0055] by Figure 1 As an example, the second voice content 145 may include a response to the first voice content 135, such as, "This is an apple." For example, the audio playback unit of the electronic device 110 may play the second voice content 145 to the user.

[0056] Therefore, in the first mode, the interaction system 100 can continuously capture multiple images using the image acquisition unit 115 mounted on the electronic device 110, and generate a response to the visual interaction request using the captured images after receiving a user's visual interaction request. Thus, the embodiments of this disclosure can achieve timely response to visual interaction requests.

[0057] Furthermore, the interaction system 100 can support the use of historically captured images to respond to users' visual interaction requests, thereby improving the efficiency of voice interaction. For example, a voice assistant application can use historical images captured five minutes ago to process the user's current visual interaction request (e.g., "What was the name of the store I saw five minutes ago?").

[0058] It should be understood that the structure and function of the various elements in the interactive system 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0059] The following section will describe in detail the specific processing flow of the interactive system 100 with reference to the attached diagram.

[0060] Example voice interaction

[0061] Figure 2 A flowchart of an example process 200 for voice interaction according to some embodiments of the present disclosure is shown. Process 200 may be implemented at electronic device 110, terminal device 120, and / or server 130. Alternatively, process 200 may also be implemented in an interactive system 100 including electronic device 110 and terminal device 120 for wireless communication. Reference is made below. Figure 1 To describe process 200.

[0062] like Figure 2 As shown, electronic device 110 can enable (202) a first mode. As an example, the first mode could be a mode that sets the image acquisition unit 115 of electronic device 110 to always-on. Alternatively, electronic device 110 may also include a second mode (e.g., a normal conversation mode). For example, the second mode could indicate that the image acquisition unit of electronic device 110 is not activated by default. The image acquisition unit 115 will only be activated when an explicit instruction from the user is received (e.g., “take a photo”). The mode of electronic device 110 can be determined based on the configuration operation of user 150. Alternatively, electronic device 110 can also switch the mode of the electronic device (e.g., from the first mode to the second mode) in response to a preset operation by the user (e.g., a voice command (e.g., “switch to the second mode”)).

[0063] As an example, the first mode of electronic device 110 can be enabled in response to an activation request from user 150. As an example, such an activation request may include a voice command (e.g., also referred to as a first voice command) received via the audio acquisition unit of electronic device 110. As an example, the first voice command may include preset keywords (e.g., "Enable first mode", "Start conversation", etc.). This disclosure is not intended to limit the specific content of the preset keywords. Thus, embodiments of this disclosure can support users in quickly enabling the first mode of electronic device 110 based on voice commands, thereby improving the efficiency of user interaction with the electronic device.

[0064] Alternatively, electronic device 110 can be associated with a voice assistant application. As an example, the voice assistant application can be installed on terminal device 120. Electronic device 110 can receive a first voice command from user 150 (e.g., voice content associated with a preset keyword, such as "activate first mode") during a conversation with the voice assistant application. For example, a user can converse with a voice assistant based on the voice assistant application. Electronic device 110 can activate its first mode in response to receiving a request to end the conversation. Alternatively, electronic device 110 can activate its first mode in response to no voice message received during a preset time period (e.g., 2 minutes) in response to a conversation with the voice assistant application. Thus, embodiments of this disclosure can support users enabling the first mode of an electronic device through conversations with a voice assistant application. Furthermore, embodiments of this disclosure can support activating the first mode of electronic device 110 after a conversation with the voice assistant application ends, effectively avoiding conflicts between the process of electronic device 110 receiving voice content in first mode and the conversation between the user and the voice assistant application, ensuring the quality of interaction between the user and electronic device 110 and between the user and the voice assistant application.

[0065] As an example, a voice assistant may include any suitable processing entity that supports interaction via prompts or similar means. In some scenarios, a voice assistant may include, but is not limited to, an agent or other assistant that supports interaction via dialogue (e.g., text or voice conversations). It should be understood that, on the one hand, a voice assistant may have a virtual avatar for interaction. On the other hand, a voice assistant may also utilize appropriate models to drive its interactive behavior, such models may include any suitable machine learning model, such as a generative model. This disclosure is not intended to limit the specific construction of a voice assistant.

[0066] Alternatively, such an activation request may include an activation operation received via an interactive interface associated with electronic device 110. Such an interactive interface may be provided, for example, by terminal device 120 communicating with electronic device 110.

[0067] The following is based on Figure 3A and Figure 3B The following is an exemplary description of process 200. It should be understood that... Figure 3A and Figure 3B The corresponding embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0068] Figure 3A and Figure 3B Example interfaces 300A and 300B according to some embodiments of the present disclosure are shown. Interfaces 300A and 300B can be, for example, provided by... Figure 1 The terminal device 120 shown is provided.

[0069] In some embodiments, such as Figure 3A As shown, terminal device 120 can present interactive interface 300A. Alternatively, terminal device 120 can trigger electronic device 110 to enable the first mode in response to receiving an activation operation from user 150 in interactive interface 300A. Alternatively, such activation operation may also include triggering (e.g., a click operation) a control (not shown) associated with the first mode in interactive interface 300A.

[0070] As an example, the interactive interface 300A may include a conversational interface (or a graphical interface of the voice assistant application) between user 150 and the voice assistant application. User 150 may interact with the voice assistant through the interactive interface 300A via text, voice, video, or by triggering interactive controls. Furthermore, the voice assistant may respond to the user's interaction. For example, the voice assistant may generate conversational messages or perform corresponding actions (e.g., calling other applications, such as playing music) in response to the user's interaction. Furthermore, the terminal device 120 may present the conversational messages generated by the voice assistant in the interactive interface 300A. For example, the conversational messages may be in the form of text, audio, or images (including image content and video content). Alternatively, the terminal device 120 may trigger the electronic device 110 to enable the first mode in response to receiving an activation operation from user 150 in the interactive interface 300A. For example, such an activation operation may include a conversational message containing preset keywords (e.g., "Enable First Mode") sent by user 150 to the voice assistant.

[0071] In some embodiments, the request to activate the first mode may further include physical operation instructions received via electronic device 110 and / or terminal device 120. For example, such physical operation instructions may be received based on sensor units (e.g., pressure sensors, capacitive touch buttons, accelerometers, etc.) configured on electronic device 110 or terminal device 120. For example, such physical operation instructions may include click operations (e.g., single-click or double-click operations, etc.) and long-press operations on electronic device 110 and / or terminal device 120.

[0072] In some embodiments, continue to refer to Figure 2 When in a first mode, the electronic device 110 can continuously capture (204) multiple images using the image acquisition unit 115. For example, the electronic device 110 can determine a shooting interval associated with the first mode and periodically capture multiple images based on the shooting interval. As an example, the electronic device 110 can determine the shooting interval for capturing multiple images based on configuration information (also referred to as second configuration information) associated with the first mode. As an example, such second configuration information can be determined based on user operations (e.g., voice commands, selection of relevant controls, etc.). For example, the shooting interval for the electronic device 110 to capture multiple images can be, for example, every 1 second, every 2 seconds, etc. This disclosure is not intended to limit the specific value of the shooting interval.

[0073] Alternatively, the electronic device 110 can determine a resolution associated with the first mode. Further, the electronic device 110 can capture at least a portion of a plurality of images based on this resolution. For example, the electronic device 110 can determine the resolution based on configuration information associated with the first mode (also referred to as first configuration information). For example, the configuration information associated with the first mode includes preset configuration information or configuration information determined by a configuration operation of the user 150. For example, the images may include picture content and / or video content. For example, for capturing video content, the resolution associated with the first mode may be, for example, a first resolution (e.g., 720P). For capturing picture content, the resolution associated with the first mode may be, for example, a second resolution (e.g., 1280×720 pixels).

[0074] Alternatively, the electronic device 110 can also determine the resolution based on a voice command from the user 150 (e.g., also known as a third voice command). For example, the electronic device 110 can update the resolution (e.g., adjust the resolution of the captured image from 1280×720 pixels to 1920×1080 pixels) in response to receiving a third voice command from the user (e.g., “Take a clearer picture”).

[0075] In some embodiments, the electronic device 110 is a headphone device or glasses device equipped with at least one camera. Thus, embodiments of this disclosure can meet the user's visual interaction needs by incorporating a camera into the headphone device or glasses device.

[0076] In some embodiments, electronic device 110 may include multiple image acquisition units (e.g., cameras). As an example, such multiple image acquisition units may include two or more image acquisition units. The multiple image acquisition units may include, for example, image acquisition unit 115. Taking electronic device 110 as an example of a headphone device, the headphone device may include a left earphone (or a first device) or a right earphone (or a second device). The left earphone and the right earphone may each be equipped with at least one image acquisition unit (e.g., a camera). Electronic device 110 may utilize a target image acquisition unit among the multiple image acquisition units to capture all the multiple images. For example, the target image acquisition unit is determined based on configuration information (also referred to as third configuration information) associated with a first mode. For example, such configuration information may instruct that when electronic device 110 is in the first mode, the image acquisition unit associated with the left earphone is used as the target image acquisition unit during a first time period (e.g., the first half hour of electronic device 110 being in the first mode), and the image acquisition unit associated with the right earphone is used as the target image acquisition unit during a second time period (e.g., after electronic device 110 has been in the first mode for half an hour).

[0077] Alternatively, the target image acquisition unit can also be determined based on the wearing status of the electronic device 110. For example, the electronic device 110 may, in response to a first device (e.g., a headphone device) where the user 150 is only wearing the electronic device 110 (e.g., a headphone device), use the image acquisition unit mounted on the left earphone as the target image acquisition unit.

[0078] In some embodiments, the electronic device 110 can take continuous pictures by alternately using a first image acquisition unit and a second image acquisition unit in the image acquisition unit. For example, during a first round of dialogue, the electronic device 110 can continuously take a first set of pictures using the first image acquisition unit among multiple image acquisition units. Furthermore, during a second round of dialogue after the first round ends, the electronic device 110 can continuously take a second set of pictures using the second image acquisition unit among multiple image acquisition units. The second image acquisition unit is different from the first image acquisition unit. For example, taking the electronic device 110 as an earphone device, the electronic device 110 can periodically take a first set of pictures during the first round of dialogue using the image acquisition unit (e.g., the first image acquisition unit) on the left earphone, and periodically take a second set of pictures during the second round of dialogue after the first round ends using the image acquisition unit (e.g., the second image acquisition unit) on the right earphone. Thus, the embodiments of this disclosure can take pictures using the same image acquisition unit in the same round of dialogue, ensuring the stability of the pictures taken during the same round of dialogue and guaranteeing the quality and relevance of the captured pictures. Furthermore, based on the embodiments of this disclosure, images are captured using different image acquisition units in different dialogue rounds. By rotating the image acquisition units, problems such as performance degradation or insufficient power due to prolonged operation of the image acquisition units can be avoided, thereby ensuring the stability and reliability of the captured images. Alternatively, the electronic device 110 can continuously capture images for a first duration using the first image acquisition unit and continuously capture images for a second duration using the second image acquisition unit. Alternatively, the first duration and the second duration can be the same or different. As an example, the first duration and the second duration can both be 30 seconds.

[0079] In some embodiments, electronic device 110 can send (206) (or transmit) multiple images captured by the image acquisition unit to terminal device 120. For example, electronic device 110 can transmit the images to terminal device 120 in real time in response to receiving images captured by the image acquisition unit. Alternatively, when the transmission speed is slower than the capture speed, electronic device 110 can store the images in a buffer unit so that the images in the buffer unit can be sent to terminal device 120 sequentially. As an example, terminal device 120 can acquire multiple images from electronic device 110, which is wirelessly connected to terminal device 120. In this way, embodiments of the present disclosure can transmit multiple images captured by electronic device to terminal device in real time in advance. Thus, before receiving a user's visual interaction request, the terminal device has already acquired the images transmitted by electronic device, thereby shortening the user's waiting time during the processing of visual interaction requests, improving the efficiency of processing visual interaction requests, and meeting the user's interaction needs.

[0080] In some embodiments, electronic device 110 may output response content to a user in response to a user's visual interaction request. Electronic device 110 may receive (210) a first voice content from user 150 using an audio acquisition unit on its own. Further, electronic device 110 may determine whether the first voice content is associated with a visual interaction request from user 150. As an example, electronic device 110 may determine that the first voice content (e.g., "Please help me see what this is") includes preset keywords (e.g., "see," "what is this," etc.) in response to determining that the first voice content is associated with a visual interaction request from user 150 (e.g., "identify the object"). Alternatively, electronic device 110 may determine whether the first voice content includes preset keywords based on a wireless communication terminal device 120.

[0081] Alternatively, electronic device 110 can transmit the first voice content to a pre-trained model, which can then determine whether the first voice content is associated with a visual interaction request from user 150, and determine the user 150's visual interaction request based on the first voice content. The deployment of the pre-trained model can be categorized into three types: the pre-trained model is deployed at electronic device 110, at terminal device 120, or at server 130. The following will provide illustrative examples of each different model deployment location.

[0082] In the first scenario, when electronic device 110 is equipped with a pre-trained model (e.g., model 140), electronic device 110 can determine (220) whether the first voice content is associated with a user's visual interaction request. For example, electronic device 110 can determine whether the first voice content is associated with a user's visual interaction request based on the pre-trained model deployed on electronic device 110. Furthermore, in response to the first voice content being determined to be associated with a user's visual interaction request, electronic device 110 can provide at least one image from a plurality of images to the pre-trained model (e.g., model 140) so that the pre-trained model can generate response content for the visual interaction request based on the first voice content and / or at least one image from the plurality of images. The response content is a second voice content or text content corresponding to the second voice content. Alternatively, electronic device 110 can send (222) a visual interaction request to terminal device 120 in response to the first voice content being determined to be associated with a user 150's visual interaction request.

[0083] Alternatively, when electronic device 110 is not equipped with a pre-trained model, electronic device 110 may directly send (212) the first voice content to terminal device 120 to determine whether the first voice content is associated with the visual interaction request of user 150 based on terminal device 120.

[0084] In the second scenario, terminal device 120 is equipped with a pre-trained model (e.g., model 140). In response to receiving first voice content, terminal device 120 can use the pre-trained model deployed on terminal device 120 to determine (218) whether the first voice content is associated with a visual interaction request from user 150. Further, after the first voice content is determined to be associated with a visual interaction request from user 150, terminal device 120 can transmit at least one image from a plurality of images to the pre-trained model. Terminal device 120 can use (224) the pre-trained model to generate response content for the visual interaction request based on the visual interaction request and at least one image from the plurality of images. The response content is second voice content or text content corresponding to the second voice content. Furthermore, terminal device 120 can send (232) the second voice content (or the text content corresponding to the second voice content) to electronic device 110.

[0085] Alternatively, terminal device 120 can determine whether the first voice content is associated with user 150's visual interaction request by utilizing other models. Such other models can be implemented, for example, as classification models (e.g., outputting classification models that include those associated with user 150 and those not associated with user 150's visual interaction request). Such classification models can be deployed, for example, in terminal device 120 or in server 130 communicating with terminal device 120. This disclosure is not intended to limit the specific implementation or training process of such classification models.

[0086] Alternatively, when terminal device 120 does not have a pre-trained model deployed, terminal device 120 can directly send (214) the first voice content to server 130 to determine whether the first voice content is associated with the visual interaction request of user 150 based on server 130. Server 130 can use a pre-trained model (e.g., model 140) deployed on server 130 to determine (216) whether the first voice content is associated with the visual interaction request of user 150, and transmit the determination result to terminal device 120.

[0087] Additionally, terminal device 120 may send (226) at least one image from a plurality of images to server 130 in response to a visual interaction request from a user 150 whose first voice content is identified as associated with the user 150. Server 130 may generate response content for the visual interaction request based on the visual interaction request and at least one image from the plurality of images using (228) a pre-trained model (e.g., model 140) deployed at server 130. The response content may be a second voice content or text content corresponding to the second voice content. Furthermore, server 130 may send (230) the response content to terminal device 120.

[0088] Additionally or alternatively, at least one of the images transmitted to the pre-trained model is determined from the multiple images based on a visual interaction request and the relevance of the multiple images. For example, if the visual interaction request associated with the first speech content (e.g., “Help me see what’s on the table”) is “Identify what’s on the table”, then at least one image could be an image containing the table among the multiple images.

[0089] Alternatively, during the generation of the second speech content using a pre-trained model, electronic device 110, terminal device 120, or server 130 may also transmit multiple images from electronic device 110 to the pre-trained model (e.g., model 140) to construct contextual information corresponding to the multiple images. In this way, the pre-trained model can obtain rich contextual information, thereby enabling it to more accurately understand the intent of user 150. As an example, the pre-trained model may be generated based on a portion of the contextual information corresponding to at least one image. For example, the multiple images may include multiple images corresponding to multiple items (e.g., "stool," "apple," "sofa," etc.). The visual interaction request corresponding to the first speech content (e.g., "What fruit is here?") is "identify the type of fruit." The second speech content generated by the pre-trained model may be generated based on partial contextual information (e.g., "an apple") corresponding to at least one image (e.g., "one apple") or text content corresponding to the second speech content.

[0090] In some embodiments, the electronic device 110 may use the audio playback unit on the electronic device 110 to play (234) second voice content.

[0091] In some embodiments, continue to refer to Figure 3A The electronic device 110 can trigger the display of a first message corresponding to the first voice content in an interactive interface associated with the electronic device 110. For example, the terminal device 120 can present an interactive interface 300A associated with the electronic device 110. The terminal device 120 can present a first message 305 corresponding to the first voice content (e.g., "Help me see what this is") in the interactive interface 300A.

[0092] In some embodiments, continue to refer to Figure 2 Terminal device 120 can present (208) one or more images from a plurality of images in an interactive application associated with electronic device 110. Alternatively, electronic device 110 can trigger the display of one or more images from a plurality of images in an interactive interface associated with electronic device 110. As an example, the one or more images may include at least one image used to generate second voice content. The one or more images may, for example, include the first image captured from a plurality of images. For example, such as Figure 3A As shown, the terminal device 120 can display image 310-1, image 310-2 and / or image 310-3 among multiple images in the interactive interface 300A.

[0093] Alternative locations, continue to consider. Figure 3A The electronic device 110 can trigger the display of a second message corresponding to the second voice content in an interactive interface associated with the electronic device 110. For example, the terminal device 120 can present a second message 315 (e.g., a description of the recognized glasses) corresponding to the second voice content in an interactive interface 300A.

[0094] Additionally or alternatively, terminal device 120 may display a reminder message 320 (e.g., "The photos will be automatically destroyed X hours after the conversation ends") in interactive interface 300A to remind user 150 that images 310-1, 310-2, and 310-3, etc., presented in interactive interface 300A will be destroyed after X hours.

[0095] In some embodiments, continue to refer to Figure 2 Terminal device 120 may, in response to reaching a preset duration (e.g., 2 hours), stop (236) displaying one or more images among a plurality of images in an interactive application associated with electronic device 110. As an example, such as Figure 3BAs shown, terminal device 120 can stop displaying one or more images in interactive interface 300B in response to displaying one or more images out of a set duration (e.g., 2 hours). Alternatively, terminal device 120 can display an indicator element 325 (e.g., "Photo has been destroyed") in the interactive interface to indicate that one or more images have stopped being displayed and have been deleted.

[0096] In some embodiments, continue to refer to Figure 2 Electronic device 110 can exit (238) the first mode. For example, electronic device 110 can exit the first mode in response to its battery level falling below a threshold (e.g., 5%). Alternatively, electronic device 110 can exit the first mode in response to a predefined condition that the wearing status of electronic device 110 does not meet (e.g., user 150 stops wearing electronic device 110 (e.g., removes headphones)). Alternatively, electronic device 110 can exit the first mode in response to a voice command (also known as a second voice command) received via the audio acquisition unit. For example, such a second voice command can be associated with predefined keywords (e.g., "exit first mode", "close dialogue mode", etc.).

[0097] Based on the process described above, embodiments of this disclosure can periodically capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode. Furthermore, embodiments of this disclosure can, after receiving first voice content associated with a user's visual interaction request, acquire and play second voice content generated by a model based on the first voice content and multiple images. Thus, the electronic device provided by embodiments of this disclosure can generate and play corresponding voice content based on the acquired image content and the user's voice content, thereby improving the efficiency of information acquisition for the user, expanding the interaction capabilities between the electronic device and the user, and meeting the user's interaction needs. In addition, since the electronic device provided in embodiments of this disclosure captures images periodically, rather than only after acquiring the user's voice content, this improves the efficiency of voice content processing and reduces the user's waiting time.

[0098] Example process

[0099] Figure 4 A flowchart illustrating an example process 400 of interaction according to some embodiments of the present disclosure is shown. Process 400 may be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 400.

[0100] like Figure 4As shown in box 410, when the electronic device 110 is in the first mode, it continuously captures multiple images using the image acquisition unit mounted on the electronic device 110.

[0101] In box 420, electronic device 110 responds to a user's visual interaction request by outputting response content to the user, wherein the response content is generated by a pre-trained model based on at least one of a plurality of images.

[0102] In some embodiments, the electronic device is equipped with an audio acquisition unit, and responding to a user's visual interaction request includes: responding to the audio acquisition unit receiving a first voice content from the user that is determined to be associated with the user's visual interaction request; and / or, the electronic device is equipped with an audio playback unit, and outputting response content to the user includes: playing a second voice content to the user through the audio playback unit, wherein the second voice content is generated by a pre-trained model based on at least one of a plurality of images.

[0103] In this way, embodiments of the present disclosure can support the use of the audio acquisition unit of an electronic device to acquire voice content and determine whether the voice content is associated with a user's visual interaction request. Furthermore, embodiments of the present disclosure can support the use of the audio playback unit of an electronic device to play the generated voice content to the user. Thus, embodiments of the present disclosure can improve the interaction efficiency between the user and the electronic device.

[0104] In some embodiments, the second speech content is generated based on the following process: in response to the first speech content being associated with a visual interaction request, a response content for the visual interaction request is generated using a pre-trained model and based on the visual interaction request and at least one of a plurality of images, the response content being the second speech content or text content corresponding to the second speech content.

[0105] In this way, embodiments of the present disclosure generate response content for visual interaction requests based on pre-trained models, which can improve the efficiency of users' visual interaction.

[0106] In some embodiments, the electronic device 110 communicates wirelessly with the terminal device 120, and the process 400 further includes: transmitting multiple images from the electronic device 110 to the terminal device 120 before the first voice content is determined to be associated with the user's visual interaction request.

[0107] In this way, embodiments of the present disclosure can transmit multiple images to the terminal device in advance, thereby shortening the user's waiting time during the processing of visual interaction requests and improving interaction efficiency.

[0108] In some embodiments, the terminal device 120 is further configured to transmit at least one of a plurality of images to a pre-trained model after the first voice content is determined to be a visual interaction request associated with the user 150.

[0109] In this way, embodiments of the present disclosure can meet the user's visual interaction needs based on at least one image processing visual interaction request using a pre-trained model.

[0110] In some embodiments, the terminal device 120 is further configured to transmit multiple images from the electronic device 110 to a pre-trained model to construct contextual information corresponding to the multiple images.

[0111] In this way, embodiments of the present disclosure can provide contextual information of multiple images to a pre-trained model, so that the pre-trained model can process the user's visual interaction requests based on the contextual information, thereby improving the accuracy of processing visual interaction requests.

[0112] In some embodiments, the second speech content is also generated based on the portion of the context information corresponding to at least one image.

[0113] In this way, embodiments of the present disclosure can generate second speech content that meets user needs based on partial context information corresponding to at least one image, thereby improving the quality of the generated second speech content.

[0114] In some embodiments, the pre-trained model is deployed on terminal device 120 or server 130 communicating with terminal device 120.

[0115] In this way, the embodiments of this disclosure can support the deployment of the model on terminal devices or servers as needed, and can meet the needs of different scenarios.

[0116] In some embodiments, process 400 further includes: in response to determining that the first voice content includes preset keywords, determining that the first voice content is associated with a user's visual interaction request; and / or, transmitting the first voice content to a pre-trained model so that the pre-trained model can determine whether the first voice content is associated with a user's visual interaction request.

[0117] In this way, embodiments of the present disclosure can determine whether the first speech content is associated with the user's visual interaction request based on preset keywords or a pre-trained model, thereby improving the recognition accuracy of visual interaction requests and meeting the user's needs.

[0118] In some embodiments, at least one image is determined from multiple images based on a visual interaction request and the relevance of the multiple images.

[0119] In this way, embodiments of the present disclosure can determine at least one image to be sent to a pre-trained model based on the relevance between a visual interaction request and multiple images, thereby reducing the task complexity of the pre-trained model in processing images and thus improving processing efficiency.

[0120] In some embodiments, periodically capturing multiple images using the image acquisition unit mounted on the electronic device 110 includes: determining a resolution associated with a first mode, and capturing at least a portion of the multiple images based on the resolution; and / or determining a shooting interval associated with the first mode, and periodically capturing multiple images based on the shooting interval.

[0121] In this way, images can be captured based on the configured resolution, ensuring the stability of image quality. Furthermore, embodiments of this disclosure can determine the shooting interval for capturing multiple images, thereby meeting the user's need to capture images at different time intervals.

[0122] In some embodiments, the first mode is enabled in response to a user's activation request, and the activation request includes: an activation operation received via an interactive interface with the electronic device 110; or a voice command; or a physical operation command.

[0123] In this way, the embodiments of this disclosure can provide users with multiple ways to enable the first mode, improving the efficiency of enabling the first mode of the electronic device.

[0124] In some embodiments, the electronic device 110 is associated with a voice assistant application, and the process 400 further includes: receiving a first voice command from a user during a conversation with the voice assistant application; and in response to receiving a request to end the conversation or no voice message being received during a preset time period, activating a first mode of the electronic device 110.

[0125] In this way, embodiments of the present disclosure can support users activating the first mode of an electronic device via voice commands from a voice assistant application, thereby improving user efficiency. Furthermore, by activating the first mode of the electronic device after a conversation ends, the present disclosure avoids conflicts between the process of the electronic device receiving voice commands in the first mode and the conversation between the user and the voice assistant application, thus ensuring the stability and reliability of the voice interaction functions provided by the electronic device.

[0126] In some embodiments, the electronic device includes a plurality of image acquisition units, and continuously capturing multiple images using the image acquisition units on the electronic device includes taking turns continuously capturing images using a first image acquisition unit and a second image acquisition unit in the image acquisition units.

[0127] In this way, embodiments of the present disclosure support shooting activities in turn through different image acquisition units on an electronic device, thereby improving shooting efficiency and shooting quality.

[0128] In some embodiments, the electronic device 110 includes a plurality of image acquisition units, and continuously capturing a plurality of images using the image acquisition units mounted on the electronic device 110 includes: capturing all the plurality of images using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with a first mode and / or the wearing state of the electronic device.

[0129] In this way, embodiments of this disclosure can determine the target acquisition unit for capturing multiple images based on configuration information, thereby meeting diverse user needs and improving efficiency. Furthermore, embodiments of this disclosure can determine the target acquisition unit for capturing multiple images based on the wearing state, adapting to the user's wearing operation and allowing the user to freely choose the wearing method, bringing convenience to the user.

[0130] In some embodiments, continuously capturing images using a first image acquisition unit and a second image acquisition unit among a plurality of image acquisition units includes: continuously capturing a first set of images using the first image acquisition unit among a plurality of image acquisition units during a first round of dialogue; and continuously capturing a second set of images using the second image acquisition unit among a plurality of image acquisition units during a second round of dialogue after the first round of dialogue has ended, wherein the second image acquisition unit is different from the first image acquisition unit; and / or continuously capturing images for a first duration using the first image acquisition unit among a plurality of image acquisition units, and continuously capturing images for a second duration using the second image acquisition unit among a plurality of image acquisition units, wherein the first duration and the second duration are the same or different.

[0131] In this way, the embodiments of this disclosure can support the use of different image acquisition units to capture images in different dialogue rounds, thereby avoiding performance degradation or insufficient power caused by continuous operation of a single image acquisition unit, and thus ensuring the stability of the image capture function. Furthermore, maintaining the use of the same image acquisition unit within the same dialogue round ensures that the capture level or effect of multiple images associated with the same dialogue round remains consistent.

[0132] In some embodiments, process 400 further includes: exiting the first mode in response to the battery level of electronic device 110 being lower than a threshold; or, if the electronic device includes a wearable device, exiting the first mode in response to the wearing state of electronic device 110 not meeting a preset condition; or exiting the first mode in response to a second voice command received via electronic device.

[0133] In this way, embodiments of the present disclosure can exit the first mode when preset conditions are met, thereby meeting the user's needs and ensuring the stability of the functions provided by the electronic device when it is in the first mode.

[0134] In some embodiments, process 400 further includes triggering the display in an interactive interface associated with electronic device 110 of: a first message corresponding to first voice content; a second message corresponding to second voice content; and one or more images from a plurality of images.

[0135] In this way, embodiments of the present disclosure can present a variety of content associated with the first mode in an interactive interface, thereby satisfying the user's viewing and interaction needs.

[0136] In some embodiments, one or more images include: at least one image for generating second speech content; and / or the first image captured from a plurality of images.

[0137] In this way, the embodiments of this disclosure can selectively determine the image content presented in the interactive interface, thereby providing users with image content that better meets their needs, and reducing the memory usage of user devices caused by the presentation of a large number of irrelevant images.

[0138] In some embodiments, the electronic device includes a wearable device; and / or, the electronic device 110 is a headphone device or glasses device equipped with at least one camera.

[0139] Thus, embodiments of this disclosure can meet the user's visual interaction needs by mounting a camera on a headphone device or glasses device.

[0140] Figure 5 A flowchart illustrating an example process 500 of interaction according to some embodiments of the present disclosure is shown. Process 500 can be implemented at terminal device 120. Reference is made below. Figure 1 To describe process 500.

[0141] like Figure 5 As shown in block 510, terminal device 120 acquires multiple images from electronic device 110 which is wirelessly connected to terminal device 120. The multiple images are continuously captured by image acquisition unit mounted on electronic device 110 when electronic device 110 is in a first mode.

[0142] In box 520, terminal device 120 responds to a user's visual interaction request by acquiring response content, wherein the response content is generated by a pre-trained model based on at least one of a plurality of images.

[0143] In box 530, terminal device 120 sends response content to electronic device 110 to output response content to user using electronic device 110.

[0144] In some embodiments, the electronic device is equipped with an audio acquisition unit, wherein responding to a user's visual interaction request includes: responding to the audio acquisition unit receiving a first voice content from the user that is determined to be associated with the user's visual request; and / or the electronic device is equipped with an audio playback unit, wherein outputting response content to the user via the electronic device includes: playing second voice content to the user via the audio playback unit, wherein the second voice content is generated by a pre-trained model based on at least one of a plurality of images.

[0145] In this way, embodiments of the present disclosure can support the use of the audio acquisition unit of an electronic device to acquire voice content and determine whether the voice content is associated with a user's visual interaction request. Furthermore, embodiments of the present disclosure can support the use of the audio playback unit of an electronic device to play the generated voice content to the user. Thus, embodiments of the present disclosure can improve the interaction efficiency between the user and the electronic device.

[0146] In some embodiments, process 500 further includes: after the first voice content is determined to be a visual interaction request associated with the user, transmitting at least one of the plurality of images to a pre-trained model.

[0147] In this way, embodiments of the present disclosure can meet the user's visual interaction needs based on at least one image processing visual interaction request using a pre-trained model.

[0148] In some embodiments, the terminal device 120 is further configured to transmit multiple images from the electronic device 110 to a pre-trained model to construct contextual information corresponding to the multiple images.

[0149] In this way, embodiments of the present disclosure can provide contextual information of multiple images to a pre-trained model, so that the pre-trained model can process the user's visual interaction requests based on the contextual information, thereby improving the accuracy of processing visual interaction requests.

[0150] In some embodiments, the second speech content is also generated based on the portion of the context information corresponding to at least one image.

[0151] In this way, embodiments of the present disclosure can generate second speech content that meets user needs based on partial context information corresponding to at least one image, thereby improving the quality of the generated second speech content.

[0152] In some embodiments, the pre-trained model is deployed on terminal device 120 or server 130 communicating with terminal device 120.

[0153] In this way, the embodiments of this disclosure can support the deployment of the model on terminal devices or servers as needed, and can meet the needs of different scenarios.

[0154] Figure 6 A flowchart illustrating an example process 600 of an interaction according to some embodiments of the present disclosure is shown. Process 600 can be implemented in an interactive system 100. The interactive system 100 includes a wireless communication electronic device 110 and a terminal device 120. Reference is made below. Figure 1 To describe process 600.

[0155] like Figure 6 As shown in box 610, when the electronic device 110 is in the first mode, the interactive system uses the image acquisition unit on the electronic device 110 to continuously capture multiple images.

[0156] In frame 620, the interactive system 100 sends multiple images from electronic device 110 to terminal device 120.

[0157] In box 630, in response to a user's visual interaction request, the interaction system 100 obtains response content from the terminal device 120, wherein the response content is generated by a pre-trained model based on at least one of a plurality of images.

[0158] In box 640, the interactive system 100 sends response content from terminal device 120 to electronic device 110 so as to use electronic device 110 to output response content to the user.

[0159] Example devices and equipment

[0160] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes.

[0161] Figure 7 A schematic structural block diagram of an example device 700 for interaction according to certain embodiments of the present disclosure is shown. Device 700 may be implemented as or included in electronic device 110. Various modules / components in device 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0162] like Figure 7 As shown, the device 700 includes: an image acquisition module 710 configured to periodically capture multiple images using an image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; and a playback module 720 configured to output response content to the user in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the multiple images.

[0163] In some embodiments, the electronic device is equipped with an audio acquisition unit, and responding to a user's visual interaction request includes: responding to the audio acquisition unit receiving a first voice content from the user that is determined to be associated with the user's visual interaction request; and / or, the electronic device is equipped with an audio playback unit, and outputting response content to the user includes: playing a second voice content to the user through the audio playback unit, wherein the second voice content is generated by a pre-trained model based on at least one of a plurality of images.

[0164] In some embodiments, the second speech content is generated based on the following process: in response to the first speech content being associated with a visual interaction request, a response content for the visual interaction request is generated using a pre-trained model and based on the visual interaction request and at least one of a plurality of images, the response content being the second speech content or text content corresponding to the second speech content.

[0165] In some embodiments, the electronic device 110 communicates wirelessly with the terminal device 120. The apparatus 700 further includes a first transmission module, which is configured to transmit multiple images from the electronic device 110 to the terminal device 120 before the first voice content is determined to be associated with the user's visual interaction request.

[0166] In some embodiments, the terminal device 120 is further configured to transmit at least one of a plurality of images to a pre-trained model after the first voice content is determined to be a visual interaction request associated with the user 150.

[0167] In some embodiments, the terminal device 120 is further configured to transmit multiple images from the electronic device 110 to a pre-trained model to construct contextual information corresponding to the multiple images.

[0168] In some embodiments, the second speech content is also generated based on the portion of the context information corresponding to at least one image.

[0169] In some embodiments, the pre-trained model is deployed on terminal device 120 or server 130 communicating with terminal device 120.

[0170] In some embodiments, the apparatus 700 further includes a determining module configured to: determine that the first voice content is associated with a user's visual interaction request in response to determining that the first voice content includes preset keywords; and / or transmit the first voice content to a pre-trained model so that the pre-trained model can determine whether the first voice content is associated with a user's visual interaction request.

[0171] In some embodiments, at least one image is determined from multiple images based on a visual interaction request and the relevance of the multiple images.

[0172] In some embodiments, the image acquisition module 710 is further configured to: determine a resolution associated with a first mode, and capture at least a portion of a plurality of images based on the resolution; and / or determine a capture interval associated with the first mode, and periodically capture a plurality of images based on the capture interval.

[0173] In some embodiments, the first mode is enabled in response to a user's activation request, and the activation request includes: an activation operation received via an interactive interface with the electronic device 110; or a voice command; or a physical operation command.

[0174] In some embodiments, the electronic device 110 is associated with a voice assistant application, and the device 700 further includes an activation module configured to: receive a first voice command from a user during a conversation with the voice assistant application; and activate a first mode of the electronic device 110 in response to receiving a request to end the conversation or if no voice message is received during a preset time period.

[0175] In some embodiments, the electronic device includes a plurality of image acquisition units, and the image acquisition module 710 is further configured to take continuous pictures in turn using a first image acquisition unit and a second image acquisition unit among the image acquisition units.

[0176] In some embodiments, the electronic device 110 includes a plurality of image acquisition units, and the image acquisition module 710 is further configured to capture all of the plurality of images using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with a first mode and / or the wearing state of the electronic device.

[0177] In some embodiments, the image acquisition module 710 is further configured to: during a first round of dialogue, continuously capture a first set of images using a first image acquisition unit among a plurality of image acquisition units; and during a second round of dialogue after the first round of dialogue ends, continuously capture a second set of images using a second image acquisition unit among a plurality of image acquisition units, the second image acquisition unit being different from the first image acquisition unit; and / or, continuously capture images for a first duration using the first image acquisition unit among a plurality of image acquisition units, and continuously capture images for a second duration using the second image acquisition unit among a plurality of image acquisition units, the first duration being the same as or different from the second duration.

[0178] In some embodiments, the device 700 further includes an exit module configured to: exit the first mode in response to the battery level of the electronic device 110 being lower than a threshold; or the electronic device includes a wearable module, and exit the first mode in response to the wearing state of the electronic device 110 not meeting a preset condition; or exit the first mode in response to a second voice command received via the electronic device.

[0179] In some embodiments, the device 700 further includes a display module configured to trigger the display of: a first message corresponding to a first voice content; a second message corresponding to a second voice content; and one or more images from a plurality of images in an interactive interface associated with the electronic device 110.

[0180] In some embodiments, one or more images include: at least one image for generating second speech content; and / or the first image captured from a plurality of images.

[0181] In some embodiments, the electronic device 110 includes a wearable device; and / or, the electronic device 110 is a headphone device or glasses device equipped with at least one camera.

[0182] Figure 8 A schematic structural block diagram of an example device 800 for interaction according to certain embodiments of the present disclosure is shown. Device 800 may be implemented as or included in terminal device 120. Various modules / components in device 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0183] like Figure 8 As shown, the device 800 includes: an image acquisition module 810 configured to acquire multiple images from an electronic device 110 that wirelessly communicates with the terminal device 120, the multiple images being continuously captured by an image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; a second receiving module 820 configured to acquire response content in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the multiple images; and a sending module 830 configured to send the response content from the terminal device 120 to the electronic device 110, so as to output the response content to the user using the electronic device 110.

[0184] In some embodiments, the electronic device is equipped with an audio acquisition unit, wherein the receiving module 820 is further configured to: respond to the audio acquisition unit receiving a first voice content of a user that is determined to be associated with a visual request of the user; and / or the electronic device is equipped with an audio playback unit, wherein the sending module 830 is further configured to: play a second voice content to the user through the audio playback unit, wherein the second voice content is generated by a pre-trained model based on at least one of a plurality of images.

[0185] In some embodiments, the apparatus 800 further includes a second transmission module configured to transmit at least one of a plurality of images to a pre-trained model after the first voice content is determined to be a visual interaction request associated with the user.

[0186] In some embodiments, the terminal device 120 is further configured to transmit multiple images from the electronic device 110 to a pre-trained model to construct contextual information corresponding to the multiple images.

[0187] In some embodiments, the second speech content is also generated based on the portion of the context information corresponding to at least one image.

[0188] In some embodiments, the pre-trained model is deployed on terminal device 120 or server 130 communicating with terminal device 120.

[0189] Figure 9 A schematic structural block diagram of an example device 900 for interaction according to certain embodiments of the present disclosure is shown. Device 900 may be implemented as or included in an interaction system 100. Interaction system 100 includes a wireless communication electronic device 110 and a terminal device 120. Various modules / components in device 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0190] like Figure 9 As shown, the device 900 includes: an image capturing module 910 configured to continuously capture multiple images using an image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; an image sending module 920 configured to send multiple images from the electronic device 110 to a terminal device 120; a voice acquisition module 930 configured to acquire response content from the terminal device 120 in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the multiple images; and a voice playback module 940 configured to send the response content from the terminal device 120 to the electronic device 110 to output the response content to the user using the wearable device 110.

[0191] The units included in devices 700, 800, or 900 can be implemented using various methods, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in devices 700, 800, or 900 may be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0192] Figure 10A block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 10 The electronic device 1000 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 10 The electronic device 1000 shown can be used to achieve Figure 1 The interactive system 100.

[0193] like Figure 10 As shown, electronic device 1000 is in the form of a general-purpose electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processing unit 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 1000.

[0194] Electronic device 1000 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 1000.

[0195] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 10 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0196] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0197] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 1000, or with any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0198] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0199] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0200] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0201] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0203] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An interaction method applied to an electronic device, the method comprising: When the electronic device is in the first mode, multiple images are continuously captured using the image acquisition unit mounted on the electronic device; as well as In response to a user's visual interaction request, response content is output to the user, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images.

2. The method according to claim 1, wherein, The electronic device is equipped with an audio acquisition unit, and the response to the user's visual interaction request includes: In response to the audio acquisition unit receiving the user's first voice content being determined to be associated with the user's visual interaction request; and / or, The electronic device is equipped with an audio playback unit, and the output of response content to the user includes: The second voice content is played to the user through the audio playback unit, wherein the second voice content is generated by the pre-trained model based on at least one of the plurality of images.

3. The method according to claim 2, wherein the second speech content is generated based on the following process: In response to the first speech content being associated with the visual interaction request, a response content for the visual interaction request is generated using the pre-trained model and based on the visual interaction request and at least one of the plurality of images, the response content being the second speech content or text content corresponding to the second speech content.

4. The method according to claim 2, wherein the electronic device communicates wirelessly with the terminal device, the method further comprising: Before the first voice content is determined to be associated with the user's visual interaction request, the plurality of images are transmitted from the electronic device to the terminal device.

5. The method according to claim 4, wherein the terminal device is further configured to: After the first voice content is determined to be associated with the user's visual interaction request, at least one of the plurality of images is transmitted to the pre-trained model.

6. The method according to claim 4, wherein the terminal device is further configured to: The multiple images are transmitted from the electronic device to the pre-trained model to construct contextual information corresponding to the multiple images.

7. The method of claim 6, wherein the second speech content is further generated based on a portion of the context information corresponding to the at least one image.

8. The method according to any one of claims 4-7, wherein the pre-trained model is deployed on the terminal device or a server communicating with the terminal device.

9. The method according to claim 2, further comprising: In response to determining that the first voice content includes preset keywords, it is determined that the first voice content is associated with the user's visual interaction request; And / or, The first voice content is transmitted to the pre-trained model so that the pre-trained model can determine whether the first voice content is associated with the user's visual interaction request.

10. The method of claim 1, wherein the at least one image is determined from the plurality of images based on the visual interaction request and the relevance of the plurality of images.

11. The method of claim 1, wherein continuously capturing multiple images using the image acquisition unit mounted on the electronic device comprises: Determine the resolution associated with the first mode, and based on the resolution, capture at least a portion of the plurality of images; And / or, Determine the shooting interval associated with the first mode, and periodically capture multiple images based on the shooting interval.

12. The method of claim 1, wherein the first mode is enabled in response to the user's activation request, and the activation request includes: An activation operation received via an interactive interface associated with the electronic device; or, Voice commands; or, Physical operation instructions.

13. The method of claim 12, wherein the electronic device is associated with a voice assistant application, the method further comprising: During a conversation with the voice assistant application, the user's first voice command is received; as well as In response to receiving a request to end the conversation or if no voice message is received in the conversation within a preset time period, the first mode of the electronic device is activated.

14. The method of claim 1, wherein the electronic device comprises a plurality of image acquisition units, and continuously capturing a plurality of images using the image acquisition units mounted on the electronic device comprises: The first image acquisition unit and the second image acquisition unit among the plurality of image acquisition units are used to take pictures continuously in turn.

15. The method of claim 1, wherein the electronic device comprises a plurality of image acquisition units, and continuously capturing a plurality of images using the image acquisition units mounted on the electronic device comprises: All of the plurality of images are captured using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with the first mode and / or the wearing status of the electronic device.

16. The method according to claim 14, wherein the continuous image capture using the first image acquisition unit and the second image acquisition unit among the plurality of image acquisition units comprises: During the first round of dialogue, the first image acquisition unit among the plurality of image acquisition units continuously captures a first set of images; as well as During the second round of dialogue after the first round of dialogue, a second set of images is continuously captured using the second image acquisition unit among the plurality of image acquisition units. The second image acquisition unit is different from the first image acquisition unit. And / or, The first image acquisition unit among the plurality of image acquisition units continuously captures images for a first duration, and the second image acquisition unit among the plurality of image acquisition units continuously captures images for a second duration, wherein the first duration and the second duration are the same or different.

17. The method according to claim 1, further comprising: In response to the electronic device's battery level falling below a threshold, the first mode is exited; or The electronic device includes a wearable device, which exits the first mode in response to the fact that the wearing state of the electronic device does not meet the preset conditions; or In response to a second voice command received via the electronic device, exit the first mode.

18. The method according to claim 2, further comprising: The trigger is displayed in the interactive interface associated with the electronic device: The first message corresponding to the first voice content; The second message corresponding to the second voice content; and One or more of the plurality of images.

19. The method of claim 18, wherein the one or more images comprise: The at least one image used to generate the second speech content; and / or The first image captured among the plurality of images.

20. The method of claim 1, wherein the electronic device comprises a wearable device; and / or, the electronic device is an earphone device or glasses device equipped with at least one camera.

21. An interaction method applied to a terminal device, the method comprising: Multiple images are acquired from an electronic device that communicates wirelessly with the terminal device, the multiple images being captured continuously using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; In response to a user's visual interaction request, obtain response content, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; as well as The response content is sent to the electronic device to output the response content to the user using the electronic device.

22. The method according to claim 21, wherein: The electronic device is equipped with an audio acquisition unit, wherein responding to a user's visual interaction request includes: In response to the audio acquisition unit receiving the user's first voice content and determining it to be associated with the user's visual request; and / or The electronic device is equipped with an audio playback unit, and the step of using the electronic device to output response content to the user includes: The audio playback unit plays second audio content to the user, wherein the second audio content is generated by the pre-trained model based on at least one of the plurality of images.

23. The method of claim 22, further comprising: After the first voice content is determined to be associated with the user's visual interaction request, at least one of the plurality of images is transmitted to the pre-trained model.

24. The method of claim 23, wherein the terminal device is further configured to: The multiple images are transmitted from the electronic device to the pre-trained model to construct contextual information corresponding to the multiple images.

25. The method of claim 24, wherein the second speech content is further generated based on a portion of the context information corresponding to the at least one image.

26. The method according to any one of claims 23-25, wherein the pre-trained model is deployed on the terminal device or a server communicating with the terminal device.

27. An interaction method applied to an interactive system, the interactive system comprising a wireless communication electronic device and a terminal device, the method comprising: When the electronic device is in the first mode, multiple images are continuously captured using the image acquisition unit mounted on the electronic device; The electronic device sends the multiple images to the terminal device; In response to a user's visual interaction request, the terminal device obtains response content, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; as well as The terminal device sends the response content to the electronic device, so that the electronic device can output the response content to the user.

28. An interaction device for use in an electronic device, the device comprising: The image acquisition module is configured to continuously capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode; as well as The playback module is configured to output response content to the user in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images.

29. An apparatus for interaction, applied to a terminal device, the apparatus comprising: The image acquisition module is configured to acquire multiple images from an electronic device that is wirelessly connected to the terminal device, the multiple images being continuously captured by an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; The second receiving module is configured to receive response content in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; as well as The sending module is configured to send the response content to the electronic device in order to output the response content to the user using the electronic device.

30. An apparatus for interaction, applied to an interactive system, the interactive system including wireless communication electronic devices and terminal devices, the apparatus comprising: The image capturing module is configured to continuously capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode; An image sending module is configured to send the plurality of images from the electronic device to the terminal device; The voice acquisition module is configured to acquire response content from the terminal device in response to a user's visual interaction request, wherein the response content is generated by a pre-trained model based on at least one of the plurality of images; as well as The voice playback module is configured to send the response content from the terminal device to the electronic device, so as to output the response content to the user using the electronic device.

31. An electronic device, comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 20, 21 to 26 or 27 when executed by the at least one processing unit.

32. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 20, 21 to 26 or 27.

Images