Method, device, equipment and storage medium for image interaction

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device and storage medium for image interaction. The method proposed herein is applied to an electronic device, and the method comprises: when the electronic device is in a first mode, capturing a plurality of images by using an image capturing unit carried by the electronic device; and actively outputting content by using a content output unit of the electronic device, wherein the content is generated by a pre-trained model based on at least one image of the plurality of images. In this way, embodiments of the present disclosure can enable the electronic device to actively provide content to the user based on the captured images, thereby meeting the user's interactive needs.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatuses, devices, and computer-readable storage media for image interaction. Background Technology

[0002] With the increasing maturity of science and technology, the use of electronic devices has gradually become a daily habit. For example, users can use electronic devices to play audio content, check the time, and measure physiological data. In addition, people can use electronic devices for interactive operations. However, the interactive operations supported by existing electronic devices are limited to simple operations such as playing and switching audio, which cannot meet users' deeper interactive needs. Summary of the Invention

[0003] In a first aspect of this disclosure, an image interaction method is provided, applied to an electronic device. The method includes: capturing multiple images using an image acquisition unit mounted on the electronic device while the electronic device is in a first mode; and actively outputting content using a content output unit of the electronic device, wherein the content is generated by a pre-trained model based on at least one of the multiple images.

[0004] In a second aspect of this disclosure, an image interaction method is provided, applied to a terminal device. The method includes: acquiring a plurality of images from an electronic device wirelessly connected to the terminal device, the plurality of images being captured using an image acquisition unit mounted on the electronic device while the electronic device is in a first mode; acquiring content, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; and sending the content to the electronic device to actively output the content using a content output unit mounted on the electronic device.

[0005] In a third aspect of this disclosure, a method for image interaction is provided, applied to an image interaction system. The image interaction system includes a wireless communication electronic device and a terminal device. The method includes: when the electronic device is in a first mode, capturing multiple images using an image acquisition unit on the electronic device; sending the multiple images from the electronic device to the terminal device; acquiring content from the terminal device, wherein the content is generated by a pre-trained model based on at least one of the multiple images; and sending the content from the terminal device to the electronic device to actively output the content using a content output unit on the electronic device.

[0006] In a fourth aspect of this disclosure, an apparatus for image interaction is provided, applied to an electronic device. The apparatus includes: an image acquisition module configured to periodically capture multiple images using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; and a first playback module configured to actively play speech content using an audio playback unit of the electronic device, wherein the speech content is generated by a pre-trained model based on at least one of the multiple images.

[0007] In a fifth aspect of this disclosure, an apparatus for image interaction is provided, applied to a terminal device. The apparatus includes: an image acquisition module configured to acquire a plurality of images from an electronic device wirelessly connected to the terminal device, the plurality of images being captured using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; a first acquisition module configured to acquire content, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; and a transmission module configured to transmit voice content to the electronic device to actively output the content using a content output unit mounted on the electronic device.

[0008] In a sixth aspect of this disclosure, an apparatus for image interaction is provided, applied to an image interaction system. The image interaction system includes a wireless communication electronic device and a terminal device. The apparatus includes: an image capturing module configured to periodically capture multiple images using an image acquisition unit on the electronic device when the electronic device is in a first mode; an image transmitting module configured to transmit the multiple images from the electronic device to the terminal device; a second acquisition module configured to acquire content from the terminal device, wherein the content is generated by a pre-trained model based on at least one of the multiple images; and a second playback module configured to transmit voice content from the terminal device to the electronic device to actively output content using a content output unit on the electronic device.

[0009] In a seventh aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the methods of the first, second, or third aspects.

[0010] In an eighth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the methods of the first, second, or third aspects.

[0011] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram is shown in which an example image interaction system according to embodiments of the present disclosure may be implemented;

[0014] Figure 2 A flowchart illustrating an example process for image interaction according to some embodiments of this disclosure is shown;

[0015] Figures 3A to 3C Example interfaces according to some embodiments of this disclosure are shown;

[0016] Figure 4 A flowchart illustrating an example process of image interaction according to some embodiments of the present disclosure is shown;

[0017] Figure 5 A flowchart illustrating an example process of image interaction according to yet another embodiment of the present disclosure is shown;

[0018] Figure 6 A flowchart illustrating an example process of image interaction according to yet another embodiment of the present disclosure is shown;

[0019] Figure 7 A schematic structural block diagram of an example device for image interaction according to some embodiments of the present disclosure is shown;

[0020] Figure 8 A schematic structural block diagram of an example device for image interaction according to further embodiments of the present disclosure is shown;

[0021] Figure 9 A schematic structural block diagram of an example device for image interaction according to further embodiments of the present disclosure is shown; and

[0022] Figure 10 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0025] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0027] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0028] As mentioned above, with the increasing maturity of science and technology, the use of electronic devices has gradually become a daily habit for people. For example, users can use electronic devices to play audio content, check the time, and measure physiological data. In addition, people can also use electronic devices for interactive operations. However, the interactive operations supported by existing electronic devices are limited to simple operations such as playing and switching audio, which cannot meet users' deeper interactive needs.

[0029] Embodiments of this disclosure propose a scheme for image interaction applied to electronic devices. According to this scheme, when the electronic device is in a first mode, multiple images can be captured using an image acquisition unit on the electronic device; and content can be actively output using a content output unit of the electronic device, wherein the content is generated by a pre-trained model based on at least one of the multiple images.

[0030] In this manner, embodiments of the present disclosure can capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode. Furthermore, embodiments of the present disclosure can utilize the electronic device to acquire and play content generated by a pre-trained model based on at least one of the multiple images. Thus, the electronic device provided by embodiments of the present disclosure can capture images, thereby enabling real-time acquisition of information about the user's surrounding environment. In addition, the electronic device provided by embodiments of the present disclosure can utilize a pre-trained model to generate content based on the captured images and actively output this content to the user, improving the efficiency of information acquisition for the user. Therefore, embodiments of the present disclosure expand the interaction capabilities between the electronic device and the user and meet the user's interaction needs.

[0031] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0032] Example Image Interaction System

[0033] Figure 1 A schematic diagram of an example image interaction system 100 according to an embodiment of the present disclosure is shown. (As follows) Figure 1 As shown, the image interaction system 100 may include an electronic device 110 and a terminal device 120. A communication connection can be established between the electronic device 110 and the terminal device 120. As an example, the electronic device 110 may connect to the terminal device 120 based on an appropriate wireless connection method; alternatively, the electronic device 110 may also establish a wired connection with the terminal device 120. For example, the communication connection may include, but is not limited to, Bluetooth connections, mobile network connections, Universal Serial Bus (USB) connections, Wireless Fidelity (WiFi) connections, etc., and the embodiments of this disclosure are not limited in this respect. Figure 1As shown, the electronic device 110 may be equipped with one or more image acquisition units 115. For example, the image acquisition unit 115 may be implemented as a camera assembly (e.g., a camera).

[0034] As an example, electronic device 110 may include wearable devices or other devices. Wearable devices may include, for example, headphone devices of suitable form, such as in-ear headphones, over-ear headphones, and headphones. This disclosure is not intended to limit the form of headphone devices.

[0035] Additionally, the headphone device may be equipped with an image acquisition unit 115 (e.g., a camera) located in an appropriate position to enable the use of the camera to capture environmental images of the user 150 when the user 150 is wearing the headphone device.

[0036] In some examples, the headphone device may include one or more cameras. For example, the left and right earbuds may each have one camera. As another example, the headphone device may also include only a single camera. For example, only the right earbud may have a camera, while the left earbud may not.

[0037] In other examples, the electronic device 110 may also be implemented in other suitable forms. For example, the electronic device 110 may include an eyeglass device equipped with one or more cameras. For example, such cameras may be located in the frame or rim of the eyeglass device.

[0038] In some embodiments, the electronic device 110 may be provided with an audio acquisition unit (e.g., a microphone) and a content output unit. The content output unit may include at least one of a screen display unit (e.g., a display screen, a projection device, etc.) or an audio playback unit 118 (e.g., a speaker). Specifically, the electronic device 110 may use the audio acquisition unit to acquire audio data and transmit the audio data to the terminal device 120. Alternatively, the electronic device 110 may also use the audio playback unit 118 to play audio data from the terminal device 120. Alternatively, the electronic device 110 may use the screen display unit to display image content and / or text content from the terminal device 120.

[0039] In some embodiments, electronic device 110 may also be associated with a voice assistant application installed on terminal device 120. For example, user 150 may use electronic device 110 to interact with the voice assistant application via voice. Additionally, as will be described below, terminal device 120 may also provide a graphical interface for the voice assistant application to provide visual interactive information associated with the voice assistant application.

[0040] In some embodiments, the terminal device 120 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 120 may also support any type of user-facing interface (such as "wearable" circuitry).

[0041] like Figure 1 As shown, the voice assistant application utilizes a pre-trained model 140 to interact with the user 150 in multiple appropriate modalities. As an example, the pre-trained model 140 can be a language model or a generative model that supports multimodal data processing.

[0042] In some embodiments, the pre-trained model 140 can be deployed on the terminal device 120. Alternatively, the pre-trained model 140 can also be deployed on the server 130. When the pre-trained model 140 is deployed on the server 130, the terminal device 120 can invoke the model 140 via a communication connection with the server 130 to achieve interaction within the voice assistant application. For example, the server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The server 130 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. The server 130 can provide background services for the voice assistant application in the terminal device 120.

[0043] In some embodiments, the electronic device 110 may have multiple operating modes. Different operating modes may be related to, for example, the operating methods of the image acquisition unit 115 and the audio playback unit 118.

[0044] Taking a headset device as an example, when the headset device is in normal conversation mode, the image acquisition unit 115 will not be activated by default. The image acquisition unit 115 will only be activated when an explicit instruction from the user is received (e.g., taking a photo). Furthermore, when the headset device is in normal conversation mode, the audio playback unit 118 will only play corresponding audio content (e.g., music content) based on the user's operation (e.g., playing music), or, after the voice assistant application receives a user's voice instruction, the audio playback unit 118 will play the voice assistant application's response to the user's voice instruction. The audio playback unit 118 will not be activated by default until a user operation or voice instruction is received.

[0045] As will be detailed below, the electronic device 110 can also support operation in a first mode. The first mode, for example, can support the image acquisition unit 115 being in a normally open state, and in this mode, the image acquisition unit 115 can capture images 125. For example, in the first mode, the image acquisition unit 115 can periodically capture images. Furthermore, the first mode can also support the content output unit actively outputting content to the user (e.g., the audio playback unit 118 actively playing voice content (or audio content) and / or the screen display unit actively displaying image content (e.g., pictures and / or videos) and / or text content). This content is generated by a pre-trained model based on at least one of the multiple images 125.

[0046] For example, when a user wears electronic device 110, the user can activate the first mode of electronic device 110. The specific process for activating the first mode will be described in detail below. For example, the user can trigger electronic device 110 to enter the first mode via voice command (e.g., “Activate first mode”).

[0047] Accordingly, during the first mode, the image acquisition unit 115 of the electronic device 110 can capture images, for example, periodically. For example, when the headphone device worn by the user is in the first mode, the camera of the headphone device can periodically capture images of the user's environment.

[0048] like Figure 1 As shown, the captured image 125 can be sent to the terminal device 120 to determine whether the image 125 meets the triggering conditions. The specific process for determining the triggering conditions will be described in detail below. For example, the triggering conditions here include the image 125 matching a preset scene. For example, the preset scene may include "supermarket", "snow scene", "rainbow", "historical building" and / or "sunset", etc. For example, if the image 125 contains "rainbow", it can be determined that the image 125 matches the preset scene (e.g., "rainbow"), thus meeting the triggering conditions.

[0049] Furthermore, when a target image among the multiple images 125 meets the triggering condition, model 140 can generate response content based on the target image. Specifically, when a target image among the multiple images 125 is determined to meet the triggering condition, the multiple images 125 or the target image among the multiple images 125 can be provided to model 140 to generate response content for that target image, such as speech content 135 or text content (e.g., text content corresponding to speech content 135). The generated text content can be converted into speech content (e.g., speech content 135) by model 140, server 130, and / or terminal device 120, for example.

[0050] As will be detailed below, the presence of a target image that meets the triggering conditions can be determined by electronic device 110, terminal device 120, server 130 and / or model 140 based on multiple captured images 125.

[0051] by Figure 1 As an example, the image interaction system 100 can use the audio playback unit 118 on the electronic device 110 to play voice content 135. The voice content 135 may include, for example, a reminder about the image 125, such as, “Look, there’s a rainbow.”

[0052] Therefore, in the first mode, the image interaction system 100 can periodically capture multiple images using the image acquisition unit 115 mounted on the electronic device 110, and generate response content using the captured images after the captured images meet the trigger conditions. Furthermore, the image interaction system 100 can actively play voice content associated with the response content to the user 150 using the audio playback unit 118 mounted on the electronic device 110. Alternatively, the image interaction system 110 can actively present image content and / or text content associated with the response content to the user 150 using the screen display unit mounted on the electronic device 110. Thus, the electronic device 110 provided in the embodiments of this disclosure can actively output content generated based on an image after capturing an image that meets the trigger conditions, thereby improving the efficiency of information acquisition for the user and meeting the user's interaction needs.

[0053] Furthermore, the image interaction system 100 can support using the historical interactions between the user 150 and the voice assistant application to determine whether the image 125 meets the trigger conditions. For example, if the historical interactions between the user and the voice assistant application are related to "pastry recommendations," then the image 125 can be determined to meet the trigger conditions if it contains "pastry shops." For example, the generated voice content could include, "This pastry shop really suits your taste!" Alternatively, the trigger conditions can also be set based on the user's voice commands. For example, a trigger condition set based on the user's voice commands could include, "Tell me when you see XX bookstore," then the image 125 can be determined to meet the trigger conditions if it contains "XX bookstore." For example, the generated voice content could include, "There's the XX bookstore you mentioned 10 meters ahead."

[0054] It should be understood that the structure and function of the various elements in the image interaction system 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0055] The following section will describe in detail the specific processing flow of the image interaction system 100 with reference to the accompanying drawings.

[0056] Example Image Interaction

[0057] Figure 2 A flowchart of an example process 200 for image interaction according to some embodiments of the present disclosure is shown. Process 200 may be implemented at electronic device 110, terminal device 120, and / or server 130. Alternatively, process 200 may also be implemented in an image interaction system 100 including electronic device 110 and terminal device 120 for wireless communication. Reference is made below. Figure 1 To describe process 200.

[0058] like Figure 2 As shown, electronic device 110 can enable (202) a first mode. As an example, the first mode can be a mode that sets the image acquisition unit 115 of electronic device 110 to always on and allows electronic device 110 to actively output content (e.g., audio playback unit 118 actively plays voice content and / or screen display unit actively presents content). Alternatively, electronic device 110 may also include a second mode (e.g., a normal dialogue mode). For example, the second mode can instruct that the image acquisition unit 115 of electronic device 110 is not activated by default. The image acquisition unit 115 will only be activated when an explicit instruction from the user is received (e.g., “take a photo”). Moreover, the second mode prohibits electronic device 110 from actively outputting content.

[0059] The mode of electronic device 110 can be determined based on the configuration operation of user 150. Alternatively, electronic device 110 can also switch the mode of electronic device (e.g., from the first mode to the second mode) in response to the user's preset operation (e.g., voice command (e.g., "switch to the second mode")).

[0060] As an example, the first mode of electronic device 110 can be activated (or woken up) in response to an activation request from user 150. As an example, such an activation request may include a voice command (e.g., also known as a second voice command) received via the audio acquisition unit of electronic device 110. As an example, the second voice command may include preset keywords (e.g., "activate first mode", "start conversation", etc.). This disclosure is not intended to limit the specific content of the preset keywords. Thus, embodiments of this disclosure can support users in quickly activating the first mode of electronic device 110 based on voice commands, thereby improving the efficiency of user interaction with the electronic device.

[0061] Alternatively, electronic device 110 may be associated with a voice assistant application. As an example, the voice assistant application may be installed on terminal device 120. Electronic device 110 may receive voice commands (e.g., voice content associated with preset keywords (e.g., "activate first mode")) from user 150 during a conversation with the voice assistant application. For example, user 150 may engage in a conversation with a voice assistant based on the voice assistant application. Electronic device 110 may activate its first mode in response to receiving a request to end the conversation. Alternatively, electronic device 110 may activate its first mode in response to no voice messages being received during a preset time period (e.g., 2 minutes) in a conversation with the voice assistant application. Thus, embodiments of this disclosure can support users enabling the first mode of an electronic device through a conversation with a voice assistant application. Furthermore, the embodiments of this disclosure can support opening the first mode of the electronic device 110 after the dialogue with the voice assistant application ends, effectively avoiding the conflict between the process of the electronic device 110 receiving voice content in the first mode and the dialogue process between the user and the voice assistant application, and ensuring the interaction quality between the user 150 and the electronic device 110 and between the user 150 and the voice assistant application.

[0062] As an example, a voice assistant may include any suitable processing entity that supports interaction via prompts or similar means. In some scenarios, a voice assistant may include, but is not limited to, an agent or other assistant that supports interaction via dialogue (e.g., text or voice conversations). It should be understood that, on the one hand, a voice assistant may have a virtual avatar for interaction. On the other hand, a voice assistant may also utilize appropriate models to drive its interactive behavior, such models may include any suitable machine learning model, such as a generative model. This disclosure is not intended to limit the specific construction of a voice assistant.

[0063] Alternatively, such an activation request may include an activation operation received via an interactive interface associated with electronic device 110. Such an interactive interface may be provided, for example, by terminal device 120 communicating with electronic device 110.

[0064] The following is based on Figures 3A to 3C The following is an exemplary description of process 200. It should be understood that... Figures 3A to 3C The corresponding embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0065] Figures 3A to 3C Example interfaces 300A and 300C according to some embodiments of the present disclosure are shown. Interfaces 300A to 300C may, for example, be derived from... Figure 1 The terminal device 120 shown is provided.

[0066] In some embodiments, such as Figure 3A As shown, terminal device 120 can present interactive interface 300A. Alternatively, terminal device 120 can trigger electronic device 110 to enable the first mode in response to receiving an activation operation from user 150 in interactive interface 300A. Alternatively, such activation operation may also include triggering (e.g., a click operation) a control (not shown) associated with the first mode in interactive interface 300A.

[0067] As an example, the interactive interface 300A may include a conversational interface (or a graphical interface of the voice assistant application) between user 150 and the voice assistant application. User 150 may interact with the voice assistant through the interactive interface 300A via text, voice, video, or by triggering interactive controls. Furthermore, the voice assistant may respond to the user's interaction. For example, the voice assistant may generate conversational messages or perform corresponding actions (e.g., calling other applications, such as playing music) in response to the user's interaction. Furthermore, the terminal device 120 may present the conversational messages generated by the voice assistant in the interactive interface 300A. For example, the conversational messages may be in the form of text, audio, or images (including image content and video content). Alternatively, the terminal device 120 may trigger the electronic device 110 to enable the first mode in response to receiving an activation operation from user 150 in the interactive interface 300A. For example, such an activation operation may include a conversational message containing preset keywords (e.g., "Enable First Mode") sent by user 150 to the voice assistant.

[0068] Alternatively, the electronic device 110 also includes a mode reminder unit. The electronic device 110 can utilize the mode reminder unit to provide a reminder signal indicating that the electronic device 110 is in a first mode. For example, such a mode reminder unit can be implemented as an LED light. The reminder signal indicating that the electronic device 110 is in the first mode may, for example, include the LED light displaying a first color (e.g., green). Alternatively, the reminder signal indicating that the electronic device 110 is not in the first mode may, for example, include the LED light being off or displaying a second color different from the first color (e.g., red).

[0069] In some embodiments, continue to refer to Figure 2 When in a first mode, electronic device 110 can capture (204) multiple images using image acquisition unit 115. As an example, electronic device 110 can periodically capture multiple images using image acquisition unit 115. As an example, electronic device 110 can determine the time interval and / or duration for capturing multiple images based on configuration information associated with the first mode (also referred to as second configuration information), and periodically capture multiple images based on the time interval and / or duration. As an example, such second configuration information can be determined based on user operations (e.g., voice commands, selection of controls related to the time interval and / or duration, etc.). For example, the time period for electronic device 110 to capture multiple images can be every 1 second, every 2 seconds, etc. The duration can indicate the duration of capturing dynamic image content (e.g., video content). For example, the duration can include 1 second, 2 seconds, 10 seconds, etc. This disclosure is not intended to limit the specific values ​​of the time period and duration.

[0070] Alternatively, the electronic device 110 can determine a resolution associated with the first mode. Further, the electronic device 110 can capture at least a portion of a plurality of images based on this resolution. For example, the electronic device 110 can determine the resolution based on configuration information associated with the first mode (also referred to as first configuration information). For example, the configuration information associated with the first mode includes preset configuration information or configuration information determined by a configuration operation of the user 150. For example, the images may include picture content and / or video content. For example, for capturing video content, the resolution associated with the first mode may be, for example, a first resolution (e.g., 720P). For capturing picture content, the resolution associated with the first mode may be, for example, a second resolution (e.g., 1280×720 pixels).

[0071] Alternatively, electronic device 110 can also determine the resolution based on voice commands from user 150. For example, electronic device 110 can update the resolution (e.g., adjust the resolution of the captured image from 1280×720 pixels to 1920×1080 pixels) in response to receiving a voice command from the user (e.g., “Take a clearer picture”).

[0072] In some embodiments, the electronic device 110 is a headphone device or glasses device equipped with at least one camera. Thus, embodiments of this disclosure can meet the user's visual interaction needs by incorporating a camera into the headphone device or glasses device.

[0073] In some embodiments, electronic device 110 may include multiple image acquisition units (e.g., cameras). These multiple image acquisition units may include, for example, image acquisition unit 115. Taking electronic device 110 as an example of a headphone device, the headphone device may include a left earphone (or a first device) or a right earphone (or a second device). The left and right earphones may each be equipped with at least one image acquisition unit (e.g., a camera). Electronic device 110 may utilize a target image acquisition unit among the multiple image acquisition units to capture all of the multiple images. For example, the target image acquisition unit is determined based on configuration information associated with a first mode (also referred to as third configuration information). For example, such configuration information may instruct that when electronic device 110 is in the first mode, the image acquisition unit associated with the left earphone is used as the target image acquisition unit during a first time period (e.g., the first half hour of electronic device 110 being in the first mode), and the image acquisition unit associated with the right earphone is used as the target image acquisition unit during a second time period (e.g., after electronic device 110 has been in the first mode for half an hour).

[0074] Alternatively, the target image acquisition unit can also be determined based on the wearing status of the electronic device 110. For example, the electronic device 110 may, in response to a first device (e.g., a headphone device) where the user 150 is only wearing the electronic device 110 (e.g., a headphone device), use the image acquisition unit mounted on the left earphone as the target image acquisition unit.

[0075] In some embodiments, the electronic device 110 can periodically capture a first set of images using a first image acquisition unit among a plurality of image acquisition units during a first round of dialogue. Furthermore, the electronic device 110 can periodically capture a second set of images using a second image acquisition unit among a plurality of image acquisition units during a second round of dialogue after the first round of dialogue. The second image acquisition unit is different from the first image acquisition unit. For example, taking the electronic device 110 as an earphone device, the electronic device 110 can periodically capture the first set of images using the image acquisition unit (e.g., the first image acquisition unit) on the left earphone during the first round of dialogue, and periodically capture the second set of images using the image acquisition unit (e.g., the second image acquisition unit) on the right earphone during the second round of dialogue after the first round of dialogue. Thus, the embodiments of this disclosure can capture images using the same image acquisition unit within the same round of dialogue, ensuring the stability of image capture within the same round of dialogue and guaranteeing the quality and relevance of the captured images. Furthermore, based on the embodiments of this disclosure, different image acquisition units are used to capture images in different dialogue rounds. By rotating the image acquisition units, problems such as performance degradation or insufficient power caused by long-term operation of the image acquisition units can be avoided, thereby ensuring the stability and reliability of the captured images.

[0076] In some embodiments, electronic device 110 can send (206) (or transmit) multiple images captured by image acquisition unit 115 to terminal device 120. For example, electronic device 110 can transmit images to terminal device 120 in real time in response to receiving images captured by image acquisition unit. Alternatively, when the transmission speed is slower than the capture speed, electronic device 110 can store images in a buffer unit so that the images in the buffer unit can be sent to terminal device 120 sequentially. As an example, terminal device 120 can acquire multiple images from electronic device 110, which is wirelessly connected to terminal device 120. In this way, embodiments of the present disclosure can transmit multiple images captured by electronic device to terminal device in real time in advance. Thus, before receiving a user's visual interaction request, the terminal device has already acquired the images transmitted by electronic device, thereby shortening the user's waiting time in the process of processing visual interaction requests, improving the efficiency of processing visual interaction requests, and meeting the user's interaction needs.

[0077] In some embodiments, multiple images captured by the electronic device 110 using its image acquisition unit 115 can be provided to a pre-trained model (e.g., model 140) so that the pre-trained model generates content based on at least one of the multiple images. The deployment of the pre-trained model can be categorized into three types: the pre-trained model is deployed at the electronic device 110, the pre-trained model is deployed at the terminal device 120, or the pre-trained model is deployed at the server 130. The following will provide exemplary descriptions for each of the different model deployment locations.

[0078] In the first scenario, when electronic device 110 is equipped with a pre-trained model (e.g., model 140), electronic device 110 may directly provide multiple images to the pre-trained model. Electronic device 110 may utilize the (215) pre-trained model to generate content (e.g., speech content and / or text content) based on at least one of the multiple images.

[0079] In the second scenario, electronic device 110 does not have a pre-trained model deployed, while terminal device 120 does. Electronic device 110 can send (206) multiple images to terminal device 120. Terminal device 120 can provide multiple images to the pre-trained model deployed at terminal device 120. Terminal device 120 can utilize (218) the pre-trained model to generate content (e.g., speech content and / or text content) based on at least one of the multiple images.

[0080] Thirdly, neither electronic device 110 nor terminal device 120 has a pre-trained model deployed, but server 130 has a pre-trained model deployed. Terminal device 120 can receive multiple images from electronic device 110 and send (210) multiple images to server 130, which communicates with terminal device 120. Further, server 130 can provide multiple images to the pre-trained model deployed on server 130. Server 130 can use (220) the pre-trained model to generate content (e.g., speech content and / or text content) based on at least one of the multiple images.

[0081] In some embodiments, before generating content based on at least one image from a plurality of images using a pre-trained model, it is necessary to determine whether a target image from the plurality of images satisfies a triggering condition. Here, the target image can refer to any one of the plurality of images. For example, this disclosure may sequentially determine whether the triggering condition is satisfied for each image from the plurality of images.

[0082] As an example, electronic device 110 can determine (214) whether a target image among multiple images satisfies the triggering condition based on a pre-trained model deployed on electronic device 110. Alternatively, terminal device 120 can determine (216) whether a target image among multiple images satisfies the triggering condition based on a pre-trained model deployed on terminal device 120. Alternatively, server 130 can determine (212) whether a target image among multiple images satisfies the triggering condition based on a pre-trained model deployed on server 130.

[0083] Alternatively, terminal device 120 can determine whether a target image among multiple images satisfies the triggering condition by utilizing other models. Such other models can be implemented, for example, as classification models (e.g., classification models that output whether a target image satisfies or does not satisfy the triggering condition). Such classification models can be deployed, for example, in terminal device 120 or in server 130 communicating with terminal device 120. This disclosure is not intended to limit the specific implementation or training process of such classification models.

[0084] In some embodiments, such triggering conditions may include, for example, matching a target image with a preset scene. As an example, the preset scene may include a scene determined by a pre-trained model or a manually set scene. For example, such preset scenes may include "supermarket," "snow scene," "rainbow," "historical building," and / or "sunset," etc. For example, if the target image contains "rainbow," then the target image can be determined to match the preset scene (e.g., "rainbow"), thus satisfying the triggering condition. Further, in response to the target image being determined to satisfy the triggering condition, the pre-trained model may generate response content based on the target image. The response content may include speech content and / or text content (e.g., text content corresponding to the speech content or other text content). Additionally, the topic of the speech content generated by the pre-trained model based on the target image is associated with the preset scene corresponding to the target image. For example, if the target image contains "rainbow," then the topic of the corresponding speech content may be associated with "rainbow." For example, such speech content may include "Look, there's a rainbow."

[0085] As an example, electronic device 110 may be associated with a voice assistant application installed on terminal device 110. Triggering conditions may include matching the target image with historical information of the voice assistant application. This historical information may, for example, indicate the historical interactions between user 150 and the voice assistant application. These historical interactions may indicate historical conversations between user 150 and the voice assistant application. For example, if the historical interactions between user 150 and the voice assistant application are related to "pastry recommendations," then if the target image contains "pastry shop," it can be determined that the target image matches the historical information of the voice assistant application. Therefore, the target image satisfies the triggering condition. Additionally, the pre-trained model associates the topic of the voice content generated based on the target image with the corresponding historical information of the target image. For example, if the target image contains "pastry shop," then the topic of the corresponding voice content can be associated with "pastry shop." For example, such voice content could include "This pastry shop suits your taste perfectly!" Alternatively, the pre-trained model can also generate voice content based on historical information. In this way, the generated voice content can include the historical interactions between user 150 and the voice assistant application. For example, user 150's historical interactions with the voice assistant application include user 150's conversation content (e.g., "I don't know where the A brand pastry shop is"). A pre-trained model can generate speech content (e.g., "Look, I found the A brand pastry shop for you") based on this conversation content and a target image (e.g., an image including "A brand pastry shop"). It should be noted that all user-related information mentioned in this disclosure (including but not limited to historical information) was obtained and used with the relevant user's knowledge and authorization.

[0086] As an example, triggering conditions can also be set based on a user's voice command (e.g., also known as a first command). For instance, a triggering condition based on a user's voice command could include "Tell me when you see XX bookstore," so the target image can be determined to meet the triggering condition if it contains "XX bookstore." For example, a pre-trained model can generate response content (e.g., "There is the XX bookstore you mentioned at the intersection ahead") based on the target image (e.g., an image containing "XX bookstore").

[0087] In some embodiments, when a pre-trained model is deployed at server 130, server 130 can send (222) response content (including voice content and / or text content) generated based on the pre-trained model deployed at server 130 to terminal device 120. Terminal device 120 can then send (224) the response content to electronic device 110.

[0088] In some embodiments, when a pre-trained model is deployed at a terminal device 120, the terminal device 120 can send (224) the response content generated based on the pre-trained model deployed at the terminal device 120 to the electronic device 110.

[0089] In some embodiments, the electronic device 110 may actively output content using the content output unit (226) (e.g., the audio playback unit 118 actively plays voice content (or audio content) and / or the screen display unit actively displays image content (e.g., pictures and / or videos) and / or text content). Thus, embodiments of this disclosure enable the electronic device 110 to generate corresponding voice content based on captured images and actively interact with the user, thereby improving the efficiency of information acquisition for the user.

[0090] In some embodiments, continue to refer to Figure 3A The electronic device 110 can trigger the display of a first message corresponding to the voice content in an interactive interface associated with the electronic device 110. For example, such an interactive interface can be provided by a screen display unit. Alternatively, such an interactive interface can be presented by a terminal device 120 that wirelessly communicates with the electronic device 110, for example, an interactive interface 300A. The terminal device 120 can present a first message 305 corresponding to the voice content (e.g., "Look, there's a rainbow") in the interactive interface 300A.

[0091] In some embodiments, continue to refer to Figure 2 Terminal device 120 can present one or more images from a plurality of images in an interactive application associated with electronic device 110. Alternatively, electronic device 110 can trigger the display of one or more images from a plurality of images in an interactive interface associated with electronic device 110. As an example, the one or more images may include a target image for generating content (e.g., voice content and / or text content). The one or more images may, for example, include the first image captured from a plurality of images. For example, as... Figure 3A As shown, terminal device 120 can display image 310 of multiple images in interactive interface 300A.

[0092] In some embodiments, multiple images captured by the electronic device 110 and / or contextual information associated with the multiple images are automatically deleted after a preset time period (e.g., 2 hours). This effectively protects user privacy and saves storage space on the user's electronic device 110 or terminal device 120.

[0093] Additionally or alternatively, terminal device 120 may display a reminder message 315 (e.g., "The photo will be automatically destroyed X hours after the conversation ends") in interactive interface 300A to remind user 150 that the image 310 presented in interactive interface 300A will be deleted after X hours.

[0094] In some embodiments, continue to refer to Figure 2 Terminal device 120 may, in response to reaching a preset duration (e.g., 2 hours), stop (234) displaying one or more images among a plurality of images in an interactive application associated with electronic device 110. As an example, such as Figure 3B As shown, terminal device 120 can stop displaying one or more images in interactive interface 300B in response to displaying one or more images out of a set time (e.g., 2 hours). Alternatively, terminal device 120 can display an indicator element 320 (e.g., "Photo has been destroyed") in the interactive interface to indicate that one or more images have stopped being displayed and have been deleted.

[0095] Additionally or alternatively, terminal device 120 (or server 130) may, in response to reaching a preset duration (e.g., 2 hours), delete multiple images acquired from electronic device 110 stored at terminal device 120 (or server 130) or a pre-trained model.

[0096] In some embodiments, continue to refer to Figure 2 Electronic device 110 can pause (228) the first mode. As an example, while electronic device 110 is in the first mode, electronic device 110 can pause the first mode in response to a pause condition to stop periodically taking pictures.

[0097] As an example, such a pause condition may include the wearing status of electronic device 110 not meeting a preset condition. For example, at least one device of electronic device 110 (e.g., the left or right earphone in a headphone device) is removed, so that electronic device 110 is not worn by user 150.

[0098] As an example, such a pause condition may include a voice command (also known as a fourth voice command) received via the audio acquisition unit of the electronic device 110. Such a voice command may include preset keywords associated with the pause first mode (e.g., "pause first mode", etc.).

[0099] As an example, such a pause condition may include electronic device 110 being triggered to provide audio content associated with the connected terminal device 120. For example, such audio content may include a ringtone for a telephone or video call (e.g., receiving an incoming call). Such audio content may also include, for example, an alarm clock.

[0100] As an example, such a pause condition may include the occlusion level of the image acquisition unit 115 being greater than a preset level (e.g., 50%). For example, when the occlusion level of the image acquisition unit 115 of the electronic device 110 is greater than 50%, the first mode of the electronic device 110 is exited to remind the user 150 that the image acquisition unit 115 is occluded.

[0101] In some embodiments, such suspension conditions may also include the electronic device 110 having less than a preset threshold (e.g., 5%), the communication connection between the electronic device 110 and the terminal device 120 being disconnected, and / or the network connection of the terminal device 120 being disconnected.

[0102] Alternatively, electronic device 110 may resume (230) the first mode in response to the pause condition no longer being met. For example, electronic device 110 may resume the first mode in response to the end of a telephone call, video call, or alarm. For example, electronic device 110 may resume the first mode in response to the wearing state of electronic device 110 meeting a preset condition (e.g., at least one device in electronic device 110 is in a wearing state). For example, electronic device 110 may resume the first mode in response to receiving a user's voice command (e.g., a voice command associated with resuming the first mode, such as "resume the first mode"). For example, electronic device 110 may resume the first mode in response to the occlusion degree of image acquisition unit 115 being less than a preset threshold (e.g., 50%).

[0103] In some embodiments, continue to refer to Figure 2 Electronic device 110 can exit (232) the first mode. For example, electronic device 110 can exit the first mode in response to its battery level falling below a threshold (e.g., 5%). Alternatively, electronic device 110 can exit the first mode in response to being worn (e.g., user 150 stops wearing electronic device 110 (e.g., removes headphones)). Alternatively, electronic device 110 can exit the first mode in response to a voice command (also known as a third voice command) received via an audio acquisition unit. For example, such a third voice command can be associated with preset keywords (e.g., "exit first mode", "close first mode", etc.). Alternatively, electronic device 110 can exit the first mode via an exit operation received in an interactive interface associated with electronic device 110 (e.g., triggering of a control associated with exiting the first mode).

[0104] In some embodiments, the electronic device 110 may also, in response to exiting the first mode, present media content and / or text content generated based on multiple images on an interactive interface associated with the electronic device 110. (Continue to refer to...) Figure 2 Terminal device 120 may, in response to electronic device 110 exiting the first mode, present (236) media content (e.g., image content and / or video content) and / or text content generated based on multiple images. As an example, such media content or text content may be a summary of multiple images captured and multiple audio contents played by electronic device 110 during the first mode. For example, such media content and / or text content may be generated by a pre-trained model based on multiple images and multiple audio contents.

[0105] In some embodiments, such as Figure 3C As shown, the terminal device 110 can present summary content 325 (e.g., including media content 325-2 and text content 325-1, etc.) generated based on multiple images in an interactive interface 300C associated with the electronic device 110.

[0106] Based on the process described above, embodiments of this disclosure can periodically capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode. Furthermore, embodiments of this disclosure can acquire and play audio content generated by a pre-trained model based on at least one of the multiple images. Thus, the electronic device provided by embodiments of this disclosure can periodically capture images, thereby acquiring information about the user's surrounding environment in real time. In addition, the electronic device provided by embodiments of this disclosure can generate audio content based on the captured images using a pre-trained model and actively play the audio content to the user, improving the efficiency of information acquisition for the user. Therefore, embodiments of this disclosure expand the interaction capabilities between the electronic device and the user and meet the user's interaction needs.

[0107] Example process

[0108] Figure 4 A flowchart illustrating an example process 400 of image interaction according to some embodiments of the present disclosure is shown. Process 400 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 400.

[0109] like Figure 4 As shown in box 410, when the electronic device 110 is in the first mode, it uses the image acquisition unit mounted on the electronic device 110 to capture multiple images.

[0110] In box 420, electronic device 110 actively outputs content using its content output unit, wherein the content is generated by a pre-trained model based on at least one of a plurality of images.

[0111] In some embodiments, the electronic device 110 includes a wearable device; and / or, the content output unit includes at least one of a screen display unit and an audio playback unit.

[0112] In this way, embodiments of the present disclosure can enable wearable devices to actively output content based on multiple captured images using a screen display unit and / or an audio playback unit, thereby improving the efficiency of user interaction with the wearable device.

[0113] In some embodiments, electronic device 110 communicates wirelessly with terminal device 120, and the content includes at least voice content.

[0114] In this way, the embodiments of this disclosure can support the electronic device 110 to actively output voice content, thereby improving the interaction efficiency between the user and the electronic device.

[0115] In some embodiments, the speech content is generated based on the following process: an electronic device 110 sends multiple images to a terminal device 120; the terminal device 120 provides multiple images to a pre-trained model; and the pre-trained model generates speech content based on at least one of the multiple images, wherein the pre-trained model is deployed on the terminal device 120 or a server 130 communicating with the terminal device 120.

[0116] In this way, embodiments of the present disclosure can use a pre-trained model to generate speech content based on multiple images, enabling the electronic device 110 to actively play speech content to the user using an audio playback unit, thereby meeting the user's interactive needs.

[0117] In some embodiments, the content is generated based on the following process: acquiring a target image from a plurality of images captured by an electronic device 110; and in response to the target image being determined to meet a triggering condition, generating response content based on the target image using a pre-trained model, the response content including voice content and / or text content.

[0118] In this way, embodiments of this disclosure generate response content for target images that meet triggering conditions based on pre-trained models, thereby satisfying user interaction needs.

[0119] In some embodiments, the triggering conditions include matching the target image with a preset scene and the topic of the response content being associated with the preset scene; and / or, the triggering conditions are set based on a user's first voice command.

[0120] In this way, embodiments of this disclosure can provide users with voice content associated with a preset scenario, thereby improving the efficiency of information acquisition for users. Furthermore, embodiments of this disclosure support users setting corresponding trigger conditions via voice commands, thereby improving user interaction efficiency.

[0121] In some embodiments, the electronic device 110 is associated with a voice assistant application, and the triggering condition includes matching the target image with historical information of the voice assistant application, the historical information indicating the user's and the voice assistant application's past interactions.

[0122] In this way, embodiments of the present disclosure can determine that a target image meets the triggering conditions based on the user's historical interactions with the voice assistant application, thereby satisfying the user's interaction needs.

[0123] In some embodiments, the voice content is also generated based on historical information.

[0124] In this way, embodiments of the present disclosure can determine voice content based on the user's and the voice assistant application's historical information, making the voice content more in line with the user's expectations or needs.

[0125] In some embodiments, process 400 further includes: in response to the electronic device 110 exiting the first mode, presenting media content and / or text content generated based on multiple images in an interactive interface associated with the electronic device 110.

[0126] In this way, the embodiments of this disclosure can generate corresponding media content and / or text content after the electronic device 110 exits the first mode, thereby recording and summarizing the interaction process of the electronic device 110 during the first mode, facilitating subsequent viewing by the user and meeting the user's needs.

[0127] In some embodiments, capturing multiple images using the image acquisition unit mounted on the electronic device 110 includes: determining a resolution associated with a first mode; capturing at least a portion of the multiple images based on the resolution; and / or determining a time interval associated with the first mode and periodically capturing multiple images based on the time interval; and / or determining a time length associated with the first mode and capturing multiple images within the time length based on the time length.

[0128] In this way, images can be captured based on the configured resolution, ensuring the stability of image quality. Furthermore, embodiments of this disclosure can determine the time period and / or duration for capturing multiple images based on configuration information, thereby enabling users to capture images using different time periods and / or durations.

[0129] In some embodiments, the first mode is enabled in response to a user’s activation request, and the activation request includes: an activation operation received via an interactive interface associated with the electronic device 110; or a second voice command received via an audio acquisition unit.

[0130] In this way, the embodiments of this disclosure can provide users with multiple ways to enable the first mode, improving the efficiency of enabling the first mode of the electronic device.

[0131] In some embodiments, the electronic device 110 includes a plurality of image acquisition units, and capturing a plurality of images using the image acquisition units mounted on the electronic device 110 includes capturing all the plurality of images using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with a first mode and / or the wearing state of the electronic device 110.

[0132] In this way, embodiments of this disclosure can determine the target acquisition unit for capturing multiple images based on configuration information, thereby meeting diverse user needs and improving efficiency. Furthermore, embodiments of this disclosure can determine the target acquisition unit for capturing multiple images based on the wearing state, adapting to the user's wearing operation and allowing the user to freely choose the wearing method, bringing convenience to the user.

[0133] In some embodiments, the electronic device includes a plurality of image acquisition units, and capturing multiple images using the image acquisition units mounted on the electronic device 110 includes: capturing a first set of images using a first image acquisition unit among the plurality of image acquisition units during a first round of dialogue; and capturing a second set of images using a second image acquisition unit among the plurality of image acquisition units during a second round of dialogue after the first round of dialogue has ended, wherein the second image acquisition unit is different from the first image acquisition unit.

[0134] In this way, the embodiments of this disclosure can support the use of different image acquisition units to capture images in different dialogue rounds, thereby avoiding performance degradation or insufficient power caused by continuous operation of a single image acquisition unit, and thus ensuring the stability of the image capture function. Furthermore, maintaining the use of the same image acquisition unit within the same dialogue round ensures that the capture level or effect of multiple images associated with the same dialogue round remains consistent.

[0135] In some embodiments, multiple images and / or contextual information associated with multiple images are automatically deleted after a preset time period.

[0136] In this way, the embodiments of this disclosure can protect user privacy and save device storage space for users.

[0137] In some embodiments, process 400 further includes: exiting the first mode in response to the battery level of electronic device 110 being below a threshold; and / or exiting the first mode in response to a third voice command received via an audio acquisition unit; and / or exiting the first mode via an exit operation received in an interactive interface associated with the electronic device.

[0138] In this way, embodiments of this disclosure can exit the first mode when preset conditions are met, thereby meeting user needs and ensuring the stability of the functions provided by the electronic device in the first mode. Furthermore, embodiments of this disclosure can exit the first mode based on user voice commands or exit operations in the interactive interface, thereby improving user interaction efficiency.

[0139] In some embodiments, process 400 further includes: while the electronic device 110 is in a first mode, pausing the first mode to stop taking photos in response to a pause condition being met, the pause condition including at least one of the following: the electronic device 110 includes a wearable device, the wearing state of the wearable device does not meet a preset condition; a fourth voice command is received via the audio acquisition unit of the electronic device; the electronic device 110 is triggered to provide audio content associated with the connected terminal device 120; the occlusion degree of the image acquisition unit is greater than a preset degree.

[0140] In this way, embodiments of the present disclosure can pause the electronic device in the first mode when the pause conditions are met, thereby meeting the user's needs and ensuring the stability of the functions provided by the electronic device in the first mode and the normal operation of other applications.

[0141] In some embodiments, process 400 further includes: triggering the display of: a first message corresponding to content; and at least one image for generating content in an interactive interface associated with electronic device 110; and / or triggering a mode reminder unit associated with electronic device to provide a reminder signal about electronic device being in a first mode.

[0142] In this way, embodiments of this disclosure can present various contents associated with the first mode in the interactive interface, thereby satisfying the user's viewing and interaction needs. Furthermore, embodiments of this disclosure can, based on reminder signals, enable the current user and other users to clearly confirm whether the electronic device 110 is in the first mode, thus satisfying the user's needs.

[0143] In some embodiments, the electronic device 110 is a headphone device or glasses device equipped with at least one camera.

[0144] Thus, embodiments of this disclosure can meet the user's visual interaction needs by mounting a camera on a headphone device or glasses device.

[0145] Figure 5 A flowchart illustrating an example process 500 of image interaction according to some embodiments of the present disclosure is shown. Process 500 can be implemented at a terminal device 120. Reference is made below. Figure 1 To describe process 500.

[0146] like Figure 5 As shown in block 510, terminal device 120 acquires multiple images from electronic device 110 which is wirelessly connected to terminal device 120. The multiple images are captured by the image acquisition unit mounted on electronic device 110 when electronic device 110 is in a first mode.

[0147] In box 520, terminal device 120 acquires content, which is generated by a pre-trained model based on at least one of a plurality of images.

[0148] In frame 530, terminal device 120 sends content to electronic device 110 to actively output content using the content output unit on electronic device 110.

[0149] In some embodiments, the electronic device 110 includes a wearable device; and / or, the content output unit includes at least one of a screen display unit and an audio playback unit.

[0150] In this way, embodiments of the present disclosure can enable wearable devices to actively output content based on multiple captured images using a screen display unit and / or an audio playback unit, thereby improving the efficiency of user interaction with the wearable device.

[0151] In some embodiments, process 500 further includes: receiving a plurality of images from electronic device 110; and a pre-trained model providing the plurality of images for generating content.

[0152] In this way, embodiments of the present disclosure can generate content based on multiple images using a pre-trained model, enabling the electronic device 110 to actively output content to the user using the content output unit, thereby meeting the user's interactive needs.

[0153] In some embodiments, process 500 further includes: acquiring a target image from a plurality of images captured by an electronic device; and, in response to the target image being determined to meet a triggering condition, providing the target image to a pre-trained model for content generation.

[0154] In this way, embodiments of this disclosure generate response content for target images that meet triggering conditions based on pre-trained models, thereby satisfying user interaction needs.

[0155] Figure 6 A flowchart of an example process 600 for image interaction according to some embodiments of the present disclosure is shown. Process 600 can be implemented in an image interaction system 100. The image interaction system 100 includes a wireless communication electronic device 110 and a terminal device 120. Reference is made below. Figure 1 To describe process 600.

[0156] like Figure 6As shown in box 610, when the electronic device 110 is in the first mode, the image interaction system 100 uses the image acquisition unit mounted on the electronic device 110 to capture multiple images.

[0157] In frame 620, the image interaction system 100 sends multiple images from electronic device 110 to terminal device 120.

[0158] In box 630, the image interaction system 100 acquires content from the terminal device 120, wherein the content is generated by a pre-trained model based on at least one of a plurality of images.

[0159] In frame 640, the image interaction system 100 sends voice content from terminal device 120 to electronic device 110 to actively output content using the content output unit on electronic device 110.

[0160] Example devices and equipment

[0161] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes.

[0162] Figure 7 A schematic structural block diagram of an example device 700 for image interaction according to certain embodiments of the present disclosure is shown. Device 700 may be implemented as or included in electronic device 110. Various modules / components in device 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0163] like Figure 7 As shown, the device 700 includes: an image acquisition module 710 configured to capture multiple images using the image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; and a first playback module 720 configured to actively output content using the content output unit of the electronic device 110, wherein the content is generated by a pre-trained model based on at least one of the multiple images.

[0164] In some embodiments, the electronic device 110 includes a wearable device; and / or, the content output unit includes at least one of a screen display unit and an audio playback unit.

[0165] In some embodiments, electronic device 110 communicates wirelessly with terminal device 120, and the content includes at least voice content.

[0166] In some embodiments, the speech content is generated based on the following process: an electronic device 110 sends multiple images to a terminal device 120; the terminal device 120 provides multiple images to a pre-trained model; and the pre-trained model generates speech content based on at least one of the multiple images, wherein the pre-trained model is deployed on the terminal device 120 or a server 130 communicating with the terminal device 120.

[0167] In some embodiments, the content is generated based on the following process: acquiring a target image from a plurality of images captured by an electronic device 110; and in response to the target image being determined to meet a triggering condition, generating response content based on the target image using a pre-trained model, wherein the response content is speech content and / or text content.

[0168] In some embodiments, the triggering conditions include matching the target image with a preset scene and the topic of the response content being associated with the preset scene; and / or, the triggering conditions are set based on a user's first voice command.

[0169] In some embodiments, the electronic device 110 is associated with a voice assistant application, and the triggering condition includes matching the target image with historical information of the voice assistant application, the historical information indicating the user's and the voice assistant application's past interactions.

[0170] In some embodiments, the voice content is also generated based on historical information.

[0171] In some embodiments, the device 700 further includes a presentation module configured to: in response to the electronic device 110 exiting a first mode, present media content and / or text content generated based on multiple images in an interactive interface associated with the electronic device 110.

[0172] In some embodiments, the image acquisition module 710 is further configured to: determine a resolution associated with a first mode; and capture at least a portion of a plurality of images based on the resolution; and / or determine a time interval associated with the first mode and periodically capture a plurality of images based on the time interval; and / or determine a time length associated with the first mode and capture a plurality of images within the time length based on the time length.

[0173] In some embodiments, the first mode is enabled in response to a user’s activation request, and the activation request includes: an activation operation received via an interactive interface associated with the electronic device 110; or a second voice command received via an audio acquisition unit.

[0174] In some embodiments, the electronic device 110 includes a plurality of image acquisition units, and the image acquisition module 710 is further configured to capture all of the plurality of images using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with a first mode and / or the wearing state of the electronic device 110.

[0175] In some embodiments, the electronic device includes a plurality of image acquisition units, and the image acquisition module 710 is further configured to: during a first round of dialogue, capture a first set of images using a first image acquisition unit among the plurality of image acquisition units; and during a second round of dialogue after the first round of dialogue ends, capture a second set of images using a second image acquisition unit among the plurality of image acquisition units, the second image acquisition unit being different from the first image acquisition unit.

[0176] In some embodiments, multiple images and / or contextual information associated with multiple images are automatically deleted after a preset time period.

[0177] In some embodiments, the device 700 further includes an exit module configured to: exit the first mode in response to the battery level of the electronic device 110 being below a threshold; and / or exit the first mode in response to a third voice command received via an audio acquisition unit and / or exit the first mode via an exit operation received in an interactive interface associated with the electronic device.

[0178] In some embodiments, the device 700 further includes a pause module configured to: pause the first mode to stop taking photos in response to a pause condition being met while the electronic device 110 is in a first mode, the pause module is configured to: pause the first mode to stop taking photos while the electronic device 110 is in a first mode, the pause condition including at least one of the following: the electronic device 110 includes a wearable device, the wearing state of the wearable device does not meet a preset condition; a fourth voice command is received via the audio acquisition unit of the electronic device 110; the electronic device 110 is triggered to provide audio content associated with the connected terminal device 120; the occlusion degree of the image acquisition unit is greater than a preset degree.

[0179] In some embodiments, the device 700 further includes a triggering module configured to: trigger the display of: a first message corresponding to content; and at least one image for generating content in an interactive interface associated with the electronic device 110; and / or trigger a mode reminder unit associated with the electronic device to provide a reminder signal about the electronic device being in a first mode.

[0180] In some embodiments, the electronic device 110 is a headphone device or glasses device equipped with at least one camera.

[0181] Figure 8A schematic structural block diagram of an example device 800 for image interaction according to certain embodiments of the present disclosure is shown. Device 800 may be implemented as or included in terminal device 120. Various modules / components in device 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0182] like Figure 8 As shown, the device 800 includes: an image acquisition module 810 configured to acquire multiple images from an electronic device 110 that wirelessly communicates with the terminal device 120, the multiple images being captured by an image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; a first acquisition module 820 configured to acquire content, wherein the content is generated by a pre-trained model based on at least one of the multiple images; and a transmission module 830 configured to transmit the content to the electronic device 110 to actively output the content using a content output unit mounted on the electronic device 110.

[0183] In some embodiments, the electronic device 110 includes a wearable device; and / or, the content output unit includes at least one of a screen display unit and an audio playback unit.

[0184] In some embodiments, the apparatus 800 further includes a first providing module configured to: receive a plurality of images from the electronic device 110; and provide the plurality of images to a pre-trained model for generating content.

[0185] In some embodiments, the apparatus 800 further includes a second providing module configured to: acquire a target image from a plurality of images captured by an electronic device; and, in response to the target image being determined to meet a triggering condition, provide the target image to a pre-trained model for content generation.

[0186] Figure 9 A schematic structural block diagram of an example device 900 for image interaction according to certain embodiments of the present disclosure is shown. Device 900 may be implemented as or included in an image interaction system 100. The image interaction system 100 includes a wireless communication electronic device 110 and a terminal device 120. Various modules / components in device 900 may be implemented by hardware, software, firmware, or any combination thereof.

[0187] like Figure 9As shown, the device 900 includes: an image capturing module 910 configured to capture multiple images using an image acquisition unit mounted on the electronic device 110 when the electronic device 110 is in a first mode; an image sending module 920 configured to send multiple images from the electronic device 110 to a terminal device 120; a second acquisition module 930 configured to acquire content from the terminal device 120, wherein the content is generated by a pre-trained model based on at least one of the multiple images; and a second playback module 950 configured to send voice content from the terminal device 120 to the electronic device 110 to actively output content using a content output unit mounted on the electronic device 110.

[0188] The units included in devices 700, 800, or 900 can be implemented using various methods, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in devices 700, 800, or 900 may be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0189] Figure 10 A block diagram of an electronic device 1000 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 10 The electronic device 1000 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 10 The electronic device 1000 shown can be used to achieve Figure 1 Image interaction system 100.

[0190] like Figure 10 As shown, electronic device 1000 is in the form of a general-purpose electronic device. Components of electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, memory 1020, storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. Processing unit 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 1000.

[0191] Electronic device 1000 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 1000.

[0192] Electronic device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 10 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1020 may include computer program product 1025 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0193] The communication unit 1040 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 1000 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0194] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 1000, or with any device that enables electronic device 1000 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0195] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0196] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0197] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0198] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0200] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for image interaction, applied to an electronic device, the method comprising: When the electronic device is in the first mode, multiple images are captured using the image acquisition unit mounted on the electronic device; as well as The electronic device actively outputs content using its content output unit, wherein the content is generated by a pre-trained model based on at least one of the plurality of images.

2. The method of claim 1, wherein the electronic device includes a wearable device; and / or, the content output unit includes at least one of a screen display unit and an audio playback unit.

3. The method according to claim 2, wherein the wearable device communicates wirelessly with the terminal device, and the content includes at least voice content.

4. The method according to claim 3, wherein the voice content is generated based on the following process: The wearable device sends the multiple images to the terminal device; The terminal device provides the multiple images to the pre-trained model; as well as The pre-trained model generates the speech content based on at least one of the plurality of images, wherein the pre-trained model is deployed on a terminal device or a server communicating with the terminal device.

5. The method of claim 1, wherein the content is generated based on the following process: Acquire the target image from the plurality of images captured by the electronic device; and In response to the target image being determined to meet the triggering condition, response content is generated based on the target image using the pre-trained model, the response content including speech content and / or text content.

6. The method according to claim 5, wherein the triggering condition includes the target image matching a preset scene, and the topic of the response content being associated with the preset scene; and / or, the triggering condition is set based on a user's first voice command.

7. The method of claim 5, wherein the electronic device is associated with a voice assistant application, and the triggering condition includes matching the target image with historical information of the voice assistant application, the historical information indicating the user's and the voice assistant application's historical interactions.

8. The method of claim 7, wherein the voice content is further generated based on the historical information.

9. The method according to claim 1, further comprising: In response to the electronic device exiting the first mode, media content and / or text content generated based on the plurality of images are presented in an interactive interface associated with the electronic device.

10. The method of claim 1, wherein capturing multiple images using the image acquisition unit mounted on the electronic device includes... Determine the resolution associated with the first mode, and based on the resolution, capture at least a portion of the plurality of images; and / or Determine the time interval associated with the first mode, and periodically capture the plurality of images based on the time interval; and / or Determine the time length associated with the first mode, and capture the plurality of images within the time length based on the time length.

11. The method of claim 1, wherein the first mode is enabled in response to a user's activation request, and the activation request includes: An activation operation received via an interactive interface associated with the electronic device; or The second voice command received via the audio acquisition unit of the electronic device.

12. The method according to claim 1, wherein the electronic device comprises a plurality of image acquisition units, and capturing a plurality of images using the image acquisition units mounted on the electronic device comprises: All of the plurality of images are captured using a target image acquisition unit among the plurality of image acquisition units, the target image acquisition unit being determined based on configuration information associated with the first mode and / or the wearing status of the electronic device.

13. The method according to claim 1, wherein the electronic device comprises a plurality of image acquisition units, and capturing a plurality of images using the image acquisition units mounted on the electronic device comprises: During the first round of dialogue, the first image acquisition unit among the plurality of image acquisition units is used to capture the first set of images; as well as During the second round of dialogue after the first round of dialogue, a second set of images is captured using a second image acquisition unit among the plurality of image acquisition units. The second image acquisition unit is different from the first image acquisition unit.

14. The method of claim 1, wherein the plurality of images and / or the context information associated with the plurality of images are automatically deleted after a preset time period.

15. The method according to claim 1, further comprising: In response to the electronic device's battery level falling below a threshold, the first mode is exited; And / or, In response to a third voice command received via the audio acquisition unit of the electronic device, exit the first mode; and / or The first mode is exited via an exit operation received in the interactive interface associated with the electronic device.

16. The method according to claim 1, further comprising: While the electronic device is in a first mode, in response to a pause condition being met, the first mode is paused to stop taking photos, the pause condition including at least one of the following: The electronic device includes a wearable device, and the wearing state of the wearable device does not meet the preset conditions; The fourth voice command received via the audio acquisition unit of the electronic device; The electronic device is triggered to provide audio content associated with the connected terminal device; The degree of occlusion of the image acquisition unit is greater than the preset degree.

17. The method according to claim 1, further comprising: Triggering the display of a first message corresponding to the content in the interactive interface associated with the electronic device; And the at least one image used to generate the content; and / or The mode reminder unit associated with the electronic device is triggered to provide a reminder signal that the electronic device is in the first mode.

18. The method of claim 2, wherein the wearable device is an earphone device or glasses device equipped with at least one camera.

19. A method for image interaction, applied to a terminal device, the method comprising: Multiple images are acquired from an electronic device that communicates wirelessly with the terminal device, the multiple images being captured using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; Obtain content, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; as well as The content is sent to the electronic device so that the content is actively output using the content output unit mounted on the electronic device.

20. The method of claim 19, wherein the electronic device comprises a wearable device; and / or, the content output unit comprises at least one of a screen display unit and an audio playback unit.

21. The method of claim 19, further comprising: Receive the plurality of images from the electronic device; as well as The pre-trained model provides the plurality of images for generating the content.

22. The method of claim 19, further comprising: Acquire the target image from the plurality of images captured by the electronic device; as well as In response to the target image being determined to meet the triggering condition, the target image is provided to the pre-trained model for generating the content.

23. A method for image interaction, applied to an image interaction system, the image interaction system including a wireless communication electronic device and a terminal device, the method comprising: When the electronic device is in the first mode, multiple images are captured using the image acquisition unit mounted on the electronic device; The electronic device sends the multiple images to the terminal device; Content is acquired by the terminal device, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; as well as The terminal device sends the content to the electronic device, so that the content is actively output by the content output unit on the electronic device.

24. An apparatus for image interaction, applied to an electronic device, the apparatus comprising: The image acquisition module is configured to capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode; as well as The first playback module is configured to actively output content using the content output unit of the electronic device, wherein the content is generated by a pre-trained model based on at least one of the plurality of images.

25. An apparatus for image interaction, applied to a terminal device, the apparatus comprising: The image acquisition module is configured to acquire multiple images from an electronic device that wirelessly communicates with the terminal device, the multiple images being captured using an image acquisition unit mounted on the electronic device when the electronic device is in a first mode; The first acquisition module is configured to acquire content, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; as well as The sending module is configured to send the content to the electronic device to actively output the content using the content output unit mounted on the electronic device.

26. An apparatus for image interaction, applied to an image interaction system, the image interaction system including wireless communication electronic devices and terminal devices, the apparatus comprising: The image capturing module is configured to capture multiple images using the image acquisition unit mounted on the electronic device when the electronic device is in a first mode; An image sending module is configured to send the plurality of images from the electronic device to the terminal device; The second acquisition module is configured to acquire content from the terminal device, wherein the content is generated by a pre-trained model based on at least one of the plurality of images; as well as The second playback module is configured to send the content from the terminal device to the electronic device, so as to actively output the content using the content output unit mounted on the electronic device.

27. An electronic device comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 18, 19 to 22 or 23 when executed by the at least one processing unit.

28. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 18, 19 to 22 or 23.

Images