Method of human-machine interaction and electronic device

Abstract

Embodiments of the present application provide a human-computer interaction method, which comprises: in response to an operation of selecting an interaction region on a first interface of an electronic device by a user, the electronic device acquires first interaction content associated with the interaction region; the electronic device acquires a first voice instruction; and the electronic device executes a first operation according to the first voice instruction and the first interaction content. Through the method, interaction content can be selected on the display screen of the electronic device, and convenient voice interaction between the electronic device and the user based on the interaction content can be realized based on an application related to the interaction content on the electronic device, thereby providing a convenient human-computer interaction mode for the user.

Description

[0001] This application is a divisional application. The original application has the application number 202411025317.0 and the original application date is July 26, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of human-computer interaction, and more specifically, to a method and electronic device for human-computer interaction. Background Technology

[0003] As smart devices become more and more widespread and their functions become more and more abundant, people are becoming more and more dependent on them in their lives and work. There are more and more scenarios in which users need to interact with the content on the screen, which makes people's need for the convenience of interacting with the screen content of smart devices increasingly higher. Summary of the Invention

[0004] This application provides a method and electronic device for human-computer interaction. Through this method and electronic device, interactive content can be selected on the display screen of the electronic device, and convenient voice interaction between the electronic device and the user can be realized based on the interactive content based on the application on the electronic device. This provides users with a convenient human-computer interaction method.

[0005] In a first aspect, a method for human-computer interaction is provided, the method comprising: responding to a user's operation of selecting an interactive area on a first interface of an electronic device, the electronic device acquiring first interactive content associated with the interactive area; the electronic device acquiring a first voice command; and the electronic device performing a first operation based on the first voice command and the first interactive content.

[0006] In this embodiment, interactive content can be selected on the display screen of the electronic device, and based on the application related to the interactive content on the electronic device, convenient voice interaction between the electronic device and the user based on the interactive content is realized, providing the user with a convenient human-computer interaction method.

[0007] In addition, in the method provided in this application embodiment, before selecting the target interactive object, it is not necessary to switch the screen of the electronic device to the selection state. The interactive content can be selected directly based on the current display screen of the electronic device, which further improves the convenience of interaction between the user and the electronic device.

[0008] Furthermore, electronic devices can adaptively adjust the selection range and determine the interactive content based on the content selected by the user, enabling precise selection of interactive content without requiring the user to frequently manually adjust the interactive area.

[0009] In conjunction with the first aspect, in one possible implementation, the electronic device acquires the first voice command by: in response to the electronic device acquiring the first interactive content, triggering the electronic device to detect the first voice command for multimodal fusion with the first interactive content.

[0010] It can be understood that in this embodiment of the application, after selecting interactive content, the electronic device is triggered to enter the state of detecting the first voice command, which is different from the state of detecting the voice command triggered when no interactive content is selected. Specifically, after selecting interactive content, the electronic device is triggered to enter the state of detecting the first voice command. In this state, the first voice command detected cannot be used independently to determine what operation to perform subsequently. Instead, it is based on multimodal fusion with the selected interactive content to determine what operation to perform subsequently.

[0011] In this embodiment, after the user selects interactive content on the screen of the electronic device, the electronic device enters a state of detecting voice commands related to the interactive content. The user does not need to perform any additional operations to trigger the electronic device to detect voice commands, which further improves the convenience of voice interaction between the user and the electronic device based on interactive content.

[0012] In conjunction with the first aspect, in one possible implementation, the electronic device acquires the first voice command by: in response to the electronic device acquiring the first interactive content, the electronic device displays a second interface, the second interface including a first control, the first control being used to trigger the electronic device to detect the first voice command that is multimodally fused with the first interactive content.

[0013] It can be understood that in this embodiment of the application, the state of detecting the first voice command triggered by the electronic device after selecting interactive content is different from the state of detecting the voice command triggered by the electronic device when no interactive content is selected. Specifically, the state of detecting the first voice command triggered by the electronic device after selecting interactive content is the state of detecting voice commands related to the interactive content.

[0014] In this embodiment of the application, when a user selects interactive content on the screen of an electronic device, a control for triggering the electronic device to enter a state of detecting voice commands related to the interactive content can be displayed on the screen. The user can use this control to trigger the electronic device to enter a state of detecting voice commands related to the interactive content, which can prevent the electronic device from mistakenly entering a state of detecting voice commands related to the interactive content.

[0015] In conjunction with the first aspect, in one possible implementation, the first control could be a navigation bar.

[0016] Optionally, when the electronic device is in a screen-on state, the first navigation bar is always displayed on the screen of the electronic device.

[0017] Optionally, the first navigation bar is always displayed on the screen of the electronic device while the electronic device is interacting with the user based on interactive content.

[0018] In this embodiment, after selecting interactive content, the electronic device can be triggered to obtain interactive instructions by touching the navigation bar corresponding to the page where the interactive content is located. After obtaining the interactive instructions, the interactive operation is performed according to the interactive instructions and the selected content, which can quickly and accurately complete the interactive task.

[0019] In conjunction with the first aspect, in one possible implementation, the electronic device acquires a first voice command, including: in response to a wake word issued by the user, triggering the electronic device to detect the first voice command for multimodal fusion with the first interactive content.

[0020] In this embodiment, after selecting interactive content, a wake word can be issued to trigger the electronic device to obtain interactive instructions. After obtaining the interactive instructions, interactive operations are performed according to the interactive instructions and the selected content, which can quickly and accurately complete the interactive task.

[0021] In conjunction with the first aspect, in one possible implementation, the first control is a voice icon.

[0022] In this embodiment, after selecting interactive content, the electronic device can be triggered to obtain interactive instructions by touching the voice icon displayed on the page where the interactive content is located. After obtaining the interactive instructions, the interactive operation is performed according to the interactive instructions and the selected content, which can quickly and accurately complete the interactive task.

[0023] In conjunction with the first aspect, in one possible implementation, the first operation is for the electronic device to display a third interface, the third interface including one or more recommended interactive services, the one or more recommended interactive services being related to the first interactive content.

[0024] In this embodiment, after selecting interactive content, the electronic device's display screen can show multiple recommended interactive services associated with the interactive content. By selecting a target recommended interactive service, the interaction between the selected interactive content and the target recommended interactive service can be realized. Users do not need to open the application corresponding to the target recommended interactive service separately, nor do they need to perform operations such as copying or saving the selected interactive content in advance. This can help users complete interactive tasks quickly and accurately.

[0025] In conjunction with the first aspect, in one possible implementation, the method further includes: in response to detecting a user clicking on a first recommended interactive service, the electronic device executing the first recommended interactive service; or, in response to detecting a user's second voice command, the electronic device executing the first recommended interactive service, the second voice command being used to instruct the selection of the first recommended interactive service; or, in response to detecting a user dragging the first interactive content to the first recommended interactive service, the electronic device executing the first recommended interactive service, the one or more recommended interactive services including the first recommended interactive service.

[0026] In this embodiment, the interaction between interactive content and recommended interactive services can be achieved by clicking the icon of the recommended interactive service, selecting the recommended interactive service via voice instruction, or dragging and dropping interactive content to the icon of the recommended interactive service. This provides users with a variety of convenient interaction methods and further enhances the user experience.

[0027] In conjunction with the first aspect, in one possible implementation, the electronic device performs a first operation based on the first voice command and the first interactive content, including: the electronic device performs the first operation based on the first voice command, the first interactive content, and the first display content, wherein the first display content is content on the first interface other than the first interactive content.

[0028] In this embodiment, the interaction result between the electronic device and the user based on the interactive content can be made not only based on the interactive content, but also with reference to other non-interactive content located on the same display interface as the interactive content, making the interaction result more accurate.

[0029] In conjunction with the first aspect, in one possible implementation, the electronic device performs a first operation based on the first voice command and the first interactive content, including: the electronic device directly performs the first operation, the first operation being used to provide a recommended interactive service, the recommended interactive service being correlated with the first interactive content, or the electronic device displays a fourth interface, the fourth interface being the interface for performing the recommended interactive service.

[0030] In this embodiment, after triggering the execution of a recommended interaction service on the interactive content, the display interface of the electronic device can be switched to the interface that executes the recommended interaction service on the interactive content, which makes it convenient for users to complete the interaction quickly and immersively, and also facilitates further editing by users; or, the electronic device can directly execute the interaction service, so that the user is unaware of the interaction process, and the display interface of the electronic device does not switch, which can avoid interrupting the user's current task process.

[0031] In conjunction with the first aspect, in one possible implementation, the method further includes: after the electronic device directly performs the first operation, displaying a first reminder on the screen of the electronic device, the first reminder being used to remind the user that the first operation has been performed.

[0032] In conjunction with the first aspect, in one possible implementation, when the first interactive content includes an image, the recommended interactive service includes one or more of image recognition, sharing, and saving services; when the first interactive content includes chat history, the recommended interactive service includes one or more of copying, sharing, saving, and editing services; when the first interactive content includes text content, the recommended interactive service includes one or more of copying, reading, saving, and editing services; when the first interactive content includes account information, the recommended interactive service includes recording and / or transferring services; when the first interactive content includes URL information, the recommended interactive service includes... The services include one or more of the following: recording service, sharing service, opening URL service, and viewing URL summary service; when the first interactive content includes address information, the recommended interactive services include one or more of the following: distance query service, navigation service, and ride-hailing service; when the first interactive content includes flight information, the recommended interactive services include one or more of the following: booking flight service, flight query service, and adding to schedule service; when the first interactive content includes appointment time information, the recommended interactive services include adding to schedule service and / or recording service; when the first interactive content includes a table, the recommended interactive services include one or more of the following: adding and saving service, copying service, and sharing service.

[0033] In conjunction with the first aspect, in one possible implementation, the method further includes: in response to the electronic device completing the first operation, triggering the electronic device to detect a third voice command for multimodal fusion with the first interactive content again; the electronic device performing a second operation based on the result of the first operation, the third voice command, and the first interactive content.

[0034] In this embodiment of the application, after the first round of voice interaction between the electronic device and the user ends, the electronic device may not end the interaction process, but instead switch to a state of continuing to detect further interaction commands, thereby enabling the electronic device and the user to achieve multi-round voice interaction based on the selected interaction content.

[0035] In conjunction with the first aspect, in one possible implementation, in response to a user's operation of selecting an interactive area on a first interface of an electronic device, the electronic device acquires first interactive content associated with the interactive area, including: in response to detecting a user's operation of circling the interactive area on the first interface, the electronic device acquires the first interactive content; or, in response to detecting a user's operation of checking the interactive area on the first interface, the electronic device acquires the first interactive content; or, in response to detecting a user's operation of long-pressing or double-clicking the interactive area on the first interface, the electronic device acquires the first interactive content; or, in response to detecting a user's operation of dragging the interactive area on the first interface, the electronic device acquires the first interactive content.

[0036] In a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the first aspect or any possible implementation thereof.

[0037] Thirdly, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method of the first aspect or any possible implementation thereof.

[0038] Fourthly, a chip is provided, wherein instructions are stored that, when executed on a device, cause the chip to perform the methods described in the first aspect or any possible implementation thereof.

[0039] Fifthly, a computer program product is provided, which stores a computer program or instructions that, when executed, implement the method in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0041] Figure 2 This is a software structure block diagram of the electronic device provided in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram of the interface corresponding to a method of interaction based on screen content;

[0043] Figure 4 This is a schematic diagram of the interface corresponding to another method of interaction based on screen content;

[0044] Figure 5This is a schematic diagram of the interface corresponding to another method of interaction based on screen content;

[0045] Figure 6 This is a schematic diagram of the interface corresponding to a method for interacting based on screen content provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0047] Figure 8 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0048] Figure 9 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0049] Figure 10 These are schematic diagrams of interfaces corresponding to the two methods for interacting with the screen based on images on the screen provided in the embodiments of this application;

[0050] Figure 11 These are schematic diagrams of interfaces corresponding to several methods for interacting with the screen based on entity content and tables on the screen, as provided in the embodiments of this application.

[0051] Figure 12 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0052] Figure 13 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0053] Figure 14 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0054] Figure 15 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0055] Figure 16 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0056] Figure 17 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0057] Figure 18 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0058] Figure 19 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0059] Figure 20 This is a schematic diagram of the interface corresponding to another method for interaction based on screen content provided in the embodiments of this application;

[0060] Figure 21 This is a schematic flowchart corresponding to a human-computer interaction method provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0062] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0063] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0064] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0065] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0066] The methods provided in this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0067] For example, Figure 1A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0070] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0071] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0074] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0075] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be housed in the same device. The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0076] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0077] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0078] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0079] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0080] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0081] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0082] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0083] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0084] The SIM card interface 195 is used to connect a SIM card. The electronic device 100 interacts with the network through the SIM card to realize functions such as making calls and data communication.

[0085] It should be understood that the phone cards in the embodiments of this application include, but are not limited to, SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated circuit cards (UICC), etc.

[0086] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0087] Figure 2 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0088] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0089] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0090] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0091] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0092] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0093] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0094] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0095] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0096] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0097] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0098] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0099] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0100] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0101] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0102] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0103] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0104] A 2D graphics engine is a graphics engine for 2D drawing.

[0105] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0106] It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

[0107] The technical solutions of this application embodiment can be applied to electronic devices with screen interaction capabilities, for example. Exemplarily, they can be applied to televisions, desktop computers, laptops, in-vehicle screens, and portable electronic devices such as mobile phones, foldable screens, tablets, cameras, camcorders, video recorders, watches, and wristbands. They can also be applied to other electronic devices with screen interaction capabilities, and to electronic devices in 5G networks or future evolved public land mobile networks (PLMNs). The main application scenario is interaction based on the content displayed on the screen of the electronic device.

[0108] As smart devices become more and more widespread and their functions become more and more abundant, people are becoming more and more dependent on them in their lives and work. There are more and more scenarios in which users need to interact with the content on the screen, which makes people's need for the convenience of interacting with the screen content of smart devices increasingly higher.

[0109] Figure 3 A schematic diagram of the interface corresponding to a method of interaction based on screen content is shown.

[0110] like Figure 3 As shown in (a), the electronic device's display interface shows a picture of a home scene, including objects such as a sofa, television, speakers, and a user; the user can switch the electronic device's display interface to the desired style by long-pressing control 301. Figure 3 The selectable interface shown in (b) is as follows; Figure 3 As shown in (c), in this selectable interface, the user can manually select the target interactive object, such as manually selecting speaker 302; after the selection is completed, the display interface of the electronic device switches to the following... Figure 3 The interface shown in (d) displays multiple interactive results related to speaker 302, such as images, prices, or purchase links of multiple identical or similar speakers.

[0111] In this method, the interactive target on the screen can only be selected after the display interface enters a specific state; and after the interactive target is selected, the interaction process ends after the search results corresponding to the interactive target are displayed. It is not possible to perform multiple rounds of interaction on the interactive target, nor can interactive question and answer be realized.

[0112] Figure 4This shows a schematic diagram of the interface corresponding to yet another method of interaction based on screen content.

[0113] like Figure 4 As shown in (a), the electronic device's display interface is located in the SMS interface. Users can switch the electronic device's display interface to the one shown in (a) by long-pressing the SMS content in the SMS interface. Figure 4 The interface shown in (b) displays multiple application icons, such as calendar, chat, email, and memos. When a user wants to edit and send the text message via email, they can drag the text message onto the email icon while holding down the long press. Furthermore, in response to the user dragging the text message onto the email icon, the electronic device's display switches to the email editing page, where the text message is displayed as edited text.

[0114] This method can only be implemented if the application is adapted to drag and drop functionality. There may be situations where some interactive content such as images, text, or videos cannot be dragged. Furthermore, it cannot select local content on the screen, and the selection of interactive objects is not precise enough. After dragging and dropping, the interaction ends after jumping to the target page. It cannot perform multiple rounds of interaction on the interactive target, nor can it realize interactive question and answer.

[0115] Figure 5 This shows a schematic diagram of the interface corresponding to yet another method of interaction based on screen content.

[0116] like Figure 5 As shown in (a), when the display interface of the electronic device is on the screen 1 interface, the display interface of the electronic device can be switched to the voice interface corresponding to screen 1. In the voice interface, a control 501 for adding the screen is displayed. When the user clicks the control 501, the display interface of the electronic device switches to the screen as shown in (a). Figure 5 The voice interaction interface shown in (b) has a dialog box 502 that displays a screenshot of screen 1. Users can click on the control 503 in the dialog box to interact with the content of screen 1.

[0117] This method does not allow interaction with specific information on the screen.

[0118] Therefore, current screen content interaction methods cannot accurately select local information on the screen, nor can they perform question-and-answer sessions based on local information on the screen.

[0119] In view of this, embodiments of this application provide a method and electronic device for human-computer interaction. Through this method and electronic device, interactive content can be selected on the display screen of the electronic device, and convenient voice interaction between the electronic device and the user can be realized based on the interactive content and applications related to the interactive content on the electronic device. For example, one or more service entry points related to the interactive content can be provided to the user, providing the user with a convenient human-computer interaction method.

[0120] In addition, in the method provided in this application embodiment, before selecting the target interactive object, it is not necessary to switch the screen of the electronic device to the selection state. The interactive content can be selected directly based on the current display screen of the electronic device, which further improves the convenience of interaction between the user and the electronic device.

[0121] Alternatively, the electronic device can adaptively adjust the selection range and determine the interactive content based on the content within the user's selection range, thereby achieving accurate selection of interactive content without requiring the user to frequently manually adjust the interactive area.

[0122] Alternatively, after the first round of voice interaction between the electronic device and the user ends, the electronic device may not end the interaction process, but instead switch to a state of continuing to detect further interaction commands, thereby enabling the electronic device and the user to achieve multi-round voice interaction based on the selected interaction content.

[0123] Alternatively, the interaction result between the electronic device and the user based on the interactive content may not only be based on the interactive content, but may also refer to other non-interactive content located on the same display interface as the interactive content, making the interaction result more accurate.

[0124] For example, Figure 6 The diagram shows an interface corresponding to a method for interacting based on screen content provided in an embodiment of this application.

[0125] like Figure 6 As shown in (a), an image 601 is displayed on the screen of an electronic device. This image 601 includes a display element 602, which may be, for example, a puppy. Figure 6 As shown in (b), when a user wishes to interact with the screen based on the display element 602, the user can select the display element 602 on the screen; after the selection is completed, the display interface of the electronic device switches to the following... Figure 6 The interface shown in (c) is such that the selection area of ​​content on the screen is determined by, for example, Figure 6The area actually selected by the user as shown in (b) is automatically adjusted to the cutout area of ​​display element 602. In addition, multiple recommended service entrances will appear on this interface. For example, optionally, the copy service, save service and share service provided in service box 603; optionally, a voice ball 604 can be displayed on this interface, which is used by the user to interact with the selected content through voice dialogue; optionally, other recommended service entrances can be displayed on this interface, such as image search (e.g., Xiaoyi recognizes objects), how to raise animals, etc.; optionally, more recommended service entrances can be viewed by clicking control 605. This application embodiment does not specifically limit the number and category of recommended service entrances displayed.

[0126] In some embodiments, it is also possible to, for example Figure 6 The interface shown in (b) displays a navigation bar 606, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0127] In one example, after a user selects a display area, they can long-press the navigation bar 606 to activate the voice function of the electronic device. The user can then interact with the selected content by voice through the electronic device.

[0128] In some embodiments, it is also possible to, for example Figure 6 The interface shown in (c) displays one or more application icons, which are associated with the selected interactive content.

[0129] In one example, the electronic device detects that a user clicks on a first application icon among one or more application icons, and then performs the first application operation on the selected content.

[0130] In one example, the electronic device detects a voice command from the user instructing the user to select the first application icon, and then performs the first application operation on the selected content.

[0131] In one example, the electronic device detects that the user drags the first interactive content to the first application icon, and the electronic device performs the first application operation on the selected content.

[0132] In some embodiments, voice interaction between the user and the electronic device based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0133] In this embodiment, when the selected interactive content is a subject, the final selected interactive content is that subject, and the selected area is automatically adjusted to the selected area of ​​that subject.

[0134] For example, Figure 7 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0135] like Figure 7 As shown in (a), an image 701 is displayed on the screen of an electronic device. The image 701 includes display elements 702 and multiple display elements 703. The display element 702 can be, for example, a puppy, and the display elements 703 can be, for example, a tree. Figure 7 As shown in (b), when a user wishes to interact with the screen based on display element 702 and display element 703, the user can select display element 702 and display element 703 on the screen; after the selection is completed, the display interface of the electronic device switches to the following... Figure 7 The interface shown in (c) is such that the selection area of ​​content on the screen is determined by, for example, Figure 7 The area actually selected by the user, as shown in (b), is automatically adjusted to the bounding rectangle of the actual selected area. Multiple recommended service entry points will appear on this interface. For example, optionally, the copy, save, and share services provided in service box 704; optionally, a voice ball 705 can be displayed on this interface, allowing the user to interact with the selected content via voice dialogue; optionally, other recommended service entry points can be displayed on this interface, such as "Little Yi Recognizes Objects" and "How to Raise Animals"; optionally, more recommended service entry points can be viewed by clicking control 706. This embodiment of the application does not specifically limit the number and category of the displayed recommended service entry points.

[0136] In some embodiments, it is also possible to, for example Figure 7 The interface shown in (b) displays a navigation bar 707, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0137] In one example, after a user selects a display area, they can long-press the navigation bar 707 to activate the voice function of the electronic device. The user can then interact with the selected content by voice through the electronic device.

[0138] In some embodiments, voice interaction between the user and the electronic device based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0139] In this embodiment, when the selected interactive content consists of multiple subjects or no subjects, the final selected interactive area is the bounding rectangle of the selected area.

[0140] For example, Figure 8This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0141] like Figure 8 As shown in (a), the screen of the electronic device displays multiple images. When a user wants to interact with the screen based on a portion of these images, the user can select a portion of the images on the screen. After the selection is completed, the display interface of the electronic device switches to the following format: Figure 8 The display interface shown in (b) has a selection area for content on the screen, as shown in [example]. Figure 8 The area actually selected by the user shown in (a) is automatically adjusted to the bounding rectangle of the actual selected area, or automatically adjusted to the bounding rectangle of multiple images included in the actual selected area. In addition, multiple recommended service entrances will appear on the interface. For example, optionally, a voice ball 801 is displayed on the interface, which is used by the user to interact with the selected content through voice dialogue. For example, optionally, other recommended service entrances can be displayed on the interface, such as copying images, sharing images, etc. For example, optionally, more recommended service entrances can be viewed by clicking the control 802. The number and category of recommended service entrances displayed in this embodiment of the application are not specifically limited.

[0142] In some embodiments, it is also possible to, for example Figure 8 The interface shown in (a) displays a navigation bar 803, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0143] In one example, after a user selects a display area, they can long-press the navigation bar 803 to activate the voice function of the electronic device. The user can then interact with the selected content by voice through the electronic device.

[0144] In some embodiments, voice interaction between the user and the electronic device based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0145] In this embodiment, when the selected interactive content consists of multiple images, the final selected interactive area is either the bounding rectangle of the actual selected area or the bounding rectangle of the multiple images included in the actual selected area.

[0146] For example, Figure 9 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0147] like Figure 9As shown in (a), the electronic device displays a chat interface on its screen. When a user wants to interact with the screen based on the chat history in this chat interface, the user can select chat history on the screen; after the selection is completed, the display interface of the electronic device switches to the following... Figure 9 The display interface shown in (b) has a selection area for content on the screen, as shown in [example]. Figure 9 The user's actual selected area shown in (a) is automatically adjusted to the bounding rectangle of the actual selected area, or automatically adjusted to the bounding rectangle of the multiple chat history boxes included in the actual selected area. Furthermore, multiple recommended service entry points will appear on this interface. For example, optionally, the copy service, save service, share service, and edit service provided in service box 901; optionally, a voice ball 902 can be displayed on this interface, which is used for the user to interact with the selected content through voice dialogue; optionally, other recommended service entry points can be displayed on this interface, such as hiding avatars and nicknames; optionally, more recommended service entry points can be viewed by clicking control 903. This embodiment of the application does not specifically limit the number and category of the displayed recommended service entry points.

[0148] In some embodiments, it is also possible to, for example Figure 9 The interface shown in (a) displays a navigation bar 904, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0149] In one example, after a user selects a display area, they can long-press the navigation bar 904 to activate the voice function of the electronic device. The user can then interact with the selected content by voice through the electronic device.

[0150] In some embodiments, voice interaction between the user and the electronic device based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0151] In this embodiment, when the selected interactive content is chat content, the final selected interactive area is the bounding rectangle of the actual selected area, or the bounding rectangle of multiple chat record boxes included in the actual selected area; and when the selected area includes an avatar, the avatar and nickname in the selected interactive content can be hidden with one click using the "Hide Avatar and Nickname" function key.

[0152] In some embodiments, a first image is displayed on the screen of an electronic device, the first image including a first display object, and a user selects a first area on the first image, the first area including all or part of the first display object.

[0153] When the area of ​​the region where the first display object intersects with the first region / MAX(area of ​​the first display object, area of ​​the first region) is greater than the first threshold, the interactive area finally determined by the electronic device is the area of ​​the first display object.

[0154] When the area of ​​the first display object intersecting with the first region / MAX (area of ​​the first display object, area of ​​the first region) is less than the first threshold, and the area of ​​the first image intersecting with the first region / MAX (area of ​​the first image, area of ​​the first region) is greater than the second threshold, the interactive area finally determined by the electronic device is the first image area.

[0155] When the area of ​​the first display object intersecting with the first region / MAX(area of ​​the first display object, area of ​​the first region) is less than the first threshold, and the area of ​​the first image intersecting with the first region / MAX(area of ​​the first image, area of ​​the first region) is less than the second threshold, the interactive area finally determined by the electronic device is the bounding rectangle area of ​​the first region.

[0156] The first threshold can be any value in the range of 0 to 1, such as 0.8 or 0.9; the second threshold can be, for example, 0.8 or 0.9.

[0157] It is understandable that the above... Figures 6 to 9 In the illustrated embodiment, the user selects interactive content on the screen by drawing circles on the screen with their knuckles. However, this does not limit the method of selecting interactive content on the screen in this embodiment. The user can also select interactive content on the screen by drawing circles on the screen with their fingers, such as by pressing hard to select or by pressing hard and then selecting; the user can also select interactive content on the screen by drawing circles on the screen with a stylus, such as by pressing hard to select or by pressing hard and then selecting; the user can also select interactive content on the screen by drawing circles on the screen with other limbs; in addition, the user can select interactive content on the screen in other possible ways, such as by using a mouse, etc., which are not limited in this embodiment.

[0158] It should be noted that: the above Figures 6 to 9 In the embodiments shown, the example is that the user selects interactive content by circling on the screen, but this does not limit the method of selecting interactive content in the embodiments of this application.

[0159] In some embodiments, users can select interactive content by checking boxes on the screen. For example, if the screen displays multiple selectable images and / or multiple selectable texts, users can select these images and / or texts to define the interactive content.

[0160] In some other embodiments, users can select interactive content by long-pressing or double-clicking content on the screen. For example, if the screen displays an article reading page containing tables, text, and images, users can long-press or double-click a table to select it as interactive content, long-press or double-click a paragraph of text to select it as interactive content, or long-press or double-click an image to select it as interactive content.

[0161] In some other embodiments, users can select interactive content by dragging and dropping content on the screen. For example, the screen displays an article reading page containing tables, text, and images. Users can select an image as interactive content by dragging it and directly drag the selected content to the target interactive application. A single drag-and-drop operation triggers both the selection of interactive content and the activation of the interactive action. For instance, after a user triggers the dragging of an image, multiple interactive application icons are displayed on the screen, such as icons for gallery, email, notes, and chat applications. If the user drags the image to the gallery icon, the image is automatically saved to the gallery. If the user drags the image to the notes icon, the image is automatically saved to the notes app. If the user drags the image to the chat application icon, the electronic device prompts the user to select the recipient to whom the image is to be shared. After the user selects the recipient, the image is sent to that recipient.

[0162] In addition, users can select interactive content in other possible ways, such as issuing instructions via voice, and the electronic device recognizing and analyzing the user's voice content, and selecting interactive content from multiple displayed contents on the screen based on the voice content analysis results.

[0163] Depend on Figures 6 to 9 As can be seen from the embodiments shown, the method provided by this application can intelligently determine the final interactive content selection area by combining the size of the user-selected area and the specific content within the selected area, thereby avoiding frequent adjustments to the selected area by the user during the selection process and improving the user experience.

[0164] For example, Figure 10The illustration shows interface diagrams corresponding to two methods for interacting with the screen based on images on the screen provided in the embodiments of this application.

[0165] like Figure 10 As shown in (a), the screen of the electronic device displays the cover image of an e-book titled "Title One". When a user wants to interact with the screen based on the e-book titled "Title One", the user can select the e-book titled "Title One". In response to the user's selection of the e-book titled "Title One", the screen of the electronic device can display service entry points related to the e-book titled "Title One". For example, optionally, service box 1001 can be displayed, which is used by the user to select services such as "save", "copy", "edit", and "graphic selection". For example, optionally, voice ball 1002 can be displayed, which is used by the user to interact with the selected content through voice dialogue. For example, optionally, other recommended service entry points, such as audiobooks and book introductions, can be displayed on the interface. For example, optionally, more service entry points related to the e-book titled "Title One" can be viewed by clicking control 1003.

[0166] In some embodiments, it is also possible to, for example Figure 10 The interface shown in (a) displays one or more application icons, which are associated with the selected content.

[0167] Among them, an exemplary operation that triggers interaction between the user and the electronic device based on selected content through one or more application icons can be found in [reference to...]. Figure 6 The description of the embodiments shown is omitted here for the sake of brevity.

[0168] In some embodiments, voice interaction between the user and the electronic device based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0169] like Figure 10As shown in (b), the screen of the electronic device displays a home leisure scene image. This home leisure scene image may include display objects such as a television, speakers, sofa, and users. When a user wants to interact with the screen based on the display object "speaker" in the home leisure scene image, the user can select the display object "speaker". In response to the user's selection of the display object "speaker", the screen of the electronic device displays service entry points related to the display object "speaker". For example, optionally, a voice ball 1004 can be displayed, which is used for the user to interact with the selected content through voice dialogue. Alternatively, other recommended service entry points can be displayed on the interface, such as Xiaoyi's object identification and product black technology. Alternatively, more service entry points related to the display object "speaker" can be viewed by clicking the control 1005.

[0170] In some embodiments, it is also possible to, for example Figure 10 The interface shown in (a) displays a navigation bar 1006, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0171] In one example, after a user selects a display area, they can long-press the navigation bar 1006 to activate the voice function of the electronic device, allowing the user to interact with the selected content through voice.

[0172] In some embodiments, it is also possible to, for example Figure 10 The interface shown in (b) displays a navigation bar 1007, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0173] In one example, after a user selects a display area, they can long-press the navigation bar 1007 to activate the voice function of the electronic device, allowing the user to interact with the selected content through voice.

[0174] In some embodiments, interaction with the selected content can also be triggered by pressing and holding the power button of the electronic device or by voice activation.

[0175] For example, Figure 11 The illustration shows interface diagrams corresponding to several methods for interacting with the screen based on content or tables provided in the embodiments of this application.

[0176] like Figure 11As shown in (a), the screen of the electronic device displays multiple text contents, such as a page explaining a contract or transaction. The multiple text contents displayed on this page include bank account information, bank information, and payee information. When a user wishes to interact with the screen based on this bank account information, bank information, and payee information (e.g., the user wishes to transfer money to the account or record the account information), the user can select the bank account information, bank information, and payee information. In response to the user's selection of the bank account information, bank information, and payee information, the screen of the electronic device displays service entry points related to the bank account information, bank information, and payee information. For example, optionally, a voice ball 1101 can be displayed, which is used for the user to interact with the selected content through voice dialogue; alternatively, the interface can display... The system displays other recommended service entry points, such as transfer and note-taking. When a user clicks the transfer service entry, the electronic device's display screen redirects to the interface for transferring funds to the selected bank account through a banking application. When a user clicks the note-taking service entry, the electronic device's display screen redirects to the recording interface in the memo. The selected bank account information, bank information, and payee information are displayed in an edited format on this recording interface. Alternatively, when a user clicks the note-taking service entry, the selected bank account information, bank information, and payee information are automatically recorded in the memo. The electronic device may notify the user on the current page that a memo has been generated, or it may not notify the user, resulting in a seamless user experience. The user can then view the memo later. Users can also click on control 1102 to view more service entry points related to the bank account information, bank information, and payee information.

[0177] In some embodiments, the above-mentioned selection of the bank account information, the opening bank information, and the payee information can be replaced by selecting the bank account information; or by selecting the bank account information and the opening bank information; or by selecting the bank account information and the payee information; or by selecting the bank account information, the opening bank information, and the payee information; the corresponding interaction process is similar to the interaction process corresponding to selecting the bank account information, the opening bank information, and the payee information.

[0178] In some embodiments, if the selected content includes bank account information and the electronic device has the China Merchants Bank application installed, then the corresponding China Merchants Bank transfer service entry will be displayed.

[0179] like Figure 11As shown in (b), the screen of the electronic device displays multiple text contents, such as a reading page of a public account article. The multiple text contents displayed on this page include URL information. When a user wants to interact with the screen based on this URL information (e.g., the user wants to jump to the URL or wants to save the URL), the user can select the URL information. In response to the user's selection of the URL information, the screen of the electronic device displays a service entry related to that URL information. For example, optionally, a voice ball 1103 is displayed, which is used for the user to interact with the selected content through voice dialogue. Alternatively, the interface may display... The device displays other recommended service entry points, such as "Open URL" and "URL Summary." When a user clicks the "Open URL" service entry, the device's display redirects to the page corresponding to that URL. When a user clicks the "URL Summary" service entry, the device's display shows a summary of the URL. Users can also click control 1104 to view more service entry points related to the URL. For example, when a user clicks the "Remember" service entry, the URL is automatically recorded in the memo. The device can either notify the user on the current page that the URL information has been recorded in the memo, or it can remain silent, providing a seamless user experience where the user can later view it in the memo.

[0180] In this context, a URL summary refers to the abstract content of the webpage corresponding to that URL. Through this summary, users can quickly understand the key points of the webpage content, similar to the abstract of a thesis. Users can also read the URL summary before opening the webpage to decide whether they need to open it.

[0181] like Figure 11As shown in (c), the screen of the electronic device displays multiple text contents, such as an event announcement page or a schedule information page. The multiple text contents displayed on this page include address information. When a user wants to interact with the screen based on this address information (e.g., the user wants to navigate to the navigation page corresponding to the address or wants to save the address), the user can select the address information. In response to the user's selection of the address information, the screen of the electronic device displays service entry points related to that address information. For example, optionally, a voice ball 1105 can be displayed, which allows the user to interact with the selected content through voice dialogue. For example, the user can ask how far the address is from the user's current location via voice. Alternatively, other recommended services can be displayed on this interface. The system includes various service entry points, such as distance lookup, navigation, and ride-hailing. When a user clicks the distance lookup service entry, the electronic device's display redirects to a map page showing the distance between the user's current location and the address. Clicking the navigation service entry redirects to a navigation page leading to the address, and clicking the ride-hailing service entry redirects to a ride-hailing page. Users can also access more service entry points related to the address by clicking control 1106. For example, clicking the "Remember" service entry automatically saves the address to the memo. The electronic device may notify the user on the current page that the address information has been saved, or it may not, providing a seamless user experience where the user can later check the memo.

[0182] like Figure 11As shown in (d), the screen of the electronic device displays multiple text contents, such as a registration order page. Among the multiple text contents displayed on this page is appointment time information. When a user wants to interact with the screen based on this appointment time information (e.g., the user wants to navigate to the calendar page corresponding to the appointment time or record the appointment time as a reminder), the user can select the appointment time information. In response to the user's selection of the appointment time information, the screen of the electronic device displays service entry points related to that appointment time information. For example, optionally, a voice ball 1107 can be displayed, which allows the user to interact with the selected content through voice dialogue. For example, the user can ask how many days are left until the appointment time via voice. Alternatively, other recommended service entry points can be displayed on this interface, such as adding an appointment or making a note. When the user clicks the "Add Appointment" service entry, the electronic device... The display screen redirects to the schedule addition page, where the appointment time information is displayed as an added schedule. Alternatively, when the user clicks the "Add Schedule" service entry, a reminder pop-up appears on the current page to inform the user that the appointment time information has been added to the schedule, or it may be automatically added to the schedule in the background without notification. This provides a seamless user experience, and the user can check it later in the notes app. When the user clicks the "Record" service entry, the electronic device's display screen redirects to the notes page, where the appointment time information is displayed as recorded information. Alternatively, when the user clicks the "Record" service entry, a reminder pop-up appears on the current page to inform the user that the appointment time information has been recorded in the notes app, or it may be automatically generated as a notes app in the background without notification. This provides a seamless user experience, and the user can check it later in the notes app. Users can also click control 1108 to view more service entry points related to the appointment time information.

[0183] In some embodiments, when the selected content does not include a specific date but includes a schedule intent, the Add Schedule service will also be displayed. For example, if the selected content includes the information "Let's have dinner together tomorrow", the Add Schedule service will be displayed accordingly. If the selected content includes the information "It will rain tomorrow", the Add Schedule service will not be displayed because there is no schedule intent.

[0184] like Figure 11As shown in (e), a chat page is displayed on the screen of the electronic device. The chat page displays multiple chat messages, including flight number information. When a user wants to interact with the screen based on this flight number information (e.g., the user wants to jump to the flight details page corresponding to the flight number or wants to book a flight corresponding to the flight number), the user can select the flight number information. In response to the user's selection of the flight number information, the screen of the electronic device displays service entry points related to the flight number information. For example, optionally, a voice ball 1109 can be displayed, which is used for the user to interact with the selected content through voice dialogue. For example, the user can ask the departure time of the flight number by voice. Alternatively, other recommended service entry points can be displayed on this interface, such as booking tickets and flight search. When the user clicks the booking ticket service entry, the display interface of the electronic device jumps to the ticket booking page corresponding to the flight number; when the user clicks the flight search service entry, the display interface of the electronic device jumps to the flight search page corresponding to the flight number. The user can also click the control 1110 to view more service entry points related to the flight number information.

[0185] like Figure 11 As shown in (f), the screen of the electronic device displays a text reading page, such as a paper reading page or a courseware browsing page. This page displays multiple text messages and tables. When a user wants to interact with the screen based on a table (e.g., the user wants to record information from the table), the user can select the table. In response to the user's selection of the table, the screen of the electronic device displays service entry points related to the table. For example, optionally, a service box 1111 can be displayed, which includes copy, save, and share services. The user can copy the table by clicking the copy service, save the table by clicking the save service, and share the table by clicking the share service. Alternatively, optionally, a voice ball 1112 can be displayed, which allows the user to interact with the selected content through voice dialogue. Alternatively, other recommended service entry points, such as table extraction and table analysis, can be displayed on this interface. The user can also click control 1113 to view more service entry points related to the table.

[0186] It is understandable that the above... Figure 10 and Figure 11In the illustrated embodiment, the user selects interactive content on the screen by drawing circles on the screen with their knuckles. However, this does not limit the method of selecting interactive content on the screen in this embodiment. The user can also select interactive content on the screen by drawing circles on the screen with their fingers, such as by pressing hard to select or by pressing hard and then selecting; the user can also select interactive content on the screen by drawing circles on the screen with a stylus; the user can also select interactive content on the screen by drawing circles on the screen with other limbs, such as by pressing hard to select or by pressing hard and then selecting with a stylus; in addition, the user can select interactive content on the screen in other possible ways, such as with a mouse, etc., which are not limited in this embodiment.

[0187] It should be noted that: the above Figure 10 and 11 In the embodiments shown, the example is that the user selects interactive content by circling on the screen, but this does not limit the method of selecting interactive content in the embodiments of this application.

[0188] In some embodiments, when a user selects a recommended service, the screen of the electronic device can remain on the current interface, and the page to which the user is redirected can be presented in the form of a pop-up window or a floating window.

[0189] Depend on Figure 10 and Figure 11 As can be seen from the embodiments shown, in the method provided by this application, when different content is selected on the screen, the page will make service function recommendations corresponding to the selected content; furthermore, the user can intelligently select any part of the content currently displayed on the screen, and the service function recommendations made are adapted to the selected part of the content, which can increase the probability of the user clicking the recommended service function; moreover, the displayed service function recommendations can also be determined by intent analysis of the selected content, which can improve the adaptability of the service function recommendations to the selected content, and further increase the probability of the user clicking the recommended service function.

[0190] For example, Figure 12 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0191] like Figure 12As shown in (a), the screen of the electronic device displays a chat interface with multiple chat messages. Each chat message received by the electronic device includes bank account information, bank information, and payee information. When a user wishes to transfer funds based on this bank account information, bank information, and payee information, the user can select these three information items. Figure 12 As shown in (b), in response to the user's selection of the bank account information, bank information, and payee information, the selected information is highlighted, and service entry points related to the bank account information, bank information, and payee information are displayed on the interface. Optionally, these service entry points may include functions such as transferring funds or copying all text. When the user clicks the transfer service entry point, the electronic device's display interface redirects to a screen as shown in (b). Figure 12 The interface shown in (c) is for transferring money to a recipient through a bank application. In this interface, the bank account information, bank information, and recipient information selected by the user have been automatically filled into the corresponding positions. The user only needs to verify the relevant information and enter the transfer amount to complete the transfer to the recipient.

[0192] For example, Figure 13 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0193] like Figure 13 As shown in (a), the electronic device displays a chat interface on its screen, showing multiple chat messages. One of the chat messages received by the electronic device is "I'm taking flight AB1234 at 6 PM the day after tomorrow." When a user wants to check the specific information of this flight, the user can select the chat message "I'm taking flight AB1234 at 6 PM the day after tomorrow." In response to the user's selection of the chat message, service entry points related to this chat message are displayed on the interface of the electronic device. Optionally, these service entry points may include flight search, keyword selection panel, etc. When the user clicks on the flight search service entry point, the display interface of the electronic device jumps to, for example,... Figure 13The interface shown in (b) displays a dialog box 1301, which shows the selected chat message "I'm taking flight AB1234 at 6 PM the day after tomorrow," the user's instruction "Flight Inquiry," and the interaction result determined by the electronic device based on the chat message "I'm taking flight AB1234 at 6 PM the day after tomorrow" and the user's instruction "Flight Inquiry." The interaction result is the information of flight AB1234 found by the electronic device. The information of flight AB1234 can be displayed in the form of a service card. When the user clicks on the service card, the display interface of the electronic device can jump to the information interface of flight AB1234 in the flight application. In this interface, the user can also continue to interact with the chat message through the voice ball 1302 or the keyboard 1303.

[0194] For example, Figure 14 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0195] like Figure 14 As shown in (a), the screen of the electronic device displays multiple text contents including bank account information, bank information, and payee information. When the user wants the electronic device to remember the bank account information, bank information, and payee information, the user can select these information. In response to the user's selection, service entry points related to the bank account information, bank information, and payee information are displayed on the interface. For example, these service entry points may optionally include transfer, remember account, etc. When the user clicks the remember account service entry point, the display interface of the electronic device jumps to, for example,... Figure 14 The interface shown in (b) displays a dialog box 1401, which shows the selected bank account information, bank information, payee information, the user's instruction "Note the account", and the interaction result determined by the electronic device based on the selected bank account information, bank information, payee information, and the user's instruction "Note the account". The interaction result is the bank account information, bank information, and payee information recorded by the electronic device. This recorded information can be displayed in the form of a service card. The electronic device can also remind the user to select the bank application that they wish to use for future transfers to this account. The user can select the bank application that they wish to use for future transfers to this account through the "Go to Edit" control on the service card.

[0196] For example, Figure 15 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0197] like Figure 15 As shown in (a), an image 1501 is displayed on the screen of the electronic device. The image 1501 includes a display element 1502, which may be, for example, a puppy. When a user wishes to interact with the screen based on the display element 1502, after selecting the display element 1502 as the interaction content, the user may, for example, click on the navigation bar 1503 in the interface. In response to the user clicking the navigation bar 1503, the display interface of the electronic device appears as follows: Figure 15 As shown in (b), optionally, the interface may display a voice recognition box 1504, in which a prompting message for guiding the user to ask a voice question may be displayed, such as "Ask about the selected object on the screen..."; Figure 15 As shown in (c), in response to a user issuing a voice question, the voice recognition box 1504 displays the user question recognized by the electronic device. For example, optionally, when the user issues a voice question, "What breed of dog is this?", the voice recognition box 1504 displays the recognized user question, "What breed of dog is this?"; then... Figure 15 As shown in (d), a dialog box 1505 can be displayed on the interface of the electronic device. In the dialog box 1505, the user's selected content and question content are displayed, as well as the electronic device's answer to the user's question.

[0198] In some embodiments, the dialog box 1505 may be displayed on the original display interface of the electronic device in the form of a floating window or a pop-up window, so as to reduce the impact of the display of the dialog box 1505 on the user's browsing of the original interface of the electronic device.

[0199] In some embodiments, voice interaction between the electronic device and the user based on selected content can also be triggered by pressing and holding the power button of the electronic device or by voice wake-up.

[0200] The navigation bar is a control displayed on the interface of an electronic device that is independent of the displayed content. The navigation bar is not canceled when the display interface of the electronic device is switched. In other words, the navigation bar can always be displayed while the screen of the electronic device is on.

[0201] In some embodiments, in response to a user's selection of interactive content, a navigation bar 1503 is triggered to be displayed on the screen of the electronic device.

[0202] In some embodiments, the navigation bar 1503 is always displayed on the screen of the electronic device throughout the entire process of the electronic device engaging in voice interaction with the user based on interactive content. Optionally, in response to detecting that the voice interaction between the electronic device and the user has ended based on interactive content, the electronic device may de-display the navigation bar 1503.

[0203] In some embodiments, the display conditions of the navigation bar can be set through the settings function of the electronic device. For example, it can be set to always be displayed, or it can be set to be displayed on the interface of certain applications, or it can be set to be displayed throughout the process of the electronic device interacting with the user based on interactive content via voice.

[0204] For example, Figure 16 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0205] like Figure 16 As shown in (a), an image 1601 is displayed on the screen of an electronic device. This image 1601 includes a display element 1602 and multiple display elements 1603. The display element 1602 can be, for example, a puppy, and the display elements 1603 can be, for example, a tree. When a user wishes to interact with the screen based on the display element 1602, after selecting the display element 1602 as the interaction content, the user can click on the voice ball 1604 in the interface. Figure 16 As shown in (b), optionally, in response to a user clicking the voice ball 1604, a voice recognition box 1605 can be displayed on the electronic device's display interface. This voice recognition box 1605 can display a prompt to guide the user to ask a voice question, such as "Ask about the selected object on the screen...". Figure 16 As shown in (c), optionally, in response to a user uttering a voice question, the voice recognition box 1605 displays the user question recognized by the electronic device. For example, when the user utters the voice question "What breed of dog is this?", the voice recognition box 1605 displays the recognized user question "What breed of dog is this?"; then as... Figure 16 As shown in (d), optionally, a dialog box 1606 is displayed on the interface of the electronic device, in which the user's selected content and question content are displayed, as well as the electronic device's answer to the user's question; based on this, as Figure 16As shown in (e), the user can continue to click on the voice ball 1604 to ask further questions based on the selected content, and the electronic device will answer the user's further questions. Alternatively, the user can reselect the displayed elements on the screen and ask further questions based on the reselected content, and the electronic device will answer the user's further questions.

[0206] In some embodiments, the dialog box 1606 may be displayed on the original display interface of the electronic device in the form of a floating window or a pop-up window, so as to reduce the impact of the display of the dialog box 1606 on the user's browsing of the original interface of the electronic device.

[0207] It can be understood that when a user selects a partial display element in an image and interacts with the screen via voice based on that selected partial display element, the response determined by the electronic device can be determined based on the selected partial display element, or it can be determined based on the selected partial display element and all display elements currently displayed on the screen (i.e., the selected display element and the unselected display elements). On this basis, it can also be determined by combining logical analysis, intent analysis, semantic analysis, etc.

[0208] In such Figure 16 (d) and such Figure 16 In the example shown in (e), for example, when a user asks "What breed of dog is this?", the electronic device searches based on the selected display element 1602 and provides the answer "This dog is ×××" based on the search results. It can also provide the user with more information about display element 1602, such as "This breed of dog comes from ×××, and has a gentle temperament...". Based on this, the user can further ask "Where is this dog?", and the electronic device determines the answer "This dog is in the woods..." based on all the display elements included in image 1601.

[0209] For further explanation of this embodiment, please refer to [link to relevant documentation]. Figures 6 to 11 The description section of the illustrated embodiment.

[0210] In this embodiment, after selecting interactive content on the screen, users can interact with the screen of the electronic device via voice based on the selected content. This enables multi-round voice interaction between the selected content and the screen, and also allows for multi-round voice interaction between the selected partial content on the screen and the screen, improving the user experience. Furthermore, when the selected content is a partial display element, the electronic device's response can be determined based on the analysis of all display elements on the page, improving the accuracy of the response.

[0211] For example, Figure 17This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0212] like Figure 17 As shown in (a), a text reading page is displayed on the screen of the electronic device. When a user wishes to interact with the screen based on the display element 1701 on the page, the user can, for example, invoke service entry points related to the display element 1701 by circling the display element 1701. Optionally, these service entry points may include the voice ball 1702, Xiaoyi's object recognition service, how to raise animals service, copy service, save service, share service, etc. Figure 17 As shown in (b), optionally, in response to a user clicking the voice ball 1702, a voice recognition box 1703 is displayed on the electronic device's display interface. This voice recognition box 1703 may display a prompt to guide the user to ask a voice question, such as "Ask about the selected object on the screen." Figure 17 As shown in (c), optionally, in response to a user uttering a voice question, the voice recognition box 1703 displays the question recognized by the electronic device. For example, when the user utters the voice question "Is this dog smart?", the voice recognition box 1703 displays the recognized question "Is this dog smart?"; then as... Figure 17 As shown in (d), optionally, a dialog box 1704 is displayed on the interface of the electronic device. In the dialog box 1704, the user's selected content (display element 1701) and question content (Is this dog smart?) are displayed, as well as the electronic device's answer to the user's question. The answer content may be, for example, "This dog has services such as music playback, whole-house smart control, and intelligent dialogue...".

[0213] In some embodiments, such as Figure 17 Based on the embodiment shown in (d), the user can continue to click on the voice ball 1702 to ask questions based on the selected display element 1701, and the electronic device will answer the user's further questions. Alternatively, the user can reselect the display element on the screen and ask questions based on the reselected content, and the electronic device will answer the user's further questions.

[0214] It can be understood that when a user selects a local display element on the screen and interacts with the screen via voice based on that selected local display element, the response determined by the electronic device can be determined based on the local display element selected by the user, or based on all display elements in the entire display interface (i.e., selected and unselected display elements), or based on the entire document content corresponding to the currently displayed document (including selected and unselected display elements, as well as other relevant content not displayed on the screen). On this basis, it can also be determined by combining logical analysis, intent analysis, semantic analysis, etc.

[0215] In one example, on a paper reading page, a user selects an image, a passage of text, or a combination of both as the content for voice interaction. The electronic device provides an answer based on the user's question, which can be determined by considering the selected content, the entire paper, or all the content currently displayed on the screen.

[0216] Similarly, in scenarios such as reading articles or novels, the responses obtained through voice interaction can be determined by the electronic device based on the selected content, combined with the overall content (including content displayed on the screen and related content not displayed on the screen) or by combining all content currently displayed on the screen.

[0217] In this embodiment, after selecting interactive content on the screen, users can interact with the screen of the electronic device via voice based on the selected content. This enables multi-round voice interaction between the selected content and the screen, and also allows for multi-round voice interaction between the selected partial content on the screen and the screen, improving the user experience. Furthermore, when the selected content is a partial display element, the electronic device's response can be determined based on the analysis of all display elements on the page, improving the accuracy of the response.

[0218] For example, Figure 18 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0219] like Figure 18 As shown in (a), a text reading page is displayed on the screen of the electronic device; when a user wishes to interact with the screen based on display element 1801 on the page, the user can select display element 1801 to determine the interaction content as display element 1801; furthermore, the user can activate the voice function using a wake word, for example, such as... Figure 18As shown in (b), optionally, the user can activate the voice function of the electronic device by using the wake word "Xiaoyi Xiaoyi," causing a voice recognition box 1802 to be displayed on the interface of the electronic device. This voice recognition box 1802 can display a guiding phrase to instruct the user to give voice commands, such as "I'm here! How can I help you?" Optionally, the electronic device can also pronounce the guiding phrase via voice. Figure 18 As shown in (c), optionally, in response to a user's voice question, the electronic device can display a dialog box 1803 on its screen after recognizing the user's question. The dialog box 1803 displays the selected content (display element 1801), the question (e.g., "Is this dog smart?"), and the answer given by the electronic device to the user's question. The answer could be, for example, "This dog has services such as music playback, whole-house smart control, and smart dialogue..." The electronic device can also give the answer via voice.

[0220] In some embodiments, after the user selects display element 1801, the voice ball may not be displayed on the display interface of the electronic device.

[0221] In some embodiments, such as Figure 18 Based on the embodiment shown in (c), for example, optionally, the user can continue to ask questions based on the selected display element 1801 or by clicking the voice ball, and the electronic device will answer the user's further questions.

[0222] It can be understood that when a user selects a local display element on the screen and interacts with the screen via voice based on that selected local display element, the response determined by the electronic device can be determined based on the local display element selected by the user, or based on all display elements in the entire display interface (i.e., selected and unselected display elements), or based on the entire document content corresponding to the currently displayed document (including selected and unselected display elements, as well as other relevant content not displayed on the screen). On this basis, it can also be determined by combining logical analysis, intent analysis, semantic analysis, etc.

[0223] In this embodiment, after selecting interactive content on the screen, the user can directly initiate voice interaction with the screen of the electronic device based on the selected interactive content by using a wake word, without the need for the user to manually click the voice ball, which can further free up the user's hands and improve the user's ease of use.

[0224] For example, Figure 19This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0225] like Figure 19 As shown in (a), a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on the display element 1901 on this page, the user can directly activate the voice function of the electronic device using a wake word, for example, such as... Figure 19 As shown in (b), optionally, for example, the user can activate the voice function of the electronic device by using the wake word "Xiaoyi Xiaoyi," causing a voice recognition box 1902 to be displayed on the interface of the electronic device. This voice recognition box 1902 can display a guiding phrase to instruct the user to give voice commands, such as "I'm here! How can I help you?" Optionally, the electronic device can also pronounce the guiding phrase via voice. Figure 19 As shown in (c), optionally, in response to a user issuing a voice question, the electronic device may display a dialog box 1903 on its screen after recognizing the user's question. In this dialog box 1903, the electronic device displays options based on the user's question, such as... Figure 19 The target interactive content (display element 1901) and the question content (e.g., is this dog smart?) extracted from the display interface shown in (a) are displayed, as well as the answer content given by the electronic device in response to the user's question. The answer content may be, for example, "This dog has services such as music playback, whole-house smart control, and smart dialogue..." The electronic device can also issue the answer content by voice.

[0226] In some embodiments, such as Figure 19 In (c) of the above, the voice ball may not be displayed on the display interface of the electronic device.

[0227] In some embodiments, such as Figure 19 Based on the embodiment shown in (c), the user can continue to ask questions based on display element 1901 or other display elements by using a wake word or clicking the voice ball, and the electronic device will provide answers to the user's further questions.

[0228] It is understandable that the answer determined by an electronic device can be determined based on all the display elements in the entire display interface. On this basis, it can also be determined by combining logical analysis, intent analysis, semantic analysis, etc.

[0229] In this embodiment, when a user wants to interact with the screen based on the local content displayed on the screen, there is no need to select the interactive content on the screen. The user can directly initiate voice interaction with the screen of the electronic device by using a wake word. The electronic device can determine the corresponding answer based on the content of the user's question and all the display elements on the screen. There is no need for the user to manually click the voice ball, which can further free the user's hands and improve the user's ease of use.

[0230] For example, Figure 20 This illustration shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided in an embodiment of this application.

[0231] like Figure 20 As shown in (a), a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on the display element 2001 on this page, the user can directly activate the voice function of the electronic device using a wake word, such as optionally, as Figure 20 As shown in (b), the user can activate the voice function of the electronic device by using the wake word "Xiao Yi Xiao Yi," causing a voice recognition box 2002 to be displayed on the interface of the electronic device. This voice recognition box 2002 can display a guiding phrase to instruct the user to give voice commands, such as "I'm here! How can I help you?" Optionally, the electronic device can also issue the guiding phrase via voice. Figure 20 As shown in (c), optionally, in response to a user's voice question, if the electronic device cannot determine the target interaction content based on the user's question, a prompt can be displayed in the voice recognition box 2002 to remind the user to circle the target interaction object, such as "Please circle the target object you need to know!"; Figure 20 As shown in (d), in response to the user's operation of selecting the target interactive object (display element 2001), a dialog box 2003 is displayed on the screen of the electronic device. In the dialog box 2003, the selected content (display element 2001) and the question content (e.g., is this dog smart?) are displayed, as well as the answer content given by the electronic device to the user's question. The answer content may be, for example, "This dog has music playback, whole-house smart control, intelligent dialogue and other service functions..." At the same time, the electronic device can also issue the answer content by voice.

[0232] In some embodiments, such as Figure 20 In (c) of the above, the voice ball may not be displayed on the display interface of the electronic device.

[0233] In some embodiments, such as Figure 20Based on the embodiment shown in (c), the user can continue to ask questions based on display element 2001 or other display elements by using a wake word or clicking the voice ball, and the electronic device will provide answers to the user's further questions.

[0234] It is understandable that the answer determined by an electronic device can be determined based on all the display elements in the entire display interface. On this basis, it can also be determined by combining logical analysis, intent analysis, semantic analysis, etc.

[0235] It should be noted that: the above Figures 15 to 20 In the illustrated embodiment, when the user manually selects interactive content, the example is taken by the user selecting interactive content by circling it on the screen. However, this does not limit the method of selecting interactive content in the embodiments of this application.

[0236] In some embodiments, when a user interacts with partial content on the screen by waking up with a wake word and through voice interaction, the user's voice interaction is not limited to voice dialogue. It can also include specific interactive actions performed by the user through voice. The interaction content can be triggered to interact with other applications through voice commands. For example, the interaction content can be recorded as a memo through voice commands; the interaction content can be sent to a friend through email, chat, or SMS through voice commands; the interaction content can be recorded as a reminder in the schedule through voice commands, etc.

[0237] It should be understood that the above Figures 6 to 20 The embodiments shown can be implemented individually or in any combination of two or more.

[0238] For example, Figure 21 A schematic flowchart illustrating a human-computer interaction method provided in an embodiment of this application is shown. Figure 21 As shown, the method includes:

[0239] S2101: In response to detecting that a user selects an interactive area on the first interface of the electronic device, the electronic device obtains first interactive content associated with the interactive area.

[0240] S2102: The electronic device receives the first voice command.

[0241] In some embodiments, in response to a user selecting an interactive area on a first interface, a voice icon is displayed on the first interface; in response to detecting a user's touch operation on the voice icon, the electronic device acquires the first voice command.

[0242] In some embodiments, the voice icon may be a voice ball.

[0243] In some embodiments, in response to a wake word issued by the user, the electronic device acquires the first voice command, wherein the wake word may be, for example, "Hey Celia!".

[0244] In some embodiments, in response to a user's touch operation on the first navigation bar, the electronic device acquires the first voice command, and the first interface includes the first navigation bar.

[0245] S2103: The electronic device performs a first operation based on the first voice command and the first interactive content.

[0246] In some embodiments, the electronic device performs the first operation based on the first voice command, the first interactive content, and the first display content, wherein the first display content is the content on the first interface of the electronic device other than the first interactive content.

[0247] In some embodiments, the first operation is for an electronic device to display one or more recommended interactive services, which are related to the first interactive content; in response to a user's selection of a first recommended interactive service, the electronic device executes the first recommended interactive service on the first interactive content, and the one or more recommended interactive services include the first recommended interactive service.

[0248] The user's action of selecting the first recommended interactive service can be a user clicking on the first recommended interactive service, a user issuing a second voice command to indicate the selection of the first recommended interactive service, or a user dragging and dropping the first interactive content to the first recommended interactive service.

[0249] In some embodiments, the electronic device may directly perform a first operation, which is used to provide a recommended interactive service that is related to the first interactive content, or the electronic device may display an interface for performing the recommended interactive service.

[0250] Optionally, after the electronic device directly performs the first operation, a first reminder can be displayed on the interface of the electronic device. The first reminder is used to remind the user that the first operation has been performed on the first interactive content.

[0251] In some embodiments, in response to the completion of the first operation by the electronic device, the electronic device is triggered to acquire and detect a voice command for multimodal fusion with the first interactive content again; the electronic device performs a second operation based on the result of the first operation, the voice command and the first interactive content.

[0252] One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0253] When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

[0254] When the modules or units described herein are implemented using software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0255] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0256] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0258] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0260] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for human-computer interaction, characterized in that, The method includes: In response to a user selecting an interactive area on the first interface of an electronic device, the electronic device acquires first interactive content associated with the interactive area. After the operation of selecting the interactive area occurs, the electronic device enters a first state; In this case, the voice commands obtained by the electronic device based on voice detection in the first state are not used independently to determine the subsequent operation of the electronic device, but are used to perform multimodal fusion with the first interaction content to determine the subsequent operation. In the first state, the electronic device obtains a first voice command based on voice detection; The electronic device performs a first operation based on the first voice command and the first interactive content. The first operation is used to provide a recommended interactive service, and the recommended interactive service is related to the first interactive content. Wherein, after the operation of selecting the interactive area occurs, the electronic device enters a first state, including: In response to the operation of selecting the interactive area, the electronic device is triggered to enter the first state; or In response to the operation of selecting the interactive area, the electronic device is triggered to display a second interface, the second interface including a first control, the first control being used to trigger the electronic device to enter the first state; In response to the user's operation on the first control, the electronic device is triggered to enter the first state; The recommendation interaction service is related to the first interaction content, including: When the first interactive content includes an image, the recommended interactive service includes one or more of image recognition service, sharing service, and saving service; or When the first interaction content includes chat history, the recommended interaction service includes one or more of the following: copy service, share service, save service, and edit service; or When the first interactive content includes text content, the recommended interactive service includes one or more of the following: copy service, read service, save service, and edit service; or When the first interactive content includes account information, the recommended interactive service includes a recording service and / or a transfer service; or When the first interactive content includes URL information, the recommended interactive service includes one or more of the following: recording service, sharing service, opening URL service, and viewing URL summary service; or When the first interactive content includes address information, the recommended interactive service includes one or more of the following: distance lookup service, navigation service, and ride-hailing service; or When the first interactive content includes flight information, the recommended interactive service includes one or more of the following: flight booking service, flight search service, and adding to schedule service; or When the first interactive content includes appointment time information, the recommended interactive service includes adding a schedule service and / or recording service; or When the first interactive content includes a table, the recommended interactive service includes one or more of the following: add and save service, copy service, and share service.

2. The method according to claim 1, characterized in that, The method further includes: The electronic device displays a third interface, which includes a first recommendation interactive service; In response to detecting a user clicking on the first recommended interactive service, the electronic device executes the first recommended interactive service; or, In response to detecting a second voice command from the user, the electronic device executes the first recommended interactive service, wherein the second voice command is used to instruct the user to select the first recommended interactive service; or... In response to detecting that a user drags the first interactive content to the first recommended interactive service, the electronic device executes the first recommended interactive service, and one or more recommended interactive services include the first recommended interactive service.

3. The method according to claim 1, characterized in that, The electronic device performs a first operation based on the first voice command and the first interactive content, including: The electronic device executes the first operation based on the first voice command, the first interactive content, and the first display content, wherein the first display content is the content on the first interface other than the first interactive content.

4. The method according to claim 1, characterized in that, The method further includes: The electronic device displays a fourth interface, which is the interface for executing the recommended interactive service.

5. The method according to claim 1, characterized in that, The method further includes: In response to the completion of the first operation by the electronic device, the electronic device is triggered to detect a third voice command for multimodal fusion with the first interactive content again; The electronic device executes the second operation based on the result of the first operation, the third voice command, and the first interactive content.

6. The method according to claim 1, characterized in that, In response to a user selecting an interactive area on a first interface of the electronic device, the electronic device acquires first interactive content associated with the interactive area, including: In response to detecting a user's selection of the interactive area on the first interface, the electronic device acquires the first interactive content; or... In response to detecting a user's action of selecting the interactive area on the first interface, the electronic device acquires the first interactive content; or... In response to detecting a user's long press or double-tap operation on the interactive area of ​​the first interface, the electronic device acquires the first interactive content; or, In response to detecting a user dragging the interactive area on the first interface, the electronic device acquires the first interactive content.

7. An electronic device, characterized in that, include: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 6.

9. A chip, characterized in that, The chip stores instructions that, when executed, implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 6.

Images