Human-computer interaction method and electronic device

Abstract

Provided in the embodiments of the present application are a human-computer interaction method. The method comprises: in response to an operation of a user selecting an interaction region on a first interface of an electronic device, the electronic device acquiring first interactive content associated with the interaction region; the electronic device acquiring a first speech instruction; and the electronic device executing a first operation on the basis of the first speech instruction and the first interactive content. By means of the method, interactive content can be selected on a display screen of an electronic device, and convenient speech interaction between the electronic device and a user on the basis of the interactive content is realized on the basis of an application on the electronic device that is related to the interactive content, thus providing the user with a human-computer interaction method that is convenient in operation.

Description

Method and electronic device for human-computer interaction

[0001] This application claims priority from the Chinese patent application No. 202411025317.0 filed on July 26, 2024, and titled "Method and electronic device for human-computer interaction", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of human-computer interaction, and more particularly, to a method and electronic device for human-computer interaction. BACKGROUND

[0003] With the increasing popularity of smart devices and the increasing richness of their functions, people are increasingly dependent on smart devices in their daily lives and work, and there are more and more scenarios in which users need to interact with the content on the screen, which makes people increasingly demand convenience in interacting with the content on the screen of smart devices. SUMMARY

[0004] The present application provides a method and electronic device for human-computer interaction, through which the interaction content can be selected on the display screen of the electronic device, and based on the application on the electronic device related to the interaction content, the convenient voice interaction between the electronic device and the user based on the interaction content can be realized, providing a convenient human-computer interaction mode for the user.

[0005] In a first aspect, a method for human-computer interaction is provided, which includes: in response to the user selecting an interaction region on a first interface of an electronic device, the electronic device acquires first interaction content associated with the interaction region; the electronic device acquires a first voice instruction; and the electronic device performs a first operation according to the first voice instruction and the first interaction content.

[0006] In embodiments of the present application, the interaction content can be selected on the display screen of the electronic device, and based on the application on the electronic device related to the interaction content, the convenient voice interaction between the electronic device and the user based on the interaction content can be realized, providing a convenient human-computer interaction mode for the user.

[0007] In addition, in the method provided by the embodiments of the present application, the screen of the electronic device can not be switched to a state to be selected before the target interaction object is selected, and the selection of the interaction content can be directly based on the current display screen of the electronic device, further improving the convenience of the interaction between the user and the electronic device.

[0008] In addition, the electronic device can adaptively adjust the selection range and determine the interaction content according to the content within the selection range of the user, and can realize accurate selection of the interaction content, without the need for the user to frequently manually adjust the interaction region.

[0009] With reference to the first aspect, in a possible implementation manner, the electronic device obtaining the first voice instruction comprises: in response to the electronic device obtaining the first interaction content, triggering the electronic device to detect the first voice instruction for multi-modal fusion with the first interaction content.

[0010] It can be understood that, in the embodiments of the present application, after the interaction content is selected, the state in which the electronic device is triggered to enter to detect the first voice instruction is different from the state in which the electronic device is triggered to detect the voice instruction in the case where the interaction content is not selected. Specifically, after the interaction content is selected, the state in which the electronic device is triggered to enter to detect the first voice instruction is a state in which the first voice instruction detected in the state cannot be used independently to determine what operation is performed subsequently, but is used to determine what operation is performed subsequently based on multi-modal fusion with the selected interaction content.

[0011] In the embodiments of the present application, after the user selects the interaction content on the screen of the electronic device, the electronic device enters the state of detecting the voice instruction related to the interaction content. The user does not need to perform another operation to trigger the electronic device to detect the voice instruction, and the convenience of the user and the electronic device for voice interaction based on the interaction content is further improved.

[0012] With reference to the first aspect, in a possible implementation manner, the electronic device obtaining the first voice instruction comprises: in response to the electronic device obtaining the first interaction content, the electronic device displays a second interface, and the second interface comprises a first control. The first control is used to trigger the electronic device to detect the first voice instruction for multi-modal fusion with the first interaction content.

[0013] It can be understood that, in the embodiments of the present application, after the interaction content is selected, the state in which the electronic device is triggered to enter to detect the first voice instruction is different from the state in which the electronic device is triggered to detect the voice instruction in the case where the interaction content is not selected. Specifically, after the interaction content is selected, the state in which the electronic device is triggered to enter to detect the first voice instruction is a state of detecting the voice instruction related to the interaction content.

[0014] In the embodiments of the present application, after the user selects the interaction content on the screen of the electronic device, the screen of the electronic device can display a control for triggering the electronic device to enter the state of detecting the voice instruction related to the interaction content. The user can trigger the electronic device to enter the state of detecting the voice instruction related to the interaction content through the control, and the electronic device can be prevented from being mistakenly entered into the state of detecting the voice instruction related to the interaction content.

[0015] With reference to the first aspect, in a possible implementation manner, the first control can be a navigation bar.

[0016] Optionally, the first navigation bar is always displayed on the display screen of the electronic device when the electronic device is in a bright screen state.

[0017] Optionally, the first navigation bar is always displayed on the display screen of the electronic device when the electronic device is in a state of interacting with the user based on the interaction content.

[0018] In the embodiments of the present application, after the interaction content is selected, the electronic device can be triggered to obtain the interaction instruction through the touch operation on the navigation bar corresponding to the page where the interaction content is located, and after the interaction instruction is obtained, the interaction operation is performed according to the interaction instruction and the selected content, so that the interaction task can be quickly and accurately completed.

[0019] In combination with the first aspect, in a possible implementation manner, the electronic device obtains the first voice instruction, including: in response to the wake-up word issued by the user, triggering the electronic device to detect the first voice instruction for multi-modal fusion with the first interaction content.

[0020] In the embodiments of the present application, after the interaction content is selected, the electronic device can be triggered to obtain the interaction instruction by issuing the wake-up word, and after the interaction instruction is obtained, the interaction operation is performed according to the interaction instruction and the selected content, so that the interaction task can be quickly and accurately completed.

[0021] In combination with the first aspect, in a possible implementation manner, the first control is a voice icon.

[0022] In the embodiments of the present application, after the interaction content is selected, the electronic device can be triggered to obtain the interaction instruction through the touch operation on the voice icon displayed on the page where the interaction content is located, and after the interaction instruction is obtained, the interaction operation is performed according to the interaction instruction and the selected content, so that the interaction task can be quickly and accurately completed.

[0023] In combination with the first aspect, in a possible implementation manner, the first operation is that the electronic device displays a third interface, and the third interface includes one or more recommended interaction services, and the one or more recommended interaction services have relevance with the first interaction content.

[0024] In the embodiments of the present application, after the interaction content is selected, the display screen of the electronic device can display a plurality of recommended interaction services associated with the interaction content, and the interaction between the selected interaction content and the target recommended interaction service can be realized through the selection operation of the target recommended interaction service, without the need for the user to separately open the application corresponding to the target recommended interaction service, and without the need for the user to perform copy, save and other operations on the selected interaction content in advance, so that the user can quickly and accurately complete the interaction task.

[0025] With reference to the first aspect, in a possible implementation manner, the method further includes: in response to detecting that the user clicks the first recommended interaction service, the electronic device executes the first recommended interaction service; or, in response to detecting a second voice instruction of the user, the electronic device executes the first recommended interaction service, the second voice instruction being used to instruct to select the first recommended interaction service; or, in response to detecting that the user drags the first interaction content to the first recommended interaction service, the electronic device executes the first recommended interaction service, the one or more recommended interaction services including the first recommended interaction service.

[0026] In the embodiments of the present application, for the interaction between the interaction content and the recommended interaction service, the interaction between the interaction content and the recommended interaction service can be realized through the operations of clicking the icon of the recommended interaction service, voice instruction to select the recommended interaction service, and dragging the interaction content to the icon of the recommended interaction service, which provides the user with multiple convenient interaction modes and further improves the user experience.

[0027] With reference to the first aspect, in a possible implementation manner, the electronic device executes a first operation according to the first voice instruction and the first interaction content, including: the electronic device executes the first operation according to the first voice instruction, the first interaction content, and first display content, the first display content being content on the first interface other than the first interaction content.

[0028] In the embodiments of the present application, the interaction result between the electronic device and the user based on the interaction content can not only be made according to the interaction content, but also can be made by referring to other non-interactive content located in the same display interface as the interaction content, so that the interaction result is more accurate.

[0029] With reference to the first aspect, in a possible implementation manner, the electronic device executes a first operation according to the first voice instruction and the first interaction content, including: the electronic device directly executes the first operation, the first operation being used to provide a recommended interaction service, the recommended interaction service having a correlation with the first interaction content, or the electronic device displays a fourth interface, the fourth interface being an interface for executing the recommended interaction service.

[0030] In the embodiments of the present application, after triggering the recommended interaction service for the interaction content, the display interface of the electronic device can be switched to the interface for executing the recommended interaction service for the interaction content, which facilitates the user to complete the interaction quickly and immersively, and is also convenient for the user to further edit; or, the electronic device can directly execute the interaction service, the user is not aware of the interaction process, and the display interface of the electronic device is not switched, which can avoid interrupting the current task process of the user.

[0031] With reference to the first aspect, in a possible implementation manner, the method further includes: after the electronic device directly performs the first operation, displaying a first reminder on a screen of the electronic device, the first reminder being used to remind the user that the first operation has been performed.

[0032] With reference to the first aspect, in a possible implementation manner, when the first interaction content includes a picture, the recommended interaction service includes one or more of a picture recognition service, a sharing service, and a saving service; when the first interaction content includes chat records, the recommended interaction service includes one or more of a copying service, a sharing service, a saving service, and an editing service; when the first interaction content includes text content, the recommended interaction service includes one or more of a copying service, a reading service, a saving service, and an editing service; when the first interaction content includes account information, the recommended interaction service includes a recording service and / or a transfer service; when the first interaction content includes website information, the recommended interaction service includes one or more of a recording service, a sharing service, a website opening service, and a website summary viewing service; when the first interaction content includes address information, the recommended interaction service includes one or more of a distance checking service and a navigation service; when the first interaction content includes flight information, the recommended interaction service includes one or more of a ticket booking service, a flight query service, and an adding schedule service; when the first interaction content includes a doctor visit time, the recommended interaction service includes an adding schedule service and / or a recording service; and when the first interaction content includes a table, the recommended interaction service includes one or more of an adding saving service, a copying service, and a sharing service.

[0033] With reference to the first aspect, in a possible implementation manner, the method further includes: in response to the electronic device completing the first operation, triggering the electronic device to detect a third voice instruction for performing multi-modal fusion with the first interaction content again; and the electronic device performing a second operation according to a result of the first operation, the third voice instruction, and the first interaction content.

[0034] In the embodiments of the present application, after the first round of voice interaction between the electronic device and the user ends, the electronic device can not end the interaction process, but switch to a state of continuing to detect further interaction instructions, so that the electronic device and the user can implement multi-round voice interaction based on the selected interaction content.

[0035] In a possible implementation manner of the first aspect, in response to the operation of the user selecting the interaction region on the first interface of the electronic device, the electronic device acquires the first interaction content associated with the interaction region, including: in response to detecting the operation of the user circling the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting the operation of the user checking the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting the operation of the user long-pressing or double-clicking the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting the operation of the user dragging the interaction region on the first interface, the electronic device acquires the first interaction content.

[0036] In a second aspect, an electronic device is provided, which includes a memory and a processor, wherein the memory is configured to store computer program code, and the processor is configured to execute the computer program code stored in the memory to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0037] In a third aspect, a computer readable storage medium is provided, which stores computer program or instructions, and when the computer program or instructions are executed, the method in the first aspect or any possible implementation manner of the first aspect is implemented.

[0038] In a fourth aspect, a chip is provided, which stores instructions, and when the instructions are executed on a device, the chip performs the method in the first aspect or any possible implementation manner of the first aspect.

[0039] In a fifth aspect, a computer program product is provided, which stores computer program or instructions, and when the computer program or instructions are executed, the method in the first aspect or any possible implementation manner of the first aspect is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0041] FIG. 2 is a software structure block diagram of an electronic device according to an embodiment of the present application;

[0042] FIG. 3 is a schematic diagram of an interface corresponding to a method of interacting based on screen content;

[0043] FIG. 4 is a schematic diagram of an interface corresponding to another method of interacting based on screen content;

[0044] FIG. 5 is a schematic diagram of an interface corresponding to another method of interacting based on screen content;

[0045] FIG. 6 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to an embodiment of the present application;

[0046] FIG. 7 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to another embodiment of the present application;

[0047] FIG. 8 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0048] FIG. 9 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0049] FIG. 10 is a schematic diagram of an interface corresponding to a method for interacting based on an image on a screen according to an embodiment of the present application;

[0050] FIG. 11 is a schematic diagram of an interface corresponding to a method for interacting based on entity content and a table on a screen according to an embodiment of the present application;

[0051] FIG. 12 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0052] FIG. 13 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0053] FIG. 14 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0054] FIG. 15 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0055] FIG. 16 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0056] FIG. 17 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0057] FIG. 18 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0058] FIG. 19 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0059] FIG. 20 is a schematic diagram of an interface corresponding to a method for interacting based on screen content according to yet another embodiment of the present application;

[0060] FIG. 21 is a schematic flowchart of a method for human-computer interaction according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application.

[0062] The technical solutions in the embodiments of the present application will be described below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application.

[0063] Hereinafter, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments, unless otherwise specified, the meaning of "plurality" is two or more than two.

[0064] The terms used in the following embodiments are only for the purpose of describing the specific embodiments, and are not intended to be limiting on the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include, for example, the expression "one or more", unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" means one, two or more than two. The term "and / or" is used to describe the relationship between the associated objects, indicating that there can be three relationships; for example, A and / or B can mean: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects.

[0065] Reference within this specification to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one or more embodiments" within this specification are not necessarily all referring to the same embodiment, although it can. In addition, the terms "comprises", "comprising", "has", "having", "includes", "including", "contains", "containing" or variations thereof are intended to mean "including but not limited to", unless expressly specified otherwise.

[0066] The method provided by the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and the like. The embodiments of the present application do not make any limitation on the specific type of electronic device.

[0067] For example, FIG. 1 shows a structural schematic diagram of an electronic device 100. The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 can include one or more processing units. For example, the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated into one or more processors.

[0070] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0071] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0072] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0074] The charging management module 140 is configured to receive charging input from a charger. The charging management module 140 can charge the battery 142 and also supply power to the electronic device through the power management module 141.

[0075] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be arranged in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be arranged in the same device. The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0076] The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0077] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 can include 1 or N display screens 194, N being a positive integer greater than 1.

[0078] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0079] The NPU is a neural-network (NN) computing processor, which processes input information quickly by drawing on the structure of a biological neural network, such as the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, the electronic device 100 can implement intelligent cognitive applications such as image recognition, face recognition, voice recognition, and text understanding, etc.

[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, etc. files are saved in the external memory card.

[0081] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various function applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one App (such as a sound playing function, an image playing function, etc.) required by a function, etc. The data storage area can store data (such as audio data, a phone book, etc.) created during use of the electronic device 100, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0082] The electronic device 100 can realize audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, etc. For example, music playing, recording, etc.

[0083] The indicator 192 can be an indicator light, which can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, etc.

[0084] The SIM card interface 195 is used to connect a SIM card. The electronic device 100 interacts with a network through the SIM card to realize functions such as calling and data communication.

[0085] It should be understood that the phone card in the embodiments of the present application includes but is not limited to a SIM card, an eSIM card, a universal subscriber identity module (USIM), a universal integrated circuit card (UICC), etc.

[0086] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 100.

[0087] FIG. 2 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application. A layered architecture divides the software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer. The application layer can include a series of application packages.

[0088] As shown in FIG. 2, the application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0089] The application framework layer provides the application layer with application programming interfaces (APIs) and programming frameworks. The application framework layer includes some pre-defined functions.

[0090] As shown in FIG. 2, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0091] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and take screenshots, etc.

[0092] The content provider is used to store and obtain data, and make the data accessible to the application. The data can include video, image, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0093] The view system includes visual controls, such as controls that display text, controls that display pictures, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays a picture.

[0094] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call state (including call connection, call hang-up, etc.).

[0095] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0096] The notification manager enables an application to display notification information in the status bar, which can be used to convey a message of the notification type, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of downloading, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of a background running application, and can also be a notification in the form of a dialog window appearing on the screen. For example, the status bar prompts text information, emits a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0097] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0098] The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0099] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0100] The system library can include multiple functional modules. For example: a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES), a 2D graphics engine (for example: SGL), etc.

[0101] The surface manager is used to manage the display subsystem, and provides a fusion of 2D and 3D layers for multiple applications.

[0102] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0103] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0104] The 2D graphics engine is a drawing engine for 2D drawing.

[0105] The kernel layer is a layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0106] It should be understood that the technical solutions in the embodiments of the present application can be used in Android, IOS, Harmony, etc.

[0107] The technical solutions of the embodiments of the present application can be applied to electronic devices with screen interaction capability. For example, the technical solutions can be applied to televisions, desktop computers, notebook computers, vehicle screens, and portable electronic devices such as mobile phones, folding screens, tablet computers, cameras, video cameras, video recorders, watches, and bracelets. The technical solutions can also be applied to other electronic devices with screen interaction capability, and can be applied to electronic devices in a 5G network or an evolved public land mobile network (PLMN) in the future, and the main application scenario can be interaction based on content displayed on the screen of an electronic device.

[0108] With the increasing popularity of smart devices and the increasing richness of functions of smart devices, people are increasingly dependent on smart devices in their daily lives and work, and there are more and more scenarios in which users need to interact with content on the screen, which makes people increasingly demand convenience in interacting with the screen content of smart devices.

[0109] FIG. 3 shows a schematic diagram of an interface corresponding to a method of interacting based on screen content.

[0110] As shown in (a) of FIG. 3, a picture of a home scene is displayed on the display interface of an electronic device, and the picture includes objects such as a sofa, a television, a sound bar, and a user. The user can switch the display interface of the electronic device to a selectable interface as shown in (b) of FIG. 3 by long-pressing a control 301. As shown in (c) of FIG. 3, in the selectable interface, the user can manually select a target interaction object, for example, the user manually selects a sound bar 302. After the selection is completed, the display interface of the electronic device is switched to the interface as shown in (d) of FIG. 3, in which a plurality of interaction results related to the sound bar 302 are displayed, for example, pictures, prices, or purchase links of a plurality of sound bars that are the same or similar.

[0111] In this method, the display interface needs to be controlled to enter a specific state before the interaction target on the screen can be selected. After the interaction target is selected, the search results corresponding to the interaction target are displayed, and the interaction process ends. The interaction target cannot be interacted with in multiple rounds, and interaction question and answer cannot be implemented.

[0112] FIG. 4 shows a schematic diagram of an interface corresponding to another method of interacting based on screen content.

[0113] As shown in Figure 4(a), the electronic device's display interface is located in the SMS interface. Users can switch the electronic device's display interface to the interface shown in Figure 4(b) by long-pressing the SMS content in the SMS interface. In this interface, multiple application icons are displayed, such as calendar application icons, chat application icons, email application icons, and memo application icons. When a user wants to edit and send the SMS content via email, the user can drag the SMS content to the email icon while holding down the long press. Furthermore, in response to the user's operation of dragging the SMS content to the email icon, the electronic device's display interface jumps to the email editing page, where the SMS content is displayed as edited text.

[0114] This method can only be implemented if the application is adapted to drag and drop functionality. There may be situations where some interactive content such as images, text, or videos cannot be dragged. Furthermore, it cannot select local content on the screen, and the selection of interactive objects is not precise enough. After dragging and dropping, the interaction ends after jumping to the target page. It cannot perform multiple rounds of interaction on the interactive target, nor can it realize interactive question and answer.

[0115] Figure 5 shows a schematic diagram of the interface corresponding to another method of interaction based on screen content.

[0116] As shown in Figure 5(a), when the display interface of the electronic device is on the screen 1, the display interface of the electronic device can be switched to the voice interface corresponding to screen 1. In the voice interface, a control 501 for adding the screen is displayed. When the user clicks the control 501, the display interface of the electronic device is switched to the voice interaction interface shown in Figure 5(b). In the voice interaction interface, a dialog box 502 is displayed, which displays a screenshot of the screen 1 interface. Based on this, the user can click the control 503 in the dialog box to have a dialogue with the content of screen 1.

[0117] This method does not allow interaction with specific information on the screen.

[0118] Therefore, current screen content interaction methods cannot accurately select local information on the screen, nor can they perform question-and-answer sessions based on local information on the screen.

[0119] In view of this, embodiments of this application provide a method and electronic device for human-computer interaction. Through this method and electronic device, interactive content can be selected on the display screen of the electronic device, and convenient voice interaction between the electronic device and the user can be realized based on the interactive content and applications related to the interactive content on the electronic device. For example, one or more service entry points related to the interactive content can be provided to the user, providing the user with a convenient human-computer interaction method.

[0120] In addition, in the method provided in the embodiments of the present application, the screen of the electronic device can not be switched to the state of being ready for selection before the target interactive object is selected, and the selection of the interactive content can be directly based on the current display screen of the electronic device, thereby further improving the convenience of the interaction between the user and the electronic device.

[0121] Further optionally, the electronic device can adaptively adjust the selection range and determine the interactive content according to the content within the selection range of the user, and the accurate selection of the interactive content can be achieved without the user frequently manually adjusting the interactive region.

[0122] Further optionally, after the first round of voice interaction between the electronic device and the user ends, the electronic device can not end the interaction process, but switch to a state of continuing to detect further interaction instructions, so that the electronic device and the user can implement multiple rounds of voice interaction based on the selected interactive content.

[0123] Further optionally, the interaction result between the electronic device and the user based on the interactive content can not only be made according to the interactive content, but also can be made in reference to other non-interactive content located in the same display interface as the interactive content, so that the interaction result is more accurate.

[0124] Exemplarily, FIG. 6 shows a schematic diagram of an interface corresponding to the method for interacting based on the screen content provided in the embodiments of the present application.

[0125] As shown in (a) of FIG. 6, the screen of the electronic device displays a picture 601, and the picture 601 includes a display element 602, which can be a puppy for example; as shown in (b) of FIG. 6, when the user wants to interact with the screen based on the display element 602, the user can circle-select the display element 602 on the screen; after the circle selection is completed, the display interface of the electronic device is switched to the interface shown in (c) of FIG. 6, in which the selection region of the content on the screen is automatically adjusted from the actual circle selection region of the user shown in (b) of FIG. 6 to the matting region of the display element 602, and a plurality of recommended service entrances appear on the interface, for example, the copy service, the save service and the share service provided in the service box 603; for example, a voice ball 604 can be displayed on the interface, which is used for the user to interact with the selected content through voice dialogue; for example, other recommended service entrances can be displayed on the interface, such as image recognition search (for example, Xia Yi recognizes objects) and how to feed, for example, more recommended service entrances can be viewed by clicking the control 605, and the number and type of the displayed recommended service entrances are not limited in the embodiments of the present application.

[0126] In some embodiments, a navigation bar 606 can also be displayed in the interface as shown in (b) of FIG. 6, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0127] In one example, after the user selects the display area, the user can long press the navigation bar 606 to wake up the voice function of the electronic device, and the user can interact with the selected content through voice interaction with the electronic device.

[0128] In some embodiments, one or more application icons associated with the selected interactive content can also be displayed in the interface as shown in (c) of FIG. 6.

[0129] In one example, the electronic device detects the user's operation of clicking a first application icon in the one or more application icons, and the electronic device performs a first application operation on the selected content.

[0130] In one example, the electronic device detects a voice instruction issued by the user to indicate selection of the first application icon, and the electronic device performs a first application operation on the selected content.

[0131] In one example, the electronic device detects the user's operation of dragging the first interactive content to the first application icon, and the electronic device performs a first application operation on the selected content.

[0132] In some embodiments, the voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or through voice wake-up.

[0133] In this embodiment, when the selected interactive content is a subject, the final selected interactive content is the subject, and the selected area is automatically adjusted to the matting area of the subject.

[0134] Exemplarily, FIG. 7 shows an interface diagram corresponding to another method for interacting based on screen content provided by the embodiments of the present application.

[0135] As shown in (a) of FIG. 7, a picture 701 is displayed on the screen of the electronic device, the picture 701 includes a display element 702, for example, a puppy, and a plurality of display elements 703, for example, trees; as shown in (b) of FIG. 7, when the user wants to interact with the screen based on the display element 702 and the display elements 703, the user can circle the display element 702 and the display elements 703 on the screen; after the circling is completed, the display interface of the electronic device is switched to the interface shown in (c) of FIG. 7, in which the selected area of the content on the screen is automatically adjusted to the circumscribed rectangular area of the actual circling area as shown in (b) of FIG. 7, and a plurality of recommended service entrances appear on the interface, for example, the copy service, the save service and the share service provided in the service box 704; for example, a voice ball 705 can be displayed on the interface, which is used for the user to interact with the selected content through voice dialogue; for example, other recommended service entrances, such as Xia Yi's object recognition and how to feed, can be displayed on the interface; for example, more recommended service entrances can be viewed by clicking the control 706, and the number and type of the displayed recommended service entrances are not limited in the embodiments of the present application.

[0136] In some embodiments, a navigation bar 707 can also be displayed on the interface as shown in (b) of FIG. 7, which is used to trigger the voice interaction between the user and the electronic device based on the selected content.

[0137] In one example, after the user circles the display area, the user can wake up the voice function of the electronic device by long pressing the navigation bar 707, and the user can interact with the selected content through voice interaction with the electronic device.

[0138] In some embodiments, the voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or through voice wake-up.

[0139] In the embodiment, when the circled interactive content is a plurality of subjects or no subject, the final selected interactive area is the circumscribed rectangular area of the circling area.

[0140] Exemplarily, FIG. 8 shows the interface diagram corresponding to another method for interacting based on the screen content provided by the embodiments of the present application.

[0141] As shown in (a) of FIG. 8, a plurality of pictures are displayed on the screen of the electronic device, when a user wants to interact with the screen based on part of the plurality of pictures, the user can circle select part of the plurality of pictures on the screen; after the circle selection is completed, the display interface of the electronic device is switched to the display interface shown in (b) of FIG. 8, in which the selected area of the content on the screen is automatically adjusted from the actual circle selection area shown in (a) of FIG. 8 to the circumscribed rectangular area of the actual circle selection area, or to the circumscribed rectangular area of the plurality of pictures included in the actual circle selection area, and a plurality of recommended service portals appear on the interface, for example, a voice ball 801 is displayed in the interface, which is used for the user to interact with the selected content through voice dialogue; for example, other recommended service portals such as copying pictures, sharing pictures, etc. can be displayed in the interface, for example, more recommended service portals can be viewed by clicking the control 802, and the number and type of the displayed recommended service portals are not limited in the embodiments of the present application.

[0142] In some embodiments, a navigation bar 803 shown in (a) of FIG. 8 can also be displayed, which is used to trigger the user to interact with the electronic device based on the selected content through voice.

[0143] In one example, after the user circles the display area, the user can wake up the voice function of the electronic device by long pressing the navigation bar 803, and the user can interact with the selected content through voice interaction with the electronic device.

[0144] In some embodiments, the user can also trigger the voice interaction with the electronic device based on the selected content through long pressing the power key of the electronic device or voice wake-up.

[0145] In this embodiment, when the circle-selected interactive content is a plurality of pictures, the finally selected interactive area is the circumscribed rectangular area of the actual circle selection area, or the circumscribed rectangular area of the plurality of pictures included in the actual circle selection area.

[0146] Exemplarily, FIG. 9 shows the interface diagram corresponding to another method for interacting based on the screen content provided by the embodiments of the present application.

[0147] As shown in (a) of FIG. 9, a chat interface is displayed on the screen of the electronic device, when a user wants to interact with the screen based on the chat record in the chat interface, the user can circle select the chat record on the screen; after the circle selection is completed, the display interface of the electronic device is switched to the display interface as shown in (b) of FIG. 9, in which the selected area of the content on the screen is automatically adjusted from the actual circle selection area as shown in (a) of FIG. 9 to the circumscribed rectangular area of the actual circle selection area, or to the circumscribed rectangular area of the multiple chat record boxes included in the actual circle selection area, and multiple recommended service entrances appear on the interface, for example, the copy service, the save service, the share service and the edit service provided in the service box 901; for example, a voice ball 902 can be displayed on the interface, which is used for the user to interact with the selected content through voice dialogue; for example, other recommended service entrances such as hiding the avatar nickname can be displayed on the interface, for example, more recommended service entrances can be viewed by clicking the control 903, and the number and category of the displayed recommended service entrances are not limited in the embodiments of the present application.

[0148] In some embodiments, a navigation bar 904 as shown in (a) of FIG. 9 can also be displayed on the interface, which is used to trigger the user to interact with the electronic device based on the selected content through voice.

[0149] In one example, after the user circles the display area, the user can wake up the voice function of the electronic device by long pressing the navigation bar 904, and the user can interact with the selected content through voice interaction with the electronic device.

[0150] In some embodiments, the user can also trigger the voice interaction with the electronic device based on the selected content through the way of long pressing the power key of the electronic device or the way of voice wake-up.

[0151] In this embodiment, when the circle-selected interaction content is chat content, the finally selected interaction area is the circumscribed rectangular area of the actual circle selection area, or the circumscribed rectangular area of the multiple chat record boxes included in the actual circle selection area; and when the circle selection area includes an avatar, the avatar nickname in the selected interaction content can also be hidden by the function key of "hiding avatar nickname".

[0152] In some embodiments, a first picture including a first display object is displayed on the screen of the electronic device, the user circles a first area on the first picture, and the first area includes all or part of the first display object.

[0153] When the area of the region where the first display object intersects the first region / MAX (the area of the first display object, the area of the first region) is greater than the first threshold value, the electronic device finally determines the interaction region as the first display object region;

[0154] When the area of the region where the first display object intersects the first region / MAX (the area of the first display object, the area of the first region) is less than the first threshold value, and the area of the region where the first picture intersects the first region / MAX (the area of the first picture, the area of the first region) is greater than the second threshold value, the electronic device finally determines the interaction region as the first picture region.

[0155] When the area of the region where the first display object intersects the first region / MAX (the area of the first display object, the area of the first region) is less than the first threshold value, and the area of the region where the first picture intersects the first region / MAX (the area of the first picture, the area of the first region) is less than the second threshold value, the electronic device finally determines the interaction region as the circumscribed rectangle region of the first region.

[0156] The first threshold value can be any value in the range of 0 to 1, for example, 0.8 or 0.9. The second threshold value can be 0.8 or 0.9, for example.

[0157] It can be understood that, in the embodiments shown in FIGS. 6 to 9, the user circles the screen to select the interactive content on the screen through the knuckles, but this does not limit the way of selecting the interactive content on the screen in the embodiments of the present application. The user can also circle the screen to select the interactive content on the screen through the fingers, for example, the way of circling the screen through the fingers after pressing, or the way of circling the screen through the fingers after pressing. The user can also circle the screen to select the interactive content on the screen through the stylus, for example, the way of circling the screen through the stylus after pressing, or the way of circling the screen through the stylus after pressing. The user can also circle the screen to select the interactive content on the screen through other body parts. In addition, the user can also select the interactive content on the screen in other possible ways, for example, through a mouse, and the embodiments of the present application do not limit this.

[0158] It should be noted that, in the embodiments shown in FIGS. 6 to 9, the user selects the interactive content by circling the screen, but this does not limit the way of selecting the interactive content in the embodiments of the present application.

[0159] In some embodiments, the user can select the interactive content by checking the content on the screen. For example, the content displayed on the screen is a plurality of checkable pictures and / or a plurality of checkable texts, and the user can select the interactive content by checking the pictures and / or texts in a targeted manner.

[0160] In some other embodiments, the user can select the interactive content by performing a long press or double-click operation on the content on the screen. For example, the content displayed on the screen is a reading page of an article, which includes tables, texts and pictures. The user can determine a table as the interactive content by performing a long press or double-click operation on the table, determine a piece of text as the interactive content by performing a long press or double-click operation on the text, and determine a picture as the interactive content by performing a long press or double-click operation on the picture.

[0161] In some other embodiments, the user can select the interactive content by performing a drag operation on the content on the screen. For example, the content displayed on the screen is a reading page of an article, which includes tables, texts and pictures. The user can select a picture as the interactive content by performing a drag operation on the picture, and can directly drag the selected content to a target interactive application. For example, after the user triggers a drag operation on a picture, the screen displays a plurality of application icons that can be interacted with, such as icons of gallery, email, memo, chat application, etc. If the user drags the picture to the icon of the gallery application, the picture is automatically saved to the gallery. If the user drags the picture to the icon of the memo application, the picture is automatically saved to the memo. If the user drags the picture to the icon of the chat application, the electronic device displays a prompt on the screen to remind the user to select an object to which the picture is to be shared. After the user selects the object to which the picture is to be shared, the picture is sent to the selected object.

[0162] In addition, the user can also select the interactive content in other possible ways. For example, the user utters an instruction voice, and the electronic device analyzes the voice content of the user and selects the interactive content from the plurality of displayed contents on the screen according to the analysis result of the voice content.

[0163] As can be seen from the embodiments shown in FIGS. 6-9, the method provided by the embodiments of the present application can intelligently determine the final interactive content selection region in combination with the size of the user's selected region and the specific content in the selected region, thereby avoiding the user from performing frequent adjustment operations on the selected region during the selection process and improving the user experience.

[0164] By way of example, FIG. 10 shows interface diagrams corresponding to two methods of interacting with the screen based on the image on the screen according to the embodiments of the present application.

[0165] As shown in (a) of FIG. 10, a cover image of an e-book with a title of "Title One" is displayed on the screen of the electronic device, when a user wants to interact with the screen based on the e-book with the title of "Title One", the user can circle the e-book with the title of "Title One", in response to the operation of the user circling the e-book with the title of "Title One", a service portal related to the e-book with the title of "Title One" can be displayed on the screen of the electronic device, for example, a service box 1001 for the user to select "save", "copy", "edit", "graphic selection" and the like services can be displayed, for example, a voice ball 1002 for the user to interact with the selected content in the form of voice dialogue can be displayed, for example, other recommended service portals such as audiobooks, book introductions, etc. can be displayed in the interface, for example, more service portals related to the e-book with the title of "Title One" can be viewed by clicking a control 1003.

[0166] In some embodiments, one or more application icons associated with the selected content can also be displayed in the interface as shown in (a) of FIG. 10.

[0167] Examples of operations for triggering interaction between the user and the electronic device based on the selected content through one or more application icons can be referred to the description of the embodiment shown in FIG. 6, which will not be repeated here for brevity.

[0168] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long-pressing the power key of the electronic device or in the voice wake-up manner.

[0169] As shown in (b) of FIG. 10, a home leisure scene image is displayed on the screen of the electronic device, which can include display objects such as a TV set, a sound box, a sofa and a user, for example, when a user wants to interact with the screen based on the display object "sound box" in the home leisure scene image, the user can circle the display object "sound box", in response to the operation of the user circling the display object "sound box", a service portal related to the display object "sound box" is displayed on the screen of the electronic device, for example, a voice ball 1004 for the user to interact with the selected content in the form of voice dialogue can be displayed, for example, other recommended service portals such as Xia Yi Recognition, product black technology, etc. can be displayed in the interface, for example, more service portals related to the display object "sound box" can be viewed by clicking a control 1005.

[0170] In some embodiments, a navigation bar 1006 can also be displayed in the interface as shown in (a) of FIG. 10, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0171] In one example, after the user selects the display area, the user can long press the navigation bar 1006 to wake up the voice function of the electronic device, and the user can interact with the selected content through voice interaction with the electronic device.

[0172] In some embodiments, a navigation bar 1007 can also be displayed in the interface as shown in (b) of FIG. 10, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0173] In one example, after the user selects the display area, the user can long press the navigation bar 1007 to wake up the voice function of the electronic device, and the user can interact with the selected content through voice interaction with the electronic device.

[0174] In some embodiments, the interaction with the selected content can also be triggered by long pressing the power button of the electronic device or through voice wake-up.

[0175] Exemplarily, FIG. 11 shows several interface schematic diagrams corresponding to the methods of interacting with the screen based on the content or table on the screen provided by the embodiments of the present application.

[0176] As shown in (a) of FIG. 11, a plurality of text contents are displayed on the screen of the electronic device, for example, a page of a contract or transaction information description can be displayed, the plurality of text contents displayed in the page include bank account information, bank information and payee information, when a user wants to interact with the screen based on the bank account information, bank information and payee information (for example, the user wants to transfer money to the account or wants to record the account information, etc.), the user can select the bank account information, bank information and payee information, in response to the operation of the user selecting the bank account information, bank information and payee information, the service portal related to the bank account information, bank information and payee information is displayed on the screen of the electronic device, for example, a voice ball 1101 can be displayed, the voice ball 1101 is used for the user to interact with the selected content through voice dialogue; for example, other recommended service portals such as transfer, record, etc. can be displayed in the interface, when the user clicks the transfer service portal, the display interface of the electronic device jumps to the interface of transferring money to the bank account in the selected content through the bank application; wherein, when the user clicks the record service portal, the display interface of the electronic device jumps to the record interface in the memo, the above selected bank account information, bank information and payee information are displayed in the record interface in the form of editing, or when the user clicks the record service portal, the above selected bank account information, bank information and payee information are automatically recorded in the memo, the electronic device can remind the user that the interaction content has generated a memo in the current page, or does not remind the user, which is not sensitive in user experience level, and the user can view it in the memo later; the user can also view more service portals related to the bank account information, bank information and payee information by clicking the control 1102.

[0177] In some embodiments, the above selecting the bank account information, bank information and payee information can be replaced by selecting the bank account information; can be replaced by selecting the bank account information and bank information; or can be replaced by selecting the bank account information and payee information; or can be replaced by selecting the bank account information, bank information and payee information; the corresponding interaction process is similar to that of selecting the bank account information, bank information and payee information.

[0178] In some embodiments, the selected content includes bank account information, and the electronic device is installed with a China Merchants Bank application, then the corresponding transfer service portal of the China Merchants Bank is displayed.

[0179] As shown in (b) of FIG. 11, a plurality of text contents are displayed on the screen of the electronic device, for example, a reading page of an article of a certain public account can be displayed, and the plurality of text contents displayed in the page include website information, when a user wants to interact with the screen based on the website information (for example, the user wants to jump to the website or wants to record the website, etc.), the user can circle the website information, in response to the operation of the user circling the website information, the screen of the electronic device displays a service entry related to the website information, for example, a voice ball 1103 is optionally displayed, which is used for the user to interact with the selected content through voice dialogue; for example, other recommended service entries can be displayed in the interface, such as opening the website, website summary, etc., wherein when the user clicks the website opening service entry, the display interface of the electronic device jumps to the page corresponding to the website, and when the user clicks the website summary service entry, the display interface of the electronic device displays the summary content corresponding to the website; the user can also view more service entries related to the website information by clicking the control 1104, for example, when the user clicks the note service entry, the website is automatically recorded in the memo, and the electronic device can remind the user that the website information has generated a memo on the current page, or not remind the user, which is not sensitive to the user experience level, and the user can view it in the memo later.

[0180] Among them, the website summary refers to the summary content of the webpage corresponding to the website, through which the user can quickly understand the key points of the webpage content corresponding to the website, which is similar to the abstract of a paper, and the user can also read the website summary before opening the webpage corresponding to the website, and then further decide whether to open the webpage corresponding to the website.

[0181] As shown in (c) of FIG. 11, a plurality of text contents are displayed on the screen of the electronic device, for example, an activity announcement page or a schedule information page can be displayed, and the plurality of text contents displayed in the page include address information, when a user wants to interact with the screen based on the address information (for example, the user wants to jump to a navigation page corresponding to the address or wants to record the address, etc.), the user can circle the address information, in response to the operation of the user circling the address information, a service entry related to the address information is displayed on the screen of the electronic device, for example, a voice ball 1105 can be displayed, which is used for the user to interact with the selected content through voice dialogue, for example, the user asks how far the address is from the current location through voice; for example, other recommended service entries can be displayed in the interface, such as distance query, navigation, and car-hailing, etc., wherein when the user clicks the distance query service entry, the display interface of the electronic device jumps to a map page, and the distance between the current location of the user and the address is displayed in the map page, when the user clicks the navigation service entry, the display interface of the electronic device jumps to a navigation page for navigation to the address; when the user clicks the car-hailing service entry, the display interface of the electronic device jumps to a car-hailing page for car-hailing to the address; the user can also view more service entries related to the address information by clicking the control 1106, for example, when the user clicks the note service entry, the address is automatically recorded in the memo, and the electronic device can remind the user that the address information has generated a memo in the current page, or does not remind the user, which is not sensitive to the user experience level, and the user can view it in the memo later.

[0182] As shown in (d) of FIG. 11, a plurality of text contents are displayed on the screen of the electronic device, for example, a registration order page can be displayed, and the plurality of text contents displayed in the page include appointment time information, when a user wants to interact with the screen based on the appointment time information (for example, the user wants to jump to a calendar page corresponding to the appointment time or record the appointment time as a reminder schedule, etc.), the user can circle the appointment time information, in response to the operation of the user circling the appointment time information, a service entry related to the appointment time information is displayed on the screen of the electronic device, for example, a voice ball 1107 can be displayed, which is used for the user to interact with the selected content through voice dialogue, for example, the user asks how many days the appointment time is through voice; for example, other recommended service entries can be displayed in the interface, for example, add schedule, note, etc., wherein when the user clicks the add schedule service entry, the display interface of the electronic device jumps to a schedule adding page, and the appointment time information is displayed as an added schedule in the page, or when the user clicks the add schedule service entry, a reminder pop-up window appears in the current page, reminding the user that the appointment time information has been added as a schedule, or it is automatically added as a schedule in the background without reminding the user, which is not sensitive to the user experience, and the user can check it in the memo later; when the user clicks the note service entry, the display interface of the electronic device jumps to a memo page, and the appointment time information is displayed as a recorded information in the page, or when the user clicks the note service entry, a reminder pop-up window appears in the current page, reminding the user that the appointment time information has been recorded in the memo, or it is automatically generated as a memo in the background without reminding the user, which is not sensitive to the user experience, and the user can check it in the memo later; the user can also view more service entries related to the appointment time information by clicking the control 1108.

[0183] In some embodiments, when the selected content does not include an explicit date, but includes a schedule intent, the add schedule service will also be displayed, for example: the selected content includes the information "Let's have dinner together tomorrow", which will correspond to the display of the add schedule service; when the selected content includes the information "It will rain tomorrow", since there is no schedule intent, the add schedule service will not be displayed.

[0184] As shown in (e) of FIG. 11, a chat page is displayed on the screen of the electronic device, the chat page includes flight number information in a plurality of chat record messages displayed in the chat page, when a user wants to interact with the screen based on the flight number information (for example, the user wants to jump to a flight detail page corresponding to the flight number or the user wants to book a flight corresponding to the flight number, etc.), the user can circle the flight number information, in response to the operation of the user circling the flight number information, a service entry related to the flight number information is displayed on the screen of the electronic device, for example, a voice ball 1109 can be displayed, the voice ball 1109 is used for the user to interact with the selected content in the form of voice dialogue, for example, the user asks the departure time of the flight number by voice; for example, other recommended service entries can be displayed in the interface, such as booking a ticket, flight query, etc., wherein when the user clicks the booking a ticket service entry, the display interface of the electronic device jumps to a ticket booking page corresponding to the flight number; when the user clicks the flight query service entry, the display interface of the electronic device jumps to a ticket query page corresponding to the flight number; the user can also view more service entries related to the flight number information by clicking the control 1110.

[0185] As shown in (f) of FIG. 11, a text reading page is displayed on the screen of the electronic device, for example, it can be a displayed thesis reading page, a courseware browsing page, etc., a plurality of text information and tables are displayed in the page, when a user wants to interact with the screen based on the table (for example, the user wants to record the table, etc.), the user can circle the table, in response to the operation of the user circling the table, a service entry related to the table is displayed on the screen of the electronic device, for example, a service box 1111 can be displayed, the service box 1111 includes a copy service, a save service, a share service, wherein the user can copy the table by clicking the copy service, also can save the table by clicking the save service, also can share the table by clicking the share service; for example, a voice ball 1112 can be displayed, the voice ball 1112 is used for the user to interact with the selected content in the form of voice dialogue; for example, other recommended service entries can be displayed in the interface, such as table extraction, table analysis, etc.; the user can also view more service entries related to the table by clicking the control 1113.

[0186] It can be understood that, in the embodiments shown in FIGS. 10 and 11, the user circles the interactive content on the screen by the knuckles, but this does not limit the way of circling the interactive content on the screen in the embodiments of the present application. The user can also circle the interactive content on the screen by the fingers, such as circling the interactive content on the screen by the fingers after pressing, or circling the interactive content on the screen by the fingers after pressing. The user can also circle the interactive content on the screen by the stylus. The user can also circle the interactive content on the screen by other body parts, such as circling the interactive content on the screen by the stylus after pressing, or circling the interactive content on the screen by the stylus after pressing. In addition, the user can also circle the interactive content on the screen by other possible ways, such as by a mouse, and the embodiments of the present application do not limit this.

[0187] It should be noted that, in the embodiments shown in FIGS. 10 and 11, the user selects the interactive content by circling on the screen, but this does not limit the way of selecting the interactive content in the embodiments of the present application.

[0188] In some embodiments, when the user selects the recommended service, the screen of the electronic device can remain in the current interface, and the page jumped to can be presented in the form of a pop-up window or a floating window.

[0189] As can be seen from the embodiments shown in FIGS. 10 and 11, in the method provided by the embodiments of the present application, when different content is selected on the screen, the page will make a service function recommendation corresponding to the selected content according to the selected content. In addition, the user can intelligently select any local content on the current display screen, and the service function recommendation made is adapted to the selected local content, which can improve the probability of the user clicking the recommended service function. In addition, the displayed service function recommendation can be determined by intent analysis of the selected content, which can improve the adaptation degree of the service function recommendation to the selected content, and further improve the probability of the user clicking the recommended service function.

[0190] Exemplarily, FIG. 12 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by the embodiments of the present application.

[0191] As shown in (a) of FIG. 12, a chat interface is displayed on the screen of the electronic device, in which multiple pieces of chat information are displayed, one piece of chat information received by the electronic device includes bank account information, bank information and payee information, when a user wants to make a transfer based on the bank account information, bank information and payee information, the user can select the bank account information, bank information and payee information; as shown in (b) of FIG. 12, in response to the operation of the user selecting the bank account information, bank information and payee information, the selected bank account information, bank information and payee information are highlighted, and service portals related to the bank account information, bank information and payee information are displayed on the interface, for example, the service portals can include transfer, copy all text, etc. When the user clicks the transfer service portal, the display interface of the electronic device jumps to the interface for transferring money to the payee through the bank application as shown in (c) of FIG. 12, in which the bank account information, bank information and payee information selected by the user have been automatically filled in the corresponding positions, the user only needs to check the related information and input the transfer amount to complete the transfer to the payee.

[0192] Exemplarily, FIG. 13 shows the interface schematic diagram corresponding to another method for interacting based on screen content provided by the embodiments of the present application.

[0193] As shown in (a) of FIG. 13, a chat interface is displayed on the screen of the electronic device, in which multiple pieces of chat information are displayed, one piece of chat information received by the electronic device is "I will take the AB1234 flight at 6 pm tomorrow", when a user wants to query the specific information of the flight, the user can select the chat information "I will take the AB1234 flight at 6 pm tomorrow", in response to the operation of the user selecting the chat information, service portals related to the chat information are displayed on the interface of the electronic device, for example, the service portals can include flight query, word selection panel, etc. When the user clicks the flight query service portal, the display interface of the electronic device jumps to the interface as shown in (b) of FIG. 13, in which a dialog box 1301 is displayed, the selected chat information "I will take the AB1234 flight at 6 pm tomorrow", the user's instruction "flight query" and the interaction result determined by the electronic device according to the chat information "I will take the AB1234 flight at 6 pm tomorrow" and the user's instruction "flight query" are displayed in the dialog box 1301, the interaction result is the information of the AB1234 flight queried by the electronic device, the information of the AB1234 flight can be displayed in the form of a service card, when the user clicks the service card, the display interface of the electronic device can jump to the information interface of the AB1234 flight in the flight application; in the interface, the user can continue to interact with the chat information through the voice ball 1302 or the keyboard 1303.

[0194] For example, FIG. 14 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0195] As shown in (a) of FIG. 14, a plurality of text contents displayed on the screen of the electronic device include bank account information, bank information and payee information. When the user wants the electronic device to remember the bank account information, bank information and payee information, the user can select the bank account information, bank information and payee information. In response to the operation of the user selecting the bank account information, bank information and payee information, service portals related to the bank account information, bank information and payee information are displayed on the interface. For example, the service portals can include transfer, record account, etc. When the user clicks the record account service portal, the display interface of the electronic device jumps to the interface shown in (b) of FIG. 14. In the interface, a dialog box 1401 is displayed. The dialog box 1401 displays the selected bank account information, bank information and payee information, the user's instruction "record account", and the interaction result determined by the electronic device based on the selected bank account information, bank information and payee information and the user's instruction "record account". The interaction result is the bank account information, bank information and payee information recorded by the electronic device. The recorded information can be displayed in the form of a service card. The electronic device can also remind the user to select a bank application that the user wants to use to transfer money to the account in the future. The user can select a bank application that the user wants to use to transfer money to the account in the future through the "go to edit" control on the service card.

[0196] For example, FIG. 15 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0197] As shown in (a) of FIG. 15, a picture 1501 is displayed on the screen of the electronic device, and the picture 1501 includes a display element 1502, which can be, for example, a puppy. When the user wants to interact with the screen based on the display element 1502, after selecting the display element 1502 as the interaction content, the user can, for example, optionally click a navigation bar 1503 in the interface. In response to the user's operation of clicking the navigation bar 1503, the display interface of the electronic device is as shown in (b) of FIG. 15. For example, optionally, a voice recognition box 1504 can be displayed on the interface, and a guide sentence for guiding the user to ask a voice question can be displayed in the voice recognition box 1504. The guide sentence can be, for example, "Ask the selected object on the screen about...". As shown in (c) of FIG. 15, in response to the user's operation of asking a voice question, the user's question recognized by the electronic device is displayed in the voice recognition box 1504. For example, when the user asks a voice question "What breed of dog is this?", the recognized voice question "What breed of dog is this?" is displayed in the voice recognition box 1504. Then, as shown in (d) of FIG. 15, the interface of the electronic device can display a dialogue box 1505, in which the user's selected content and the question content are displayed, and the answer content given by the electronic device to the user's question is displayed.

[0198] In some embodiments, the dialogue box 1505 described above can be displayed in the form of a floating window or a pop-up window on the original display interface of the electronic device, so as to reduce the impact of the display of the dialogue box 1505 on the user's browsing of the original interface of the electronic device.

[0199] In some embodiments, the voice interaction between the electronic device and the user based on the selected content can also be triggered by pressing the power button of the electronic device for a long time or by voice wake-up.

[0200] The navigation bar refers to a control displayed in the interface of the electronic device independently of the display content. When the display interface of the electronic device is switched, the navigation bar is not cancelled. That is, during the bright screen period of the electronic device, the navigation bar can be displayed at all times.

[0201] In some embodiments, in response to the user's operation of selecting the interaction content, the electronic device displays the navigation bar 1503 on the screen.

[0202] In some embodiments, during the entire process of the voice interaction between the electronic device and the user based on the interaction content, the navigation bar 1503 is always displayed on the screen of the electronic device. For example, in response to detecting that the voice interaction between the electronic device and the user based on the interaction content is ended, the electronic device can cancel the display of the navigation bar 1503.

[0203] In some embodiments, the display condition of the navigation bar can be set through a setting function of the electronic device, for example, the navigation bar can be set to be always displayed, or the navigation bar can be set to be displayed on the interface of certain applications, or the navigation bar can be set to be displayed during the whole process of voice interaction between the electronic device and the user based on the interactive content.

[0204] For example, FIG. 16 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0205] As shown in (a) of FIG. 16, a picture 1601 is displayed on the screen of the electronic device, the picture 1601 includes a display element 1602 and a plurality of display elements 1603, the display element 1602 can be a puppy for example, and the display elements 1603 can be trees for example; when the user wants to interact with the screen based on the display element 1602, the user can click a voice ball 1604 in the interface after selecting the display element 1602 as the interactive content; as shown in (b) of FIG. 16, for example, in response to the operation of the user clicking the voice ball 1604, a voice recognition box 1605 can be displayed on the display interface of the electronic device, and a guide sentence for guiding the user to ask a voice question can be displayed in the voice recognition box 1605, for example, the guide sentence can be "Ask the selected object on the screen about its situation..."; as shown in (c) of FIG. 16, for example, in response to the operation of the user asking a voice question, the voice recognition box 1605 displays the voice question recognized by the electronic device, for example, when the voice question asked by the user is "What breed of dog is this?", the voice recognition box 1605 displays the recognized voice question "What breed of dog is this?"; then as shown in (d) of FIG. 16, for example, a dialogue box 1606 is displayed on the interface of the electronic device, in the dialogue box, the user's selected content and the question content are displayed, and the answer content given by the electronic device to the question of the user is also displayed; on this basis, as shown in (e) of FIG. 16, the user can further ask questions based on the selected content by continuously clicking the voice ball 1604, and the electronic device gives answers to the further questions of the user, or the user can select the display element on the screen again to further ask questions based on the reselected content, and the electronic device gives answers to the further questions of the user.

[0206] In some embodiments, the dialogue box 1606 described above can be displayed in the form of a floating window or a pop-up window on the original display interface of the electronic device, so as to reduce the influence of the display of the dialogue box 1606 on the user's browsing of the original interface of the electronic device.

[0207] It can be understood that when the user selects a partial display element in the picture and performs voice interaction with the screen based on the selected partial display element, the answer determined by the electronic device can be determined according to the selected partial display element, or can be determined in combination with all display elements (i.e., selected display elements and unselected display elements) displayed on the current screen with the selected partial display element as the center, and on this basis, can be determined in combination with logical analysis, intent analysis, semantic analysis, and the like.

[0208] In the example shown in (d) in FIG. 16 and (e) in FIG. 16, for example, when the user asks, "What breed of dog is this?", the electronic device performs a search according to the selected display element 1602, and gives an answer "The breed of this dog is ×××" according to the search result, and can also provide the user with more information about the display element 1602, such as "The breed of dog is derived from ×××, and has a gentle temperament..."; on this basis, the user can further ask, "Where is this dog?", and the electronic device determines an answer "The dog is in the woods..." according to all display elements included in the picture 1601.

[0209] More explanations about this embodiment can be found in the description of the embodiments shown in FIGS. 6-11.

[0210] In this embodiment, after selecting the interaction content on the screen, the user can interact with the screen of the electronic device through voice based on the selected interaction content, which can realize multi-round voice interaction between the selected content and the screen, and can realize multi-round voice interaction between the selected partial content on the screen and the screen, thereby improving the user experience. In addition, when the selected content is a partial display element, the answer of the electronic device can be determined based on the analysis of all display elements on the page, which can improve the accuracy of the answer.

[0211] Exemplarily, FIG. 17 shows an interface diagram corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0212] As shown in (a) of FIG. 17, a text reading page is displayed on the screen of the electronic device; when the user wants to interact with the screen based on a display element 1701 in the page, the user can, for example, invoke service entrances related to the display element 1701 by circling the display element 1701, which may, for example, include a voice ball 1702, a small art object recognition service, a how to feed service, a copy service, a save service, a share service, and the like; as shown in (b) of FIG. 17, for example, in response to the user's operation of clicking the voice ball 1702, a voice recognition box 1703 is displayed on the display interface of the electronic device, in which a guide sentence for guiding the user to ask a voice question may be displayed, which may be, for example, "Ask the selected object on the screen about the situation"; as shown in (c) of FIG. 17, for example, in response to the user's operation of asking a voice question, the voice recognition box 1703 displays the question recognized by the electronic device, for example, when the voice question asked by the user is "Is this dog smart?", the question recognized by the electronic device is "Is this dog smart?"; then as shown in (d) of FIG. 17, for example, the interface of the electronic device displays a dialogue box 1704, in which the user's circling content (display element 1701) and the question content (Is this dog smart?) are displayed, and the answer content given by the electronic device to the user's question is displayed, which may be, for example, "This dog has music playing, whole-house intelligent control, intelligent conversation, and the like."

[0213] In some embodiments, based on the embodiment shown in (d) of FIG. 17, the user can further ask questions based on the selected display element 1701 by continuing to click the voice ball 1702, and the electronic device gives answers to the user's further questions, or the user can select the display element on the screen again to further ask questions based on the reselected content, and the electronic device gives answers to the user's further questions.

[0214] It can be understood that when the user selects a local display element in the screen and interacts with the screen based on the selected local display element, the answer determined by the electronic device may be determined according to the selected local display element, may be determined according to all display elements in the entire display interface (i.e., selected display elements and unselected display elements), may be determined according to the entire document content corresponding to the currently displayed document (including selected display elements and unselected display elements, and other related content not displayed on the screen), and may be determined in combination with logical analysis, intent analysis, semantic analysis, and the like.

[0215] In one example, in a thesis reading page, the user selects a picture, a piece of text, or a combination of a piece of text and a picture in the page as the interactive content for voice interaction. The answer given by the electronic device based on the user's question can be determined based on the selected content, the content of the entire thesis, or the content displayed on the current screen.

[0216] Similarly, in an article reading scenario, a novel reading scenario, etc., the answer obtained through voice interaction can be determined by the electronic device based on the selected content, the overall content (including the content displayed on the screen and the related content not displayed on the screen), or the content displayed on the current screen.

[0217] In this embodiment, after selecting the interactive content on the screen, the user can interact with the screen of the electronic device through voice based on the selected interactive content. The multi-round voice interaction between the selected content and the screen can be implemented, and the multi-round voice interaction between the local content on the screen and the screen can be implemented, thereby improving the user experience. In addition, when the selected content is a locally displayed element, the answer of the electronic device can be determined based on the analysis of all the displayed elements of the page, thereby improving the accuracy of the answer.

[0218] Exemplarily, FIG. 18 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0219] As shown in (a) of FIG. 18, a text reading page is displayed on the screen of the electronic device; when the user wants to interact with the screen based on a display element 1801 in the page, the user can circle the display element 1801 to determine the interaction content as the display element 1801; further, the user can wake up the voice function through a wake-up word, for example, as shown in (b) of FIG. 18, for example, the user can wake up the voice function of the electronic device through the wake-up word "Xiaoyi Xiaoyi", so that a voice recognition box 1802 is displayed on the interface of the electronic device, in which a guide sentence for guiding the user to issue a voice instruction can be displayed, which can be "I'm here! Is there anything I can help you with?", and meanwhile, for example, the electronic device can also issue the guide word in the form of voice; as shown in (c) of FIG. 18, for example, in response to the operation of the user issuing a voice question, the electronic device can display a dialogue box 1803 on the screen of the electronic device after recognizing the question of the user, in which the circled content (display element 1801) and the question content (for example: Is this dog smart?) are displayed, and the answer content given by the electronic device to the question of the user is also displayed, which can be "This dog has music playing, whole-house intelligent control, intelligent conversation and other service functions…", and meanwhile, the electronic device can also issue the answer content in the form of voice.

[0220] In some embodiments, after the user circles the display element 1801, the voice ball can not be displayed on the display interface of the electronic device.

[0221] In some embodiments, based on the embodiment shown in (c) of FIG. 18, for example, the user can further ask questions based on the selected display element 1801 or reselect other display elements by continuing to use the wake-up word or click the voice ball, and the electronic device gives answers to the further questions of the user.

[0222] It can be understood that when the user selects a local display element in the screen and interacts with the screen based on the selected local display element, the answer determined by the electronic device can be determined according to the selected local display element, can be determined according to all display elements in the entire display interface (i.e., the selected display element and the unselected display element), can be determined according to the entire document content corresponding to the currently displayed document (including the selected display element and the unselected display element, and other related content not displayed on the screen), and can be determined in combination with logical analysis, intent analysis, semantic analysis, etc.

[0223] In this embodiment, after selecting the interactive content on the screen, the user can directly initiate voice interaction with the screen of the electronic device based on the selected interactive content through the wake-up word, without the need for the user to manually click the voice ball, which can further free the user's hands and improve the user's convenience.

[0224] For example, FIG. 19 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0225] As shown in (a) of FIG. 19, a text reading page is displayed on the screen of the electronic device; when the user wants to interact with the screen based on the display element 1901 in the page, the user can directly wake up the voice function of the electronic device through the wake-up word, for example, as shown in (b) of FIG. 19, for example, the user can wake up the voice function of the electronic device through the wake-up word "Xiaoyi Xiaoyi", so that the voice recognition box 1902 is displayed on the interface of the electronic device, and a guide sentence for guiding the user to issue a voice instruction can be displayed in the voice recognition box 1902, for example, the guide sentence can be "I'm here! Do you need any help?", and simultaneously, for example, the electronic device can also issue the guide word through voice; as shown in (c) of FIG. 19, for example, in response to the operation of the user issuing a voice question, after recognizing the question of the user, the electronic device can display a dialogue box 1903 on the screen of the electronic device, in which the target interactive content (display element 1901) extracted from the display interface shown in (a) of FIG. 19 according to the question content of the user and the question content (for example: Is this dog smart?) are displayed, and the answer content given by the electronic device to the question of the user is also displayed, for example, the answer content can be "This dog has music playing, whole-house intelligent control, intelligent conversation, etc. Service functions…", and simultaneously, the electronic device can also issue the answer content through voice.

[0226] In some embodiments, in (c) of FIG. 19, the voice ball can not be displayed on the display interface of the electronic device.

[0227] In some embodiments, based on the embodiment shown in (c) of FIG. 19, the user can further ask questions based on the display element 1901 or other display elements through the wake-up word or by clicking the voice ball, and the electronic device gives answers to the further questions of the user.

[0228] It can be understood that the answer determined by the electronic device can be determined according to all display elements in the entire display interface, and on this basis, logical analysis, intent analysis, semantic analysis, etc. can also be combined.

[0229] In this embodiment, when the user wants to interact with the screen based on the local content displayed on the screen, the user does not need to select the interaction content on the screen, but can directly initiate voice interaction between the screen of the electronic device by the wake-up word, the electronic device can determine the corresponding answer content according to the content asked by the user and all display elements displayed on the screen, and the user does not need to manually click the voice ball, which can further free the user's hands and improve the user's use convenience.

[0230] Exemplarily, FIG. 20 shows a schematic diagram of an interface corresponding to another method for interacting based on screen content provided by an embodiment of the present application.

[0231] As shown in (a) of FIG. 20, a text reading page is displayed on the screen of the electronic device; when the user wants to interact with the screen based on the display element 2001 in the page, the user can directly wake up the voice function of the electronic device by the wake-up word, for example, optionally, as shown in (b) of FIG. 20, the user can wake up the voice function of the electronic device by the wake-up word "Xiao Yi Xiao Yi", so that the voice recognition box 2002 is displayed on the interface of the electronic device, and a guide sentence for guiding the user to issue a voice instruction can be displayed in the voice recognition box 2002, for example, "I am here! What do you need help with?", and optionally, the electronic device can also issue the guide word in the form of voice; as shown in (c) of FIG. 20, for example, optionally, in response to the operation that the user issues a voice question, if the electronic device cannot determine the target interaction content based on the content of the user's question, a guide sentence reminding the user to select the target interaction object can be displayed in the voice recognition box 2002, for example, "Please select the target object you want to know!"; as shown in (d) of FIG. 20, in response to the operation that the user selects the target interaction object (the display element 2001), a dialog box 2003 is displayed on the screen of the electronic device, and the selected content (the display element 2001) and the question content (for example, "Is this dog smart?") are displayed in the dialog box 2003, and the answer content given by the electronic device to the question of the user is also displayed, for example, "This dog has music playing, whole-house intelligent control, intelligent conversation and other service functions…", and the electronic device can also issue the answer content in the form of voice.

[0232] In some embodiments, as shown in (c) of FIG. 20, the voice ball can not be displayed on the display interface of the electronic device.

[0233] In some embodiments, based on the embodiment shown in (c) of FIG. 20, the user can further ask questions based on the display element 2001 or other display elements by continuing to use the wake-up word or click the voice ball, and the electronic device gives answers to the further questions of the user.

[0234] It can be understood that the answer determined by the electronic device can be determined according to all display elements in the entire display interface, and on this basis, logical analysis, intent analysis, semantic analysis, etc. can also be combined to determine.

[0235] It should be noted that in the embodiments shown in FIGS. 15-20, when the user manually selects the interactive content, the user selects the interactive content by circling on the screen is taken as an example for illustration, but this does not constitute a limitation on the selection method of the interactive content in the embodiments of the present application.

[0236] In some embodiments, when the user interacts with the local content on the screen through the wake-up word and voice interaction, the user's interaction behavior through voice is not limited to voice dialogue, but can also include specific interaction behaviors of the user through voice, and the voice instruction can be used to trigger the interaction between the interactive content and other applications, for example: the voice instruction can be used to record the interactive content as a memo; the voice instruction can be used to send the interactive content to friends through email or chat or short message application; the voice instruction can be used to record the interactive content as a reminder schedule, etc.

[0237] It should be understood that the embodiments shown in FIGS. 6-20 can be implemented alone or in combination of any two or more.

[0238] Exemplarily, FIG. 21 shows a schematic flowchart of a method of human-computer interaction provided by an embodiment of the present application. As shown in FIG. 21, the method comprises:

[0239] S2101: In response to detecting that the user selects an interactive region on a first interface of an electronic device, the electronic device acquires first interactive content associated with the interactive region according to the interactive region.

[0240] S2102: The electronic device acquires a first voice instruction.

[0241] In some embodiments, in response to the user's operation of selecting the interactive region on the first interface, a voice icon is displayed on the first interface; and in response to detecting a touch operation of the user on the voice icon, the electronic device acquires the first voice instruction.

[0242] In some embodiments, the voice icon can be a voice ball.

[0243] In some embodiments, in response to the user issuing a wake-up word, the electronic device acquires the first voice instruction, wherein the wake-up word can be, for example, "Xiao Yi Xiao Yi!".

[0244] In some embodiments, in response to a touch operation of the user on a first navigation bar, the electronic device acquires the first voice instruction, and the first interface includes the first navigation bar.

[0245] S2103: The electronic device executes a first operation according to the first voice instruction and the first interaction content.

[0246] In some embodiments, the electronic device executes the first operation according to the first voice instruction, the first interaction content, and first display content, where the first display content is content on a first interface of the electronic device other than the first interaction content.

[0247] In some embodiments, the first operation is that the electronic device displays one or more recommended interaction services that have a correlation with the first interaction content; in response to an operation of a user selecting a first recommended interaction service, the electronic device executes the first recommended interaction service on the first interaction content, and the one or more recommended interaction services include the first recommended interaction service.

[0248] The operation of the user selecting the first recommended interaction service can be an operation of the user clicking the first recommended interaction service, can also be an operation of the user issuing a second voice instruction for indicating selection of the first recommended interaction service, and can also be an operation of the user dragging the first interaction content to the first recommended interaction service.

[0249] In some embodiments, the electronic device can directly execute the first operation for providing a recommended interaction service that has a correlation with the first interaction content, or display an interface for executing the recommended interaction service.

[0250] Optionally, after the electronic device directly executes the first operation, the electronic device can display a first reminder on an interface of the electronic device, where the first reminder is for reminding the user that the first operation has been executed on the first interaction content.

[0251] In some embodiments, in response to the electronic device executing the first operation, the electronic device is triggered to acquire a voice instruction for performing multi-modal fusion with the first interaction content again; the electronic device executes a second operation according to a result of the first operation, the voice instruction, and the first interaction content.

[0252] One or more of the modules or units described herein can be implemented in software, hardware, or a combination thereof. When any of the modules or units are implemented in software, the software is in the form of computer program instructions and is stored in a memory, and a processor can be used to execute the program instructions and implement the above method flows. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, and the like computing devices running software, each of which can include one or more cores for executing software instructions to perform computations or processing. The processor can be built-in to a SoC (system on chip) or an application specific integrated circuit (ASIC), or can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform computations or processing, the processor can further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or a logic circuit implementing specialized logic operations.

[0253] When any of the modules or units described herein are implemented in hardware, the hardware can be any one or a combination of a CPU, a microprocessor, a DSP, an MCU, an artificial intelligence processor, an ASIC, a SoC, an FPGA, a PLD, a specialized digital circuit, a hardware accelerator, or a non-integrated discrete device, which can run necessary software or be independent of software to perform the above method flows.

[0254] When the modules or units described in the specification are implemented by using software, the modules or units can be implemented in a form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, the whole or part of the flow or function described in the embodiments of the present application is generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0255] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0257] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be realized by other ways. For example, the above described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0259] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0260] The functions, if realized in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0261] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of human-machine interaction, characterized by, The method comprises: In response to an operation of a user selecting an interaction region on a first interface of an electronic device, the electronic device acquires first interaction content associated with the interaction region; The electronic device acquires a first voice instruction; The electronic device executes a first operation according to the first voice instruction and the first interaction content.

2. The method of claim 1, wherein, The electronic device acquires the first voice instruction, comprising: In response to the electronic device acquiring the first interaction content, the electronic device triggers detection of the first voice instruction for multi-modal fusion with the first interaction content.

3. The method of claim 1, wherein, The electronic device acquires the first voice instruction, comprising: In response to the electronic device acquiring the first interaction content, the electronic device displays a second interface, the second interface comprising a first control for triggering the electronic device to detect the first voice instruction for multi-modal fusion with the first interaction content.

4. The method of claim 1, wherein, The electronic device acquires a first voice instruction, comprising: In response to a wake-up word uttered by a user, the electronic device triggers detection of the first voice instruction for multi-modal fusion with the first interaction content.

5. The method according to any one of claims 1 to 4, characterized in that, The first operation is that the electronic device displays a third interface, the third interface comprising one or more recommended interaction services, The one or more recommended interaction services have relevance with the first interaction content.

6. The method of claim 5, wherein, The method further comprises: In response to detecting an operation of a user clicking a first recommended interaction service, the electronic device executes the first recommended interaction service; or In response to detecting a second voice instruction of a user, the electronic device executes the first recommended interaction service, the second voice instruction being used to instruct selection of the first recommended interaction service; or In response to detecting an operation of a user dragging the first interaction content to the first recommended interaction service, the electronic device executes the first recommended interaction service, the one or more recommended interaction services comprising the first recommended interaction service.

7. The method according to any one of claims 1 to 6, characterized in that, The electronic device executes a first operation according to the first voice instruction and the first interaction content, comprising: The electronic device executes the first operation according to the first voice instruction, the first interaction content, and first display content, the first display content being content on the first interface other than the first interaction content.

8. The method according to any one of claims 1 to 7, characterized in that, The electronic device executes a first operation according to the first voice instruction and the first interaction content, comprising: The electronic device directly executes the first operation, the first operation being used to provide a recommended interaction service, the recommended interaction service having relevance with the first interaction content, or The electronic device displays a fourth interface, the fourth interface being an interface for executing the recommended interaction service.

9. The method of claim 5, 6 or 8, wherein: when the first interaction content comprises a picture, the recommended interaction service comprises one or more of a picture recognition service, a sharing service, and a saving service; when the first interaction content comprises chat records, the recommended interaction service comprises one or more of a copying service, a sharing service, a saving service, and an editing service; When the first interaction content comprises text content, the recommended interaction service comprises one or more of a copy service, a reading service, a saving service, and an editing service; When the first interaction content comprises account information, the recommended interaction service comprises a recording service and / or a transfer service; When the first interaction content comprises website information, the recommended interaction service comprises one or more of a recording service, a sharing service, a website opening service, and a website summary viewing service; When the first interaction content comprises address information, the recommended interaction service comprises one or more of a distance checking service, a navigation service, and a taxi service; When the first interaction content comprises flight information, the recommended interaction service comprises one or more of a ticket booking service, a flight query service, and a schedule adding service; When the first interaction content comprises a visit time, the recommended interaction service comprises a schedule adding service and / or a recording service; When the first interaction content comprises a table, the recommended interaction service comprises one or more of a saving service, a copy service, and a sharing service.

10. The method according to any one of claims 1 to 9, characterized in that, The method further comprises: in response to the electronic device completing the first operation, triggering the electronic device to detect a third voice instruction for re-fusing the first interaction content in a multi-modal manner; in accordance with a result of the first operation, the third voice instruction, and the first interaction content, the electronic device performs a second operation.

11. The method according to any one of claims 1 to 10, characterized in that, In response to the user selecting an interaction region on a first interface of the electronic device, the electronic device acquires first interaction content associated with the interaction region, comprising: in response to detecting that the user circles the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting that the user checks the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting that the user long-presses or double-clicks the interaction region on the first interface, the electronic device acquires the first interaction content; or, in response to detecting that the user drags the interaction region on the first interface, the electronic device acquires the first interaction content.

12. An electronic device, comprising: comprise: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs comprise instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions, and when the program or instructions are executed, the method of any one of claims 1 to 11 is implemented.

14. A chip, characterized by The chip stores instructions, and when the instructions are executed, the method of any one of claims 1 to 11 is implemented.

15. A computer program product, characterised in that, The computer program product stores a program or instructions, and when the program or instructions are executed, the method of any one of claims 1 to 11 is implemented.

Images