Human-computer interaction method and electronic equipment

Abstract

The embodiment of the invention provides a man-machine interaction method, which comprises the following steps that: in response to an operation of selecting an interaction area on a first interface of electronic equipment by a user, the electronic equipment acquires first interaction content associated with the interaction area; the electronic equipment obtains a first voice instruction; and the electronic equipment executes a first operation according to the first voice instruction and the first interaction content. Through the method, the interaction content can be selected on the display screen of the electronic equipment, convenient voice interaction based on the interaction content between the electronic equipment and the user is realized based on the application related to the interaction content on the electronic equipment, and a man-machine interaction mode convenient to operate is provided for the user.

Description

[0001] This application is a divisional application. The application number of the original application is 202411025317.0, and the original application date is July 26, 2024. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The embodiments of the present application relate to the field of human-computer interaction, and more specifically, to a human-computer interaction method and electronic device. Background Art

[0003] As smart devices become more and more popular and their functions become more and more abundant, people are becoming more and more dependent on smart devices in their lives and work. There are more and more scenarios in which users need to interact with the content on the screen, which makes people have an increasing demand for the convenience of interacting with the screen content of smart devices. Summary of the invention

[0004] The present application provides a human-computer interaction method and an electronic device, through which interactive content can be selected on the display screen of the electronic device, and based on applications related to the interactive content on the electronic device, convenient voice interaction based on the interactive content can be achieved between the electronic device and the user, thereby providing the user with a human-computer interaction method that is easy to operate.

[0005] In a first aspect, a method for human-computer interaction is provided, the method comprising: in response to a user selecting an interaction area on a first interface of an electronic device, the electronic device obtains first interaction content associated with the interaction area; the electronic device obtains a first voice instruction; and the electronic device performs a first operation according to the first voice instruction and the first interaction content.

[0006] In an embodiment of the present application, interactive content can be selected on the display screen of an electronic device, and based on applications related to the interactive content on the electronic device, convenient voice interaction based on the interactive content can be achieved between the electronic device and the user, providing the user with a convenient human-computer interaction method.

[0007] In addition, in the method provided in the embodiment of the present application, before selecting the target interactive object, it is not necessary to switch the screen of the electronic device to the waiting state, and the interactive content can be selected directly based on the current display screen of the electronic device, further improving the convenience of interaction between the user and the electronic device.

[0008] In addition, the electronic device can adaptively adjust the selection range and determine the interactive content according to the content within the user's selection range, and can achieve accurate selection of interactive content without the user having to manually adjust the interactive area frequently.

[0009] In combination with the first aspect, in a possible implementation, the electronic device obtains the first voice instruction, including: in response to the electronic device obtaining the first interactive content, triggering the electronic device to detect the first voice instruction for multimodal fusion with the first interactive content.

[0010] It can be understood that: in the embodiment of the present application, after the interactive content is selected, the state of detecting the first voice instruction that is triggered to enter the electronic device is different from the state of detecting the voice instruction that is triggered when the interactive content is not selected. Specifically, after the interactive content is selected, the electronic device is triggered to enter the state of detecting the first voice instruction. In this state, the first language instruction detected cannot be used independently to determine what operation to perform subsequently, but is based on multimodal fusion with the selected interactive content to determine what operation to perform subsequently.

[0011] In an embodiment of the present application, after the user selects interactive content on the screen of the electronic device, the electronic device enters a state of detecting voice commands related to the interactive content. The user does not need to perform additional operations to trigger the electronic device to detect voice commands, which further improves the convenience of voice interaction between the user and the electronic device based on the interactive content.

[0012] In combination with the first aspect, in a possible implementation, the electronic device obtains the first voice instruction, including: in response to the electronic device obtaining the first interactive content, the electronic device displays a second interface, the second interface includes a first control, and the first control is used to trigger the electronic device to detect the first voice instruction for multimodal fusion with the first interactive content.

[0013] It can be understood that in the embodiment of the present application, after the interactive content is selected, the state of detecting the first voice command that is triggered by the electronic device is different from the state of detecting the voice command that is triggered by the electronic device when the interactive content is not selected. Specifically, after the interactive content is selected, the state of detecting the first voice command that is triggered by the electronic device is the state of detecting voice commands related to the interactive content.

[0014] In an embodiment of the present application, when a user selects interactive content on the screen of an electronic device, a control for triggering the electronic device to enter a state of detecting voice commands related to the interactive content can be triggered to be displayed on the screen of the electronic device. The user can use the control to trigger the electronic device to enter a state of detecting voice commands related to the interactive content, thereby preventing the electronic device from mistakenly entering a state of detecting voice commands related to the interactive content.

[0015] In combination with the first aspect, in a possible implementation manner, the first control may be a navigation bar.

[0016] Optionally, when the electronic device is in a screen-on state, the first navigation bar is always displayed on the display screen of the electronic device.

[0017] Optionally, when the electronic device is in a state of interacting with a user based on interactive content, the first navigation bar is always displayed on the display screen of the electronic device.

[0018] In an embodiment of the present application, after selecting interactive content, the electronic device can be triggered to obtain interactive instructions by touching the navigation bar corresponding to the page where the interactive content is located. After obtaining the interactive instructions, the interactive operation is performed according to the interactive instructions and the selected content, so that the interactive task can be completed quickly and accurately.

[0019] In combination with the first aspect, in a possible implementation, the electronic device obtains a first voice instruction, including: in response to a wake-up word issued by a user, triggering the electronic device to detect the first voice instruction for multimodal fusion with the first interactive content.

[0020] In an embodiment of the present application, after selecting the interactive content, the electronic device can be triggered to obtain the interactive instructions by issuing a wake-up word, and after obtaining the interactive instructions, the interactive operation is performed according to the interactive instructions and the selected content, so that the interactive task can be completed quickly and accurately.

[0021] In combination with the first aspect, in a possible implementation manner, the first control is a voice icon.

[0022] In an embodiment of the present application, after selecting interactive content, the electronic device can be triggered to obtain interactive instructions by touching the voice icon displayed on the page where the interactive content is located. After obtaining the interactive instructions, the interactive operation is performed according to the interactive instructions and the selected content, so that the interactive task can be completed quickly and accurately.

[0023] In combination with the first aspect, in a possible implementation manner, the first operation is displaying a third interface on the electronic device, where the third interface includes one or more recommended interactive services, and the one or more recommended interactive services are correlated with the first interactive content.

[0024] In an embodiment of the present application, after selecting interactive content, a plurality of recommended interactive services associated with the interactive content can be displayed on the display screen of the electronic device. By selecting a target recommended interactive service, interaction between the selected interactive content and the target recommended interactive service can be achieved. The user does not need to separately open the application corresponding to the target recommended interactive service, nor does the user need to copy, save, etc. the selected interactive content in advance, thereby helping the user to complete interactive tasks quickly and accurately.

[0025] In combination with the first aspect, in a possible implementation, the method also includes: in response to detecting an operation of a user clicking on a first recommended interactive service, the electronic device executes the first recommended interactive service; or, in response to detecting a second voice instruction of the user, the electronic device executes the first recommended interactive service, and the second voice instruction is used to indicate the selection of the first recommended interactive service; or, in response to detecting an operation of a user dragging the first interactive content to the first recommended interactive service, the electronic device executes the first recommended interactive service, and the one or more recommended interactive services include the first recommended interactive service.

[0026] In an embodiment of the present application, the interaction between interactive content and recommended interactive services can be achieved by clicking the icon of the recommended interactive service, selecting the recommended interactive service through voice instructions, and dragging the interactive content to the icon of the recommended interactive service, thereby providing users with a variety of convenient interaction methods and further enhancing the user experience.

[0027] In combination with the first aspect, in a possible implementation method, the electronic device performs a first operation according to the first voice command and the first interactive content, including: the electronic device performs the first operation according to the first voice command, the first interactive content and the first display content, and the first display content is the content on the first interface except the first interactive content.

[0028] In an embodiment of the present application, the interaction result between the electronic device and the user based on the interactive content can be made not only based on the interactive content, but also with reference to other non-interactive content located on the same display interface as the interactive content, so that the interaction result is more accurate.

[0029] In combination with the first aspect, in a possible implementation method, the electronic device performs a first operation based on the first voice command and the first interactive content, including: the electronic device directly executes the first operation, the first operation is used to provide a recommended interactive service, and the recommended interactive service is related to the first interactive content, or the electronic device displays a fourth interface, which is an interface for executing the recommended interactive service.

[0030] In an embodiment of the present application, after triggering the execution of a recommended interactive service for interactive content, the display interface of the electronic device can switch to an interface for executing the recommended interactive service for interactive content, so that the user can complete the interaction quickly and immersively, and also facilitate further editing by the user; alternatively, the electronic device can directly execute the interactive service, the user is unaware of the interaction process, and the display interface of the electronic device does not switch, thereby avoiding interrupting the user's current task progress.

[0031] In combination with the first aspect, in a possible implementation, the method further includes: after the electronic device directly executes the first operation, displaying a first reminder on the screen of the electronic device, where the first reminder is used to remind the user that the first operation has been executed.

[0032] In combination with the first aspect, in a possible implementation, when the first interactive content includes a picture, the recommended interactive service includes one or more of a picture recognition service, a sharing service, and a saving service; when the first interactive content includes a chat record, the recommended interactive service includes one or more of a copy service, a sharing service, a saving service, and an editing service; when the first interactive content includes text content, the recommended interactive service includes one or more of a copy service, a reading service, a saving service, and an editing service; when the first interactive content includes account information, the recommended interactive service includes a recording service and / or a transfer service; when the first interactive content includes website information, the recommended interactive service includes one or more of a copy service, a reading service, a saving service, and an editing service. The services include one or more of a recording service, a sharing service, a URL opening service, and a URL summary viewing service; when the first interactive content includes address information, the recommended interactive services include one or more of a distance checking service, a navigation service, and a taxi service; when the first interactive content includes flight information, the recommended interactive services include one or more of a ticket booking service, a flight query service, and a schedule adding service; when the first interactive content includes medical consultation time information, the recommended interactive services include a schedule adding service and / or a recording service; when the first interactive content includes a table, the recommended interactive services include one or more of a save adding service, a copying service, and a sharing service.

[0033] In combination with the first aspect, in a possible implementation, the method also includes: in response to the electronic device completing the first operation, triggering the electronic device to detect a third voice instruction for performing multimodal fusion with the first interactive content again; the electronic device performs a second operation based on the result of the first operation, the third voice instruction and the first interactive content.

[0034] In an embodiment of the present application, after the first round of voice interaction between the electronic device and the user ends, the electronic device may not end the interaction process, but instead switch to a state of continuing to detect further interaction instructions, thereby allowing the electronic device and the user to achieve multiple rounds of voice interaction based on the selected interaction content.

[0035] In combination with the first aspect, in a possible implementation, in response to a user selecting an interaction area on a first interface of an electronic device, the electronic device obtains first interaction content associated with the interaction area, including: in response to detecting that the user circles the interaction area on the first interface, the electronic device obtains the first interaction content; or, in response to detecting that the user checks the interaction area on the first interface, the electronic device obtains the first interaction content; or, in response to detecting that the user long presses or double-clicks the interaction area on the first interface, the electronic device obtains the first interaction content; or, in response to detecting that the user drags the interaction area on the first interface, the electronic device obtains the first interaction content.

[0036] In a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0037] In a third aspect, a computer-readable storage medium is provided, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method in the above-mentioned first aspect or any possible implementation manner of the first aspect is implemented.

[0038] In a fourth aspect, a chip is provided, in which instructions are stored. When the instructions are executed on a device, the chip executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0039] In a fifth aspect, a computer program product is provided, in which a computer program or instructions are stored. When the computer program or instructions are executed, the method in the above-mentioned first aspect or any possible implementation manner of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0041] Figure 2 is a software structure block diagram of the electronic device provided in the embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of an interface corresponding to a method of interacting based on screen content;

[0043] Figure 4 This is a schematic diagram of an interface corresponding to another method of interacting based on screen content;

[0044] Figure 5This is a schematic diagram of an interface corresponding to another method of interacting based on screen content;

[0045] Figure 6 This is a schematic diagram of an interface corresponding to a method for interaction based on screen content provided in an embodiment of the present application;

[0046] Figure 7 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0047] Figure 8 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0048] Fig. 9 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0049] Fig.10 It is a schematic diagram of the interface corresponding to two methods of interacting with the screen based on the image on the screen provided in the embodiments of the present application;

[0050] Fig.11 It is a schematic diagram of an interface corresponding to several methods of interacting with a screen based on entity content and tables on the screen provided in an embodiment of the present application;

[0051] Fig.12 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0052] Fig.13 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0053] Fig.14 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0054] Fig.15 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0055] Fig.16 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0056] Fig.17 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0057] Fig.18 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0058] Fig.19 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0059] Fig. 20 This is a schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application;

[0060] Fig.21 It is a schematic flow chart corresponding to a human-computer interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solution in the present application will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0062] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" or "multiple" refers to two or more than two.

[0063] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0064] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0065] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "one embodiment", "some embodiments", "another embodiment", "some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0066] The method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0067] For example, Figure 11 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0070] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0071] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0074] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0075] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device. The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor.

[0076] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0077] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0078] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0079] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0081] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one App required for a function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0082] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0083] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0084] The SIM card interface 195 is used to connect a SIM card. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.

[0085] It should be understood that the phone card in the embodiments of the present application includes but is not limited to a SIM card, an eSIM card, a universal subscriber identity module (USIM), a universal integrated circuit card (UICC), and the like.

[0086] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0087] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer. The application layer can include a series of application packages.

[0088] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0089] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0090] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0091] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0092] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0093] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0094] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0095] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0096] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0097] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0098] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0099] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0100] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0101] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0102] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0103] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0104] A 2D graphics engine is a drawing engine for 2D drawings.

[0105] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0106] It should be understood that the technical solutions in the embodiments of the present application can be used in Android, IOS, Hongmeng and other systems.

[0107] The technical solution of the embodiment of the present application can be applied to electronic devices with screen interaction capabilities. For example, it can be applied to televisions, desktop computers, laptops, car screens, portable electronic devices such as mobile phones, folding screens, tablet computers, cameras, video cameras, video recorders, watches, bracelets, other electronic devices with screen interaction capabilities, electronic devices in 5G networks or electronic devices in future evolved public land mobile communication networks (PLMN), etc. The main application scenario can be based on the interaction of the content displayed on the screen of the electronic device.

[0108] As smart devices become more and more popular and their functions become more and more abundant, people are becoming more and more dependent on smart devices in their lives and work. There are more and more scenarios in which users need to interact with the content on the screen, which makes people have an increasing demand for the convenience of interacting with the screen content of smart devices.

[0109] Figure 3 A schematic diagram of an interface corresponding to a method for interacting based on screen content is shown.

[0110] like Figure 3 As shown in (a) of FIG. 1 , a picture of a home scene is displayed on the display interface of the electronic device, and the picture includes objects such as a sofa, a television, a speaker, and a user; the user can switch the display interface of the electronic device to the following by long pressing the control 301: Figure 3 The circleable interface shown in (b) in FIG. Figure 3 As shown in (c) in the circleable interface, the user can manually circle the target interactive object, for example, the user manually circles the speaker 302; after the circle is completed, the display interface of the electronic device switches to Figure 3 The interface shown in (d) in the figure displays multiple interaction results related to the speaker 302, for example, pictures, prices or purchase links of multiple identical or similar speakers may be displayed.

[0111] In this method, the display interface needs to be controlled to enter a specific state before the interactive target on the screen can be selected; and after the interactive target is selected, the interactive process ends after the search results corresponding to the interactive target are displayed. It is impossible to perform multiple rounds of interaction on the interactive target, and it is impossible to implement interactive question and answer.

[0112] Figure 4A schematic diagram of an interface corresponding to yet another method for interacting based on screen content is shown.

[0113] like Figure 4 As shown in (a) of FIG. 1 , the display interface of the electronic device is located in the text message interface. The user can switch the display interface of the electronic device to the text message interface by long pressing the text message content in the text message interface. Figure 4 The interface shown in (b) in the figure displays multiple application icons, for example, a calendar application icon, a chat application icon, an email application icon and a memo application icon. When the user wishes to edit and send the text message content via email, the user can drag the text message content to the email icon while keeping a long press. Furthermore, in response to the user dragging the text message content to the email icon, the display interface of the electronic device jumps to the editing page of the email, and the text message content is displayed as edited text on the editing page.

[0114] This method can only be implemented when the application is adapted to the drag function. Some interactive content such as pictures, texts or videos may not be draggable; in addition, it is impossible to select local content on the screen, and the selection of interactive objects is not accurate enough; after dragging is completed, the interaction ends after jumping to the target page, and it is impossible to perform multiple rounds of interaction on the interactive target, and it is impossible to implement interactive question and answer.

[0115] Figure 5 A schematic diagram of an interface corresponding to yet another method for interacting based on screen content is shown.

[0116] like Figure 5 As shown in (a) of FIG. 1 , when the display interface of the electronic device is located at the interface of screen 1, the display interface of the electronic device can be switched to the voice interface corresponding to screen 1. In the voice interface, a control 501 for adding the screen is displayed. When the user clicks the control 501, the display interface of the electronic device is switched to Figure 5 The voice interaction interface shown in (b) in the figure displays a dialog box 502, in which a screenshot of the interface of screen 1 is displayed. On this basis, the user can click on the control 503 in the dialog box to communicate with the content of screen 1.

[0117] In this method, it is not possible to interact with local information on the screen.

[0118] Therefore, in the current screen content interaction method, it is impossible to achieve accurate selection of local information on the screen, nor is it possible to ask and answer questions about the local information on the screen.

[0119] In view of this, an embodiment of the present application provides a method and electronic device for human-computer interaction, through which interactive content can be selected on the display screen of the electronic device, and based on applications related to the interactive content on the electronic device, convenient voice interaction based on the interactive content can be implemented between the electronic device and the user. For example, one or more service entrances related to the interactive content are provided to the user, providing the user with a convenient human-computer interaction method.

[0120] In addition, in the method provided in the embodiment of the present application, before selecting the target interactive object, it is not necessary to switch the screen of the electronic device to the waiting state, and the interactive content can be selected directly based on the current display screen of the electronic device, further improving the convenience of interaction between the user and the electronic device.

[0121] Further optionally, the electronic device can adaptively adjust the selection range and determine the interactive content according to the content within the selection range of the user, thereby enabling accurate selection of the interactive content without the user having to manually adjust the interactive area frequently.

[0122] Further optionally, after the first round of voice interaction between the electronic device and the user ends, the electronic device may not end the interaction process, but instead switch to a state of continuing to detect further interaction instructions, thereby allowing multiple rounds of voice interaction between the electronic device and the user based on the selected interaction content.

[0123] Further optionally, the interaction result between the electronic device and the user based on the interactive content may be made not only according to the interactive content, but also with reference to other non-interactive content located on the same display interface as the interactive content, so that the interaction result is more accurate.

[0124] For example, Figure 6 A schematic diagram of an interface corresponding to a method for interaction based on screen content provided in an embodiment of the present application is shown.

[0125] like Figure 6 As shown in (a) of FIG. 1 , a picture 601 is displayed on the screen of the electronic device, and the picture 601 includes a display element 602, and the display element 602 may be, for example, a puppy; Figure 6 As shown in (b) of FIG. 1 , when the user wishes to interact with the screen based on the display element 602, the user can circle the display element 602 on the screen; after the circle is completed, the display interface of the electronic device switches to the following Figure 6 In the interface shown in (c) of FIG. 1 , the selection area of ​​the content on the screen is as follows: Figure 6The area actually circled by the user shown in (b) is automatically adjusted to the cutout area of ​​the display element 602, and a plurality of recommended service entrances will appear on the interface, such as, optionally, the copy service, save service and share service provided in the service box 603; for example, optionally, a voice ball 604 can be displayed in the interface, and the voice ball 604 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entrances can be displayed in the interface, such as image search (for example: small art identification), how to raise, etc. For example, optionally, more recommended service entrances can be viewed by clicking the control 605. The embodiment of the present application does not specifically limit the number and category of the displayed recommended service entrances.

[0126] In some embodiments, it is also possible to Figure 6 The interface shown in (b) displays a navigation bar 606, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0127] In one example, after the user circles the display area, he or she may long press the navigation bar 606 to wake up the voice function of the electronic device, and the user may interact with the selected content through voice interaction with the electronic device.

[0128] In some embodiments, it is also possible to Figure 6 One or more application icons are displayed in the interface shown in (c), and the one or more application icons are associated with the selected interactive content.

[0129] In one example, the electronic device detects that a user clicks on a first application icon among one or more application icons, and the electronic device performs a first application operation on the selected content.

[0130] In one example, the electronic device detects a voice instruction issued by a user to instruct selection of the first application icon, and the electronic device performs a first application operation on the selected content.

[0131] In one example, the electronic device detects an operation of a user dragging the first interactive content to the first application icon, and the electronic device performs a first application operation on the selected content.

[0132] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0133] In this embodiment, when the circled interactive content is a subject, the interactive content finally selected is the subject, and the circled area is automatically adjusted to the selection area of ​​the subject.

[0134] For example, Figure 7 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0135] like Figure 7 As shown in (a) in FIG. 1 , a picture 701 is displayed on the screen of the electronic device, and the picture 701 includes a display element 702 and multiple display elements 703. The display element 702 may be, for example, a puppy, and the display element 703 may be, for example, a tree; Figure 7 As shown in (b) of FIG. 1 , when the user wishes to interact with the screen based on the display element 702 and the display element 703, the user can circle the display element 702 and the display element 703 on the screen; after the circle is completed, the display interface of the electronic device switches to the following Figure 7 In the interface shown in (c) of FIG. 1 , the selection area of ​​the content on the screen is as follows: Figure 7 The actual selected area of ​​the user shown in (b) is automatically adjusted to the circumscribed rectangular area of ​​the actual selected area, and multiple recommended service entrances will appear on the interface, such as, optionally, the copy service, save service and share service provided in the service box 704; for example, optionally, a voice ball 705 can be displayed in the interface, and the voice ball 705 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entrances can be displayed in the interface, such as small art objects, how to raise, etc. For example, optionally, more recommended service entrances can be viewed by clicking the control 706. The embodiment of the present application does not specifically limit the number and category of the displayed recommended service entrances.

[0136] In some embodiments, it is also possible to Figure 7 The interface shown in (b) displays a navigation bar 707, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0137] In one example, after the user circles the display area, he or she can long press the navigation bar 707 to wake up the voice function of the electronic device. The user can interact with the electronic device through voice to interact with the selected content.

[0138] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0139] In this embodiment, when the circled interactive content is a plurality of subjects or has no subject, the interactive area finally selected is the circumscribed rectangular area of ​​the circled area.

[0140] For example, Figure 8A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0141] like Figure 8 As shown in (a) of FIG. 1 , a plurality of pictures are displayed on the screen of the electronic device. When the user wishes to interact with the screen based on some of the pictures, the user can circle some of the pictures on the screen. After the circle selection is completed, the display interface of the electronic device switches to the following example: Figure 8 The display interface shown in (b) in FIG. 1 is a display interface in which the content selection area on the screen is as follows: Figure 8 The actual selected area of ​​the user shown in (a) is automatically adjusted to the circumscribed rectangular area of ​​the actual selected area, or is automatically adjusted to the circumscribed rectangular area of ​​multiple pictures included in the actual selected area, and multiple recommended service entrances will appear on the interface. For example, optionally, a voice ball 801 is displayed in the interface, and the voice ball 801 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entrances can be displayed in the interface, such as copying pictures, sharing pictures, etc. For example, optionally, more recommended service entrances can be viewed by clicking control 802. The embodiment of the present application does not specifically limit the number and category of the displayed recommended service entrances.

[0142] In some embodiments, it is also possible to Figure 8 The interface shown in (a) displays a navigation bar 803, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0143] In one example, after the user circles the display area, he or she may long press the navigation bar 803 to wake up the voice function of the electronic device, and the user may interact with the selected content through voice interaction with the electronic device.

[0144] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0145] In this embodiment, when the circled interactive content is multiple pictures, the interactive area finally selected is the circumscribed rectangular area of ​​the actual circled area, or the circumscribed rectangular area of ​​the multiple pictures included in the actual circled area.

[0146] For example, Fig. 9 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0147] like Fig. 9As shown in (a) of FIG. 1 , a chat interface is displayed on the screen of the electronic device. When a user wishes to interact with the screen based on a chat record in the chat interface, the user can circle the chat record on the screen. After the circle is completed, the display interface of the electronic device switches to the following example: Fig. 9 The display interface shown in (b) in FIG. 1 is a display interface in which the content selection area on the screen is as follows: Fig. 9 The actual circled area of ​​the user shown in (a) is automatically adjusted to the circumscribed rectangular area of ​​the actual circled area, or is automatically adjusted to the circumscribed rectangular area of ​​multiple chat record boxes included in the actual circled area, and multiple recommended service entrances will appear on the interface, such as, optionally, the copy service, save service, share service and edit service provided in the service box 901; for example, optionally, a voice ball 902 can be displayed in the interface, and the voice ball 902 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entrances can be displayed in the interface, such as hiding the avatar nickname, etc. For example, optionally, more recommended service entrances can be viewed by clicking the control 903. The embodiment of the present application does not specifically limit the number and category of the displayed recommended service entrances.

[0148] In some embodiments, it is also possible to Fig. 9 The interface shown in (a) displays a navigation bar 904, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0149] In one example, after the user circles the display area, he or she may long press the navigation bar 904 to wake up the voice function of the electronic device, and the user may interact with the selected content through voice interaction with the electronic device.

[0150] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0151] In this embodiment, when the circled interactive content is chat content, the final selected interactive area is the circumscribed rectangular area of ​​the actual circled area, or the circumscribed rectangular area of ​​multiple chat record boxes included in the actual circled area; and when the circled area includes an avatar, the avatar nickname in the selected interactive content can also be hidden with one click through the "Hide Avatar Nickname" function key.

[0152] In some embodiments, a first picture is displayed on a screen of an electronic device, the first picture includes a first display object, and a user circles a first area on the first picture, the first area includes all or part of the first display object.

[0153] When the area of ​​the region where the first display object intersects the first region / MAX (area of ​​the first display object, area of ​​the first region) is greater than the first threshold, the electronic device finally determines the interaction region as the first display object region;

[0154] When the area of ​​the region where the first display object intersects the first region / MAX(area of ​​the first display object, area of ​​the first region) is less than the first threshold, and the area of ​​the region where the first picture intersects the first region / MAX(area of ​​the first picture, area of ​​the first region) is greater than the second threshold, the electronic device finally determines the interaction region to be the first picture region;

[0155] When the area of ​​the area where the first display object intersects with the first area / MAX(area of ​​the first display object, area of ​​the first area) is less than the first threshold, and the area of ​​the area where the first picture intersects with the first area / MAX(area of ​​the first picture, area of ​​the first area) is less than the second threshold, the electronic device finally determines the interaction area as the circumscribed rectangular area of ​​the first area.

[0156] The first threshold value may be any value in the range of 0 to 1, for example, 0.8 or 0.9; the second threshold value may be 0.8 or 0.9, for example.

[0157] It is understandable that the above Figures 6 to 9 In the embodiment shown, the user circles the interactive content on the screen by drawing circles on the screen with the knuckles, but this does not constitute a limitation on the manner of circling the interactive content on the screen in the embodiments of the present application. The user can also circle the interactive content on the screen by drawing circles on the screen with the fingers, such as by pressing the finger hard to circle or by pressing the finger hard and then circling; the user can also circle the interactive content on the screen by drawing circles on the screen with a stylus, such as by pressing the stylus hard to circle or by pressing the stylus hard and then circling; the user can also circle the interactive content on the screen by drawing circles on the screen with other parts of the body; in addition, the user can also circle the interactive content on the screen in other possible ways, such as by using a mouse, etc., which is not limited in the embodiments of the present application.

[0158] It should be noted that the above Figures 6 to 9 In the illustrated embodiment, the user selects interactive content by circling on the screen as an example for explanation, but this does not limit the method of selecting interactive content in the embodiments of the present application.

[0159] In some embodiments, the user can select interactive content by checking the content on the screen. For example, if the content displayed on the screen is a plurality of pictures and / or texts that can be checked, the user can select the checked content as interactive content by checking these pictures and / or texts in a targeted manner.

[0160] In some other embodiments, the user can select interactive content by long pressing or double-clicking the content on the screen. For example, the content displayed on the screen is a reading page of an article, which contains tables, text, and pictures. The user can determine the table as interactive content by long pressing or double-clicking a table, and can also determine a paragraph of text as interactive content by long pressing or double-clicking the paragraph, and can also determine the picture as interactive content by long pressing or double-clicking the picture.

[0161] In some other embodiments, the user can select interactive content by dragging the content on the screen. For example, the content displayed on the screen is a reading page of an article, which contains tables, text, and pictures. The user can select a picture as interactive content by dragging the picture, and can directly drag the selected content to the target interactive application. The selection of interactive content and the triggering of interactive operations are achieved through a single drag. For example, after the user triggers the dragging of a picture, multiple interactive application icons are displayed on the screen, such as icons of applications such as gallery, email, memo, and chat applications. If the user drags the picture to the gallery application icon, the picture is automatically saved to the gallery; if the user drags the picture to the memo application icon, the picture is automatically saved to the memo; if the user drags the picture to the chat application icon, the electronic device reminds the user through the screen display to select the object with which he wants to share the picture. After the user selects the object with which he wants to share the picture, the picture is sent to the sharing object.

[0162] In addition, the user may also select interactive content through other possible methods, such as the user giving an instruction voice, the electronic device recognizing and analyzing the user's voice content, and selecting interactive content from multiple display contents on the screen based on the voice content analysis results.

[0163] Depend on Figures 6 to 9 It can be seen from the illustrated embodiment that the method provided by the embodiment of the present application can intelligently determine the final interactive content selection area based on the size of the area circled by the user and the specific content in the circled area, thereby avoiding the user from frequently adjusting the circled area during the selection process and improving the user experience.

[0164] For example, Fig.10Interface schematic diagrams corresponding to two methods of interacting with a screen based on an image on the screen provided in an embodiment of the present application are shown.

[0165] like Fig.10 As shown in (a), a cover image of an electronic book titled "Title One" is displayed on the screen of the electronic device. When a user wishes to interact with the screen based on the electronic book titled "Title One", the user can circle the electronic book "Title One". In response to the user's operation of circling the electronic book "Title One", a service entry related to the electronic book "Title One" can be displayed on the screen of the electronic device. For example, a service box 1001 can be displayed, and the service box 1001 is used for the user to select services such as "save", "copy", "edit", and "graphic selection". For example, a voice ball 1002 can be displayed, and the voice ball 1002 is used for the user to interact with the selected content through voice dialogue; for example, other recommended service entries, such as audio books, book introductions, etc., can be displayed in the interface. For example, more service entries related to the electronic book "Title One" can be viewed by clicking control 1003.

[0166] In some embodiments, it is also possible to Fig.10 One or more application icons are displayed in the interface shown in (a), and the one or more application icons are associated with the selected content.

[0167] Among them, the exemplary operation of triggering the interaction between the user and the electronic device based on the selected content through one or more application icons can be referred to in Figure 6 The description of the illustrated embodiment will not be repeated here for the sake of brevity.

[0168] In some embodiments, voice interaction between the user and the electronic device based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0169] like Fig.10As shown in (b), a family leisure scene image is displayed on the screen of the electronic device. The family leisure scene image may include display objects such as a television, a speaker, a sofa and a user. When the user wishes to interact with the screen based on the display object "speaker" in the family leisure scene image, the user may circle the display object "speaker". In response to the user's operation of circling the display object "speaker", a service entrance related to the display object "speaker" is displayed on the screen of the electronic device. For example, optionally, a voice ball 1004 may be displayed, and the voice ball 1004 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entrances may be displayed in the interface, such as small art objects, product black technology, etc. For example, optionally, more service entrances related to the display object "speaker" may be viewed by clicking control 1005.

[0170] In some embodiments, it is also possible to Fig.10 The interface shown in (a) displays a navigation bar 1006, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0171] In one example, after the user circles the display area, he or she may long press the navigation bar 1006 to wake up the voice function of the electronic device, and the user may interact with the selected content through voice interaction with the electronic device.

[0172] In some embodiments, it is also possible to Fig.10 The interface shown in (b) displays a navigation bar 1007, which is used to trigger voice interaction between the user and the electronic device based on the selected content.

[0173] In one example, after the user circles the display area, he or she may long press the navigation bar 1007 to wake up the voice function of the electronic device, and the user may interact with the selected content through voice interaction with the electronic device.

[0174] In some embodiments, the interaction with the selected content can also be triggered by long pressing the power button of the electronic device or by voice wake-up.

[0175] For example, Fig.11 Interface schematic diagrams corresponding to several methods of interacting with the screen based on the content or table on the screen provided in the embodiments of the present application are shown.

[0176] like Fig.11As shown in (a) in the figure, a plurality of text contents are displayed on the screen of the electronic device, for example, a page of a contract or transaction information description may be displayed, and the plurality of text contents displayed on the page include bank account information, bank opening information and payee information. When a user wishes to interact with the screen based on the bank account information, bank opening information and payee information (for example, the user wishes to transfer money to the account or wishes to record the account information, etc.), the user may circle the bank account information, bank opening information and payee information. In response to the user's operation of circling the bank account information, bank opening information and payee information, a service portal related to the bank account information, bank opening information and payee information is displayed on the screen of the electronic device. For example, a voice ball 1101 may be displayed, and the voice ball 1101 is used for the user to interact with the selected content through voice dialogue; for example, a Other recommended service entrances are displayed, such as transfer, note, etc. When the user clicks on the transfer service entrance, the display interface of the electronic device jumps to the interface of the bank account in the selected content through the bank application; wherein, when the user clicks on the note service entrance, the display interface of the electronic device jumps to the recording interface in the memo, and the above-mentioned selected bank account information, account opening bank information and payee information are displayed in an edited form on the recording interface, or, when the user clicks on the note service entrance, the above-mentioned selected bank account information, account opening bank information and payee information are automatically recorded in the memo, and the electronic device can remind the user on the current page that a memo has been generated for the interactive content, or it may not remind the user, and the user experience is imperceptible, and the user can check it in the memo later; the user can also view more service entrances related to the bank account information, account opening bank information and payee information by clicking on control 1102.

[0177] In some embodiments, the above-mentioned circled selection of the bank account information, the opening bank information and the payee information can also be replaced by circled selection of the bank account information; can also be replaced by circled selection of the bank account information and the opening bank information; or can be replaced by circled selection of the bank account information and the payee information; or can be replaced by circled selection of the bank account information, the opening bank information and the payee information; the corresponding interaction process is similar to the interaction process corresponding to circled selection of the bank account information, the opening bank information and the payee information.

[0178] In some embodiments, the selected content includes bank account information, and the China Merchants Bank application is installed on the electronic device, then the corresponding transfer service entrance of China Merchants Bank is displayed.

[0179] like Fig.11As shown in (b) in the figure, a plurality of text contents are displayed on the screen of the electronic device, for example, a reading page of a public account article can be displayed, and the plurality of text contents displayed on the page include website information. When the user wishes to interact with the screen based on the website information (for example, the user wishes to jump to the website or wishes to record the website, etc.), the user can circle the website information. In response to the user's operation of circling the website information, a service entrance related to the website information is displayed on the screen of the electronic device, for example, a voice ball 1103 is optionally displayed, and the voice ball 1103 is used for the user to interact with the selected content through voice dialogue; for example, optionally, a Other recommended service entrances are displayed, such as opening a URL, a URL summary, etc., wherein, when the user clicks to open the URL service entrance, the display interface of the electronic device jumps to the page corresponding to the URL, and when the user clicks on the URL summary service entrance, the summary content corresponding to the URL is displayed on the display interface of the electronic device; the user can also view more service entrances related to the URL information by clicking on control 1104. For example, when the user clicks on the note service entrance, the URL is automatically recorded in the memo, and the electronic device can remind the user on the current page that a memo has been generated for the URL information, or not remind the user, and the user experience is imperceptible, and the user can check it in the memo later.

[0180] Among them, the URL summary refers to the summary content of the web page corresponding to the URL. Through the URL summary, the user can quickly understand the main points of the content of the web page corresponding to the URL. Its function is similar to the abstract of a paper. Before opening the web page corresponding to the URL, the user can read the URL summary first, and then further decide whether to open the web page corresponding to the URL.

[0181] like Fig.11As shown in (c) in the figure, a plurality of text contents are displayed on the screen of the electronic device, for example, an activity announcement page or a travel schedule information page may be displayed, and the plurality of text contents displayed on the page include address information. When the user wishes to interact with the screen based on the address information (for example, the user wishes to jump to the navigation page corresponding to the address or wishes to record the address, etc.), the user may circle the address information. In response to the user's operation of circling the address information, a service portal related to the address information is displayed on the screen of the electronic device. For example, a voice ball 1105 may be displayed, and the voice ball 1105 is used for the user to interact with the selected content by way of voice dialogue, for example, the user may ask by voice how far the address is from the user's current location. For example, other recommended services may be displayed in the interface. Service entrances, such as checking distance, navigation, and taking a taxi, wherein, when the user clicks on the distance checking service entrance, the display interface of the electronic device jumps to the map page, which displays the distance between the user's current location and the address, and when the user clicks on the navigation service entrance, the display interface of the electronic device jumps to the navigation page for navigating to the address; when the user clicks on the taxi service entrance, the display interface of the electronic device jumps to the taxi page for taking a taxi to the address; the user can also view more service entrances related to the address information by clicking on control 1106. For example, when the user clicks on the note service entrance, the address is automatically recorded in the memo, and the electronic device can remind the user on the current page that a memo has been generated for the address information, or not remind the user, and the user experience is imperceptible, and the user can check it in the memo later.

[0182] like Fig.11As shown in (d) in the figure, multiple text contents are displayed on the screen of the electronic device, for example, a registration order page can be displayed, and the multiple text contents displayed in the page include consultation time information. When the user wishes to interact with the screen based on the consultation time information (for example, the user wishes to jump to the calendar page corresponding to the consultation time or record the consultation time as a reminder schedule, etc.), the user can circle the consultation time information. In response to the user's operation of circling the consultation time information, a service portal related to the consultation time information is displayed on the screen of the electronic device. For example, optionally, a voice ball 1107 can be displayed, and the voice ball 1107 is used for the user to interact with the selected content through voice dialogue, for example, the user asks by voice how many days are left before the consultation time; for example, optionally, other recommended service portals can be displayed in the interface, such as adding a schedule, making a note, etc., wherein, when the user clicks on the add schedule service portal, the electronic device's The display interface jumps to the schedule adding page, in which the consultation time information is displayed on the page as an added schedule, or when the user clicks on the add schedule service entrance, a reminder pop-up window appears on the current page, reminding the user that the consultation time information has been added as a schedule, or it is automatically added as a schedule in the background without reminding the user, and the user experience is not noticeable, and the user can check it in the memo later; when the user clicks on the note service entrance, the display interface of the electronic device jumps to the memo page, and the consultation time information is displayed on the page as recorded information, or when the user clicks on the note service entrance, a reminder pop-up window appears on the current page, reminding the user that the consultation time information has been recorded in the memo, or it is automatically generated as a memo in the background without reminding the user, and the user experience is not noticeable, and the user can check it in the memo later; the user can also view more service entrances related to the consultation time information by clicking control 1108.

[0183] In some embodiments, when the selected content does not include a specific date but includes a schedule intention, the Add Schedule Service will also be displayed. For example, when the selected content includes the message "Let's have dinner together tomorrow", the Add Schedule Service will be displayed accordingly. When the selected content includes the message "It will rain tomorrow", the Add Schedule Service will not be displayed because there is no schedule intention.

[0184] like Fig.11As shown in (e) in the figure, a chat page is displayed on the screen of the electronic device, and the multiple chat record messages displayed in the chat page include flight number information. When the user wants to interact with the screen based on the flight number information (for example, the user wants to jump to the flight details page corresponding to the flight number or the user wants to book a flight corresponding to the flight number, etc.), the user can circle the flight number information. In response to the user's operation of circling the flight number information, the screen of the electronic device displays a service entry related to the flight number information. For example, optionally, a voice ball 1109 can be displayed, and the voice ball 1109 is used for the user to interact with the selected content through voice dialogue. For example, the user asks the departure time of the flight number through voice; for example, optionally, other recommended service entries can be displayed in the interface, such as booking air tickets, flight inquiry, etc., wherein, when the user clicks on the air ticket booking service entry, the display interface of the electronic device jumps to the air ticket booking page corresponding to the flight number; when the user clicks on the flight inquiry service entry, the display interface of the electronic device jumps to the air ticket inquiry page corresponding to the flight number; the user can also view more service entries related to the flight number information by clicking on the control 1110.

[0185] like Fig.11 As shown in (f), a text reading page is displayed on the screen of the electronic device, for example, it may be a paper reading page, a courseware browsing page, etc., and multiple text information and tables are displayed in the page. When the user wants to interact with the screen based on the table (for example, the user wants to record the table, etc.), the user can circle the table. In response to the user's operation of circling the table, a service entry related to the table is displayed on the screen of the electronic device. For example, optionally, a service box 1111 can be displayed, and the service box 1111 includes a copy service, a save service, and a share service, wherein the user can copy the table by clicking the copy service, save the table by clicking the save service, and share the table by clicking the share service; for example, optionally, a voice ball 1112 can be displayed, and the voice ball 1112 is used for the user to interact with the selected content through voice dialogue; for example, optionally, other recommended service entries can be displayed in the interface, such as table extraction, table analysis, etc.; the user can also view more service entries related to the table by clicking the control 1113.

[0186] It is understandable that the above Fig.10 and Fig.11In the embodiment shown, the user circles the interactive content on the screen by drawing circles on the screen with the knuckles, but this does not constitute a limitation on the manner of circling the interactive content on the screen in the embodiments of the present application. The user can also circle the interactive content on the screen by drawing circles on the screen with the fingers, such as by pressing the finger hard to circle or by pressing the finger hard and then circling; the user can also circle the interactive content on the screen by drawing circles on the screen with a stylus; the user can also circle the interactive content on the screen by drawing circles on the screen with other body parts, such as by pressing the stylus hard to circle or by pressing the stylus hard and then circling; in addition, the user can also circle the interactive content on the screen in other possible ways, such as by a mouse, etc., which is not limited in the embodiments of the present application.

[0187] It should be noted that the above Fig.10 and 11 In the illustrated embodiment, the user selects interactive content by circling on the screen as an example for explanation, but this does not limit the method of selecting interactive content in the embodiments of the present application.

[0188] In some embodiments, after the user selects a recommended service, the screen of the electronic device can remain in the current interface, and the page jumped to can be presented in a pop-up window or a floating window.

[0189] Depend on Fig.10 and Fig.11 It can be seen from the illustrated embodiment that in the method provided by the embodiment of the present application, when different contents are selected on the screen, the page will make service function recommendations corresponding to the selected contents based on the selected contents; and, the user can intelligently select any local content on the currently displayed screen, and the service function recommendations made are adapted to the selected local content, which can increase the probability that the user clicks on the recommended service function; and, the displayed service function recommendations can also be determined by performing intent analysis on the selected content, which can improve the adaptability of the service function recommendations to the selected content, and further increase the probability that the user clicks on the recommended service function.

[0190] For example, Fig.12 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0191] like Fig.12As shown in (a) in the figure, a chat interface is displayed on the screen of the electronic device, and multiple chat messages are displayed in the chat interface. A chat message received by the electronic device includes bank account information, account opening bank information and payee information. When the user wants to transfer money based on the bank account information, account opening bank information and payee information, the user can circle the bank account information, account opening bank information and payee information; Fig.12 As shown in (b) of FIG. 1 , in response to the user's operation of circling the bank account information, the account opening bank information and the payee information, the circled bank account information, the account opening bank information and the payee information are highlighted, and service entrances related to the bank account information, the account opening bank information and the payee information are displayed on the interface. For example, these service entrances may include transfer, copy all text, etc.; when the user clicks on the transfer service entrance, the display interface of the electronic device jumps to the following figure: Fig.12 (c) shows an interface for transferring money to a payee through a bank application. In this interface, the bank account information, opening bank information and payee information circled by the user have been automatically filled in the corresponding positions. The user only needs to check the relevant information and enter the transfer amount to complete the transfer to the payee.

[0192] For example, Fig.13 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0193] like Fig.13 As shown in (a) in the figure, a chat interface is displayed on the screen of the electronic device, and multiple chat messages are displayed in the chat interface. One chat message received by the electronic device is "I will take flight AB1234 at 6 pm the day after tomorrow". When the user wants to query the specific information of the flight, the user can circle the chat message "I will take flight AB1234 at 6 pm the day after tomorrow". In response to the user's operation of circling the chat message, service entrances related to the chat message are displayed on the interface of the electronic device. For example, these service entrances may include flight query, word selection panel, etc.; when the user clicks on the flight query service entrance, the display interface of the electronic device jumps to Fig.13The interface shown in (b) in the figure, in which a dialog box 1301 is displayed, wherein the dialog box 1301 displays the selected chat message "I will take flight AB1234 at 6 pm the day after tomorrow", the user's instruction "flight query", and the interaction result determined by the electronic device based on the chat message "I will take flight AB1234 at 6 pm the day after tomorrow" and the user's instruction "flight query", wherein the interaction result is the information of flight AB1234 retrieved by the electronic device, and the information of flight AB1234 can be displayed in the form of a service card. When the user clicks on the service card, the display interface of the electronic device can jump to the information interface of flight AB1234 in the flight application; in this interface, the user can also continue to interact with the chat message through the voice ball 1302 or the keyboard 1303.

[0194] For example, Fig.14 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0195] like Fig.14 As shown in (a) in the figure, the multiple text contents displayed on the screen of the electronic device include bank account information, account opening bank information and payee information. When the user wants the electronic device to remember the bank account information, account opening bank information and payee information, the user can circle the bank account information, account opening bank information and payee information. In response to the user's operation of circling the bank account information, account opening bank information and payee information, service entrances related to the bank account information, account opening bank information and payee information are displayed on the interface. For example, these service entrances may include transfer, account record, etc.; when the user clicks on the account record service entrance, the display interface of the electronic device jumps to Fig.14 The interface shown in (b) in the figure, in which a dialog box 1401 is displayed, wherein the dialog box 1401 displays the selected bank account information, the bank information and the payee information, the user's instruction "remember the account", and the interaction result determined by the electronic device based on the selected bank account information, the bank information and the payee information and the user's instruction "remember the account", the interaction result is the bank account information, the bank information and the payee information recorded by the electronic device, the recorded information can be displayed in the form of a service card, the electronic device can also remind the user to select the bank application that he or she wishes to use to transfer money to this account in the future, and the user can select the bank application that he or she wishes to use to transfer money to this account in the future through the "go to edit" control on the service card.

[0196] For example, Fig.15 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0197] like Fig.15 As shown in (a) of FIG. 1 , a picture 1501 is displayed on the screen of the electronic device, and the picture 1501 includes a display element 1502, and the display element 1502 may be, for example, a puppy; when the user wishes to interact with the screen based on the display element 1502, after selecting the display element 1502 as the interactive content, for example, the user may optionally click on a navigation bar 1503 in the interface; in response to the user clicking on the navigation bar 1503, the display interface of the electronic device is as shown in FIG. Fig.15 As shown in (b) in the figure, for example, optionally, a voice recognition box 1504 may be displayed on the interface, and a guide for guiding the user to ask a voice question may be displayed in the voice recognition box 1504, and the guide may be, for example, "Ask about the selected object on the screen..."; Fig.15 As shown in (c) of FIG. 1 , in response to the user issuing a voice question, the voice recognition box 1504 displays the user question recognized by the electronic device. For example, when the user issues a voice question “What breed of dog is this?”, the voice recognition box 1504 displays the recognized user question “What breed of dog is this?”; and then Fig.15 As shown in (d) in FIG. 1 , a dialog box 1505 may be displayed on the interface of the electronic device, in which the user's circled content and question content are displayed, as well as the answer content given by the electronic device to the user's question.

[0198] In some embodiments, the dialog box 1505 may be displayed on the original display interface of the electronic device in the form of a floating window or a pop-up window, so as to reduce the impact of the display of the dialog box 1505 on the user's browsing of the original interface of the electronic device.

[0199] In some embodiments, voice interaction between the electronic device and the user based on the selected content can also be triggered by long pressing the power button of the electronic device or voice wake-up.

[0200] Among them, the navigation bar refers to a control displayed in the interface of the electronic device that is independent of the displayed content. When the display interface of the electronic device is switched, the navigation bar is not cancelled. In other words, the navigation bar can always be displayed while the screen of the electronic device is on.

[0201] In some embodiments, in response to the user selecting an interactive content, a navigation bar 1503 is triggered to be displayed on the screen of the electronic device.

[0202] In some embodiments, during the entire process of the electronic device performing voice interaction with the user based on the interactive content, the navigation bar 1503 is always displayed on the screen of the electronic device. Optionally, in response to detecting that the voice interaction between the electronic device and the user based on the interactive content has ended, the electronic device can cancel the display of the navigation bar 1503.

[0203] In some embodiments, the display conditions of the navigation bar can be set through the setting function of the electronic device. For example, it can be set to be displayed all the time, it can be set to be displayed in the interface of certain applications, and it can also be set to be displayed during the entire process of voice interaction between the electronic device and the user based on interactive content.

[0204] For example, Fig.16 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0205] like Fig.16 As shown in (a) of FIG. 1 , a picture 1601 is displayed on the screen of the electronic device, and the picture 1601 includes a display element 1602 and multiple display elements 1603. The display element 1602 may be, for example, a puppy, and the display element 1603 may be, for example, a tree. When a user wishes to interact with the screen based on the display element 1602, after selecting the display element 1602 as the interactive content, the user may click on the voice ball 1604 in the interface. Fig.16 As shown in (b) of FIG. 1 , for example, optionally, in response to the user clicking the voice ball 1604, a voice recognition box 1605 may be displayed on the display interface of the electronic device, and a guide for guiding the user to ask a voice question may be displayed in the voice recognition box 1605, and the guide may be, for example, "Ask about the selected object on the screen..."; Fig.16 As shown in (c) in FIG. 1 , for example, optionally, in response to the user issuing a voice question, the voice recognition box 1605 displays the user question recognized by the electronic device. For example, when the user issues a voice question “What breed of dog is this?”, the voice recognition box 1605 displays the recognized user question “What breed of dog is this?”; and then Fig.16 As shown in (d) in FIG. 1 , for example, optionally, a dialog box 1606 is displayed on the interface of the electronic device, in which the user's circled content and question content are displayed, and the answer content given by the electronic device to the user's question is displayed; on this basis, Fig.16As shown in (e), the user can continue to click the voice ball 1604 to further ask questions based on the selected content, and the electronic device will give answers to the user's further questions. Alternatively, the user can select the display elements on the screen again and further ask questions based on the re-selected content, and the electronic device will give answers to the user's further questions.

[0206] In some embodiments, the dialog box 1606 may be displayed on the original display interface of the electronic device in the form of a floating window or a pop-up window, so as to reduce the impact of the display of the dialog box 1606 on the user's browsing of the original interface of the electronic device.

[0207] It can be understood that when a user selects a local display element in a picture and conducts voice interaction with the screen based on the selected local display element, the answer determined by the electronic device can be determined based on the local display element selected by the user, or it can be determined based on the selected local display element and combined with all display elements currently displayed on the screen (i.e., selected display elements and unselected display elements). On this basis, it can also be determined in combination with logical analysis, intent analysis, semantic analysis, etc.

[0208] In such Fig.16 (d) and Fig.16 In the example shown in (e), for example, optionally, when the user asks "What breed of dog is this?", the electronic device searches according to the selected display element 1602, and gives an answer "The breed of this dog is ×××" according to the search result, and can also provide the user with more information about the display element 1602, such as "This breed of dog comes from ×××, and has a gentle temperament..."; on this basis, the user can further ask "Where is this dog?", and the electronic device determines the answer "This dog is in the woods..." according to all the display elements included in the picture 1601.

[0209] For more explanation of this embodiment, please refer to Figures 6 to 11 Description section of the illustrated embodiment.

[0210] In this embodiment, after selecting interactive content on the screen, you can interact with the screen of the electronic device through voice based on the selected interactive content, and can realize multiple rounds of voice interaction between the selected content and the screen. In addition, you can perform multiple rounds of voice interaction between the selected local content on the screen and the screen, thereby improving the user experience; in addition, when the selected content is a local display element, the answer of the electronic device can be determined based on the analysis of all display elements of the page, which can improve the accuracy of the answer.

[0211] For example, Fig.17A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0212] like Fig.17 As shown in (a) of FIG. 1 , a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on a display element 1701 in the page, the user may, for example, circle the display element 1701 to invoke a service entry associated with the display element 1701. For example, these service entries may include a voice ball 1702, a small art object service, a how-to-feed service, a copy service, a save service, a share service, etc.; Fig.17 As shown in (b) of FIG. 1 , for example, optionally, in response to the user clicking the voice ball 1702, a voice recognition box 1703 is displayed on the display interface of the electronic device, and a guide for guiding the user to ask a voice question may be displayed in the voice recognition box 1703, and the guide may be, for example, "Ask about the selected object on the screen". Fig.17 As shown in (c) of FIG. 1 , for example, optionally, in response to the user's voice questioning operation, the voice recognition box 1703 displays the question recognized by the electronic device. For example, when the user's voice question is "Is this dog smart?", the voice recognition box 1703 displays the recognized question "Is this dog smart?"; then Fig.17 As shown in (d), for example, optionally, a dialog box 1704 is displayed on the interface of the electronic device, in which the user's circled content (display element 1701) and question content (Is this dog smart?) are displayed, as well as the answer content given by the electronic device to the user's question, and the answer content may be, for example, "This dog has service functions such as music playback, whole-house smart control, and smart conversation..."

[0213] In some embodiments, Fig.17 Based on the embodiment shown in (d), the user can continue to click the voice ball 1702 to further ask questions based on the selected display element 1701, and the electronic device gives answers to the user's further questions. Alternatively, the user can select the display element on the screen again and further ask questions based on the re-selected content, and the electronic device gives answers to the user's further questions.

[0214] It can be understood that when a user selects a local display element on the screen and performs voice interaction with the screen based on the selected local display element, the answer determined by the electronic device can be determined based on the local display element selected by the user, or based on all display elements in the entire display interface (i.e., selected display elements and unselected display elements), or based on the entire document content corresponding to the currently displayed document (including selected display elements and unselected display elements, as well as other related content not displayed on the screen). On this basis, it can also be determined in combination with logical analysis, intent analysis, semantic analysis, etc.

[0215] In one example, on a paper reading page, a user selects a picture or a paragraph of text on the page, or a combination of a paragraph of text and a picture, as interactive content for voice interaction. The answer given by the electronic device based on the user's question can be centered on the selected content and determined in combination with the content of the entire paper or all the content currently displayed on the screen.

[0216] Similarly, in article reading scenarios, novel reading scenarios, etc., the answers obtained by voice interaction can be determined by the electronic device based on the selected content, combined with the overall content (including the content displayed on the screen and related content not displayed on the screen) or combined with all the content currently displayed on the screen.

[0217] In this embodiment, after selecting interactive content on the screen, you can interact with the screen of the electronic device through voice based on the selected interactive content, and can realize multiple rounds of voice interaction between the selected content and the screen. In addition, you can perform multiple rounds of voice interaction between the selected local content on the screen and the screen, thereby improving the user experience; in addition, when the selected content is a local display element, the answer of the electronic device can be determined based on the analysis of all display elements of the page, which can improve the accuracy of the answer.

[0218] For example, Fig.18 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0219] like Fig.18 As shown in (a) of FIG. 1 , a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on a display element 1801 in the page, the user can circle the display element 1801 to determine that the interactive content is the display element 1801; further, the user can wake up the voice function through a wake-up word, for example, Fig.18As shown in (b) in the figure, for example, optionally, the user can wake up the voice function of the electronic device through the wake-up word "Xiao Yi Xiao Yi", so that a voice recognition box 1802 is displayed on the interface of the electronic device, and a guide word for guiding the user to issue a voice instruction can be displayed in the voice recognition box 1802, and the guide word can be, for example, "I'm here! Is there anything I can help you with?" At the same time, for example, optionally, the electronic device can also issue the guide word in a voice manner; Fig.18 As shown in (c), for example, optionally, in response to a user asking a voice question, the electronic device may display a dialog box 1803 on the screen of the electronic device after recognizing the user's question, in which the circled content (display element 1801) and the question content (for example: Is this dog smart?) are displayed, as well as the answer content given by the electronic device to the user's question, which may be, for example, "This dog has service functions such as music playback, whole-house smart control, and smart conversation..." At the same time, the electronic device may also issue the answer content by voice.

[0220] In some embodiments, after the user circles the display element 1801, the voice ball may not be displayed on the display interface of the electronic device.

[0221] In some embodiments, Fig.18 On the basis of the embodiment shown in (c), for example, optionally, the user can continue to ask questions based on the selected display element 1801 or reselect other display elements by continuing to use the wake-up word or clicking the voice ball, and the electronic device gives answers to the user's further questions.

[0222] It can be understood that when a user selects a local display element on the screen and performs voice interaction with the screen based on the selected local display element, the answer determined by the electronic device can be determined based on the local display element selected by the user, or based on all display elements in the entire display interface (i.e., selected display elements and unselected display elements), or based on the entire document content corresponding to the currently displayed document (including selected display elements and unselected display elements, as well as other related content not displayed on the screen). On this basis, it can also be determined in combination with logical analysis, intent analysis, semantic analysis, etc.

[0223] In this embodiment, after selecting interactive content on the screen, the user can directly initiate voice interaction with the screen of the electronic device based on the selected interactive content by means of a wake-up word. The user does not need to manually click on the voice ball, which can further free the user's hands and improve the user's convenience.

[0224] For example, Fig.19A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0225] like Fig.19 As shown in (a) of FIG. 1 , a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on the display element 1901 in the page, the user can directly wake up the voice function of the electronic device through a wake-up word, for example, Fig.19 As shown in (b) in the figure, for example, optionally, the user can wake up the voice function of the electronic device through the wake-up word "Xiao Yi Xiao Yi", so that a voice recognition box 1902 is displayed on the interface of the electronic device, and a guide word for guiding the user to issue a voice instruction can be displayed in the voice recognition box 1902, and the guide word can be, for example, "I'm here! Is there anything I can help you with?" At the same time, for example, optionally, the electronic device can also issue the guide word in a voice manner; Fig.19 As shown in (c) of FIG. 1 , for example, optionally, in response to the user's voice questioning operation, the electronic device may display a dialog box 1903 on the screen of the electronic device after recognizing the user's question, in which the electronic device displays the following information based on the user's question content: Fig.19 The target interactive content (display element 1901) and question content (for example: Is this dog smart?) are extracted from the display interface shown in (a), and the answer content given by the electronic device to the user's question is displayed. The answer content may be, for example, "This dog has service functions such as music playback, whole-house smart control, and smart conversation..." At the same time, the electronic device can also issue the answer content by voice.

[0226] In some embodiments, Fig.19 In (c), the voice ball may not be displayed on the display interface of the electronic device.

[0227] In some embodiments, Fig.19 Based on the embodiment shown in (c), the user can continue to ask questions based on display element 1901 or other display elements by continuing to use the wake-up word or clicking the voice ball, and the electronic device will answer the user's further questions.

[0228] It can be understood that the answer determined by the electronic device can be determined based on all display elements in the entire display interface. On this basis, it can also be determined in combination with logic analysis, intent analysis, semantic analysis, etc.

[0229] In this embodiment, when the user wants to interact with the screen based on the local content displayed on the screen, there is no need to select the interactive content on the screen. The user can directly initiate voice interaction with the screen of the electronic device by means of a wake-up word. The electronic device can determine the corresponding answer content based on the content of the user's question and all the display elements displayed on the screen. The user does not need to manually click on the voice ball, which can further free the user's hands and improve the user's convenience.

[0230] For example, Fig. 20 A schematic diagram of an interface corresponding to another method for interaction based on screen content provided in an embodiment of the present application is shown.

[0231] like Fig. 20 As shown in (a) of FIG. 1 , a text reading page is displayed on the screen of the electronic device; when the user wishes to interact with the screen based on the display element 2001 in the page, the user can directly wake up the voice function of the electronic device through the wake-up word, for example, optionally, as Fig. 20 As shown in (b) of FIG. 1 , the user can wake up the voice function of the electronic device by using the wake-up word “Xiao Yi Xiao Yi”, so that a voice recognition box 2002 is displayed on the interface of the electronic device. In the voice recognition box 2002, a guide word for guiding the user to issue a voice instruction can be displayed. The guide word can be, for example, “I’m here! Is there anything I can help you with?” At the same time, the electronic device can also optionally issue the guide word by voice; Fig. 20 As shown in (c) in the figure, for example, optionally, in response to the user's voice questioning operation, if the electronic device cannot determine the target interactive content based on the user's question content, a guide message to remind the user to circle the target interactive object may be displayed in the voice recognition box 2002, for example, "Please circle the target object you need to know!"; Fig. 20 As shown in (d), in response to the user's operation of circling the target interactive object (display element 2001), a dialog box 2003 is displayed on the screen of the electronic device. In the dialog box 2003, the circled content (display element 2001) and the question content (for example: Is this dog smart?) are displayed, as well as the answer content given by the electronic device to the user's question. The answer content may be, for example, "This dog has service functions such as music playback, whole-house smart control, and smart conversation..." At the same time, the electronic device can also issue the answer content by voice.

[0232] In some embodiments, Fig. 20 In (c), the voice ball may not be displayed on the display interface of the electronic device.

[0233] In some embodiments, Fig. 20Based on the embodiment shown in (c), the user can continue to ask questions based on display element 2001 or other display elements by continuing to use the wake-up word or clicking the voice ball, and the electronic device will give answers to the user's further questions.

[0234] It can be understood that the answer determined by the electronic device can be determined based on all display elements in the entire display interface. On this basis, it can also be determined in combination with logic analysis, intent analysis, semantic analysis, etc.

[0235] It should be noted that the above Figures 15 to 20 In the illustrated embodiment, when the user manually selects interactive content, the user selects the interactive content by circling on the screen as an example for explanation, but this does not limit the method of selecting the interactive content in the embodiments of the present application.

[0236] In some embodiments, when a user interacts with local content on the screen by waking up with a wake-up word and interacting with voice, the user's interactive behavior through voice is not limited to voice dialogue, but may also include the user's specific interactive behavior in the form of voice, and may trigger the interaction between the interactive content and other applications through voice commands, for example: recording the interactive content as a memo through voice commands; sending the interactive content to friends via email, chat, text messaging and other applications through voice commands; recording the interactive content as a reminder schedule through voice commands, etc.

[0237] It should be understood that the above Figures 6 to 20 The embodiments shown may be implemented individually or in combination of any two or more.

[0238] For example, Fig.21 FIG. 1 is a schematic flow chart of a method for human-computer interaction provided in an embodiment of the present application. Fig.21 As shown, the method includes:

[0239] S2101: In response to detecting that a user selects an interaction area on a first interface of an electronic device, the electronic device obtains first interaction content associated with the interaction area based on the interaction area.

[0240] S2102: The electronic device obtains a first voice command.

[0241] In some embodiments, in response to a user selecting an interactive area on a first interface, a voice icon is displayed on the first interface; in response to detecting a user touch operation on the voice icon, the electronic device obtains the first voice instruction.

[0242] In some embodiments, the speech icon may be a speech ball.

[0243] In some embodiments, in response to a wake-up word issued by a user, the electronic device obtains the first voice instruction, where the wake-up word may be, for example, "Xiaoyi Xiaoyi!".

[0244] In some embodiments, in response to a user's touch operation on the first navigation bar, the electronic device obtains the first voice instruction, and the first interface includes the first navigation bar.

[0245] S2103: The electronic device performs a first operation according to the first voice instruction and the first interactive content.

[0246] In some embodiments, the electronic device performs the first operation according to the first voice command, the first interactive content, and the first display content, wherein the first display content is the content on the first interface of the electronic device except the first interactive content.

[0247] In some embodiments, the first operation is that the electronic device displays one or more recommended interactive services, and the one or more recommended interactive services are correlated with the first interactive content; in response to the user selecting the first recommended interactive service operation, the electronic device executes the first recommended interactive service on the first interactive content, and the one or more recommended interactive services include the first recommended interactive service.

[0248] Among them, the operation of the user selecting the first recommended interactive service may be an operation of the user clicking on the first recommended interactive service, or an operation of the user issuing a second voice command to indicate the selection of the first recommended interactive service, or an operation of the user dragging the first interactive content to the first recommended interactive service.

[0249] In some embodiments, the electronic device may directly execute a first operation for providing a recommended interactive service, the recommended interactive service being related to the first interactive content, or the electronic device may display an interface for executing the recommended interactive service.

[0250] Optionally, after the electronic device directly performs the first operation, a first reminder may be displayed on the interface of the electronic device, where the first reminder is used to remind the user that the first operation has been performed on the first interactive content.

[0251] In some embodiments, in response to the electronic device completing the first operation, the electronic device is triggered to acquire and detect a voice instruction for performing multimodal fusion with the first interactive content again; the electronic device performs a second operation based on the result of the first operation, the voice instruction and the first interactive content.

[0252] One or more of the modules or units described herein can be implemented by software, hardware or a combination of the two. When any of the above modules or units are implemented in software, the software exists in the form of computer program instructions and is stored in a memory, and a processor can be used to execute the program instructions and implement the above method flow. The processor may include but is not limited to at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller (MCU), or an artificial intelligence processor and other types of computing devices that run software, each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor may be built into an SoC (system on chip) or an application specific integrated circuit (ASIC), or it may be an independent semiconductor chip. In addition to the core used to execute software instructions for operations or processing in the processor, necessary hardware accelerators may be further included, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or a logic circuit that implements a dedicated logic operation.

[0253] When the modules or units described in this document are implemented in hardware, the hardware can be any one or any combination of a CPU, a microprocessor, a DSP, an MCU, an artificial intelligence processor, an ASIC, a SoC, an FPGA, a PLD, a dedicated digital circuit, a hardware accelerator, or a non-integrated discrete device, which can run the necessary software or be independent of the software to execute the above method flow.

[0254] When the modules or units described herein are implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0255] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0260] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0261] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for human-computer interaction, characterized in that: The method comprises: In response to a user selecting an interactive area on a first interface of an electronic device, the electronic device acquires first interactive content associated with the interactive area, and the electronic device starts voice detection; The electronic device acquires a first voice instruction based on the voice detection; The electronic device performs a first operation according to the first voice instruction and the first interactive content.

2. The method according to claim 1, characterized in that The electronic device starts detecting a first voice instruction for performing multimodal fusion with the first interactive content, including: In response to the electronic device acquiring the first interactive content, the electronic device is triggered to detect the first voice instruction.

3. The method according to claim 1, characterized in that The electronic device starts detecting a first voice instruction for performing multimodal fusion with the first interactive content, including: The electronic device displays a second interface, the second interface includes a first control, and the first control is used to trigger the electronic device to detect the first voice instruction for multimodal fusion with the first interactive content.

4. The method according to claim 1, characterized in that The electronic device starts detecting a first voice instruction for performing multimodal fusion with the first interactive content, including: In response to a wake-up word issued by a user, the electronic device is triggered to detect the first voice instruction for multimodal fusion with the first interactive content.

5. The method according to any one of claims 1 to 4, characterized in that The first operation is that the electronic device displays a third interface, wherein the third interface includes one or more recommended interactive services. The one or more recommended interactive services are correlated with the first interactive content.

6. The method according to claim 5, characterized in that The method further comprises: In response to detecting that a user clicks on a first recommended interactive service, the electronic device executes the first recommended interactive service; or, In response to detecting a second voice instruction of the user, the electronic device executes the first recommended interactive service, wherein the second voice instruction is used to instruct selection of the first recommended interactive service; or, In response to detecting an operation of a user dragging the first interactive content to the first recommended interactive service, the electronic device executes the first recommended interactive service, and the one or more recommended interactive services include the first recommended interactive service.

7. The method according to any one of claims 1 to 6, characterized in that The electronic device performs a first operation according to the first voice instruction and the first interactive content, including: The electronic device performs the first operation according to the first voice instruction, the first interactive content and the first display content, where the first display content is the content on the first interface except the first interactive content.

8. The method according to any one of claims 1 to 7, characterized in that The electronic device performs a first operation according to the first voice instruction and the first interactive content, including: The electronic device directly performs the first operation, the first operation is used to provide a recommended interactive service, and the recommended interactive service is related to the first interactive content, or The electronic device displays a fourth interface, where the fourth interface is an interface for executing the recommendation interactive service.

9. The method according to claim 5, 6 or 8, characterized in that: When the first interactive content includes a picture, the recommended interactive service includes one or more of a picture recognition service, a sharing service, and a saving service; When the first interactive content includes a chat record, the recommended interactive service includes one or more of a copy service, a sharing service, a saving service, and an editing service; When the first interactive content includes text content, the recommended interactive service includes one or more of a copy service, a reading service, a saving service, and an editing service; When the first interactive content includes account information, the recommended interactive service includes a recording service and / or a transfer service; When the first interactive content includes website information, the recommended interactive service includes one or more of a recording service, a sharing service, an opening website service, and a viewing website summary service; When the first interactive content includes address information, the recommended interactive service includes one or more of a distance check service, a navigation service, and a taxi service; When the first interactive content includes flight information, the recommended interactive service includes one or more of a ticket booking service, a flight query service, and a schedule adding service; When the first interactive content includes consultation time information, the recommended interactive service includes adding a schedule service and / or a recording service; When the first interactive content includes a table, the recommended interactive service includes one or more of an adding and saving service, a copying service, and a sharing service.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: In response to the electronic device completing the first operation, triggering the electronic device to detect a third voice instruction for performing multimodal fusion with the first interactive content again; The electronic device performs a second operation according to the result of the first operation, the third voice instruction and the first interactive content.

11. The method according to any one of claims 1 to 10, characterized in that In response to a user selecting an interactive area on a first interface of the electronic device, the electronic device acquiring first interactive content associated with the interactive area, including: In response to detecting an operation of a user circling the interactive area on the first interface, the electronic device acquires the first interactive content; or, In response to detecting that the user selects the interactive area on the first interface, the electronic device acquires the first interactive content; or, In response to detecting that the user long presses or double-clicks the interaction area on the first interface, the electronic device acquires the first interaction content; or, In response to detecting an operation of a user dragging the interactive area on the first interface, the electronic device acquires the first interactive content.

12. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 11 is implemented.

14. A chip, characterized in that: Instructions are stored in the chip, and when the instructions are executed, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The computer program product stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 11 is implemented.

Images