Interaction method and electronic equipment

Abstract

The invention discloses an interaction method and electronic equipment, the electronic equipment can support a user to start a natural language interaction function through a long-press operation and display a natural language interaction entrance, and after a voice / text instruction input by the user through a natural language is detected, the electronic equipment can display the instruction at the natural language interaction entrance, so that the interaction effect is improved. After the instruction is input, the electronic equipment can determine whether the instruction has a default object or a referral, and if yes, the electronic equipment can extract and recognize the interaction object from the interface content at the position of the long-press operation of the user or at the position of the long-press operation of the user and the position nearby the long-press operation of the user so as to determine the interaction object. And the interaction object is substituted into an input instruction for understanding and execution. Thus, through fusion of the touch operation and natural language interaction, selection and further operation of the interaction object can be efficiently achieved, the problem that natural language is inconvenient to express the interaction object is solved, and the man-machine interaction efficiency and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an interaction method and electronic device. Background Art

[0002] With the development of electronic devices, more and more electronic devices can provide natural language interaction functions, such as voice interaction functions. The voice interaction function can support electronic devices to receive user voice commands and understand and execute the voice commands.

[0003] However, in some scenarios where an interactive object needs to be processed (for example, editing a flower in a picture containing multiple flowers), the user is sometimes unable to accurately express the interactive object through voice alone, and the electronic device is unable to accurately and quickly recognize the user's intention. Multiple rounds of voice interaction may be required to complete the task, resulting in a poor user experience. Summary of the invention

[0004] The embodiments of the present application provide an interaction method and an electronic device, which can efficiently realize the selection and further operation of interaction objects through the integration of user touch operation and natural language interaction, solve the problem of inconvenience in expressing interaction objects in natural language, and improve human-computer interaction efficiency and user experience.

[0005] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to an electronic device, wherein the electronic device includes a screen, and the method includes: the electronic device displays a first user interface of a first application; the electronic device detects a first operation of a user on the first user interface, and the position of the first operation on the screen is a first position; in response to the first operation, the electronic device turns on a natural language interaction function, and the natural language interaction function is used for the electronic device and the user to interact through natural language instructions; the electronic device detects that the user inputs a first natural language instruction, and determines that the natural language interaction object is a first interaction object based on the first natural language instruction and the first position, and the first interaction object is included in the first user interface; the electronic device executes the operation corresponding to the first natural language instruction on the first interaction object.

[0006] The first application may be any application installed on the electronic device. It is worth noting that system programs such as the desktop are also applications. The first user interface may be any user interface provided by the first application. The first operation may be, for example, a single-finger or multi-finger long-press operation (such as a long-press operation performed by a user's finger or a touch device), a click operation, etc. The first position may be any position on the screen. The first natural language instruction may be a natural language instruction input by a user through a natural language interaction function. The first interactive object may be an object for natural language interaction, such as a picture, video, text, file, application or application icon, card, and other objects.

[0007] By implementing the method provided in the first aspect, the electronic device can support the user to start the natural language interaction function through the first operation. After detecting the voice / text command input by the user through natural language, the electronic device can extract and identify the interactive object of the interface content at the location of the user's first operation or at the location of the user's first operation and its vicinity to determine the interactive object, and substitute the interactive object into the input command for understanding and execution. In this way, through the integration of touch operation and natural language interaction, the selection of interactive objects and further operations can be efficiently realized, solving the problem of inconvenience in expressing interactive objects in natural language, and improving human-computer interaction efficiency and user experience.

[0008] In a possible implementation, the first operation is a long press operation.

[0009] In this way, the user can trigger the electronic device to start the natural language interaction function through a long press operation, and can trigger the electronic device to select an interaction object, thereby solving the problem of inconvenience in expressing the interaction object in natural language.

[0010] In one possible implementation, the electronic device determines that a natural language interaction object is a first interaction object based on a first natural language instruction and a first position, specifically including: when the first natural language instruction does not include the first keyword, the first keyword is a keyword used to indicate a natural language interaction object, and the electronic device determines that the first interaction object is an object that can be identified at a first position on the first user interface; or, when the first natural language instruction includes the first keyword, the electronic device determines that the natural language interaction object indicated by the first keyword is the first interaction object.

[0011] The first keyword may be a word used to specify an interactive object (such as "this", "it", "that", etc.), or a word used to indicate the name of the interactive object.

[0012] That is to say, after the first natural language instruction is input, the electronic device can determine whether there is a default object in the first natural language instruction (that is, the first natural language instruction does not include words used to indicate the name of the interactive object) or a pronoun (that is, the first natural language instruction includes words such as this, that, it, etc. used to refer to the interactive object). If so, the electronic device can determine the interactive object based on the first position; if not, the electronic device can determine that the natural language interactive object indicated by the first keyword is the first interactive object, and can perform a corresponding operation based on the first natural language instruction.

[0013] In a possible implementation, after the natural language interaction function is enabled on the electronic device, the method further includes: the electronic device displays a natural language interaction entrance, where the natural language interaction entrance is used to display a first natural language instruction.

[0014] In this way, the electronic device can prompt the user that the natural language interaction function has been turned on by displaying the natural language interaction entrance, and after the user enters a natural language instruction, the electronic device can display the instruction on the natural language interaction entrance so that the user can see the instruction he or she entered in time, thereby improving the user experience.

[0015] In a possible implementation, the natural language interaction entrance is located at the first position or near the first position.

[0016] In this way, the natural language interaction entrance is placed wherever the first position is, which can provide users with a better visual experience and facilitate users to find and use the natural language interaction entrance, thereby facilitating interaction between the electronic device and the user through natural language commands.

[0017] In one possible implementation, after the natural language interaction function is turned on in the electronic device, the method further includes: if the electronic device does not detect the user inputting a natural language instruction within a first preset time period, the electronic device outputs a first prompt, and the first prompt is used to prompt the user to input a recognizable natural language instruction corresponding to the first interactive object.

[0018] The first preset time period may be a preset time period (eg, 2 seconds, 3 seconds, etc.) after the electronic device starts detecting a natural language command input by the user.

[0019] In this way, when the user does not know how to perform natural language interaction, the electronic device can output corresponding prompts (such as voice prompts or text prompts) so that the user can quickly learn how to perform natural language interaction, thereby improving interaction efficiency and user experience.

[0020] In one possible implementation, the first natural language instruction includes any one of a first voice instruction or a first text instruction, and the method also includes: the electronic device detects a second operation of the user on the natural language interaction entrance; the electronic device detects that the user inputs the first natural language instruction, specifically including: in response to the second operation, the electronic device detects the first voice instruction input by the user through a microphone, or the electronic device calls up and displays a virtual keyboard and detects the first text instruction input by the user through the virtual keyboard.

[0021] The second operation may be, for example, a click operation, the first language instruction may be a voice instruction input by the user through a microphone, the first text instruction may be a text instruction input by the user through a virtual keyboard, and the second operation may be, for example, a click operation, etc.

[0022] In this way, during the natural language interaction between the electronic device and the user, the electronic device can not only support the user's voice input, but also support the user's text input. The interaction method is more flexible and can be suitable for users with different input habits.

[0023] In a possible implementation, after the electronic device detects that the user inputs the first natural language instruction, the method further includes: if the electronic device detects that the user cancels the first operation, the electronic device no longer detects the natural language instruction input by the user.

[0024] In this way, after the user cancels the first operation, the electronic device can determine that the user has completed inputting the natural language instruction, so that the next operation can be executed.

[0025] In a possible implementation, the method also includes: when the electronic device detects that the user long presses and drags the first interactive object, the electronic device reduces the display of the natural language interaction entrance, or cancels the display of the natural language interaction entrance; when the electronic device detects that the first interactive object does not move again within a second preset time period, the electronic device resumes displaying the natural language interaction entrance.

[0026] The second preset time period may be a period of time (eg, 2 seconds, 3 seconds, etc.) after the user stops dragging the first interactive object.

[0027] In this way, when the user long presses and drags the first interactive object, it may mean that the user currently has no intention of conducting natural language interaction. In this case, the electronic device can reduce the display of or cancel the display of the natural language interaction entrance to reduce the obstruction of the interface content by the display of the natural language interaction entrance. After the user stops dragging the first interactive object, it may mean that the user currently has the intention of conducting natural language interaction. In this case, the electronic device can resume displaying the natural language interaction entrance so as to make timely preparations for natural language interaction.

[0028] In one possible implementation, the first natural language instruction does not include keywords that directly indicate the name of the interactive object. Since the object that currently needs to be interacted with has been determined through the first operation, or the first operation and other recognition operations, the user does not need to excessively consider how to clearly describe the location or name of the object when entering the natural language instruction so as to facilitate accurate recognition by the electronic device. It is only necessary to verbally use some words that indirectly refer to the object, such as "this" and "that", or even directly describe the instruction to be operated without indicating the object. In this way, it avoids the user spending too much time describing an object that is difficult to describe, improves the convenience of the user's natural language instruction interaction, and helps to improve the user's use of natural language instructions.

[0029] In a possible implementation, the first interactive object is a first picture or the first content in the first picture, the first content is a part of the content in the first picture, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: when the first natural language instruction is used to indicate the intention to send the first interactive object to the second application, the electronic device sends the first interactive object from the first application to the second application; or, when the first natural language instruction is used to indicate the intention to erase, the electronic device erases the first content from the first picture; or, when the first natural language instruction is used to indicate the intention to replace the first content with the second content, the electronic device replaces the first content with the second content; or, when the first natural language instruction is used to indicate the intention to replace the first content with the second content When the natural language instruction is used to indicate the intention to insert a third content, the electronic device inserts the third content into the first picture; or, when the first natural language instruction is used to indicate the intention to identify, the electronic device identifies the first interactive object and displays the identification result; or, when the first natural language instruction is used to indicate the intention to beautify, the electronic device beautifies the first interactive object, wherein the beautification intention includes one or more of the following intentions: beautification, body beautification, blurring, mosaic, graffiti, adding borders / stickers / marks; or, when the first natural language instruction is used to indicate the intention to adjust parameters, the electronic device adjusts the parameters of the first interactive object, wherein the parameters include one or more of the following: filter style, brightness, saturation, sharpness.

[0030] Among them, the first picture may be a picture in the first application, the first content may be a part of the content in the first picture (for example, the cutout body), the second application may be an application different from the first application, the second content may be content searched by the electronic device to replace the first content, and the third content may be content searched by the electronic device to be inserted into the first picture.

[0031] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on pictures (such as sending, erasing, replacing, inserting, identifying, beautifying, style transfer, parameter adjustment, etc.).

[0032] In one possible implementation, the first interactive object is a first video, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate a sending intention, the electronic device sends the first video from a first application to a second application; or, when the first natural language instruction is used to indicate a cropping intention, the electronic device crops the first video and saves the cropped first video segment or first video screen.

[0033] The first video may be a video in a first application, and the second application may be an application different from the first application.

[0034] In this way, through natural language interaction, electronic devices can easily and quickly implement various operations on videos (such as sending and cropping video content).

[0035] In one possible implementation, the first interactive object is a first chat entry, which includes a chat message. The electronic device performs an operation corresponding to a first natural language instruction on the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to display a summary of unread messages included in the chat message, the electronic device searches for the unread messages, and generates and displays a summary of the unread messages based on the unread messages; or, when the first natural language instruction is used to indicate an intention to query relevant information of the chat message, the electronic device queries and displays relevant information, wherein the relevant information includes any one of the following: the content of the chat message, the sender of the chat message.

[0036] The first chat entry may be a chat entry in a first application (eg, a social application).

[0037] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on chat messages (such as displaying unread message summaries, question and answer queries, etc.).

[0038] In one possible implementation, the first interactive object is an icon of a first document, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to display a content summary of the first document, the electronic device obtains the content of the first document, and generates and displays a summary of the content based on the content; or, when the first natural language instruction is used to indicate an intention to display notes in the first document, the electronic device queries and displays notes in the first document, wherein the notes include one or more of the following: marks, annotations; or, when the first natural language instruction is used to indicate an intention to translate the content of the first document, the electronic device translates the content of the first document and displays the translation result; or, when the first natural language instruction is used to indicate an intention to perform format conversion / beautification on the first document, the electronic device performs format conversion / beautification on the first document, generates and displays the first document after format conversion / beautification.

[0039] The first document may be a document on an electronic device.

[0040] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on documents (such as displaying document summaries, displaying notes in documents, translating document content, converting / beautifying document formats, etc.).

[0041] In one possible implementation, the first interactive object is a first text, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to collect the first text, the electronic device collects the first text; or, when the first natural language instruction is used to indicate an intention to query the first text, the electronic device queries the first text and displays the query results, which include relevant information of the first text.

[0042] The first text may be text in the first application.

[0043] In this way, through natural language interaction, electronic devices can easily and quickly implement various operations on text (such as text collection, text query, etc.).

[0044] In one possible implementation, the first interactive object is a second text, the first natural language instruction includes a third text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: the electronic device combines the second text and the third text into one or more fourth texts; the electronic device replaces the second text with the fourth text, and displays the fourth text.

[0045] The second text may be text to be modified in the first application, the third text may be text input by a user through natural language, and the fourth text is a semantically coherent text generated by combining the second text and the third text.

[0046] In this way, through natural language interaction, electronic devices can easily and quickly implement content modification operations on text.

[0047] In a possible implementation, when there are multiple fourth texts, before the electronic device replaces the second text with the fourth text, the method further includes: the electronic device outputs a second prompt, where the second prompt is used to prompt the user to select any one of the multiple fourth texts.

[0048] The second prompt may be a prompt output by the electronic device in the form of voice or text.

[0049] In this way, it is convenient for the user to select a newly generated text (ie, the fourth text) in time to modify the content of the original text (ie, the first text).

[0050] In one possible implementation, the first interactive object is a fifth text, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to modify the text format of the fifth text, the electronic device modifies the text format of the fifth text; wherein the text format includes one or more of the following: font, font size, color, alignment, title style, body style, bold, italic, and underline.

[0051] The fifth text may be text to be modified in the first application.

[0052] In this way, through natural language interaction, electronic devices can easily and quickly implement format modification operations on text.

[0053] In a second aspect, an embodiment of the present application provides an interaction method, which is applied to an electronic device, and the method includes: the electronic device displays a first user interface, which includes a first page of a first document; the electronic device detects a first operation of a user on the first page; in response to the first operation, the electronic device turns on a natural language interaction function, and the natural language interaction function is used for the electronic device and the user to interact through natural language instructions; the electronic device detects that the user inputs a first natural language instruction; the electronic device generates a first batch of annotations based on the first natural language instruction, and displays the first batch of annotations on the first user interface.

[0054] Among them, the first user interface can be a user interface after the electronic device enters the annotation mode, the first document can be a document on the electronic device, the first page can be a page in the first document, the first operation can be, for example, a long press operation, etc., the first natural language instruction can be a natural language instruction input by the user through the natural language interaction function, and the first annotation can be an annotation generated by the electronic device based on the first natural language instruction input by the user.

[0055] By implementing the method provided in the second aspect, when the electronic device opens a certain document and enters the annotation mode, the electronic device can support the user to trigger the electronic device to start the natural language interaction function through, for example, a long press operation (for example, a long press operation performed by the user's finger or a touch device), and can add annotations to the document based on the natural language interaction function, which is convenient and fast, simplifies user operations, and improves user experience.

[0056] In a possible implementation, the first operation is a long press operation.

[0057] In this way, the user can trigger the electronic device to start the natural language interaction function through a long press operation, which is convenient and quick.

[0058] In one possible implementation, the position of the first operation on the screen of the electronic device is the first position, and the method also includes: the electronic device identifies whether there is an interactive object at the first position, and if not, the electronic device displays the first batch of annotations at the first position; if so, the electronic device displays the first batch of annotations at a second position, and the second position is the same as or different from the first position.

[0059] The first position may be any position on the screen, and the second position may be a position on the screen that is the same as or different from the first position.

[0060] In an embodiment of the present application, when there is no interactive object at the first position, the electronic device may display the annotation at the first position; when there is an interactive object at the first position, the electronic device may still display the annotation at the first position, or may display the annotation at a position different from the first position.

[0061] In a possible implementation, the second position is a center position of the interactive object.

[0062] In this way, when there is an interactive object at the first position, the electronic device can display the annotation at the center of the position where the interactive object is located, which is more aesthetically pleasing.

[0063] In a possible implementation, the method further includes: the electronic device adds a mark to the interactive object, and displays the marked interactive object, wherein the mark includes: a highlight mark.

[0064] In this way, when there is an interactive object at the first position, the electronic device can add a mark (such as a highlight mark, etc.) to the interactive object to highlight the interactive object generating the annotation for the user to view. The mark style added to the interactive object is not limited.

[0065] In one possible implementation, the method also includes: the electronic device detects a second operation of the user on the first page; in response to the second operation, the electronic device turns on the natural language interaction function, and determines the distance between the location of the second operation and the location of the first annotation; the electronic device detects that the user inputs a second natural language instruction, and generates a second annotation based on the second natural language instruction; when the distance is less than or equal to a first preset distance threshold, the electronic device replaces the first annotation with a second annotation, and displays the second annotation on the first user interface; or, when the distance is greater than a second preset distance threshold, the electronic device adds and displays the second annotation on the first user interface; or, when the distance is greater than the first preset distance threshold and less than or equal to the second preset distance threshold, the electronic device replaces the first annotation with the second annotation based on the user's selection operation and displays the second annotation on the first user interface, or, adds and displays the second annotation on the first user interface.

[0066] Among them, the second operation can be used to edit the annotation (such as a long press operation, etc.), the second natural language instruction can be a natural language instruction input by the user through the natural language interaction function, the second annotation can be an annotation generated based on the second natural language instruction input by the user, the first preset distance threshold can be a distance threshold between the location of the second operation preset by the electronic device and the location of the first batch of annotations, and the second preset distance threshold can be another distance threshold between the location of the second operation preset by the electronic device and the location of the first batch of annotations.

[0067] In this way, under different preset distance thresholds, the electronic device can determine whether to directly replace the first annotation with the second annotation, or to directly insert the second annotation into the first document, or to determine whether to replace the first annotation with the second annotation or to insert the second annotation into the first document based on the user's selection.

[0068] In one possible implementation, the method also includes: the electronic device detects a third operation performed by the user on the first batch of annotations; in response to the third operation, the electronic device displays a first option, and the first option is used to delete the first batch of annotations; the electronic device detects a fourth operation performed by the user on the first option, and in response to the fourth operation, the electronic device deletes and cancels the display of the first batch of annotations; or, the electronic device detects a fifth operation performed by the user to long press and drag the first batch of annotations to a third position, and in response to the fifth operation, the electronic device displays the first batch of annotations at the third position.

[0069] Among them, the third operation may be an operation for editing the annotation (such as a click operation, etc.), the first option may be an option for deleting the first batch of annotations, the fourth operation may be, for example, an operation in which the user clicks on the first option, the fifth operation may be, for example, an operation in which the user long presses and drags the annotation, and the third position may be a position on the screen that is different from the first position and the second position.

[0070] In this way, the electronic device can support users to perform editing operations such as deletion and movement on the generated annotations, which is convenient and quick.

[0071] In a possible implementation, the second operation is a long press operation, and the third operation is a single click operation.

[0072] In a possible implementation, the method also includes: the electronic device detects a sixth operation of the user, the sixth operation includes any one of the following: switching pages, scrolling pages, zooming in on pages, and zooming out on pages; in response to the sixth operation, the electronic device displays the first content in the first user interface, the first content is the content in the first document, and displays annotations related to the first content.

[0073] The sixth operation may be an operation for switching, scrolling, zooming in, zooming out, etc. of a page.

[0074] In this way, when the page content in the first document changes, if the current content corresponds to an annotation, the electronic device can also display the annotation in real time for the user to view.

[0075] In one possible implementation, the first natural language instruction includes any one of a first voice instruction or a first text instruction, and the electronic device detects that a user inputs the first natural language instruction, specifically including: the electronic device detects the first voice instruction input by the user through a microphone; or the electronic device calls up and displays a virtual keyboard, and detects the first text instruction input by the user through the virtual keyboard.

[0076] In this way, during the natural language interaction between the electronic device and the user, the electronic device can not only support the user's voice input, but also support the user's text input. The interaction method is more flexible and can be suitable for users with different input habits.

[0077] In a third aspect, an embodiment of the present application provides an electronic device, which includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes a method in any possible implementation of the first aspect or the second aspect above.

[0078] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are run on an electronic device, the electronic device executes a method in any possible implementation of the first aspect or the second aspect mentioned above.

[0079] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the method described in any possible implementation of the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1A-1E It is a user interface schematic diagram for realizing a quick sharing function of picture content in a group of cutout flow application scenarios provided by an embodiment of the present application;

[0081] Figure 1F-1H It is a user interface schematic diagram for realizing the quick collection / copying function of picture content in a group of cutout flow application scenarios provided by an embodiment of the present application;

[0082] Figure 1I-1J It is a user interface schematic diagram of a group of whole-image transfer application scenarios provided by an embodiment of the present application;

[0083] Figure 1K It is a display style diagram of a natural language interaction portal provided in an embodiment of the present application;

[0084] Figure 2A-2D It is a user interface schematic diagram of a group of natural language interaction entrances provided by an embodiment of the present application that are gradually reduced in size as the interaction object moves;

[0085] Figure 3A-3C This is a schematic diagram of a user interface for implementing a function of erasing picture content provided by an embodiment of the present application;

[0086] Figure 4A-4B It is a schematic diagram of a user interface for implementing a function of replacing picture content provided by an embodiment of the present application;

[0087] Figure 5A-5D It is a schematic diagram of a user interface for realizing a function of recognizing image content provided by an embodiment of the present application;

[0088] Figure 6 This is a schematic diagram of a user interface for implementing a picture beautification function provided in an embodiment of the present application;

[0089] Figure 7 This is a schematic diagram of a user interface for implementing a picture style transfer function provided in an embodiment of the present application;

[0090] Figure 8A-8F A schematic diagram of a user interface for performing corresponding operations on pictures in a social application provided in an embodiment of the present application;

[0091] Figure 9A-9E It is a schematic diagram of a user interface for realizing a quick sharing function of video content provided by an embodiment of the present application;

[0092] Figure 10A-10B is a schematic diagram of a user interface for implementing a video content cropping function provided in an embodiment of the present application;

[0093] Figure 11A-11B is another user interface schematic diagram for implementing a video content cropping function provided by an embodiment of the present application;

[0094] Figures 12A-12D It is a schematic diagram of a user interface for realizing a function of displaying unread message summaries provided by an embodiment of the present application;

[0095] Figure 12E-12F It is a schematic diagram of a user interface for implementing a question-and-answer query function for chat messages provided in an embodiment of the present application;

[0096] Figures 13A-13D A schematic diagram of a user interface for displaying a document summary provided by an embodiment of the present application;

[0097] Figure 13E-13F is a schematic diagram of a set of user interfaces for realizing the function of displaying notes in a document provided by an embodiment of the present application;

[0098] Figure 13G-13H A schematic diagram of a user interface for implementing a document content translation function provided in an embodiment of the present application;

[0099] Figure 14A-14E This is a schematic diagram of a user interface for implementing a text collection function provided in an embodiment of the present application;

[0100] Figure 14F-14H is a schematic diagram of a user interface for implementing a text query function provided by an embodiment of the present application;

[0101] Figure 15A-15N It is a schematic diagram of a user interface for implementing a text content modification function provided in an embodiment of the present application;

[0102] Figure 16A-16O A user interface diagram of a group of functions of adding annotations, displaying annotations, and editing annotations in a document provided by an embodiment of the present application;

[0103] Fig.17 It is a flowchart of an interactive method provided in an embodiment of the present application;

[0104] Fig.18 is a schematic diagram of a software architecture of an electronic device 100 provided in an embodiment of the present application;

[0105] Fig.19 It is a structural diagram of an electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0106] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0107] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0108] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0109] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0110] With the development of science and technology, the way of human-computer interaction has also changed. It has become possible to transform from machine language interaction to natural language interaction. Machine language interaction is mainly completed by clicking on interface controls or physical buttons through keyboard and mouse or touch. Sometimes a task requires multiple levels of jumps to complete, which is cumbersome. Natural language interaction is mainly completed through the user's natural language input (such as language input or text input) combined with powerful natural language processing capabilities. It can avoid level jumps and go straight to the point in one step. Therefore, compared with machine language interaction, natural language interaction can bring users a more convenient and intelligent experience.

[0111] Currently, more and more electronic devices can provide natural language interaction functions, which can support electronic devices to receive language / text instructions from users through natural language input (such as voice input or text input), and can support electronic devices to understand and execute the instructions.

[0112] However, in some scenarios where an interactive object needs to be processed (for example, editing a flower in a picture containing multiple flowers), the user is sometimes unable to accurately express the interactive object through voice / text alone, and the electronic device is unable to accurately and quickly recognize the user's intention. Multiple rounds of natural language interaction may be required to complete the task, resulting in low interaction efficiency and poor user experience.

[0113] Based on the above-mentioned problem of inconvenience in expressing interactive objects in natural language, the embodiment of the present application provides an interactive method, which can be applied to an electronic device 100 that can provide a natural language interactive function. The electronic device 100 can support the user to start the natural language interactive function through a long press operation, and display the natural language interactive entrance. Further, the electronic device 100 can detect the voice / text instruction input by the user through natural language (such as voice input or text input), and display the instruction at the natural language interactive entrance. After the instruction is input, the electronic device 100 can determine whether the instruction has a default object or a pronoun. If so, the electronic device 100 can extract and identify the interactive object of the interface content at the location where the user long presses the operation or at the location where the user long presses the operation and its vicinity, thereby determining the interactive object, and substituting the interactive object into the input instruction for understanding and execution, thereby completing the task specified by the user. In this way, through the integration of touch operation and natural language interaction, the selection of interactive objects and further operations can be efficiently realized, solving the problem of inconvenience in expressing interactive objects in natural language, improving the efficiency of human-computer interaction, and improving user experience.

[0114] It should be noted that the electronic device 100 in the embodiment of the present application may be equipped with Or portable electronic devices with other operating systems, such as mobile phones, tablet computers, smart watches, smart bracelets, AR devices, VR devices, etc., can also be non-portable electronic devices such as laptop computers with touch-sensitive surfaces or touch panels, desktop computers with touch-sensitive surfaces or touch panels, etc. The embodiment of the present application does not limit the type of electronic device 100.

[0115] The interaction method provided in the embodiment of the present application can be applied in a variety of scenarios. The following introduces the interaction method provided in the embodiment of the present application in combination with the user interface and several typical application scenarios.

[0116] It should be noted that the following application scenarios are only a few typical application scenarios. The interaction method provided in the embodiments of the present application can also be applied to many other scenarios, and the embodiments of the present application are not limited to this.

[0117] 1. Display application scenarios

[0118] "Display application scenario" may refer to an application scenario in which the interactive method provided in the embodiment of the present application is used to perform corresponding operations (such as sending, editing, etc.) on display information such as pictures and videos.

[0119] 1. Perform corresponding operations on the picture

[0120] (1) Sending picture content

[0121] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to send the entire picture or a part of the picture (also referred to as an object in the picture) to other electronic devices (such as a friend's electronic device), other applications installed on the electronic device 100, etc., thereby realizing the quick sharing / collection / copying function of the picture content.

[0122] 1) Cut-out transfer (i.e. sending a part of the picture)

[0123] Figure 1A-1E A set of user interfaces for realizing the quick sharing function of picture content in the cutout flow application scenario is exemplified.

[0124] Take a picture in the gallery application as an example, see Figure 1A , Figure 1A The exemplary user interface 110 may be a user interface of a gallery application. The user interface 110 may include a picture 111 , and the picture 111 may be a picture that the user is browsing.

[0125] In the embodiment of the present application, the user can trigger the electronic device 100 to perform intelligent cutout and display a thumbnail of the entire image by performing a long press operation on the image (for example, a long press operation performed by the user's finger or a touch device). For example, if the interface content at the location of the long press operation or at the location of the long press operation and its vicinity includes the cutout subject, then the electronic device 100 can perform intelligent cutout and display the cutout subject; for another example, if the interface content at the location of the long press operation or at the location of the long press operation and its vicinity does not include the cutout subject, then the electronic device 100 can display a thumbnail of the entire image.

[0126] Continue reading Figure 1A , exemplarily, the electronic device 100 can detect a user's long press operation on the picture 111. Assuming that the location of the long press operation is the location of the left flower in the picture 111, in response to the long press operation, the electronic device 100 can identify whether the interface content at the location of the long press operation or at the location of the long press operation and its vicinity includes a cutout subject. If, for example, the location of the long press operation includes the left flower in the picture 111, the electronic device 100 can determine that the cutout subject is the left flower in the picture 111. Further, referring to Figure 1B , the electronic device 100 can highlight the cutout subject 113 (i.e., the flower on the left side of the picture 111) to facilitate the user to know whether the cutout subject identified by the electronic device 100 is consistent with the user's intended cutout subject. In the embodiment of the present application, the cutout subject identified by the electronic device 100 can be an interactive object selected by the user for subsequent natural language interaction.

[0127] In an embodiment of the present application, the above-mentioned long press operation can also trigger the electronic device 100 to start the natural language interaction function.

[0128] Continue reading Figure 1B After the electronic device 100 starts the natural language interaction function, it can display a natural language interaction entrance, which can be used to prompt the user that the electronic device 100 has started the natural language interaction function and can also be used for subsequent natural language interaction. Figure 1B The natural language interaction portal 112 is shown as an example.

[0129] Optionally, a logo 112a may be displayed on the natural language interaction entrance 112, and the logo 112a may be used to prompt the user that the natural language interaction function of the electronic device 100 has been activated.

[0130] Optionally, a prompt 112b may be displayed on the natural language interaction entrance 112, and the prompt 112b may be used to prompt the user to input a natural language instruction related to the interaction object. Figure 1B As shown in the example, the user selects a flower as the interactive object, and the prompt 112b can be a prompt for prompting the user to input a natural language instruction related to the flower (for example, you can ask "what kind of flower is this"). It is easy to understand that the prompt can be generated by the electronic device 100 based on the identified interactive object, that is, the generated prompt can be a prompt related to the interactive object, and the prompt is a natural language instruction corresponding to the interactive object and can be recognized by the electronic device 100.

[0131] Optionally, the prompt 112b may also be output in the form of voice broadcast.

[0132] In some embodiments, after the electronic device 100 starts the natural language interaction function, it can start to detect the natural language command input by the user in real time, regardless of whether the user cancels the long press operation. Figure 1B The operation (eg, click operation) of the exemplary identification 112a or the user's Figure 1B After an operation (eg, a click operation) at any position on the exemplary natural language interaction portal 112 , detection of natural language instructions input by the user can be started.

[0133] In some embodiments, the prompt displayed on the natural language interaction entrance or the prompt announced by voice (for example, prompt 112b) may be displayed / voice announced only when the electronic device 100 does not detect the user's voice command within a preset time period after the natural language interaction function is started.

[0134] In some embodiments, after the electronic device 100 starts detecting voice commands input by the user, if the electronic device 100 detects that the user has not canceled the long press operation (i.e., has been long pressing the screen of the electronic device 100), then the electronic device 100 may continue to detect the voice commands input by the user; if the electronic device 100 detects that the user has canceled the long press operation (for example, the user's finger or touch device has left the screen of the electronic device 100), then the electronic device 100 may no longer detect the voice commands input by the user (i.e., end collecting voice input).

[0135] In some embodiments, after the electronic device 100 detects the voice command input by the user, regardless of whether the user cancels the long press operation, if the electronic device 100 does not detect that the user continues to input the voice command within a preset period of time, then the electronic device 100 can confirm that the voice command input is complete. In other embodiments, after the electronic device 100 detects the voice command input by the user, if the electronic device 100 detects that the user cancels the long press operation, then the electronic device 100 can confirm that the voice command input is complete.

[0136] In the embodiment of the present application, the natural language instruction input by the user may be a voice instruction or a text instruction. The voice instruction may be input by the user through the microphone on the electronic device 100, and the text instruction may be input by the user through the virtual keyboard on the electronic device 100. It is easy to understand that when the current application supports virtual keyboard input, the natural language instruction input by the user may be a voice instruction or a text instruction; when the current application does not support virtual keyboard input, the natural language instruction input by the user may only be a voice instruction.

[0137] See also Figure 1C , taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to share the interactive object to a certain application through a voice instruction. Assume that the user wants to share to a chat group named "Happy Family" in the social application "Changlian" Figure 1B The cutout subject 113 in the interactive object is then detected by the electronic device 100. The electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "Open Changlian Happy Family"), and can display the detected relevant voice instructions input by the user in the form of text (for example, the text 112d "Open Changlian Happy Family").

[0138] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Figure 1B The exemplary logo 112a is replaced by Figure 1C The logo 112c is shown as an example. The logo 112c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0139] Further, see Figure 1C After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to open the "Changlian" application and display the user interface corresponding to the chat group with the group chat name "Happy Family".

[0140] Further, see Figure 1D For example, the electronic device 100 may display a user interface 120 corresponding to the chat group "Happy Family", and the electronic device 100 may support the user to long press and drag in the user interface 120 Figure 1B Further, after the electronic device 100 detects that the user cancels the long press and drag operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), in response to the operation, refer to Figure 1E , the electronic device 100 can Figure 1B The interactive object of the cutout subject 113 in the image is sent to the chat group named "Happy Family" in the social application "Changlian", so that the quick sharing function of the image content is realized.

[0141] In the embodiment of the present application, Figures 1A to 1D During the whole process of pressing the long button, the user may not cancel the long button operation. Figure 1D Previously, the user could also cancel the long press operation, in which case the electronic device 100 would no longer display Figure 1D , directly display Figure 1E , that is, the cutout body 113 is directly sent to the chat group named "Happy Family" in the social application "Changlian".

[0142] It should be noted that the voice instruction "Open Changlian Happy Family" for implementing the quick sharing function of the picture content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the quick sharing function of the picture content needs to meet certain conditions, under which the electronic device 100 can determine where the user wants to share the selected interactive object. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating where the user wants to share the selected interactive object, such as verbs such as "open", "jump to", "share to", and nouns such as the name of the application, the name of the contact / chat group, etc.

[0143] Figure 1F-1H A set of user interfaces for implementing quick collection / copying functions of image content in a cutout transfer application scenario is exemplified.

[0144] Similarly, the interactive object selected by the user is Figure 1B Taking the cutout subject 113 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Figure 1A and Figure 1B The description will not be repeated here.

[0145] See also Figure 1F , taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to collect / copy the interactive object to a certain application through the voice instruction. Figure 1B If the interactive object 113 of the cutout body is collected / copied to the "Memo" application, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "copy this to the memo") and can display the detected relevant voice instructions input by the user in the form of text (for example, text 112e "copy this to the memo").

[0146] Further, see Figure 1F The electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to open the "Memo" application and display the corresponding user interface.

[0147] Further, see Figure 1G For example, the electronic device 100 may display a user interface 130, wherein the user interface 130 may be a user interface corresponding to an existing note in the memo (e.g., a note named "XX Flower"), or a newly created note, which is not limited in the present embodiment of the application. The electronic device 100 may support the user to long press and drag in the user interface 130. Figure 1Bthe interaction object of the subject 113 for matting in it. Further, after the electronic device 100 detects that the user cancels the long - press and drag operation (such as the operation that the user's finger or touch device leaves the screen of the electronic device 100), in response to this operation, refer to Figure 1H , the electronic device 100 can Figure 1B add the interaction object of the subject 113 for matting in it to the note named "XX Flower". Thus, the quick collection / copy function of the picture content is realized.

[0148] In the embodiment of the present application, during the whole process from Figure 1F to Figure 1G , the user can keep not canceling the long - press operation. In some embodiments, before displaying Figure 1G , the user can also cancel the long - press operation. In this case, the electronic device 100 can no longer display Figure 1G , and directly display Figure 1H , that is, directly add the subject 113 for matting to the note named "XX Flower".

[0149] It should be noted that the voice command "copy this to the memo" for realizing the quick collection / copy function of the picture content above is only exemplary and should not limit the present application. It is easy to understand that the voice command for realizing the quick collection / copy function of the picture content above needs to meet certain conditions. Under this condition, the electronic device 100 can determine where the user wants to collect / copy the selected interaction object. Among them, the above - mentioned "meeting certain conditions" can mean that the voice command includes some keywords for indicating where the user wants to collect / copy the selected interaction object, such as verbs like "open", "collect to", "copy to", etc., or nouns like the name of the application program, etc.

[0150] It should be noted that the present application embodiment does not limit the display style of the natural - language interaction entry; the present application embodiment also does not limit the display position of the natural - language interaction entry. For example, it can be at the position where the long - press operation is located; or for another example, it can be at a position near the position where the long - press operation is located; and so on.

[0151] It should be noted that the meaning represented by the "natural - language interaction entry" has been recorded in the embodiment of the present application, and its name does not constitute any limitation to the embodiment of the present application. In some other embodiments of the present application, the natural - language interaction entry can also be called a natural - language (or voice / text) recognition box / prompt box / recognition bar / prompt bar / recognition window / prompt window, etc.

[0152] It should be noted that in the interaction method provided in the embodiment of the present application, the operation of the user selecting the interaction object and triggering the electronic device 100 to start the natural language interaction function can be the same operation. In the embodiment of the present application, the operation is only taken as a long press operation as an example, which should not constitute a limitation on the present application. In other embodiments of the present application, the operation can also be other operations.

[0153] It should be noted that in the interaction method provided in the embodiment of the present application, whether the user wants to trigger the electronic device 100 to start the natural language interaction function at the same time after selecting the interaction object can be set by the user himself or by the system default setting of the electronic device 100, and the embodiment of the present application does not limit this.

[0154] 2) Entire picture transfer (i.e. sending the entire picture)

[0155] See also Fig. 1I The user can also trigger the electronic device 100 to display a thumbnail of the entire picture by long pressing the picture (for example Fig. 1I The thumbnail 114 shown as an example can also trigger the electronic device 100 to start the natural language interaction function to complete the whole picture flow. Among them, the specific process of the whole picture flow is similar to the aforementioned cutout flow, and can refer to the aforementioned related content, which will not be repeated here.

[0156] The specific process of the above-mentioned whole picture transfer is only based on a picture that the user is browsing. In some embodiments, the whole picture transfer can also be performed for pictures that the user has browsed before. Figure 1J , Figure 1J The exemplary user interface 140 may be a user interface in a gallery application that is displayed before browsing a certain picture, and the user interface may include thumbnails of one or more pictures. After the user selects a thumbnail of a certain picture, the electronic device 100 may also be triggered to start the natural language interaction function, and the entire picture flow is completed through the natural language interaction function. The specific process is similar to the aforementioned entire picture flow for a certain picture that the user is browsing, and will not be repeated here.

[0157] In some embodiments, after the electronic device 100 detects that the user has selected and completed the interactive object and starts the natural language interaction function to display the natural language interaction entrance, if the electronic device 100 detects that the user has long pressed and dragged the interactive object, then in response to this operation, the electronic device 100 may move the natural language interaction entrance along with the interactive object.

[0158] Optionally, if the electronic device 100 detects that the displacement of the interactive object caused by the user's long press and drag is greater than a preset displacement threshold, the electronic device 100 may reduce the display of the natural language interaction entrance, or cancel the display of the natural language interaction entrance. If the electronic device 100 does not detect the displacement of the interactive object within a preset time period, the electronic device 100 may resume displaying the natural language interaction entrance.

[0159] For example, see Figure 2A Taking the interactive object as thumbnail 114 as an example, the electronic device 100 can detect the user's operation of long pressing and dragging the thumbnail 114, and in response to the operation, refer to Figure 2B , Figure 2C The electronic device 100 can move the natural language interaction entrance 112 along with the thumbnail 114, and can also move the natural language interaction entrance 112 as shown in FIG. Figure 2B , Figure 2C The display gradually decreases until it is as shown Figure 2D Cancel the display as shown.

[0160] Optionally, while the natural language interaction portal moves and shrinks along with the interactive object, the electronic device 100 may continue to detect voice commands input by the user (i.e., continue to receive sound), or may no longer detect voice commands input by the user (i.e., stop receiving sound); when the natural language interaction portal is cancelled, the electronic device 100 may no longer detect voice commands input by the user; when the natural language interaction portal is restored to display, the electronic device 100 may continue to detect voice commands input by the user.

[0161] In some embodiments, see Figure 1K While displaying the above-mentioned natural language interaction entrance (such as the natural language interaction entrance 112), the electronic device 100 can also display one or more functional options (such as the "subject segmentation" option, the "smart erase" option, the "background quick change" option, etc.), and display the above-mentioned natural language interaction entrance and the one or more functional options in a scenario-based menu bar 141 at the same time. The user can trigger the electronic device 100 to perform the operation corresponding to the functional option by clicking the corresponding functional option, which is convenient and quick.

[0162] (2) Erase the image content

[0163] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to erase a certain part of the picture (also referred to as a certain object in the picture) from the picture, thereby realizing the erasing function of the picture content.

[0164] Similarly, the interactive object selected by the user is Figure 1BTaking the cutout subject 113 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Figure 1A and Figure 1B The description will not be repeated here.

[0165] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to erase the interactive object from the picture through the voice instruction. Figure 3A Assuming that the user wants to erase the interactive object 113 of the cutout body from the picture, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "erase this"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 112f "erase this").

[0166] Further, see Figure 3A After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to erase the interactive object of the cutout subject 113 from the picture.

[0167] Furthermore, the electronic device 100 can use a related algorithm to erase the interactive object 113 from the image. Figure 3B , the electronic device 100 may no longer display the interactive object, the cutout subject 113, in the picture. Thus, the function of erasing the picture content is realized.

[0168] It should be noted that the voice instruction "erase this" for implementing the function of erasing the image content is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for implementing the function of erasing the image content needs to meet certain conditions, under which the electronic device 100 can determine the position of the object in the image that the user wants to erase. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the erasing object and the erasing action.

[0169] In some embodiments, if the electronic device 100 cannot determine the specific intention of the user through the voice command input by the user and the location of the long press operation, then the electronic device 100 can further prompt the user to perform the corresponding operation through the natural language interaction portal in the form of voice broadcast or text prompts, so as to determine the specific intention of the user. Figure 3CIf the electronic device 100 cannot determine the position of the object in the picture that the user wants to erase, then the electronic device 100 can prompt the user, for example, displaying a prompt 112g "Where do you want to erase, please click the position you want to erase". After detecting that the user performs the corresponding operation according to the prompt 112g, the electronic device 100 can then erase the object at the corresponding position in the picture.

[0170] (3) Image content replacement

[0171] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to replace a part of the picture (also referred to as an object in the picture) (i.e., replace the content of a part of the picture with other content), thereby realizing the function of replacing the picture content.

[0172] Similarly, the interactive object selected by the user is Figure 1B Taking the cutout subject 113 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Figure 1A and Figure 1B The description will not be repeated here.

[0173] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to replace the interactive object through the voice instruction. Figure 4A Assuming that the user wants to replace the interactive object 113 of the cutout body (which can be called the replaced object) with other objects (which can be called the replacement object), the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "replace this with leaves") and can display the detected relevant voice instructions input by the user in the form of text (for example, text 112h "replace this with leaves").

[0174] Further, see Figure 4A After the above-mentioned voice instruction is input, the electronic device 100 can also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to replace the interactive object of the cutout body 113 with other objects (for example, replace the interactive object of the cutout body 113 with leaves).

[0175] Furthermore, the electronic device 100 can search for replacement objects locally or in the cloud. After the search is completed, the electronic device 100 can select a replacement object by default (or support the user to select a replacement object from multiple replacement objects that have been searched), and replace the interactive object of the cutout subject 113 with the selected replacement object through a relevant algorithm. After the replacement is completed, refer to Figure 4B, the electronic device 100 may no longer display the interactive object of the cutout subject 113 in the picture, but display the selected replacement object (such as the replacement object 115 of the leaf) at the location of the original cutout subject 113 in the picture. Thus, the function of replacing the picture content is realized.

[0176] It should be noted that the voice instruction "Replace this with leaves" for implementing the function of replacing the image content is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for implementing the function of replacing the image content needs to meet certain conditions, under which the electronic device 100 can determine what the user wants to replace the replaced object with. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the replaced object, the replacement object, and the replacement action.

[0177] In some embodiments, the electronic device 100 may also support the user to insert content into the picture. It is easy to understand that if the user wants to insert content into the picture, then, similar to the above-mentioned picture content replacement, the user can enter a voice instruction related to content insertion. After the language instruction is entered, the electronic device 100 can analyze the above-mentioned voice instruction, thereby determining that the operation corresponding to the above-mentioned voice instruction is to insert content into the picture. Further, the electronic device 100 can search for insertion objects locally or in the cloud. After the search is completed, the electronic device 100 can select an insertion object by default (or support the user to select an insertion object from multiple insertion objects that have been searched), and insert the above-mentioned selected insertion object into the picture through a related algorithm (for example, the insertion object can be inserted into the user's long press position in the picture). It is easy to understand that the above-mentioned voice instructions related to content insertion need to meet certain conditions, under which the electronic device 100 can determine what content the user wants to insert into the picture. Among them, the above-mentioned "certain conditions" can refer to the voice instruction including certain keywords for indicating insertion objects, insertion actions, and insertion positions.

[0178] In some embodiments, the electronic device 100 may also support the user to complete the content when some content is missing in the picture. It is easy to understand that if the user wants to complete the content in the picture, then, similar to the above-mentioned picture content replacement, the user can enter a voice command related to content completion (for example, a voice command "complete the missing content in the picture"). After the voice command is entered, the electronic device 100 can analyze the above-mentioned voice command to determine that the operation corresponding to the above-mentioned voice command is to complete the content in the picture. Further, the electronic device 100 can determine the content that needs to be completed through a relevant algorithm and complete the content in the picture. It is easy to understand that the above-mentioned voice command related to content completion needs to meet certain conditions, under which the electronic device 100 can determine that the user wants to complete the content in the picture. Among them, the above-mentioned "certain conditions" can refer to the voice command including certain keywords for indicating the completion action.

[0179] (4) Image content recognition (e.g., recognition of objects or facial images in images)

[0180] By implementing the interaction method provided in the embodiment of the present application, the electronic device 100 can support the recognition of the interaction object selected by the user in the picture, thereby realizing the recognition function of the picture content.

[0181] Similarly, the interactive object selected by the user is Figure 1B Taking the cutout subject 113 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Figure 1A and Figure 1B The description will not be repeated here.

[0182] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to use the voice instruction to let the electronic device 100 recognize the interactive object. Figure 5A , assuming that the user wants the electronic device 100 to identify the cutout subject 113, then the electronic device 100 can detect the relevant voice instructions input by the user (such as voice "what flower is this" or "what is this", etc.), and can display the detected relevant voice instructions input by the user in the form of text (such as text 112i "what flower is this" or "what is this", etc.).

[0183] Further, see Figure 5A After the above-mentioned voice instruction is input, the electronic device 100 can also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to identify the cutout subject 113.

[0184] Furthermore, the electronic device 100 can identify the cutout subject 113 through a related algorithm. After the identification is completed, refer to Figure 5B , the electronic device 100 can display the recognition result 116, wherein the recognition result 116 can include the name of the cutout subject 113 (e.g., "XX Flower"), and can also include a brief introduction to the cutout subject 113. The brief introduction to the cutout subject 113 can be queried by the electronic device 100 locally or in the cloud. It should be noted that the embodiment of the present application does not limit the display style (e.g., card style, ordinary text style, etc.) and display position of the recognition result 116.

[0185] In some embodiments, the electronic device 100 may also support the identification of a face image in a picture. Figure 5C , assuming that the interactive object selected by the user is the cutout subject 117, the cutout subject 117 may include a face image, and assuming that the user wants the electronic device 100 to recognize the cutout subject 117, then the electronic device 100 may detect the relevant voice command input by the user (e.g., voice "Is this singer XXX?" or "Who is this?", etc.), and may display the relevant voice command input by the user that has been detected in the form of text (e.g., text 112j "Is this singer XXX?" or "Who is this?", etc.). Further, continue to refer to Figure 5C After the voice command is input, the electronic device 100 can also analyze the voice command to determine that the operation corresponding to the voice command is to identify the cutout subject 117. Further, the electronic device 100 can identify the cutout subject 117 through a relevant algorithm. After the identification is completed, refer to Figure 5D , the electronic device 100 can display the recognition result 118, wherein the recognition result 118 can include the identity of the cutout subject 117 (e.g., "singer XXX"), and can also include a brief introduction to the cutout subject 117. The brief introduction to the cutout subject 117 can be queried by the electronic device 100 locally or in the cloud. It should be noted that the embodiment of the present application does not limit the display style (e.g., card style, ordinary text style, etc.) and display position of the recognition result 118.

[0186] It is easy to understand that the above-mentioned voice instructions related to picture content recognition need to meet certain conditions, under which the electronic device 100 can determine the user's intention to let the electronic device 100 perform picture content recognition. Among them, the above-mentioned "certain conditions" can mean that the voice instructions include certain keywords for indicating recognition actions (such as query, recognition, what, who, etc.).

[0187] (5) Image beautification (e.g., applying makeup to the face of a person in the image)

[0188] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to beautify the picture.

[0189] The interactive object selected by the user Figure 5C For example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the above-mentioned Figure 1A and Figure 1B The description will not be repeated here.

[0190] Taking the natural language command input by the user as a voice command as an example, the electronic device 100 can support the user to use the voice command to let the electronic device 100 beautify the interactive object (for example, beautify the face). Figure 6 , assuming that the user asks the electronic device 100 to perform beautification on the cutout body 117 (for example, apply lipstick to the cutout body 117), then the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "apply a bean paste-colored lipstick"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 112k "apply a bean paste-colored lipstick").

[0191] Further, see Figure 6 After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to apply a bean paste-colored lipstick to the cutout body 117 .

[0192] Furthermore, the electronic device 100 can perform intelligent cutout to obtain the cutout body 117, and identify the lip position of the cutout body 117 through a relevant algorithm, and add bean paste color at this position, and display it, thereby achieving the beauty and facial effect desired by the user; or, the electronic device 100 can also identify the facial image at the position where the user long presses through a relevant algorithm, and perform feature analysis on the facial image, determine the position of the lips in the facial image, and add bean paste color at this position, and display it, thereby achieving the beauty and facial effect desired by the user.

[0193] In some embodiments, when the cutout cannot be achieved through a long press operation, the electronic device 100 can also automatically identify the facial image in the picture and perform corresponding beauty and facial operations in combination with the voice commands input by the user.

[0194] It is easy to understand that the above-mentioned voice instructions related to picture beautification need to meet certain conditions, under which the electronic device 100 can determine how the user wants to beautify the picture. Among them, the above-mentioned "certain conditions" can mean that the voice instructions include certain keywords for indicating picture beautification (such as applying lipstick, thin face, big eyes, skin grinding, beauty, body beauty, mosaic, graffiti, adding frame / sticker / marker, blur, etc.).

[0195] (6) Image style transfer

[0196] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to transfer the style of the entire picture or a part of the picture. Among them, style transfer can refer to adding a filter effect to the entire picture or a part of the picture to change its original style.

[0197] In the scenario of image style transfer, the specific process of the user selecting an interactive object and activating the natural language interaction function of the electronic device 100 can refer to the aforementioned Figure 1A and Figure 1B The description will not be repeated here.

[0198] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to use the voice instruction to let the electronic device 100 perform style transfer on the interactive object. Figure 7 Taking the interactive object being the entire picture as an example, the electronic device 100 can detect relevant voice instructions input by the user (such as the voice "a little more retro"), and can display the detected relevant voice instructions input by the user in the form of text (such as text 112l "a little more retro").

[0199] Further, see Figure 7 After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to add a retro filter to the entire picture.

[0200] Furthermore, the electronic device 100 can add a retro filter to the above picture through a relevant algorithm and display it, thereby achieving the retro effect desired by the user.

[0201] It is easy to understand that the above-mentioned voice instructions related to picture style transfer need to meet certain conditions, under which the electronic device 100 can determine how the user wants to transfer the style of the picture. Among them, the above-mentioned "certain conditions" can mean that the voice instructions include certain keywords for indicating picture style transfer (such as fresh, beautiful, literary, fashionable, natural, retro, black and white, Chinese style, European and American, Japanese style, etc.).

[0202] In some embodiments, the electronic device 100 may also support the user to adjust parameters of the image (for example, adjusting parameters such as brightness, contrast, saturation, and sharpness of the image). The specific process of parameter adjustment is similar to the specific process of style transfer described above, and will not be repeated here.

[0203] The above application scenarios for performing corresponding operations on pictures are all introduced by taking pictures in the gallery application as an example. The examples are not limited to pictures in the gallery application, but can also be pictures in other applications. Figure 8A-8E The following uses a picture in a social application as an example to introduce the process of performing corresponding operations on the picture.

[0204] In an embodiment of the present application, a user can trigger the electronic device 100 to display a menu bar corresponding to the above picture (i.e., the interactive object) by performing a long press operation on the picture (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0205] See also Fig. 8A , Fig. 8A The user interface 120 shown in the example may be a user interface of a social application (such as a Changlian application) (such as a user interface corresponding to a chat group with a group chat name of "Happy Family"), which may include one or more chat records (such as the chat record in picture 121).

[0206] Continue reading Fig. 8A For example, the electronic device 100 may detect a user's long press operation on the image 121. In response to the long press operation, the electronic device 100 may display Figure 8B The user interface 150 is shown as an example, and the natural language interaction function is enabled.

[0207] See also Figure 8B The user interface 150 may include a natural language interaction entry 151, which may be used to prompt the user that the electronic device 100 has started the natural language interaction function and may also be used for subsequent natural language interaction. The user interface 150 may also include a menu bar 152. The user interface 150 may also include Fig. 8A As shown in picture 121.

[0208] Optionally, a logo 151 a may be displayed on the natural language interaction entrance 151 , and the logo 151 a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0209] Optionally, a prompt 151b may be displayed on the natural language interaction entrance 151, and the prompt 151b may be used to prompt the user to input a natural language instruction related to the interaction object. Fig. 8A As shown in the example, the interactive object selected by the user is picture 121, and the electronic device 100 can recognize that the picture 121 includes flowers. The prompt 151b can be a prompt for prompting the user to enter a natural language instruction related to flowers (for example, you can ask "what kind of flower is this"). Optionally, the electronic device 100 can also automatically recommend functions according to the recognition type, and display corresponding function options in the above menu bar 152, such as "AI recognition of objects", etc. The user can trigger the electronic device 100 to display the recognition result by clicking on the function option.

[0210] Optionally, the prompt 151b may also be output in the form of voice broadcast.

[0211] Continue reading Figure 8B The menu bar 152 may include one or more options (such as a “copy to album” option, an “edit” option, a “share” option, etc.), and these one or more options may be used to perform corresponding operations on the picture 121 .

[0212] Generally, social applications support virtual keyboard input. Therefore, in the embodiment of the present application, the natural language command input by the user can be a voice command or a text command. The voice command can be input by the user through the microphone on the electronic device 100, and the text command can be input by the user through the virtual keyboard on the electronic device 100.

[0213] The embodiment of the present application does not limit the triggering method of the electronic device 100 to detect whether the user inputs a voice command or a text command.

[0214] In some embodiments, the triggering method for the electronic device 100 to detect the user input voice command may be that the electronic device 100 starts the natural language interaction function (that is, after starting the natural language interaction function, the user input voice command begins to be detected). In other embodiments, the triggering method for the electronic device 100 to detect the user input voice command may also be that the electronic device 100 detects that the user has directed Figure 8B An operation (eg, a click operation) of the marker 151a is shown as an example.

[0215] For example, see Figure 8C Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What flower is this"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 151d "What flower is this").

[0216] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Figure 8B The logo 151a shown in the exemplary embodiment is replaced by Figure 8C The logo 151c is shown as an example. The logo 151c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0217] Optionally, continue to Figure 8C During the process of the user inputting a voice command, the menu bar 152 can be hidden.

[0218] Further, see Figure 8C After the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to identify the flowers included in the interactive object.

[0219] Furthermore, the electronic device 100 can identify the interactive object through a related algorithm. After the identification is completed, refer to Fig.8D , the electronic device 100 may display the recognition result 153, wherein the recognition result 153 may include the name of the flower included in the interactive object (e.g., "XX flower"), and may also include a brief introduction to the flower. It should be noted that the embodiment of the present application does not limit the display style (e.g., card style, ordinary text style, etc.) and display position of the recognition result 153.

[0220] In some embodiments, the electronic device 100 detects that the user inputs a text instruction, which is triggered by the electronic device 100 detecting that the user is Figure 8B The operation of the mark 151a shown in the exemplary embodiment (such as a click operation), in other embodiments, the triggering mode of the electronic device 100 detecting the user inputting the text instruction can also be that the electronic device 100 detects the user's Figure 8B An operation (eg, a click operation) at any position on the natural language interaction portal 151 is shown as an example.

[0221] For example, see Figure 8B If the user wants to input a text instruction through the virtual keyboard, the electronic device 100 can detect the user's Figure 8B In response to the operation (eg, a click operation) of the exemplary identification 151a, refer to Fig. 8E The electronic device 100 can call up a virtual keyboard, and the user can enter text instructions through the virtual keyboard. After the text instructions are entered, the electronic device 100 can analyze the text instructions and execute the operations corresponding to the text instructions. The specific process is similar to the analysis and execution process of the aforementioned voice instructions, and will not be repeated here.

[0222] It should be noted that the embodiment of the present application does not limit the display style of the natural language interaction entrance; the embodiment of the present application does not limit the display position of the natural language interaction entrance. For example, it can be displayed above / below the menu bar; for another example, it can also be displayed in the menu bar (for example Fig.8F shown); and so on.

[0223] It should be noted that in the interactive method provided in the embodiment of the present application, the operation of triggering the electronic device 100 to display the menu bar corresponding to the interactive object (such as a picture) and the operation of triggering the electronic device 100 to start the natural language interactive function can be the same operation. In the embodiment of the present application, the operation is only taken as a long press operation as an example, which should not constitute a limitation on the present application. In other embodiments of the present application, the operation can also be other operations.

[0224] It should be noted that in the interaction method provided in the embodiment of the present application, whether to simultaneously trigger the electronic device 100 to start the natural language interaction function after triggering the electronic device 100 to display the menu bar corresponding to the interactive object (such as a picture) can be set by the user independently or by the system default setting of the electronic device 100, and the embodiment of the present application does not limit this.

[0225] It is easy to understand that in the embodiment of the present application, the electronic device 100 can realize various editing functions of the picture (such as picture content erasing, picture content replacement / insertion / completion, picture content recognition, picture beautification, picture style transfer, picture parameter adjustment, etc.) through relevant image processing algorithm models (such as DALL-E2, Midjourny, etc.). Among them, DALL-E2 is an artificial intelligence image generator based on natural language, which can create high-quality images and works of art based on text descriptions of natural language, and can realize various image generation / editing functions such as image repair and image expansion; among them, Midjourey is also an artificial intelligence image generator based on natural language, which can generate realistic and diverse images based on text descriptions of natural language, and can realize functions such as color adjustment, resolution adjustment, size adjustment, style transfer, etc.

[0226] 2. Perform corresponding operations on the video

[0227] (1) Video content delivery

[0228] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to send videos to other electronic devices (such as friends' electronic devices), other applications installed on the electronic device 100, etc., thereby realizing the quick sharing function of video content.

[0229] Figure 9A-9EA set of user interfaces for implementing a quick sharing function of video content is exemplified.

[0230] Take a video in the Gallery application as an example. Fig. 9A , Fig. 9A The exemplary user interface 210 may be a user interface of a gallery application. The user interface 210 may include a video 211 . The video 211 may be a video that the user has viewed before playing it.

[0231] In an embodiment of the present application, a user can trigger the electronic device 100 to start a natural language interaction function by performing a long press operation on a video (for example, a long press operation performed by the user's finger or a touch device), and can also trigger the video to be used as an interaction object for subsequent natural language interaction selected by the user.

[0232] Continue reading Fig. 9A For example, the electronic device 100 may detect a user's long press operation on the video 211, and in response to the long press operation, refer to Fig. 9B , the electronic device 100 can start the natural language interaction function, and the electronic device 100 can also determine that the interaction object selected by the user is the video 211.

[0233] Continue reading Fig. 9B After starting the natural language interaction function, the electronic device 100 may display a natural language interaction entrance 212. The natural language interaction entrance 212 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0234] Optionally, a logo 212a may be displayed on the natural language interaction entrance 212, and the logo 212a may be used to prompt the user that the natural language interaction function of the electronic device 100 has been activated.

[0235] Optionally, a prompt 212b may be displayed on the natural language interaction entrance 212, and the prompt 212b may be used to prompt the user to input a natural language instruction related to the interaction object. Fig. 9B As shown in the example, the user selects a video as the interactive object, and the prompt 212b can be a prompt for prompting the user to input natural language instructions related to the video (for example, you can ask "what video is this").

[0236] Optionally, the prompt 212b may also be output in the form of voice broadcast.

[0237] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the aforementioned picture content sending, which will not be repeated here.

[0238] See also Fig. 9C , taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to share the interactive object to a certain application through a voice instruction. Assume that the user wants to share to a chat group named "Happy Family" in the social application "Changlian" Fig. 9A For example, if the interactive object is the video 211 in the video, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "Open Changlian Happy Family"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 212d "Open Changlian Happy Family").

[0239] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Fig. 9B The exemplary logo 212a is replaced by Fig. 9C The logo 212c is shown as an example. The logo 212c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0240] Further, see Fig. 9C After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to open the "Changlian" application and display the user interface corresponding to the chat group with the group chat name "Happy Family".

[0241] Further, see Fig.9D For example, the electronic device 100 may display a user interface 120 corresponding to the chat group "Happy Family", and the electronic device 100 may support the user to long press and drag in the user interface 120 Fig. 9A Further, after the electronic device 100 detects that the user cancels the long press and drag operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), in response to the operation, refer to Fig.9E , the electronic device 100 can Fig. 9A The interactive object of video 211 is sent to the chat group named "Happy Family" in the social application "Changlian", so that the quick sharing function of video content is realized.

[0242] In the embodiment of the present application, FIG. 9A to FIG. 9D During the whole process of pressing the long button, the user may not cancel the long button operation. Fig.9D Previously, the user could also cancel the long press operation, in which case the electronic device 100 would no longer display Fig.9D , directly display Fig.9E , that is, to send the video 211 directly to the chat group named "Happy Family" in the social application "Changlian".

[0243] It should be noted that the voice instruction "Open Changlian Happy Family" for implementing the quick sharing function of video content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the quick sharing function of video content needs to meet certain conditions, under which the electronic device 100 can determine where the user wants to share the selected interactive object. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating where the user wants to share the selected interactive object, such as verbs such as "open", "jump to", "share to", and nouns such as the name of the application, the name of the contact / chat group, etc.

[0244] It is easy to understand that, similar to the aforementioned sending of picture content, the interactive method provided in the embodiment of the present application can also realize the quick collection / copy function of video content. The specific process can refer to the aforementioned related content of realizing the quick collection / copy function of picture content, which will not be repeated here.

[0245] (2) Video content (e.g., specific clips, specific frames) cropping

[0246] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to crop the video content (such as a specific segment), thereby realizing the cropping function of the video content.

[0247] Similarly, the interactive object selected by the user is Fig. 9A Taking the exemplary video 211 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Fig. 9A and Fig. 9B The description will not be repeated here.

[0248] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to crop the interactive object video 211 through the voice instruction to obtain a certain video segment in the video 211.

[0249] See also Fig. 10A Assuming that the user wants to obtain a video clip from 0 to 30 seconds in video 211, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "intercept a video clip from 0 to 30 seconds"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 212e "intercept a video clip from 0 to 30 seconds").

[0250] Further, see Fig. 10A After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to capture a video clip from 0 to 30 seconds in the video 211 .

[0251] Furthermore, the electronic device 100 can intercept the video segment of 0 to 30 seconds in the video 211 through the relevant algorithm. After the interception is completed, refer to Fig. 10B , the electronic device 100 can also save the captured video clips. So far, the cropping function of the video content is realized.

[0252] It should be noted that the voice instruction "cut a video clip of 0 to 30 seconds" for implementing the cropping function of the video content is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for implementing the cropping function of the video content needs to meet certain conditions, under which the electronic device 100 can determine which video clip the user wants to crop / cut. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the cropping action (such as "crop", "cut", etc.), the position and duration of the cropped video clip.

[0253] Above Figure 10A-10B The process of cropping a video clip in a video before playing is introduced. For a video being played, the electronic device 100 can also support the user to crop a certain video clip in the video. The specific process is similar to cropping a video clip in a video before playing, and will not be repeated here. Here, for a video being played, the electronic device 100 can also support intercepting a certain video screen in the video. Figure 11A-11B To introduce.

[0254] See also Fig.11A Assuming that the user wants to capture the frame at 00 minutes and 30 seconds in the video 211, the interactive object selected by the user can be the frame at 00 minutes and 30 seconds in the video 211. The specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Fig. 9A and Fig. 9B The electronic device 100 can detect the relevant voice command input by the user (for example, the voice "capture this frame"), and can display the relevant voice command input by the user that has been detected in the form of text (for example, the text 212f "capture this frame").

[0255] Further, see Fig.11AAfter the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to capture the frame at 00 minutes and 30 seconds in the video 211.

[0256] Furthermore, the electronic device 100 can intercept the frame at 00 minutes and 30 seconds in the video 211 through a related algorithm. After the interception is completed, refer to Fig. 11B The electronic device 100 can also save the captured video screen.

[0257] In some embodiments, when a user performs a long press operation on a playing video, the electronic device 100 can pause playing the video upon detecting the long press operation, and the video screen displayed during the pause can be the video screen corresponding to the playback progress bar when the user performs the long press operation.

[0258] In other embodiments, for a video being played, the electronic device 100 may also support the user to capture the video screen at any time point corresponding to the playback progress bar, rather than the video screen corresponding to the playback progress bar when the current user performs a long press operation. For example, the voice command input by the user instructs the electronic device 100 to capture the video screen at a certain time point, and the electronic device 100 can capture the video screen at that time point based on the voice command.

[0259] In other embodiments, for a video before playing, the electronic device 100 may also support the user to capture the video screen at any time point corresponding to the play progress bar. For example, after the user performs a long press operation on the video before playing to trigger the electronic device 100 to start the natural language interaction function, the user can enter a voice command to instruct the electronic device 100 to capture the video screen at which time point. Further, after detecting the voice command, the electronic device 100 can capture the video screen at the corresponding time point based on the voice command.

[0260] It can be understood that the interactive method provided in the embodiments of the present application is not limited to the above-mentioned exemplary operations on pictures and videos, and more operations can be performed on pictures and videos. For example, as long as the operations on pictures and videos can be implemented by methods in the prior art (for example, by clicking on corresponding options in the menu bar, etc., to implement more operations on pictures and videos), the interactive methods provided in the embodiments of the present application can also be implemented.

[0261] 2. Container Application Scenarios

[0262] "Container-type application scenarios" may refer to application scenarios in which the interactive method provided in the embodiments of the present application is used to perform corresponding operations (such as displaying unread message summaries, question and answer queries, etc.) on information displayed in list form (such as chat messages, etc.); it may also refer to application scenarios in which the interactive method provided in the embodiments of the present application is used to perform corresponding operations (such as displaying document summaries, displaying notes in documents, full-text translation, question and answer queries, format conversion, format beautification, etc.) on information displayed in grid form (such as documents, etc.).

[0263] 1. Perform corresponding operations on chat messages (or chat records)

[0264] (1) Display unread message summaries

[0265] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to view unread message summaries. In this way, when there are multiple unread messages, the user can quickly obtain the main information to be conveyed by these multiple unread messages by viewing the unread message summaries. The user does not need to view the unread messages one by one, which is convenient and fast, simplifies user operations, and improves user experience.

[0266] Take a social application as an example, see Fig. 12A , Fig. 12A The exemplary user interface 310 may be a user interface of the social application, and the user interface 310 may include one or more chat items (such as chat item 311, chat item 312, etc.). The user clicks on a chat item to view one or more chat messages that may be included in the chat item.

[0267] In an embodiment of the present application, a user can trigger the electronic device 100 to start a natural language interaction function by performing a long press operation on a chat item (for example, a long press operation performed by the user's finger or a touch device), and can also trigger the chat item to be selected as an interaction object for subsequent natural language interaction by the user.

[0268] Continue reading Fig. 12A Taking group chat messages as an example, it can be seen that chat item 311 is a chat item corresponding to group chat XXX. From chat item 311, it can be seen that there are multiple unread messages in group chat XXX. Assuming that the user wants to quickly know the main information to be conveyed by these multiple unread messages, then the electronic device 100 can detect the user's long press operation on chat item 311. In response to the long press operation, refer to Fig. 12B , the electronic device 100 can start the natural language interaction function, and the electronic device 100 can also determine that the interaction object selected by the user is the chat entry 311.

[0269] Continue reading Fig. 12BAfter starting the natural language interaction function, the electronic device 100 may display a natural language interaction entry 313. The natural language interaction entry 313 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0270] Optionally, a logo 313a may be displayed on the natural language interaction entrance 313, and the logo 313a may be used to prompt the user that the electronic device 100 has started the natural language interaction function.

[0271] Optionally, a prompt 313b may be displayed on the natural language interaction entrance 313, and the prompt 313b may be used to prompt the user to input a natural language instruction related to the interaction object. Fig. 12B As shown in the example, the user selects the chat item 311 as the interactive object, and the prompt 313b may be a prompt for prompting the user to input a natural language instruction related to the chat item (for example, you may say "show unread message summary").

[0272] Optionally, the prompt 313b may also be output in the form of voice broadcast.

[0273] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0274] See also Fig. 12C Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "show unread message summary"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 313d "show unread message summary").

[0275] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Fig. 12B The exemplary logo 313a is replaced by Fig. 12C The logo 313c is shown as an example. The logo 313c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0276] Further, see Fig. 12C After the above voice instruction is input, the electronic device 100 can also analyze the above voice instruction to determine that the operation corresponding to the above voice instruction is to display the unread message summary in the chat entry 311.

[0277] Furthermore, the electronic device 100 may search for unread messages in the chat entry 311, and generate an unread message digest through a relevant algorithm. After the unread message digest is generated, refer to Fig.12D , the electronic device 100 can display the generated unread message summary 314. So far, the display function of the unread message summary is completed.

[0278] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the unread message summary 314.

[0279] It should be noted that the voice instruction "display unread message summary" for realizing the display function of the unread message summary is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the display function of the unread message summary needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the unread message summary. Among them, the above-mentioned satisfying "certain conditions" may mean that the voice instruction includes certain keywords for indicating the viewing of the unread message summary (for example, keywords such as "display", "view", "unread message", "new message", "summary", "outline", "brief description", "introduction", "what", etc.).

[0280] (2) Question and answer query

[0281] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 and the user can conduct question-and-answer queries on chat messages (for example, querying which contact in a chat entry sent what voice / text message, which contact sent what file, etc.). In this way, the user only needs to click on the chat entry display interface (for example, Fig. 12A The user interface 310 shown can quickly obtain specific chat information without the user having to enter the chat interface corresponding to the chat entry to view it. This is convenient and fast, simplifies user operations, and improves user experience.

[0282] Similarly, the interactive object selected by the user is Fig. 12A Taking the chat entry 311 as an example, the specific process of the user selecting the interaction object and starting the natural language interaction function of the electronic device 100 can refer to the above-mentioned Fig. 12A and Fig. 12B The description will not be repeated here.

[0283] Also, taking the natural language command input by the user as a voice command as an example, see Fig.12E, assuming that the user wants to know what voice / text message Lily, a group member in the group chat XXX corresponding to the chat entry 311, has sent in the group chat, then the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "what Lily said"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 313e "what Lily said").

[0284] Further, see Fig.12E After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to display what voice / text message Lily, a group member in the group chat XXX corresponding to the chat entry 311, has sent in the group chat.

[0285] Further, the electronic device 100 can query the voice / text message sent by the group member Lily in the group chat XXX corresponding to the chat entry 311 in the group chat. After the query is completed, refer to Fig.12F , the electronic device 100 may display the query result 315, which may include the voice / text message sent by Lily in the group chat XXX that the electronic device 100 has queried.

[0286] It should be noted that the voice / text messages sent by Lily in the group chat XXX queried above may be all voice / text messages sent by Lily in the group chat XXX that have been read and unread by the user, or may be only the voice / text messages sent by Lily in the group chat XXX that have not been read by the user. This embodiment of the present application is not limited to this.

[0287] Optionally, for the voice message found in the above search, the electronic device 100 may convert the voice message into a text message and then display it in the search result, so that it is convenient for the user to view.

[0288] It is easy to understand that if the user wants to know other information sent by a certain contact, for example, what files a certain contact has sent, then the electronic device 100 can display the files sent by the contact in the above query result 315.

[0289] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style, it can also be a normal text style, etc.) and display position of the above-mentioned query result 315.

[0290] It should be noted that the above-mentioned voice instruction "What did Lily say" for question-and-answer queries on chat messages is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the above-mentioned voice instruction for question-and-answer queries on chat messages needs to meet certain conditions, under which the electronic device 100 can determine which specific chat information in the chat message the user wants to view. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating which specific chat information in the chat message to view (such as the contact's nickname, "say", "ask", "send", "what", "which" and other keywords).

[0291] It should be noted that social applications generally support virtual keyboard input. Therefore, in the embodiments of the present application, the natural language instructions input by the user can be voice instructions or text instructions. The above description is only based on the example of the user inputting voice instructions. The specific process of the user inputting text instructions can refer to the relevant content in the aforementioned display application scenario, which will not be repeated here.

[0292] It should be noted that the above only takes the corresponding operations on chat messages as an example to illustrate the interaction method provided in the embodiment of the present application. It is not limited to this. The interaction method provided in the embodiment of the present application can also perform corresponding operations on other types of messages (such as advertising, news and other push messages). The specific process is similar to the corresponding operations on chat messages, and will not be repeated here.

[0293] It should be noted that the above-mentioned corresponding operations on chat messages are only explained by taking the two operations of displaying unread message summaries and question and answer queries as examples. It is not limited to these two operations, and there may be more operations. The embodiments of the present application do not limit this.

[0294] 2. Perform corresponding operations on the document

[0295] (1) Display document summary

[0296] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to view document summaries without opening the documents. In this way, when it is necessary to obtain the main information of multiple documents, users can quickly obtain the main information that the documents want to convey without opening each document in turn. This is convenient, fast and efficient, simplifies user operations and improves user experience.

[0297] For example, see Fig.13A The user interface 410 may include one or more document icons (eg, icon 411 of document 5, etc.), and the user may open the corresponding document by clicking on the document icon.

[0298] In an embodiment of the present application, a user can trigger the electronic device 100 to start a natural language interaction function by performing a long press operation on the document icon (for example, a long press operation performed by the user's finger or a touch device), and can also trigger the document icon to be used as an interaction object selected by the user for subsequent natural language interaction.

[0299] Continue reading Fig.13A , assuming that the user wants to view the summary of document 5, then the electronic device 100 can detect the user's long press operation on the icon 411 of document 5, and in response to the long press operation, refer to Fig. 13B , the electronic device 100 can start the natural language interaction function, and the electronic device 100 can also determine that the interaction object selected by the user is the icon 411 of the document 5.

[0300] Continue reading Fig. 13B After starting the natural language interaction function, the electronic device 100 may display a natural language interaction entrance 412. The natural language interaction entrance 412 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0301] Optionally, a logo 412a may be displayed on the natural language interaction entrance 412, and the logo 412a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0302] Optionally, a prompt 412b may be displayed on the natural language interaction entrance 412, and the prompt 412b may be used to prompt the user to input a natural language instruction related to the interaction object. Fig. 13B As shown in the example, the user selects the icon 411 of the document 5 as the interactive object, and the prompt 412b can be a prompt for prompting the user to input a natural language instruction related to the document (for example, you can say "what document is this").

[0303] Optionally, the prompt 412b may also be output in the form of voice broadcast.

[0304] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0305] See also Fig. 13CTaking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "what is this document" or "show the summary of this document", etc.), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 412d "what is this document" or "show the summary of this document", etc.).

[0306] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Fig. 13B The exemplary logo 412a is replaced by Fig. 13C The logo 412c is shown as an example. The logo 412c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0307] Further, see Fig. 13C After the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to display the summary of the document 5.

[0308] Furthermore, the electronic device 100 can obtain the content of the document 5 (such as the title, the text, etc.), and generate a summary of the document 5 through a relevant algorithm. After the summary of the document 5 is generated, refer to Fig.13D , the electronic device 100 may display the generated result 413, which may include the summary of the document 5, and optionally, may also include the title of the document 5. So far, the display function of the document summary is completed.

[0309] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above result 413.

[0310] It should be noted that the voice instruction "What document is this" for implementing the display function of the document summary is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the display function of the document summary needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the document summary. Among them, the above-mentioned satisfying "certain conditions" may mean that the voice instruction includes certain keywords for indicating viewing the document summary (for example, keywords such as "display", "view", "what document", "summary", "outline", "brief description", and "introduction").

[0311] (2) Display notes in the document

[0312] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to view the notes (such as marks, annotations, etc.) in the document without opening the document. In this way, when there are multiple documents, the user can quickly view the notes in the document without opening each document in turn, which is convenient, fast and efficient, simplifies user operations and improves user experience.

[0313] Similarly, the interactive object selected by the user is Fig.13A Taking the icon 411 of the exemplary document 5 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Fig.13A and Fig. 13B The description will not be repeated here.

[0314] Also, taking the natural language command input by the user as a voice command as an example, see Fig.13E , assuming that the user wants to view the notes in document 5, the electronic device 100 can detect the relevant voice instructions input by the user (such as the voice "What is marked in the document"), and can display the detected relevant voice instructions input by the user in the form of text (such as text 412e "What is marked in the document").

[0315] Further, see Fig.13E After the above voice instruction is input, the electronic device 100 can also analyze the above voice instruction to determine that the operation corresponding to the above voice instruction is to display the notes in the document 5.

[0316] Further, the electronic device 100 can obtain the document 5 and search for notes in the document 5. After the search is completed, refer to Fig.13F , the electronic device 100 may display the query result 414, which may include the notes in the queried document 5. Fig.13F As can be seen from the figure, the notes in document 5 include highlight marks and comments. So far, the display function of the notes in the document has been realized.

[0317] In some embodiments, the electronic device 100 may also save the query result 414, for example, the query result 414 may be saved in a memo, so that the user can view it in real time.

[0318] In some embodiments, the electronic device 100 may also support the user to add notes to the document. For example, the user may input voice instructions related to adding notes, and the electronic device 100 may add corresponding notes to the document based on the voice instructions.

[0319] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style, it can also be a normal text style, etc.) and the display position of the above-mentioned query result 414.

[0320] It should be noted that the voice instruction "What is marked in the document" for realizing the function of displaying the notes in the document is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for realizing the function of displaying the notes in the document needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the notes in the document. Among them, the above-mentioned satisfying "certain conditions" may mean that the voice instruction includes certain keywords for indicating viewing the notes in the document (such as keywords such as "mark", "mark", "annotate", "what", "note", etc.).

[0321] (3) Document content translation

[0322] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to translate the content of the document and view the translated content without opening the document. In this way, when the user wants to translate the content of the document, the translation of the document content can be completed without the user opening the document and manually performing the translation operation. This is convenient, fast and efficient, simplifies user operations and improves user experience.

[0323] Similarly, the interactive object selected by the user is Fig.13A Taking the icon 411 of the exemplary document 5 as an example, the specific process of the user selecting the interactive object and starting the natural language interactive function of the electronic device 100 can refer to the aforementioned Fig.13A and Fig. 13B The description will not be repeated here.

[0324] Also, taking the natural language command input by the user as a voice command as an example, see Figure 13G , assuming that the user wants to translate the content of document 5 into English, then the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "translate the full text into English"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 412f "translate the full text into English").

[0325] Further, see Figure 13G After the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to translate the full text of document 5 into English and display the translated full text of document 5.

[0326] Furthermore, the electronic device 100 can obtain the full text of the document 5 and translate the full text into English. After the translation is completed, refer to Fig.13H , the electronic device 100 may display the translation result 415, which may include the full text content of the document 5 translated into English. So far, the document content translation function is realized.

[0327] In some embodiments, the electronic device 100 may also save the translation result 415 as a new document, so that the user can view it in real time.

[0328] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above-mentioned translation result 415.

[0329] It should be noted that the voice instruction "Translate the full text into English" for implementing the document content translation function is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for implementing the document content translation function needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to translate the document content. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating that the document is to be translated into full text (such as the name of the language to be translated / converted into, "translation", "conversion" and other keywords).

[0330] (4) Document format conversion / beautification

[0331] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to convert or beautify the format of a document without opening the document (for example, beautifying the document format into a format that is easy for users to read and has both pictures and text), which is convenient and fast, simplifies user operations, and improves user experience.

[0332] Among them, the specific process of converting or beautifying the format of the document is similar to the specific process of displaying the document summary, displaying the notes in the document, and translating the document content, except that the voice commands input by the user are different, which will not be repeated here.

[0333] It is easy to understand that when a user wants to convert the format or beautify the format of a document, the user can input voice commands related to format conversion or beautification (for example, "convert the document format to XX format", etc.), and the electronic device 100 can convert the format or beautify the format of the document based on the voice command. After completing the format conversion or beautification, the electronic device 100 can display the document after format conversion or beautification for the user to view.

[0334] (5) Question and answer query

[0335] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 and the user can conduct question-and-answer queries on documents (for example, querying whether a certain document includes certain content, etc.). In this way, the user can quickly obtain relevant information about the document without opening the document, which is convenient and fast, simplifies user operations, and improves user experience.

[0336] Among them, the specific process of conducting question and answer queries on documents is similar to the specific process of displaying document summaries, displaying notes in documents, and translating document contents, except that the voice commands input by the user are different, which will not be repeated here.

[0337] It is easy to understand that when a user wants to conduct a question and answer query on a document, the user can input a voice command related to the question and answer query (for example, "Does the document include XXX content?", etc.), and the electronic device 100 can obtain the corresponding document based on the voice command and query the relevant information of the document indicated by the voice command. After the query is completed, the electronic device 100 can display the query results for the user to view.

[0338] It should be noted that, in the case where the application providing the user interface 410 supports virtual keyboard input, the natural language instruction input by the user can be not only a voice instruction but also a text instruction. The above description is only based on the example of the user inputting a voice instruction. The specific process of the user inputting a text instruction can refer to the relevant content in the aforementioned display application scenario, which will not be repeated here.

[0339] It should be noted that the above-mentioned corresponding operations on the document are only illustrated by taking the five operations of displaying document summary, displaying notes in the document, translating document content, converting / beautifying document format, and question and answer query as examples. It is not limited to these five operations and can also be more operations. The embodiments of the present application do not limit this.

[0340] 3. Operational Application Scenarios

[0341] "Operational application scenarios" may refer to application scenarios in which the interactive method provided in the embodiments of the present application is used to perform corresponding operations (such as querying, collecting, editing, displaying summaries, etc.) on text (such as ordinary text, or special text such as links, names, numbers, email addresses, dates, addresses, etc.).

[0342] 1. Text Collection

[0343] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to quickly collect selected text or selected text and its context, realize the quick text collection function, which is convenient and fast, simplifies user operations, and improves user experience.

[0344] In an embodiment of the present application, a user can trigger the electronic device 100 to select the corresponding text and display the menu bar corresponding to the text by performing a long press operation on the text (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0345] Take a reading application as an example, see Fig.14A , Fig.14A The exemplary user interface 510 may be a user interface of the reading application, and the user interface 510 may include one or more text paragraphs (eg, text paragraph 511 ).

[0346] Continue reading Fig.14A Assuming that the user wants to collect text, the electronic device 100 can detect a certain position (for example, Fig.14A In response to the long press operation, refer to Fig. 14B , the electronic device 100 can select the corresponding text (for example, the text "X"), and the electronic device 100 can also display a menu bar 513, which can include one or more options (for example, a "search" option, a "copy" option, a "select all" option, a "translate" option, etc.), and these one or more options can be used to perform corresponding operations on the selected text. The electronic device 100 can also start a natural language interaction function.

[0347] It should be noted that the range of text selected by the electronic device 100 through the long press operation can be set according to preset rules. For example, the text selected by the electronic device 100 can be only the text at the location where the long press operation is located, or it can be a larger range of text than the location where the long press operation is located. The embodiments of the present application are not limited to this.

[0348] Continue reading Fig. 14B After starting the natural language interaction function, the electronic device 100 may display a natural language interaction entrance 512. The natural language interaction entrance 512 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0349] Optionally, a logo 512a may be displayed on the natural language interaction entrance 512, and the logo 512a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0350] Optionally, a prompt 512b may be displayed on the natural language interaction entrance 512, and the prompt 512b may be used to prompt the user to input a natural language instruction related to the interactive object (such as the selected text or the selected text and its context), for example Fig. 14B The prompt 512b is shown as an example (for example, you can say "Collect this paragraph", "Collect this sentence", "Collect this page" or "Collect this chapter").

[0351] Optionally, the prompt 512b may also be output in the form of voice broadcast.

[0352] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0353] See also Fig. 14C Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to collect the text paragraph 511, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "collect this paragraph"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 512d "collect this paragraph").

[0354] Optionally, after the electronic device 100 starts detecting the voice command input by the user, Fig. 14B The exemplary logo 512a is replaced by Fig. 14C The logo 512c is shown as an example. The logo 512c can be used to prompt the user that the electronic device 100 is detecting a voice command input by the user.

[0355] Further, see Fig. 14C After the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to collect the text segment (ie, text segment 511) where the selected text is located.

[0356] Furthermore, the electronic device 100 can obtain the text segment (ie, text segment 511) where the selected text is located by using a related text detection and recognition algorithm. After the acquisition is completed, refer to Fig.14D , the electronic device 100 may display a window 513, which may include the acquired text paragraph to be collected (ie, the text paragraph 511). Further, the electronic device 100 may collect the acquired text paragraph to be collected. Fig.14E The electronic device 100 may also display a window 514, which may include information indicating the collection path (eg, "Collect to Memo") and the acquired text paragraphs to be collected. Thus, the text collection function is completed.

[0357] It is easy to understand that the above-mentioned collection path can be a system preset of the electronic device 100, or can be set by the user independently, and the embodiment of the present application is not limited to this.

[0358] In some embodiments, the above-mentioned window 513 and window 514 may not be displayed.

[0359] It is easy to understand that the above-mentioned text detection and recognition algorithm can be one or more, and the embodiment of the present application does not limit the specific type of the algorithm. Exemplarily, the above-mentioned text detection and recognition algorithm can be, for example, an optical character recognition (OCR) algorithm. In an embodiment of the present application, the electronic device 100 can give priority to selecting the text currently selected by the user as the interactive object. Further, the scope of the interactive object can be expanded in combination with the semantics of the user's instruction (for example, the text selected by the user is only a part of the text in a certain text paragraph, and the semantics of the user's instruction is the entire text paragraph, then the interactive object can be expanded from the above-mentioned part of the text to the entire text paragraph). That is to say, if the electronic device 100 determines that the interactive object is only the selected text based on the user instruction and the text currently selected by the user, then the electronic device 100 can perform the operation corresponding to the above instruction on the selected text; if the electronic device 100 determines that the interactive object includes not only the text currently selected by the user but also its context based on the user instruction and the text currently selected by the user, then the electronic device 100 can detect and identify the context of the selected text. For example, if the interactive object is the entire text paragraph where the text currently selected by the user is located, then the electronic device 100 can determine the above text paragraph through the paragraph recognition rules in the OCR algorithm (such as the blank line / paragraph identifier of the text, the alignment of the text, specific identifiers, font style, etc.). It is easy to understand that the above is only explained by taking the recognition of a paragraph of text as an example. The recognition of a sentence of text, a page of text, a chapter of text, etc. is similar to the above recognition of a paragraph of text, and will not be repeated here.

[0360] It should be noted that the voice instruction "Collect this paragraph" for implementing the text collection function is merely exemplary and should not constitute a limitation on the present application. It is easy to understand that the voice instruction for implementing the text collection function needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to collect the text. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating text collection (such as keywords such as "collection", "this paragraph", "this sentence", "this page", "this chapter", "this article", etc.).

[0361] 2. Text query

[0362] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to quickly query the information related to the selected text, realize the quick text query function, which is convenient and fast, simplifies the user operation, and improves the user experience.

[0363] The specific process of the user selecting text and activating the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection, and can refer to the aforementioned Figure 14A-14B The description will not be repeated here.

[0364] See also Fig.14F , assuming that the text selected by the user is "XXX Book", the electronic device 100 can recognize that the text is the name of the book. Therefore, after the electronic device 100 starts the natural language interaction function, the electronic device 100 can optionally display a prompt 512e (for example, "You can ask this for more information") on the natural language interaction entrance 512. The prompt 512e can be used to prompt the user to enter natural language instructions related to the selected text.

[0365] Also, taking the natural language command input by the user as a voice command as an example, see Figure 14G Assuming that the user wants to know the introduction of the book "XXX Book" that has been selected above, the electronic device 100 can detect the relevant voice instructions input by the user (for example, voice "show the introduction of this book"), and can display the detected relevant voice instructions input by the user in the form of text (for example, text 512f "show the introduction of this book").

[0366] Further, see Figure 14G After the voice command is input, the electronic device 100 may also analyze the voice command to determine that the operation corresponding to the voice command is to display the introduction of the book corresponding to the text "XXX Books" selected by the user.

[0367] Furthermore, the electronic device 100 can query the introduction of the book corresponding to the text "XXX Book" selected by the user locally or in the cloud. After the query is completed, refer to Fig.14H , the electronic device 100 can display the query result 515, and the query result 515 can include the introduction of the book corresponding to the text "XXX Books" selected by the user. So far, the text query function is realized.

[0368] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style, it can also be a normal text style, etc.) and display position of the above-mentioned query result 515.

[0369] It should be noted that the above only uses the example that the text selected by the user is text related to the book name to illustrate the text query function, and is not limited to this. Other texts (such as text related to audio and video names, article names, game names, application names, etc.) can also be used. The embodiments of the present application do not limit this.

[0370] It should be noted that the above description only uses a certain reading application as an example to illustrate the text collection and text query functions, and is not limited to this. It can also be other applications that can support text display and text selection, and the embodiments of the present application do not limit this.

[0371] It should be noted that the above only takes ordinary text as an example to illustrate the process of using the interactive method provided in the embodiment of the present application to perform corresponding operations on text. It is not limited to ordinary text. The interactive method provided in the embodiment of the present application is also applicable to special texts (such as links, names, numbers, emails, dates, addresses, etc.). For example, for a link, by selecting the link and entering the relevant natural language instructions, a summary of the specific content corresponding to the link can be displayed; for another example, for a name / number (such as a mobile phone number, a personal account, etc.), by selecting the name / number and entering the relevant natural language instructions, communication and social interaction (such as collecting the name / number, adding the name / number to the contacts in the address book, etc.) / order tracking, etc. can be performed; for another example, for an email address, by selecting the email address and entering the relevant natural language instructions, an email can be sent; for another example, for a date, by selecting the date and entering the relevant natural language instructions, a reminder / alarm can be set; for another example, for an address, by entering the address and the relevant natural language instructions, the navigation / ticketing / weather information corresponding to the address can be queried. It is easy to understand that, similar to the above-mentioned scenario of performing corresponding operations on ordinary text, in the scenario of performing corresponding operations on special text, the electronic device 100 can also display prompts related to the special text, such as prompts 512b and prompts 512e.

[0372] It should be noted that, in the case where the application providing the user interface 510 supports virtual keyboard input, the natural language instruction input by the user can be not only a voice instruction but also a text instruction. The above description is only based on the example of the user inputting a voice instruction. The specific process of the user inputting a text instruction can refer to the relevant content in the aforementioned display application scenario, which will not be repeated here.

[0373] 3. Text Editing

[0374] (1) Text content modification (such as content insertion, deletion, replacement, etc.)

[0375] In order to modify the text content through the natural language interaction function, the user is currently required to input natural language instructions that can accurately express the user's modification intention and modify the text. Usually, the instruction needs to include the text to be modified, the modification method, and the modified target text. For example, the text before modification is "Today I am in Beijing", and the modified text is "Today I am in Shanghai". Then, the natural language instruction input by the user can be, for example, "Change Beijing to Shanghai". It can be seen that the instruction includes "Beijing", the text to be modified, "Change to", a keyword used to describe the modification method, and "Shanghai", the modified target text. That is to say, the above-mentioned relatively complex and cumbersome instructions need to be input to complete the text content modification.

[0376] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user inputting an instruction that only includes the modified target text, and the electronic device 100 can automatically generate one or more semantically coherent modification results based on the instruction and the text before modification, thereby achieving the purpose of completing the text content modification in a simple and natural way, which is convenient and fast and improves the user experience. The following is a detailed introduction in conjunction with the user interface schematic diagram.

[0377] For text content modification, common modification requirements include the following three types, which are introduced in detail below.

[0378] 1) Iterative optimization of expression (for example, the text before modification is "I want to make a plan", and the text after modification is "I want to make a systematic fitness plan")

[0379] In an embodiment of the present application, a user can trigger the electronic device 100 to select the corresponding text by performing a long press operation on the text that needs to be modified (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start a natural language interaction function.

[0380] The specific process of the user selecting the text to be modified and starting the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection, and can refer to the aforementioned Figure 14A-14B The description will not be repeated here.

[0381] For the modification requirement of "iterative optimization of expression", one or more semantically coherent modification results may be generated during the modification process, which are described in detail below.

[0382] a. Generate multiple semantically coherent modification results

[0383] Take an application that supports text content modification (such as a memo application) as an example. Fig.15AThe user interface 610 may be a user interface of the application, and the user interface 610 may include text that needs to be modified (for example, the text "The weather is very good today, suitable for outdoor activities."), and may also include a natural language interaction entrance 611 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0384] Among them, the text that needs to be modified selected by the user may include only the text to be modified (for example, the text "The weather is very good today"), or it may include both the text to be modified (for example, the text "The weather is very good today") and the context of the text to be modified (for example, the text "suitable for outdoor activities"), and the embodiments of the present application are not limited to this.

[0385] Continue reading Fig.15A Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to add text content related to expressing "Today's weather is better than yesterday's" to the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "better than yesterday"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 611a "better than yesterday").

[0386] Further, see Fig.15A Taking the example that the text selected by the user to be modified is "The weather is very good today", after the above voice instruction is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice instruction (for example, the voice "better than yesterday"), so as to determine that the operation corresponding to the above voice instruction is to add the text "better than yesterday" to the text selected by the user to be modified "The weather is very good today" to generate a semantically coherent modification result.

[0387] Furthermore, the electronic device 100 can generate a semantically coherent modification result through a related algorithm. After the modification result is generated, refer to Fig. 15B , the electronic device 100 may generate and display a variety of semantically coherent modification results, such as modification result 612 "Today's weather is better than yesterday's", modification result 613 "Today's weather is very good, better than yesterday's". In this case, the electronic device 100 can support the user to select one of the modification results.

[0388] Optionally, the electronic device 100 may prompt the user to select one of the modification results by voice announcement or text prompt (eg, prompt 614 “Which modification result do you want to select”).

[0389] In a possible implementation, the user may select one of the modification results by inputting a voice command, see Fig. 15C, the user can input a voice-related voice command (e.g., voice "first"), and the electronic device 100 can display the detected voice command input by the user in the form of text (e.g., text 611b "first"). Further, after the voice command is input, the electronic device 100 can analyze the voice command to determine that the operation corresponding to the voice command is to modify the text before modification according to the first modification result (e.g., modification result 612). Further, refer to Fig.15D , the electronic device 100 can modify the text before modification according to the modification result 612 and display the modified text (for example, the text "Today's weather is better than yesterday's and suitable for outdoor activities.").

[0390] In another possible implementation, continue to refer to Fig. 15B The user can also select one of the modification results by clicking. For example, if the user wants to select modification result 612, the user can click on modification result 612. After the electronic device 100 detects the click operation, it can modify the text before modification according to modification result 612 and display the modified text (for example, the text "Today's weather is better than yesterday's, which is suitable for outdoor activities.").

[0391] b. Generate a semantically coherent modification result

[0392] Taking an application that supports text content modification (such as a memo application) as an example, see Fig.15E The user interface 610 may be a user interface of the application, and the user interface 610 may include text that needs to be modified (such as the text "I want to make a plan"). The user interface 610 may also include a natural language interaction entrance 615 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0393] Continue reading Fig.15E Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to add text content related to expressing "systematic fitness plan" in the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "systematic fitness"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 615a "systematic fitness").

[0394] Further, see Fig.15ETaking the example that the text selected by the user to be modified is "I want to make a plan", after the above voice instruction is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice instruction (for example, the voice "systematic fitness"), so as to determine that the operation corresponding to the above voice instruction is to add the text "systematic fitness" to the text selected by the user to be modified "I want to make a plan" to generate a semantically coherent modification result.

[0395] Furthermore, the electronic device 100 can generate a semantically coherent modification result through a related algorithm. After the modification result is generated, refer to Fig.15F , the electronic device 100 may only generate one semantically coherent modification result. In this case, the electronic device 100 can directly modify the text before modification according to the generated semantically coherent modification result and display the modified text (for example, "I want to make a systematic fitness plan").

[0396] Optionally, see Figure 15G , the electronic device 100 displays Fig.15F Before the modified text is shown, the generated semantically coherent modification result (for example, modification result 616 "I want to make a systematic fitness plan") may be displayed first, and the electronic device 100 will display the modified text after detecting that the user confirms the modification through voice command or click operation on the modification result 616. Fig.15F The modified text is shown. In this way, the user can be given more options. If the user is not satisfied with the modification result, he can also re-enter the relevant instructions to generate a new modification result.

[0397] 2) Automatic speech recognition (ASR) errors caused by noise and accent (for example, the text before modification is "Play me a song with upgraded wallpaper", and the text after modification is "Play me a song with rare characters")

[0398] In an embodiment of the present application, a user can trigger the electronic device 100 to select the corresponding text by performing a long press operation on the text that needs to be modified (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start a natural language interaction function.

[0399] The specific process of the user selecting the text to be modified and starting the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection, and can refer to the aforementioned Figure 14A-14B The description will not be repeated here.

[0400] Taking an application that supports text content modification (such as a memo application) as an example, see Fig.15HThe user interface 610 may be a user interface of the application, and the user interface 610 may include text that needs to be modified (for example, the text "A song was played today"). The user interface 610 may also include a natural language interaction entry 617 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0401] Continue reading Fig.15H Taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to replace "upgrade wallpaper" with "uncommon characters" in the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, voice "uncommon characters"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 617a "uncommon characters").

[0402] Further, see Fig.15H Taking the text selected by the user to be modified as "A song that promotes wallpaper was played today" as an example, after the above voice command is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice command (for example, the voice "uncommon characters"), so as to determine that the operation corresponding to the above voice command is to replace "promoting wallpaper" with "uncommon characters" in the text selected by the user to be modified as "A song that promotes wallpaper was played today".

[0403] Furthermore, the electronic device 100 can generate a semantically coherent modification result through a related algorithm. After the modification result is generated, refer to Fig.15I , the electronic device 100 may only generate one semantically coherent modification result. In this case, the electronic device 100 can directly modify the text before modification according to the generated semantically coherent modification result and display the modified text (for example, "A song with rare characters was played today").

[0404] Optionally, when the electronic device 100 generates one or more semantically coherent modification results, the electronic device 100 displays Fig.15I Before displaying the modified text, one or more semantically coherent modification results (such as Fig.15J The electronic device 100 displays the modified text after detecting that the user confirms the modification result to be selected through voice command or click operation on a modification result. In this way, the user can be given more choices. If the user is not satisfied with the modification result, he can also re-enter the relevant command to generate a new modification result.

[0405] The above voice command only takes "rare characters" as an example. In some embodiments, for homophonic characters or characters with similar pronunciations, in order to avoid errors, the above voice command can be a more specific voice command. For example, for the command with the intention of indicating adding "rare characters", the above voice command "rare characters" can be a more specific voice command, such as "rare characters, the character for 'life', the character for 'out-of-the-way', the character for 'font'".

[0406] 3) Slips of the tongue and filler words generated during the natural language input process (for example, the original text is "I heard that it will rain tomorrow", and the modified text is "I heard that it will rain tomorrow")

[0407] During the natural language input process, users' colloquial expressions usually generate filler words and slips of the tongue. Although there are filler words and slips of the tongue, the original meaning of the input sentence is completely retained. Therefore, the electronic device 100 can identify and correct colloquial errors through technologies such as large language models (LLMs). Among them, the above large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of natural language text. The above large language model can handle various natural language tasks, such as text classification, question answering, dialogue, etc.

[0408] In the embodiments of the present application, when the user performs a long press operation on the text to be modified (for example, a long press operation performed by the user's finger or a touch device), it can trigger the electronic device 100 to select the corresponding text. In addition, it can also trigger the electronic device 100 to start the natural language interaction function, so as to automatically identify and correct colloquial errors.

[0409] The following will separately introduce in detail the specific process of the electronic device 100 automatically identifying and correcting colloquial errors for the filler words and slips of the tongue in the text to be modified.

[0410] a. Filler words in the text to be modified

[0411] Taking a certain application program that supports text content modification (such as a memo application program) as an example, refer to Figure 15K , the user interface 610 can be a user interface of this application program. The user interface 610 may include the text to be modified (such as the text "Today, um, the weather is very good"). It can be seen that the text contains the filler word "um".

[0412] Continue to refer to Figure 15K, assuming that the user wants to delete the interjection words (such as "hmm") in the text to be modified, then the electronic device 100 can detect the user's operation of selecting the text to be modified (such as a long press operation). In response to the operation, the electronic device 100 can select the corresponding text and automatically identify and delete the interjection words in the text through a related algorithm, thereby generating a modification result 615 "The weather is very good today". The embodiment of the present application does not limit the display form of the modification result 615, for example, it can be displayed on Figure 15K In the natural language interaction entrance shown.

[0413] It can be seen that in the embodiment of the present application, the electronic device 100 can automatically identify and delete tone words in the text selected by the user without the user inputting natural language instructions, which is convenient and fast and can provide the user with an intelligent experience.

[0414] In one possible implementation, see Figure 15L After generating the above modification result 615, the electronic device 100 can directly modify the text before modification according to the modification result and display the modified text (for example, "The weather is very good today").

[0415] In another possible implementation, the electronic device 100 may also display the modified result 615 after detecting that the user confirms the modification through a voice command or a click operation on the modified result 615. Figure 15L Modified text shown.

[0416] b. Slips of the tongue in the text that need to be corrected

[0417] Taking an application that supports text content modification (such as a memo application) as an example, see Figure 15M The user interface 610 may be a user interface of the application program, and the user interface 610 may include text to be modified (e.g., the text "I am on a business trip to Beijing today, no, it is Shanghai, I am doing some research with some college teachers."). It can be seen that there is a slip of the tongue in the text, that is, the city I am on a business trip to today is Shanghai, not Beijing.

[0418] Continue reading Figure 15M , assuming that the user wants to correct a slip of the tongue in the text to be modified, then the electronic device 100 can detect the user's operation of selecting the text to be modified (such as a long press operation). In response to the operation, the electronic device 100 can select the corresponding text and can automatically identify and correct the slip of the tongue in the text through a related algorithm, thereby generating a modification result 616 "I am on a business trip to Shanghai today". The embodiment of the present application does not limit the display form of the modification result 616, for example, it can be displayed on Figure 15M In the natural language interaction entrance shown.

[0419] It can be seen that in the embodiment of the present application, the electronic device 100 can automatically identify and correct the slip of the tongue in the text selected by the user without the user inputting natural language instructions, which is convenient and fast and can provide the user with an intelligent experience.

[0420] In one possible implementation, see Fig.15N After generating the above modification result 616, the electronic device 100 can directly modify the text before modification according to the modification result and display the modified text (for example, "I am on a business trip to Shanghai today, and I am doing some discussions with several college teachers.").

[0421] In another possible implementation, the electronic device 100 may also display the modified result 616 after detecting that the user confirms the modification through a voice command or a click operation on the modified result 616. Fig.15N Modified text shown.

[0422] In some embodiments, during the natural language input process, the electronic device 100 can automatically identify and correct spoken errors. That is, the text generated by the natural language input is the text after the spoken errors are corrected. In this way, there is no need for subsequent users to actively select text and modify the text content, which simplifies user operations and improves text input efficiency.

[0423] It is easy to understand that the above-mentioned text content modification through natural language interaction function can be implemented by the electronic device 100 based on an input method application (such as the Xiaoyi input method application).

[0424] In some embodiments, the interactive method provided in the embodiments of the present application is not limited to the above-mentioned text content modification. The interactive method can also realize text format modification through a natural language interaction function. It is easy to understand that the process is similar to the above-mentioned text content modification, except that the natural language instructions input by the user are different. In the text format modification scenario, the user can input a natural language instruction to indicate the text format modification, wherein the above-mentioned text format may include but is not limited to one or more of the following: font, font size, color, alignment, style (title / text), bold, italic, and underline.

[0425] (2) Add, display, and edit comments in a document

[0426] By implementing the interaction method provided in the embodiment of the present application, when the electronic device 100 opens a certain document and enters the annotation mode, the electronic device 100 can support the user to trigger the electronic device 100 to start the natural language interaction function through a long press operation (for example, a long press operation performed by the user's finger or a touch device), and can add annotations to the document based on the natural language interaction function, edit annotations, etc., which is convenient and fast, simplifies user operations, and improves user experience.

[0427] Let's first combine Figures 16A-16G The specific process of adding annotations to a document is introduced by way of example.

[0428] See also Fig.16A The user interface 710 may be a user interface when the electronic device 100 opens a document (eg, a document 711 regarding an annual work summary) and enters an annotation mode.

[0429] In the embodiment of the present application, assuming that the user wants to add a comment to the current page of the document, the electronic device 100 can detect the user's long press operation on the current page of the document 711. In response to the long press operation, the electronic device 100 can start the natural language interaction function and can identify whether there is an interactive object / subject (such as an image, text, etc.) at the location of the long press operation in the current page of the document 711. If not, the electronic device 100 can generate an annotation based on the voice command input by the user, and can display the generated annotation at the location of the long press operation; if yes, the electronic device 100 can not only generate an annotation based on the voice command input by the user, but also highlight the interactive object, and can also display the generated annotation at other locations other than the location of the long press operation (such as the center position of the interactive object). The following introduces these two situations respectively.

[0430] For example, see Fig.16A , assuming that there is no interactive object at the location where the user long presses, for example Fig.16A The position of the long press operation is blank, that is, there is no interactive object. Then, after the electronic device 100 starts the natural language interaction function, refer to Fig. 16B, the electronic device 100 can display a comment box 712 at the location of the above-mentioned long press operation, and can also display a logo 712b. The logo 712b can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user. Assuming that the comment the user wants to add is "It is good to add a subtitle here", then the electronic device 100 can detect the relevant voice command input by the user (for example, the voice "It is good to add a subtitle here"), and can display the detected relevant voice command input by the user in the form of text (for example, text 712a "It is good to add a subtitle here"). Furthermore, after the above-mentioned voice command is input, if the electronic device 100 detects that the user cancels the long press operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), then the electronic device 100 can generate an annotation based on the above-mentioned voice command. After the annotation is generated, refer to Fig. 16C , the electronic device 100 can display the generated annotation (for example, annotation 713) at the location where the long press operation is performed.

[0431] For example, see Fig.16D , assuming that there is an interactive object at the location where the user long presses, for example Fig.16D The text "year" is present at the position of the long press operation, that is, there is an interactive object. Then, after the electronic device 100 starts the natural language interaction function, refer to Fig.16E , the electronic device 100 can display a comment box 714 at the location of the above-mentioned long press operation, and can also display a logo 714b. The logo 714b can be used to prompt the user that the electronic device 100 is detecting a voice instruction input by the user. Assuming that the comment the user wants to add is "Enlarge the title font size", the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "Enlarge the title font size"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 714a "Enlarge the title font size"). Furthermore, after the above-mentioned voice instruction is input, if the electronic device 100 detects that the user cancels the long press operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), then the electronic device 100 can generate an annotation based on the above-mentioned voice instruction. After the annotation is generated, refer to Fig.16F It can be seen that the electronic device 100 can not only display the generated annotations (for example, displaying the annotation 715 in the center of the text “Annual Work Summary”), but also highlight the interactive object corresponding to the annotation (for example, the text “Annual Work Summary”).

[0432] It should be noted that the above-mentioned interactive object can be the object at the location where the long press operation is performed, or it can be the object after the object at the location where the long press operation is performed is expanded, and this embodiment of the application does not limit this. Fig.16DFor example, if the object at the long press position is the word "year", the expanded object can be the word "annual work summary". Figure 16D-16F The illustrated embodiment uses the expanded object of the text "Annual Work Summary" as the interactive object.

[0433] In the embodiment of the present application, after the electronic device 100 starts the natural language interaction function in response to the user's long press operation, if the electronic device 100 does not detect the user inputting a voice command within a period of time, and detects that the long press operation is cancelled (for example, the user's finger or touch device leaves the screen of the electronic device 100), then, refer to Figure 16G The electronic device 100 can bring up a virtual keyboard and display a comment box 712 , and the user can input text instructions through the virtual keyboard to add comments in the comment box 712 .

[0434] In some embodiments, the generated annotations may be displayed in a pierceable floating layer above the application, so that the user's operation on the application page content is not affected.

[0435] In the embodiment of the present application, after adding an annotation to a document, when the page content of the document changes (for example, the page content changes due to page cutting / scrolling / moving / zooming in / reducing operations), the electronic device 100 determines that the page content is stably displayed (for example, after a preset time period of stopping page cutting / scrolling / moving / zooming in / reducing operations), if the currently displayed page content corresponds to the added annotation, then the electronic device 100 can further determine whether to display the annotation based on a preset rule. The following is a detailed description.

[0436] In the aforementioned process of adding annotations, the electronic device 100 can record the annotation screenshot image (the image can include the page content and the annotated image) and the coordinate information of the annotation display position. After the annotation is added and the page content changes, the electronic device 100 can obtain the historical annotation screenshot image and analyze the historical annotation screenshot image. If the currently displayed page content is similar to the page content in a historical annotation screenshot image, then the electronic device 100 can find the annotation corresponding to the page content, and can determine the coordinate information of the updated display position of the annotation based on the display position and display ratio information of the page content and its corresponding annotations in the historical annotation screenshot image, and the display position and display ratio information of the currently displayed page content. After that, the electronic device 100 can display the annotation at the corresponding position of the screen based on the coordinate information of the updated display position of the annotation; if the currently displayed page content is not similar to the page content in one or more historical annotation screenshot images, or the currently displayed page content is similar to the page content in multiple historical annotation screenshot images, then the electronic device 100 may not display the annotation.

[0437] It is easy to understand that whether the currently displayed page content is similar to the page content in the historical annotated screenshot image can be measured using statistical knowledge in mathematics. For example, the "confidence" indicator can be used to measure. If the confidence is higher than a preset threshold, it can be said that the currently displayed page content is similar to the page content in a historical annotated screenshot image. If the confidence is lower than a preset threshold, it can be said that the currently displayed page content is not similar to the page content in a historical annotated screenshot image.

[0438] For example, see Fig.16H , taking the currently displayed page content as “Summary” as an example, assuming that the electronic device 100 determines that the currently displayed page content is consistent with the page content in a certain historical annotation screenshot image (e.g. Fig. 16C The page content shown in the "Annual Work Summary" is similar to the above historical annotation screenshot image. Fig. 16C Taking the annotated screenshot image shown in as an example, the electronic device 100 can Fig.16H Display Fig. 16C As shown in the comment 713 "It would be nice to add a subtitle here".

[0439] It is easy to see that Fig.16H The page content shown can be Fig. 16C The enlarged page content (for example Fig.16H The page content in is only displayed in magnified form. Fig. 16C (see “Summary” in the “Annual Work Summary” shown in ).

[0440] For example, see Fig.16H, also taking the current displayed page content as "Summary" as an example, it is easy to understand that if the document contains Fig. 16C If the page contents of one or more other pages other than the page shown also include the content of "Summary" and are all added with corresponding annotations, then the electronic device 100 may determine that the page content currently displayed is similar to the page contents in the multiple historical annotated screenshot images, then the electronic device 100 may no longer display the content in the current page. Fig.16H Display comments in .

[0441] It is easy to understand that if the electronic device 100 determines Fig.16H If the page content currently displayed in the image is not similar to the page content in one or more historical annotation screenshot images, then the electronic device 100 may no longer display the page content in the image. Fig.16H Display comments in .

[0442] In the embodiment of the present application, in the above two cases where the annotation is not displayed, the electronic device 100 can also support the user to view the annotation. Fig.16H Option 716a is displayed in the exemplary window 716. After the electronic device 100 detects the user's operation (e.g., a click operation) on option 716a, it can display part or all of the annotations added to the document in a window or card, etc., for the user to view. In some embodiments, if the currently displayed page content has a corresponding annotation added, the user can also click on the annotation displayed in the window or card, etc., to trigger the electronic device 100 to Fig.16H Display the annotation as shown.

[0443] In the embodiment of the present application, after adding annotations to a document, the electronic device 100 can also support the user to perform editing operations on the annotations (such as deleting the annotations, moving the annotations, modifying the annotation content, etc.), which will be described in detail below.

[0444] In the embodiment of the present application, the electronic device 100 may support the user to trigger the electronic device 100 to delete or move an annotation through a single-click operation.

[0445] For example, see Fig.16I Taking the annotation 713 as an example, the electronic device 100 may detect a single click operation of the user on the annotation 713, and in response to the single click operation, refer to Fig.16J , the electronic device 100 may display option 717, which may be used to delete the annotation 713. In this case, the annotation 713 is in an editable state. If the user wants to delete the annotation 713, the electronic device 100 may detect a single click operation of the user on option 717, and in response to the single click operation, the electronic device 100 may delete and cancel the display of the annotation 713. Fig.16J, if the user wants to move the annotation 713, then the electronic device 100 can detect the operation that the user long - presses and drags the annotation 713 to a certain position. In response to this operation, the electronic device 100 can display the annotation 713 at this position and no longer display the annotation 713 at the position before the movement.

[0446] In some embodiments, the user can click on other positions outside the annotation 713 to trigger the electronic device 100 to cancel the display of the above - mentioned option 717. In this way, the annotation 713 can no longer be in the above - mentioned editable state.

[0447] It should be noted that if the annotation is displayed on a layer that can be penetrated above the application, in order to prevent the editing operation on the annotation from affecting the content of the application page below the layer, therefore, the above - mentioned click operation can be a certain specific click operation, such as a two - finger click operation, etc.

[0448] In the embodiments of the present application, the electronic device 100 can support the user to trigger the electronic device 100 to modify the annotation content or insert a new annotation by long - pressing and inputting natural language instructions (such as voice instructions, text instructions).

[0449] Exemplarily, refer to Figure 16K , Figure 16K In the currently displayed page of the document 711 shown, there is one annotation 713. After the electronic device 100 detects the long - press operation on the currently displayed page of the document 711, in response to this long - press operation, the electronic device 100 can activate the natural language interaction function and can receive the natural language instructions input by the user through the activated natural language interaction function. The electronic device 100 can also obtain the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713, and can determine whether the distance d between the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713 is less than or equal to a preset distance threshold d1. If so (i.e., d <= d1), then the electronic device 100 can replace the annotation content of the annotation 713 based on the above - input natural language instructions, thus completing the modification of the annotation content; if not (i.e., d > d1), then the electronic device 100 can further determine whether the distance d between the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713 is less than or equal to a preset distance threshold d2. If so (i.e., d1 < d <= d2), then the electronic device 100 can prompt the user whether to insert a new annotation or replace the annotation content of the annotation 713 based on the above - input natural language instructions in the form of voice broadcast or interface display, and can perform corresponding operations based on the user's selection; if not (i.e., d > d2), then the electronic device 100 can insert a new annotation based on the above - input natural language instructions.

[0450] In some embodiments, when the currently displayed page includes multiple annotations, the electronic device 100 may obtain the annotation closest to the location of the long-press operation for comparison.

[0451] The following describes the specific process of the electronic device 100 modifying the annotation content or inserting a new annotation with reference to the user interface by way of example.

[0452] Continue to refer to Figure 16K , assuming that the distance d between the coordinates of the location where the user's long-press operation is located and the coordinates of annotation 713 is less than or equal to a preset distance threshold d1 (i.e., d <= d1), then, referring to Figure 16L , the electronic device 100 may detect relevant voice commands input by the user (such as the voice "Add a cover"), and may display the detected relevant voice commands input by the user in text form (such as the text 718 "Add a cover"). Further, after the above voice command is input, if the electronic device 100 detects that the user cancels the long-press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then, the electronic device 100 may replace the annotation content of annotation 713 based on the above voice command. After the replacement is completed, referring to Figure 16M , the electronic device 100 may display the replaced annotation. It can be seen that the annotation content of annotation 713 is replaced from the original "Add a subtitle here" to "Add a cover". Thus, the modification of the annotation content is completed.

[0453] Continue to refer to Figure 16K , assuming that the distance d between the coordinates of the location where the user's long-press operation is located and the coordinates of annotation 713 is greater than the preset distance threshold d1 and less than or equal to the preset distance threshold d2 (i.e., d1 < d <= d2), then, referring to Fig.16N , the electronic device 100 may detect relevant voice commands input by the user (such as the voice "Add a cover"), and may display the detected relevant voice commands input by the user in text form (such as the text 718 "Add a cover"). Further, after the above voice command is input, if the electronic device 100 detects that the user cancels the long-press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then, the electronic device 100 may also display option 719 and option 720. Among them, option 719 may be used to prompt the user to insert a new annotation, and option 720 may be used to prompt the user to replace the annotation content of annotation 713. If the electronic device 100 detects that the user selects to insert a new annotation through a voice command or a touch operation (such as a click operation on option 719), then, referring to Fig.16O, the electronic device 100 can insert a new annotation (such as annotation 721) at the corresponding position (such as the position of the long press operation). If the electronic device 100 detects that the user chooses to replace the annotation through voice command or touch operation (such as a click operation on option 720), then, refer to Figure 16M , the electronic device 100 can replace the annotation content of the annotation 713 from the original "It is better to add a subtitle here" to "Add a cover" and display the replaced annotation.

[0454] In the above-mentioned annotation editing scenario, after the electronic device 100 starts the natural language interaction function in response to the user's long press operation, if the electronic device 100 does not detect the user inputting a voice command within a period of time, and detects that the user cancels the long press operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), then, similarly to the above-mentioned annotation adding scenario, Figure 16G In that case, the electronic device 100 can also call up a virtual keyboard to support the user to input text instructions through the virtual keyboard. After the input is completed, the electronic device 100 can complete the annotation editing based on the input text instructions. It is easy to understand that the completion of annotation editing based on the input text instructions is just a different way of inputting instructions. The other processes are the same as the above-mentioned process of completing annotation editing based on voice instructions, which will not be repeated here.

[0455] Fig.17 The specific process of an interaction method provided in an embodiment of the present application is exemplified.

[0456] like Fig.17 As shown, the method can be applied to the electronic device 100. The specific steps of the method are described in detail below:

[0457] S1701. The electronic device 100 detects a user operation (eg, a long press operation), and in response to the user operation, the electronic device 100 activates a natural language interaction function.

[0458] Among them, the above-mentioned user operation can be a touch operation of the user on the screen of the electronic device 100. In the embodiment of the present application, the touch operation can trigger the electronic device 100 to start the natural language interaction function and display the natural language interaction entrance, so that the electronic device 100 can subsequently detect the natural language instructions (such as voice instructions or text instructions) input by the user through the natural language interaction function, and can display the natural language instructions at the natural language interaction entrance. The touch operation can also trigger the electronic device 100 to select an interactive object, so that the corresponding operation can be performed on the interactive object based on the above-mentioned input natural language instructions.

[0459] In the embodiments of the present application, the above user operation is only a long press operation (such as a long press operation on a picture, video, text, chat item, document icon, document page, etc.) as an example, and should not constitute a limitation on the present application. In other embodiments of the present application, the above user operation may also be other operations.

[0460] Among them, the specific process of triggering the electronic device 100 to start the natural language interaction function and display the natural language interaction entrance and select the interaction object through the above-mentioned user operation (such as a long press operation) can refer to the relevant text description in the aforementioned user interface embodiment, which will not be repeated here.

[0461] S1702: The electronic device 100 detects that a user inputs a natural language instruction, and determines an interaction object based on the natural language instruction and the user operation.

[0462] Specifically, after starting the natural language interaction function, the electronic device 100 can use the natural language interaction function to detect the natural language instructions input by the user. After the natural language instructions are input, the electronic device 100 can determine whether there is a default object (that is, the natural language instruction does not include the interactive object) or a pronoun (that is, the natural language instruction includes words such as this, it, etc. used to refer to the interactive object) in the natural language instruction. If so (that is, the natural language instruction does not include keywords used to indicate the interactive object), the electronic device 100 can extract and identify the interactive object at the location of the above-mentioned user operation (such as a long press operation) or the interface content at and near the location, thereby determining the interactive object. If not (that is, the natural language instruction includes keywords used to indicate the interactive object), the electronic device 100 can execute the following step S1703.

[0463] In some embodiments, when there is no default object in the natural language instruction (ie, the natural language instruction includes an interactive object), the electronic device 100 may perform corresponding operations on the interactive object according to the interactive object included in the natural language instruction.

[0464] For example, assuming that the natural language instruction is “brighten the picture”, it can be seen that the interactive object included in the natural language instruction is “picture”, then the electronic device 100 can increase the brightness of the picture through a relevant algorithm.

[0465] In other embodiments, when the interactive object included in the natural language instruction is different from the interactive object selected by triggering the electronic device 100 through the above-mentioned user operation (such as a long press operation), the electronic device 100 may perform corresponding operations on the interactive object according to the interactive object included in the natural language instruction.

[0466] Exemplarily, assuming that the natural language instruction is "brighten the picture", it can be seen that the interactive object included in the natural language instruction is "picture", and the interactive object selected by the electronic device 100 triggered by the above user operation is the cutout subject in the picture (such as a flower). In this case, the electronic device 100 can use a relevant algorithm to increase the overall brightness of the picture, rather than just the brightness of the cutout subject in the picture. That is to say, the interactive object determined by the electronic device 100 is the above-mentioned entire picture, rather than the cutout subject in the picture.

[0467] In other embodiments, when there is a default object or pronoun in the natural language instruction, the electronic device 100 may prioritize the smallest unit object (such as a control, a cutout body, a text segmentation, etc.) that can be identified at the location of the above-mentioned user operation (such as a long press operation) as the interactive object, and then the electronic device 100 may determine whether to expand the scope of the interactive object based on the above-mentioned natural language instruction. If so, the scope of the interactive object may be expanded to determine a new interactive object, and the operation corresponding to the above-mentioned natural language instruction may be performed on the new interactive object.

[0468] Exemplarily, assuming that the natural language instruction is "erase", it can be seen that there is a default object in the natural language instruction. Then, the electronic device 100 can use the smallest unit object that can be identified at the location where the above user operation is located as the interactive object, such as using the cutout subject in a certain picture as the interactive object. Further, the electronic device 100 can erase the cutout subject in the picture through a relevant algorithm.

[0469] Exemplarily, assuming that the natural language instruction is "erase this", it can be seen that the natural language instruction includes the pronoun "this", then the electronic device 100 can also use the smallest unit object that can be identified at the location of the above user operation as an interactive object, such as using the cutout subject in a certain picture as an interactive object. Further, the electronic device 100 can erase the cutout subject in the picture through a relevant algorithm.

[0470] Exemplarily, assuming that the natural language instruction is "copy this paragraph to the memo", it can be seen that the natural language instruction includes the pronoun "this paragraph". Assuming that the interactive object selected by the electronic device 100 triggered by the above user operation is a part of the text in a text paragraph (that is, the selected text), then the electronic device 100 can determine the scope of the interactive object to be expanded based on the "this paragraph" in the above natural language instruction. Further, the electronic device 100 can identify the context of the above selected text through relevant algorithms (such as text detection and recognition algorithms such as OCR), thereby obtaining the entire text paragraph where the above selected text is located. Further, the electronic device 100 can determine that the new interactive object is the entire text paragraph where the above selected text is located, and can copy the entire text paragraph where the above selected text is located to the memo.

[0471] S1703. The electronic device 100 responds to the above-mentioned natural language instruction based on the above-mentioned interactive object.

[0472] Specifically, after determining the interactive object, the electronic device 100 can analyze and understand the natural language instruction input by the user in combination with the interactive object. After the analysis and understanding, the electronic device 100 can respond to the natural language instruction, that is, perform the operation corresponding to the natural language instruction for the interactive object. For example, perform corresponding operations on display information such as pictures and videos (such as sending, editing, etc.); for another example, perform corresponding operations on information displayed in a list form (such as chat messages, etc.) (such as displaying unread message summaries, question and answer queries, etc.), and perform corresponding operations on information displayed in a grid form (such as documents, etc.) (such as displaying document summaries, displaying notes in documents, full text translation, question and answer queries, format conversion, format beautification, etc.); for another example, perform corresponding operations on text (such as ordinary text, or special text such as links, names, numbers, email addresses, dates, addresses, etc.) (such as query, collection, editing, displaying summary, etc.); for another example, add annotations to documents, display annotations, edit annotations, etc.; and so on.

[0473] Among them, the specific process of the electronic device 100 responding to the natural language instructions input by the user can refer to the relevant text description in the aforementioned user interface embodiment, which will not be repeated here.

[0474] The following introduces a software structure of an electronic device 100 provided in an embodiment of the present application.

[0475] Fig.18 The software structure of an electronic device 100 provided in an embodiment of the present application is exemplified.

[0476] like Fig.18As shown, the software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0477] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library, and the kernel layer.

[0478] The application layer can include a series of application packages.

[0479] like Fig.18 As shown, the application package may include camera, gallery, calendar, call, map, voice assistant, WLAN, Bluetooth, music, video, short message, input method (not shown in the figure), memo (not shown in the figure), mailbox (not shown in the figure), Changlian (not shown in the figure, i.e. a social application) and other applications.

[0480] Among them, the voice assistant can be an application with voice / text command recognition function. For example, after detecting the user's voice input, voice recognition technology, natural language processing technology, etc. can be used to identify the user's intention and respond in a targeted manner (such as voice answering, starting an application, changing device configuration, etc.). Among them, the electronic device 100 can always turn on the voice assistant, or it can start the voice assistant after detecting user operation (such as long pressing an interface element / button operation, etc.), or it can start the voice assistant after detecting a wake-up word, which is not limited here.

[0481] The above-mentioned voice / text instructions may refer to voice / text for controlling the electronic device 100 to perform one or more operations.

[0482] It should be noted that the name of the voice assistant application is merely a term used in the embodiments of the present application. Its meaning has been recorded in the embodiments of the present application, and its name does not constitute any limitation to the embodiments of the present application.

[0483] In the embodiment of the present application, the electronic device 100 can be triggered to start the natural language interaction function by the user's touch operation on the screen of the electronic device 100 (for example, a long press operation on a picture, video, text, chat item, document icon, document page, etc. displayed on the screen of the electronic device 100), so that the electronic device 100 can subsequently detect the natural language instruction (such as voice instruction or text instruction) input by the user through the natural language interaction function. In addition, the above touch operation can also trigger the electronic device 100 to select an interactive object, so that the corresponding operation can be performed on the interactive object based on the natural language instruction input by the user. For details, please refer to the relevant text descriptions in the above embodiments, which will not be repeated here.

[0484] In an embodiment of the present application, the natural language interaction function initiated by the electronic device 100 may be a system capability provided by the electronic device 100. In some embodiments, the system capability may be implemented based on a voice assistant application or an input method application.

[0485] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0486] like Fig.18 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0487] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0488] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0489] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0490] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0491] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0492] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0493] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0494] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0495] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0496] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0497] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0498] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0499] A 2D graphics engine is a drawing engine for 2D drawings.

[0500] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, Bluetooth driver, and sensor driver.

[0501] Apart from Fig.18 In addition to the examples shown, the solution provided in this application can also be applied to other types of operating systems, such as Hongmeng operating system, Apple operating system, Windows operating system, etc.

[0502] The following is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0503] Fig.19 The structure of an electronic device 100 provided in an embodiment of the present application is exemplarily shown.

[0504] like Fig.19 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0505] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0506] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0507] In the embodiment of the present application, the processor 110 can be used to respond to user operations (such as long press operations) to select interactive objects, start natural language interaction functions, and can also be used to analyze, understand, and execute natural language instructions input by users, etc. For details, please refer to the relevant text descriptions in the above embodiments, which will not be repeated here.

[0508] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0509] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0510] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0511] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0512] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0513] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0514] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0515] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0516] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0517] The USB interface 130 is an interface that complies with the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0518] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0519] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.

[0520] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0521] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0522] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0523] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0524] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0525] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0526] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0527] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0528] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0529] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0530] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, etc. of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0531] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0532] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0533] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0534] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0535] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0536] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0537] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0538] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0539] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0540] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0541] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0542] In the embodiment of the present application, the electronic device 100 can convert the sound signal into an electrical signal through the microphone 170C. For example, when the electronic device 100 receives a voice command input by the user, the user can make a sound by approaching the microphone 170C with his mouth so that the sound is input into the microphone 170C.

[0543] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0544] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0545] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0546] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0547] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0548] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0549] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0550] It should be understood that Fig.19 The electronic device 100 shown is only an example, and the electronic device 100 may have more Fig.19 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Fig.19 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0551] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0552] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0553] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interaction method, applied to an electronic device, wherein the electronic device includes a screen, characterized in that: The method comprises: The electronic device displays a first user interface of a first application; The electronic device detects a first operation of the user on the first user interface, where the position of the first operation on the screen is a first position; In response to the first operation, the electronic device starts a natural language interaction function, where the natural language interaction function is used for the electronic device to interact with the user through natural language instructions; The electronic device detects that a user inputs a first natural language instruction, and determines, based on the first natural language instruction and the first position, that the natural language interaction object is a first interaction object, wherein the first interaction object is included in the first user interface; The electronic device executes an operation corresponding to the first natural language instruction on the first interactive object.

2. The method according to claim 1, characterized in that: The first operation is a long press operation.

3. The method according to claim 1 or 2, characterized in that: The electronic device determines, based on the first natural language instruction and the first position, that the natural language interaction object is a first interaction object, specifically including: In a case where the first natural language instruction does not include a first keyword, the first keyword is a keyword used to indicate the natural language interaction object, and the electronic device determines that the first interaction object is an object that can be identified at the first position on the first user interface; or, In a case where the first natural language instruction includes the first keyword, the electronic device determines that the natural language interaction object indicated by the first keyword is the first interaction object.

4. The method according to any one of claims 1 to 3, characterized in that: After the electronic device starts the natural language interaction function, the method further includes: The electronic device displays a natural language interaction portal, and the natural language interaction portal is used to display the first natural language instruction.

5. The method according to claim 4, characterized in that The natural language interaction entrance is located at the first position or near the first position.

6. The method according to any one of claims 1 to 5, characterized in that: After the electronic device starts the natural language interaction function, the method further includes: If the electronic device does not detect that the user inputs a natural language instruction within a first preset time period, the electronic device outputs a first prompt, where the first prompt is used to prompt the user to input a recognizable natural language instruction corresponding to the first interactive object.

7. The method according to any one of claims 4 to 6, characterized in that: The first natural language instruction includes any one of a first voice instruction or a first text instruction, and the method further includes: The electronic device detects a second operation of the user on the natural language interaction portal; The electronic device detects that a user inputs a first natural language instruction, specifically comprising: In response to the second operation, the electronic device detects the first voice instruction input by the user through the microphone, or the electronic device calls up and displays a virtual keyboard and detects the first text instruction input by the user through the virtual keyboard.

8. The method according to any one of claims 1 to 6, characterized in that: After the electronic device detects that the user inputs a first natural language instruction, the method further includes: If the electronic device detects that the user cancels the first operation, the electronic device no longer detects the natural language instructions input by the user.

9. The method according to any one of claims 4 to 8, characterized in that: The method further comprises: When the electronic device detects that the user long presses and drags the first interactive object, the electronic device reduces the display of the natural language interaction entrance, or cancels the display of the natural language interaction entrance; When the electronic device detects that the first interactive object does not move again within a second preset time period, the electronic device resumes displaying the natural language interaction entrance.

10. The method according to any one of claims 1 to 9, characterized in that: The first natural language instruction does not include keywords that directly indicate the location or name of the interactive object.

11. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first picture or a first content in the first picture, the first content is a portion of the content in the first picture, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to send the first interactive object to a second application, the electronic device sends the first interactive object from the first application to the second application; or, When the first natural language instruction is used to indicate an erasing intention, the electronic device erases the first content from the first picture; or, In a case where the first natural language instruction is used to indicate an intention to replace the first content with a second content, the electronic device replaces the first content with the second content; or, When the first natural language instruction is used to indicate an intention to insert a third content, the electronic device inserts the third content into the first picture; or, In a case where the first natural language instruction is used to indicate a recognition intention, the electronic device recognizes the first interactive object and displays a recognition result; or, In the case where the first natural language instruction is used to indicate a beautification intention, the electronic device beautifies the first interactive object, wherein the beautification intention includes one or more of the following intentions: beautification, body beautification, blurring, mosaic, graffiti, and adding a frame / sticker / marker; or, In a case where the first natural language instruction is used to indicate an intention to adjust parameters, the electronic device performs parameter adjustment on the first interactive object, wherein the parameters include one or more of the following: filter style, brightness, saturation, and sharpness.

12. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first video, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate a sending intention, the electronic device sends the first video from the first application to a second application; or, In case the first natural language instruction is used to indicate a cropping intention, the electronic device crops the first video and saves the cropped first video segment or first video frame.

13. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is the first chat item, the first chat item includes a chat message, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to display a summary of an unread message included in the chat message, the electronic device searches for the unread message, and generates and displays a summary of the unread message based on the unread message; or, In a case where the first natural language instruction is used to indicate an intention to query relevant information of the chat message, the electronic device queries and displays the relevant information, wherein the relevant information includes any one of the following: content of the chat message, sender of the chat message.

14. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is an icon of a first document, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to display a content summary of the first document, the electronic device obtains the content of the first document, and generates and displays a summary of the content based on the content; or, In the case where the first natural language instruction is used to indicate an intention to display the notes in the first document, the electronic device queries and displays the notes in the first document, wherein the notes include one or more of the following: marks, annotations; or, In a case where the first natural language instruction is used to indicate an intention to translate the content of the first document, the electronic device translates the content of the first document and displays a translation result; or, In a case where the first natural language instruction is used to indicate an intention to perform format conversion / beautify the first document, the electronic device performs format conversion / beautify the first document, generates and displays the first document after format conversion / beautification.

15. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to collect the first text, the electronic device collects the first text; or, In a case where the first natural language instruction is used to indicate an intention to query the first text, the electronic device queries the first text and displays a query result, wherein the query result includes relevant information of the first text.

16. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a second text, the first natural language instruction includes a third text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: The electronic device combines the second text and the third text into one or more fourth texts; The electronic device replaces the second text with the fourth text and displays the fourth text.

17. The method according to claim 16, characterized in that In the case that there are multiple fourth texts, before the electronic device replaces the second text with the fourth text, the method further includes: The electronic device outputs a second prompt, where the second prompt is used to prompt the user to select any one of the multiple fourth texts.

18. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a fifth text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to modify a text format of the fifth text, the electronic device modifies the text format of the fifth text; The text format includes one or more of the following: font, font size, color, alignment, title style, body style, bold, italic, and underline.

19. An interactive method, applied to an electronic device, characterized in that: The method comprises: The electronic device displays a first user interface, wherein the first user interface includes a first page of a first document; The electronic device detects a first operation of a user on the first page; In response to the first operation, the electronic device starts a natural language interaction function, where the natural language interaction function is used for the electronic device to interact with the user through natural language instructions; The electronic device detects that a user inputs a first natural language instruction; The electronic device generates a first batch of annotations based on the first natural language instruction, and displays the first batch of annotations on the first user interface.

20. The method according to claim 19, characterized in that The first operation is a long press operation.

21. The method according to claim 19 or 20, characterized in that The position of the first operation on the screen of the electronic device is a first position, and the method further includes: The electronic device identifies whether there is an interactive object at the first position. If not, the electronic device displays the first batch of annotations at the first position. If so, the electronic device displays the first batch of annotations at a second position, which is the same as or different from the first position.

22. The method according to claim 21, characterized in that The second position is a center position of the interactive object.

23. The method according to claim 21 or 22, characterized in that The method further comprises: The electronic device adds a mark to the interactive object and displays the interactive object after the mark is added, wherein the mark includes: a highlight mark.

24. The method according to any one of claims 19 to 23, characterized in that: The method further comprises: The electronic device detects a second operation of the user on the first page; In response to the second operation, the electronic device starts the natural language interaction function and determines the distance between the location of the second operation and the location of the first batch of annotations; The electronic device detects that a user inputs a second natural language instruction, and generates a second annotation based on the second natural language instruction; When the distance is less than or equal to a first preset distance threshold, the electronic device replaces the first annotation with the second annotation and displays the second annotation on the first user interface; or, When the distance is greater than a second preset distance threshold, the electronic device adds and displays the second annotation on the first user interface; or, When the distance is greater than the first preset distance threshold and less than or equal to the second preset distance threshold, the electronic device replaces the first annotation with the second annotation based on the user's selection operation and displays the second annotation on the first user interface, or adds and displays the second annotation on the first user interface.

25. The method according to any one of claims 19 to 24, characterized in that: The method further comprises: The electronic device detects a third operation of the user on the first batch of bets; In response to the third operation, the electronic device displays a first option, where the first option is used to delete the first batch of bets; The electronic device detects a fourth operation of the user on the first option, and in response to the fourth operation, the electronic device deletes and cancels display of the first batch of annotations; or, The electronic device detects a fifth operation of a user long pressing and dragging the first batch of annotations to a third position, and in response to the fifth operation, the electronic device displays the first batch of annotations at the third position.

26. The method according to claim 24 or 25, characterized in that The second operation is a long press operation, and the third operation is a single click operation.

27. The method according to any one of claims 19 to 26, characterized in that: The method further comprises: The electronic device detects a sixth operation of the user, where the sixth operation includes any one of the following: switching pages, scrolling pages, zooming in on a page, and zooming out on a page; In response to the sixth operation, the electronic device displays first content on the first user interface, where the first content is content in the first document, and displays annotations related to the first content.

28. The method according to any one of claims 19 to 27, characterized in that: The first natural language instruction includes any one of a first voice instruction or a first text instruction, and the electronic device detects that a user inputs the first natural language instruction, specifically including: The electronic device detects the first voice instruction input by the user through a microphone; or, The electronic device calls up and displays a virtual keyboard, and detects the first text instruction input by a user through the virtual keyboard.

29. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in any one of claims 1-18 or 19-28.

30. A computer storage medium, characterized in that The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 18 or 19 to 28.

Images