Interaction method and electronic device

Abstract

The present application discloses an interaction method and an electronic device. The electronic device can support a user to enable a natural language interaction function by means of a long-press operation and can display a natural language interaction entry; upon detecting a voice / text instruction inputted by a user through a natural language, the electronic device can display the instruction at the natural language interaction entry; once the input of the instruction is completed, the electronic device can determine whether the instruction has a default object or a demonstrative pronoun; and if yes, the electronic device can perform interaction object extraction and identification on interface content at the position where the user long-press operation is performed or at and near the position where the user long-press operation is performed, so as to determine an interaction object, and substitute the interaction object into the inputted instruction for understanding and executing. In this way, by means of fusion of a touch operation and natural language interaction, the selection and further operation for an interaction object can be efficiently realized, and the problem of inconvenience in expressing the interaction object by using a natural language is solved, thereby improving the human-computer interaction efficiency and user experience.

Description

Interactive method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 14, 2023, with application number 202311028609.5 and application name “Interactive Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an interaction method and electronic device. Background Art

[0003] With the development of electronic devices, more and more electronic devices can provide natural language interaction functions, such as voice interaction functions. The voice interaction function can support electronic devices to receive user voice commands and understand and execute the voice commands.

[0004] However, in some scenarios where an interactive object needs to be processed (for example, editing a flower in a picture containing multiple flowers), users are sometimes unable to accurately express the interactive object through voice alone, and the electronic device is unable to accurately and quickly recognize the user's intention. Multiple rounds of voice interaction may be required to complete the task, resulting in a poor user experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an interaction method and an electronic device, which can efficiently realize the selection and further operation of interactive objects through the integration of user touch operations and natural language interactions, solve the problem of inconvenience in expressing interactive objects in natural language, and improve human-computer interaction efficiency and user experience.

[0007] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to an electronic device, the electronic device including a screen, the method including: the electronic device displays a first user interface of a first application; the electronic device detects a first operation of a user on the first user interface, and the position of the first operation on the screen is a first position; in response to the first operation, the electronic device turns on a natural language interaction function, and the natural language interaction function is used for interaction between the electronic device and the user through natural language instructions; the electronic device detects that the user inputs a first natural language instruction, and determines that the natural language interaction object is a first interaction object based on the first natural language instruction and the first position, and the first interaction object is included in the first user interface; the electronic device executes the operation corresponding to the first natural language instruction on the first interaction object.

[0008] The first application may be any application installed on the electronic device. It is worth noting that system applications such as the desktop are also applications. The first user interface may be any user interface provided by the first application. The first operation may be, for example, a single-finger or multi-finger long press operation (such as a long press operation performed by a user's finger or a touch device), a click operation, etc. The first position may be any position on the screen. The first natural language instruction may be a natural language instruction input by the user through a natural language interaction function. The first interactive object may be an object for natural language interaction, such as a picture, video, text, file, application or application icon, card, or other object.

[0009] By implementing the method provided in the first aspect, the electronic device can support the user to activate the natural language interaction function through a first operation. After detecting the voice / text instruction input by the user through natural language, the electronic device can extract and identify the interactive object from the interface content at the location of the user's first operation or at the location of the user's first operation and its vicinity, thereby determining the interactive object, and substitute the interactive object into the input instruction for understanding and execution. In this way, through the integration of touch operation and natural language interaction, the selection and further operation of the interactive object can be efficiently realized, solving the problem of inconvenience in expressing the interactive object in natural language, and improving the efficiency of human-computer interaction and user experience.

[0010] In a possible implementation, the first operation is a long press operation.

[0011] In this way, the user can trigger the electronic device to start the natural language interaction function through a long press operation, and can trigger the electronic device to select an interaction object, thereby solving the problem of inconvenience in expressing the interaction object in natural language.

[0012] In one possible implementation, the electronic device determines that the natural language interaction object is the first interaction object based on the first natural language instruction and the first position, specifically including: when the first natural language instruction does not include the first keyword, the first keyword is a keyword used to indicate the natural language interaction object, and the electronic device determines that the first interaction object is an object that can be identified at the first position on the first user interface; or, when the first natural language instruction includes the first keyword, the electronic device determines that the natural language interaction object indicated by the first keyword is the first interaction object.

[0013] The first keyword may be a word used to specify an interactive object (such as "this", "it", "that", etc.), or a word used to indicate the name of the interactive object.

[0014] That is to say, after the first natural language instruction is input, the electronic device can determine whether there is a default object in the first natural language instruction (that is, the first natural language instruction does not include words used to indicate the name of the interactive object) or a reference word (that is, the first natural language instruction includes words such as this, that, it, etc. used to refer to the interactive object). If so, the electronic device can determine the interactive object based on the first position. If not, the electronic device can determine that the natural language interactive object indicated by the first keyword is the first interactive object, and can perform corresponding operations based on the first natural language instruction.

[0015] In a possible implementation, after the natural language interaction function is enabled on the electronic device, the method further includes: the electronic device displays a natural language interaction entrance, where the natural language interaction entrance is used to display a first natural language instruction.

[0016] In this way, the electronic device can prompt the user that the natural language interaction function has been turned on by displaying the natural language interaction entrance, and after the user enters a natural language instruction, the electronic device can display the instruction on the natural language interaction entrance so that the user can see the instruction he or she entered in time, thereby improving the user experience.

[0017] In a possible implementation, the natural language interaction entrance is located at the first position or near the first position.

[0018] In this way, the natural language interaction entrance is placed wherever the first position is, which can provide users with a better visual experience, facilitate users to find and use the natural language interaction entrance, and thus facilitate interaction between electronic devices and users through natural language commands.

[0019] In one possible implementation, after the electronic device turns on the natural language interaction function, the method also includes: if the electronic device does not detect the user inputting a natural language instruction within a first preset time period, the electronic device outputs a first prompt, and the first prompt is used to prompt the user to input a recognizable natural language instruction corresponding to the first interactive object.

[0020] The first preset time period may be a preset time period (eg, 2 seconds, 3 seconds, etc.) after the electronic device begins to detect a natural language instruction input by the user.

[0021] In this way, when the user does not know how to perform natural language interaction, the electronic device can output corresponding prompts (such as voice prompts or text prompts) so that the user can quickly know how to perform natural language interaction, thereby improving interaction efficiency and user experience.

[0022] In one possible implementation, the first natural language instruction includes any one of a first voice instruction or a first text instruction, and the method also includes: the electronic device detects a second operation of the user on the natural language interaction entrance; the electronic device detects the user inputting the first natural language instruction, specifically including: in response to the second operation, the electronic device detects the first voice instruction input by the user through the microphone, or the electronic device calls up and displays a virtual keyboard and detects the first text instruction input by the user through the virtual keyboard.

[0023] The second operation may be, for example, a click operation, the first language instruction may be a voice instruction input by the user through a microphone, the first text instruction may be a text instruction input by the user through a virtual keyboard, and the second operation may be, for example, a click operation, etc.

[0024] In this way, during the natural language interaction between the electronic device and the user, the electronic device can not only support the user's voice input, but also support the user's text input. The interaction method is more flexible and can be suitable for users with different input habits.

[0025] In a possible implementation, after the electronic device detects that the user inputs the first natural language instruction, the method further includes: if the electronic device detects that the user cancels the first operation, the electronic device no longer detects the natural language instruction input by the user.

[0026] In this way, after the user cancels the first operation, the electronic device can determine that the user has completed inputting the natural language instruction, and thus can execute the next operation.

[0027] In one possible implementation, the method also includes: when the electronic device detects that the user long presses and drags the first interactive object, the electronic device shrinks the display of the natural language interaction entrance, or cancels the display of the natural language interaction entrance; when the electronic device detects that the first interactive object does not move again within a second preset time period, the electronic device resumes displaying the natural language interaction entrance.

[0028] The second preset time period may be a period of time (eg, 2 seconds, 3 seconds, etc.) after the user stops dragging the first interactive object.

[0029] In this way, when the user long presses and drags the first interactive object, it may indicate that the user currently has no intention of conducting natural language interaction. Then, the electronic device can reduce the display of or cancel the display of the natural language interaction entrance to reduce the obstruction of the interface content by the display of the natural language interaction entrance. After the user stops dragging the first interactive object, it may indicate that the user currently has the intention of conducting natural language interaction. Then, the electronic device can restore the display of the natural language interaction entrance so as to make timely preparations for natural language interaction.

[0030] In one possible implementation, the first natural language instruction does not include keywords that directly indicate the name of the interactive object. Since the object that currently needs to be interacted with has been determined through the first operation, or the first operation and other recognition operations, the user does not need to overly consider how to clearly describe the location or name of the object when entering the natural language instruction so that the electronic device can accurately identify it. It only needs to verbally use some words that indirectly refer to the object, such as "this" and "that", or even directly describe the instruction to be operated without indicating the object. In this way, users are prevented from spending too much time describing an object that is difficult to describe, which improves the convenience of user natural language instruction interaction and helps to improve the user's use of natural language instructions.

[0031] In one possible implementation, the first interactive object is a first picture or the first content in the first picture, the first content is a part of the content in the first picture, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: when the first natural language instruction is used to indicate the intention to send the first interactive object to the second application, the electronic device sends the first interactive object from the first application to the second application; or, when the first natural language instruction is used to indicate the intention to erase, the electronic device erases the first content from the first picture; or, when the first natural language instruction is used to indicate the intention to replace the first content with the second content, the electronic device replaces the first content with the second content; or, when the first natural language instruction is used to indicate the intention to replace the first content with the second content When the natural language instruction is used to indicate the intention to insert third content, the electronic device inserts the third content into the first picture; or, when the first natural language instruction is used to indicate the intention to identify, the electronic device identifies the first interactive object and displays the identification result; or, when the first natural language instruction is used to indicate the intention to beautify, the electronic device beautifies the first interactive object, wherein the beautification intention includes one or more of the following intentions: beautification, body beautification, blurring, mosaic, graffiti, adding borders / stickers / marks; or, when the first natural language instruction is used to indicate the intention to adjust parameters, the electronic device adjusts the parameters of the first interactive object, wherein the parameters include one or more of the following: filter style, brightness, saturation, and sharpness.

[0032] Among them, the first picture can be a picture in the first application, the first content can be a part of the content in the first picture (such as the cutout body), the second application can be an application different from the first application, the second content can be content searched by the electronic device to replace the first content, and the third content can be content searched by the electronic device to be inserted into the first picture.

[0033] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on pictures (such as sending, erasing, replacing, inserting, identifying, beautifying, style transfer, parameter adjustment, etc.).

[0034] In one possible implementation, the first interactive object is a first video, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate a sending intention, the electronic device sends the first video from the first application to the second application; or, when the first natural language instruction is used to indicate a cropping intention, the electronic device crops the first video and saves the cropped first video clip or first video screen.

[0035] The first video may be a video in a first application, and the second application may be an application different from the first application.

[0036] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on videos (such as sending and cropping video content).

[0037] In one possible implementation, the first interactive object is a first chat entry, which includes a chat message. The electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to display a summary of unread messages included in the chat message, the electronic device searches for the unread message, and generates and displays a summary of the unread message based on the unread message; or, when the first natural language instruction is used to indicate an intention to query relevant information of the chat message, the electronic device queries and displays relevant information, wherein the relevant information includes any one of the following: the content of the chat message, the sender of the chat message.

[0038] The first chat entry may be a chat entry in a first application (eg, a social application).

[0039] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on chat messages (such as displaying unread message summaries, question and answer queries, etc.).

[0040] In one possible implementation, the first interactive object is an icon of the first document, and the electronic device executes an operation corresponding to the first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate the intention of displaying a content summary of the first document, the electronic device obtains the content of the first document, and generates and displays a summary of the content based on the content; or, when the first natural language instruction is used to indicate the intention of displaying notes in the first document, the electronic device queries and displays notes in the first document, wherein the notes include one or more of the following: marks, annotations; or, when the first natural language instruction is used to indicate the intention of translating the content of the first document, the electronic device translates the content of the first document and displays the translation result; or, when the first natural language instruction is used to indicate the intention of format conversion / beautification of the first document, the electronic device performs format conversion / beautification on the first document, generates and displays the first document after format conversion / beautification.

[0041] The first document may be a document on an electronic device.

[0042] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on documents (such as displaying document summaries, displaying notes in documents, translating document content, converting / beautifying document formats, etc.).

[0043] In one possible implementation, the first interactive object is a first text, and the electronic device executes an operation corresponding to a first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to collect the first text, the electronic device collects the first text; or, when the first natural language instruction is used to indicate an intention to query the first text, the electronic device queries the first text and displays the query results, which include relevant information about the first text.

[0044] The first text may be text in a first application.

[0045] In this way, through natural language interaction, electronic devices can easily and quickly perform various operations on text (such as text collection, text query, etc.).

[0046] In one possible implementation, the first interactive object is the second text, the first natural language instruction includes the third text, and the electronic device performs the operation corresponding to the first natural language instruction for the first interactive object, specifically including: the electronic device combines the second text and the third text into one or more fourth texts; the electronic device replaces the second text with the fourth text, and displays the fourth text.

[0047] The second text may be text to be modified in the first application, the third text may be text input by a user through natural language, and the fourth text is a semantically coherent text generated by combining the second text and the third text.

[0048] In this way, through natural language interaction, electronic devices can easily and quickly perform content modification operations on text.

[0049] In a possible implementation, when there are multiple fourth texts, before the electronic device replaces the second text with the fourth text, the method further includes: the electronic device outputting a second prompt, where the second prompt is used to prompt the user to select any one of the multiple fourth texts.

[0050] The second prompt may be a prompt output by the electronic device in the form of voice or text.

[0051] In this way, the user can conveniently select a newly generated text (ie, the fourth text) in time to modify the content of the original text (ie, the first text).

[0052] In one possible implementation, the first interactive object is a fifth text, and the electronic device executes an operation corresponding to the first natural language instruction for the first interactive object, specifically including: when the first natural language instruction is used to indicate an intention to modify the text format of the fifth text, the electronic device modifies the text format of the fifth text; wherein the text format includes one or more of the following: font, font size, color, alignment, title style, body style, bold, italic, and underline.

[0053] The fifth text may be text to be modified in the first application.

[0054] In this way, through natural language interaction, electronic devices can easily and quickly perform format modification operations on text.

[0055] In a second aspect, an embodiment of the present application provides an interaction method, applied to an electronic device, the method comprising: the electronic device displays a first user interface, the first user interface including a first page of a first document; the electronic device detects a first operation of the user on the first page; in response to the first operation, the electronic device turns on a natural language interaction function, the natural language interaction function being used for interaction between the electronic device and the user through natural language instructions; the electronic device detects that the user inputs a first natural language instruction; the electronic device generates a first batch of annotations based on the first natural language instruction, and displays the first batch of annotations on the first user interface.

[0056] Among them, the first user interface can be a user interface after the electronic device enters the annotation mode, the first document can be a document on the electronic device, the first page can be a page in the first document, the first operation can be, for example, a long press operation, etc., the first natural language instruction can be a natural language instruction input by the user through the natural language interaction function, and the first annotation can be an annotation generated by the electronic device based on the first natural language instruction input by the user.

[0057] By implementing the method provided in the second aspect, when an electronic device opens a document and enters the annotation mode, the electronic device can support the user to trigger the electronic device to start the natural language interaction function through, for example, a long press operation (for example, a long press operation performed by the user's finger or touch device), and can add annotations to the document based on the natural language interaction function, which is convenient and fast, simplifies user operations, and improves user experience.

[0058] In a possible implementation, the first operation is a long press operation.

[0059] In this way, users can trigger the electronic device to start the natural language interaction function through a long press operation, which is convenient and quick.

[0060] In one possible implementation, the position of the first operation on the screen of the electronic device is the first position, and the method also includes: the electronic device identifies whether there is an interactive object at the first position, and if not, the electronic device displays the first batch of annotations at the first position; if so, the electronic device displays the first batch of annotations at a second position, and the second position is the same as or different from the first position.

[0061] The first position may be any position on the screen, and the second position may be a position on the screen that is the same as or different from the first position.

[0062] In an embodiment of the present application, when there is no interactive object at the first position, the electronic device may display the annotation at the first position; when there is an interactive object at the first position, the electronic device may still display the annotation at the first position, or may also display the annotation at a position different from the first position.

[0063] In a possible implementation, the second position is a center position of the interactive object.

[0064] In this way, when there is an interactive object at the first position, the electronic device can display the annotation at the center of the position where the interactive object is located, which is more aesthetically pleasing.

[0065] In a possible implementation, the method further includes: the electronic device adding a mark to the interactive object, and displaying the marked interactive object, wherein the mark includes a highlight mark.

[0066] In this way, when there is an interactive object at the first position, the electronic device can add a mark (such as a highlight mark) to the interactive object to highlight the interactive object generating the annotation for the user to view. The mark style added to the interactive object is not limited.

[0067] In one possible implementation, the method also includes: the electronic device detects a second operation of the user on the first page; in response to the second operation, the electronic device turns on the natural language interaction function, and determines the distance between the location of the second operation and the location of the first annotation; the electronic device detects that the user inputs a second natural language instruction, and generates a second annotation based on the second natural language instruction; when the distance is less than or equal to a first preset distance threshold, the electronic device replaces the first annotation with a second annotation, and displays the second annotation on the first user interface; or, when the distance is greater than a second preset distance threshold, the electronic device adds and displays the second annotation on the first user interface; or, when the distance is greater than the first preset distance threshold and less than or equal to the second preset distance threshold, the electronic device replaces the first annotation with the second annotation based on the user's selection operation and displays the second annotation on the first user interface, or, adds and displays the second annotation on the first user interface.

[0068] Among them, the second operation can be used to edit the annotation (such as a long press operation, etc.), the second natural language instruction can be a natural language instruction input by the user through the natural language interaction function, the second annotation can be an annotation generated based on the second natural language instruction input by the user, the first preset distance threshold can be a distance threshold between the location of the second operation preset by the electronic device and the location of the first batch of annotations, and the second preset distance threshold can be another distance threshold between the location of the second operation preset by the electronic device and the location of the first batch of annotations.

[0069] In this way, under different preset distance thresholds, the electronic device can determine whether to directly replace the first annotation with the second annotation, or whether to directly insert the second annotation into the first document, or whether to determine based on the user's choice whether to replace the first annotation with the second annotation or insert the second annotation into the first document.

[0070] In one possible implementation, the method also includes: the electronic device detects a third operation of the user on the first batch of annotations; in response to the third operation, the electronic device displays a first option, and the first option is used to delete the first batch of annotations; the electronic device detects a fourth operation of the user on the first option, and in response to the fourth operation, the electronic device deletes and cancels the display of the first batch of annotations; or, the electronic device detects a fifth operation of the user long pressing and dragging the first batch of annotations to a third position, and in response to the fifth operation, the electronic device displays the first batch of annotations at the third position.

[0071] Among them, the third operation can be an operation for editing the annotation (such as a click operation, etc.), the first option can be an option for deleting the first batch of annotations, the fourth operation can be, for example, an operation in which the user clicks on the first option, the fifth operation can be, for example, an operation in which the user long presses and drags the annotation, and the third position can be a position on the screen that is different from the first position and the second position.

[0072] In this way, the electronic device can support users to perform editing operations such as deletion and movement on the generated annotations, which is convenient and fast.

[0073] In a possible implementation, the second operation is a long press operation, and the third operation is a single click operation.

[0074] In one possible implementation, the method also includes: the electronic device detects a sixth operation of the user, the sixth operation including any one of the following: switching pages, scrolling pages, zooming in on pages, and zooming out on pages; in response to the sixth operation, the electronic device displays first content on the first user interface, the first content is the content in the first document, and displays annotations related to the first content.

[0075] The sixth operation may be an operation for switching, scrolling, zooming in, zooming out, etc. of a page.

[0076] In this way, when the page content in the first document changes, if the current content corresponds to an annotation, the electronic device can also display the annotation in real time for the user to view.

[0077] In one possible implementation, the first natural language instruction includes any one of a first voice instruction or a first text instruction, and the electronic device detects that the user inputs the first natural language instruction, specifically including: the electronic device detects the first voice instruction input by the user through a microphone; or the electronic device calls up and displays a virtual keyboard, and detects the first text instruction input by the user through the virtual keyboard.

[0078] In this way, during the natural language interaction between the electronic device and the user, the electronic device can not only support the user's voice input, but also support the user's text input. The interaction method is more flexible and can be suitable for users with different input habits.

[0079] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method in any possible implementation of the first aspect or the second aspect above.

[0080] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are run on an electronic device, the electronic device executes the method in any possible implementation of the first or second aspect above.

[0081] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method described in any possible implementation of the first or second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figures 1A-1E are schematic diagrams of user interfaces for implementing a quick sharing function for image content in a set of cutout and transfer application scenarios provided by an embodiment of the present application;

[0083] 1F-1H are schematic diagrams of user interfaces for implementing quick collection / copying functions for image content in a set of cutout transfer application scenarios provided by an embodiment of the present application;

[0084] FIG1I-FIG1J are schematic diagrams of user interfaces in a set of whole-image transfer application scenarios provided by an embodiment of the present application;

[0085] FIG1K is a schematic diagram of a display style of a natural language interaction portal provided in an embodiment of the present application;

[0086] Figures 2A-2D are schematic diagrams of user interfaces in which a group of natural language interaction portals are gradually reduced in size as an interactive object moves, provided by an embodiment of the present application;

[0087] 3A-3C are schematic diagrams of a set of user interfaces for implementing a function of erasing image content, provided in an embodiment of the present application;

[0088] 4A-4B are schematic diagrams of a set of user interfaces for implementing a function of replacing image content provided by an embodiment of the present application;

[0089] 5A-5D are schematic diagrams of a set of user interfaces for implementing a picture content recognition function provided by an embodiment of the present application;

[0090] FIG6 is a schematic diagram of a user interface for implementing a picture beautification function provided in an embodiment of the present application;

[0091] FIG7 is a schematic diagram of a user interface for implementing an image style transfer function provided by an embodiment of the present application;

[0092] 8A-8F are schematic diagrams of a set of user interfaces for performing corresponding operations on pictures in a social application provided by an embodiment of the present application;

[0093] 9A-9E are schematic diagrams of a set of user interfaces for implementing a quick sharing function of video content provided in an embodiment of the present application;

[0094] 10A-10B are schematic diagrams of a set of user interfaces for implementing a video content cropping function provided in an embodiment of the present application;

[0095] 11A-11B are schematic diagrams of another set of user interfaces for implementing a video content cropping function provided by an embodiment of the present application;

[0096] 12A-12D are schematic diagrams of a set of user interfaces for implementing the function of displaying unread message summaries, provided in an embodiment of the present application;

[0097] 12E-12F are schematic diagrams of a set of user interfaces for implementing a question-and-answer query function for chat messages, provided in an embodiment of the present application;

[0098] 13A-13D are schematic diagrams of a set of user interfaces for displaying a document summary, provided in an embodiment of the present application;

[0099] 13E-13F are schematic diagrams of a set of user interfaces for implementing the function of displaying notes in a document, provided in an embodiment of the present application;

[0100] 13G-13H are schematic diagrams of a set of user interfaces for implementing a document content translation function provided in an embodiment of the present application;

[0101] 14A-14E are schematic diagrams of a set of user interfaces for implementing a text collection function according to an embodiment of the present application;

[0102] 14F-14H are schematic diagrams of a set of user interfaces implementing a text query function provided in an embodiment of the present application;

[0103] 15A-15N are schematic diagrams of a set of user interfaces for implementing a text content modification function provided in an embodiment of the present application;

[0104] 16A-16O are schematic diagrams of a set of user interfaces for implementing the functions of adding annotations, displaying annotations, and editing annotations in a document, provided in an embodiment of the present application;

[0105] FIG17 is a flow chart of an interaction method provided in an embodiment of the present application;

[0106] FIG18 is a schematic diagram of a software architecture of an electronic device 100 provided in an embodiment of the present application;

[0107] FIG19 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0108] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0109] It should be understood that the terms "first," "second," and the like in the specification, claims, and drawings of this application are used to distinguish between different objects, rather than to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0110] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0111] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0112] With the development of science and technology, the way of human-computer interaction has also changed. It has become possible to switch from machine language interaction to natural language interaction. Machine language interaction is mainly completed by clicking on interface controls or physical buttons through keyboard and mouse or touch. Sometimes a task requires multiple levels of jump to complete, which is cumbersome. Natural language interaction is mainly completed through the user's natural language input (such as voice input or text input) combined with powerful natural language processing capabilities. It can avoid hierarchical jumps and go directly to the next step. Therefore, compared with machine language interaction, natural language interaction can bring users a more convenient and intelligent experience.

[0113] Currently, more and more electronic devices can provide natural language interaction functions, which can support electronic devices to receive language / text instructions from users through natural language input (such as voice input or text input), and can support electronic devices to understand and execute the instructions.

[0114] However, in some scenarios where an interactive object needs to be processed (for example, editing a flower in a picture containing multiple flowers), users are sometimes unable to accurately express the interactive object through voice / text alone, and the electronic device is unable to accurately and quickly recognize the user's intention. Multiple rounds of natural language interaction may be required to complete the task, resulting in low interaction efficiency and poor user experience.

[0115] Based on the above-mentioned problem of inconvenience in expressing interactive objects in natural language, an embodiment of the present application provides an interactive method that can be applied to an electronic device 100 that can provide a natural language interactive function. The electronic device 100 can support the user to start the natural language interactive function through a long press operation and display a natural language interactive entrance. Further, the electronic device 100 can detect the voice / text instruction input by the user through natural language (such as voice input or text input) and display the instruction at the natural language interactive entrance. After the instruction is input, the electronic device 100 can determine whether the instruction has a default object or a referent. If so, the electronic device 100 can extract and identify the interactive object of the interface content at the location where the user long presses the operation or at the location where the user long presses the operation and its vicinity, thereby determining the interactive object and substituting the interactive object into the input instruction for understanding and execution, thereby completing the task specified by the user. In this way, through the integration of touch operation and natural language interaction, the selection of interactive objects and further operations can be efficiently realized, solving the problem of inconvenience in expressing interactive objects in natural language, improving human-computer interaction efficiency, and improving user experience.

[0116] It should be noted that the electronic device 100 in the embodiment of the present application may be equipped with Or portable electronic devices with other operating systems, such as mobile phones, tablet computers, smart watches, smart bracelets, AR devices, VR devices, etc., can also be non-portable electronic devices such as laptop computers with touch-sensitive surfaces or touch panels, desktop computers with touch-sensitive surfaces or touch panels, etc. The embodiments of the present application do not limit the type of electronic device 100.

[0117] The interaction method provided in the embodiment of the present application can be applied in a variety of scenarios. The following introduces the interaction method provided in the embodiment of the present application in combination with the user interface and several typical application scenarios.

[0118] It should be noted that the following application scenarios are only a few typical application scenarios. The interaction method provided in the embodiments of the present application can also be applied to many other scenarios, and the embodiments of the present application are not limited to this.

[0119] 1. Display Application Scenarios

[0120] "Display application scenarios" may refer to application scenarios in which the interactive method provided in the embodiments of the present application is used to perform corresponding operations (such as sending, editing, etc.) on display information such as pictures and videos.

[0121] 1. Perform corresponding operations on the picture

[0122] (1) Sending picture content

[0123] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to send the entire picture or a part of the picture (also referred to as an object in the picture) to other electronic devices (such as a friend's electronic device), other applications installed on the electronic device 100, etc., thereby realizing the quick sharing / collection / copying function of the picture content.

[0124] 1) Cutout transfer (i.e. sending a part of the picture)

[0125] FIG. 1A to FIG. 1E exemplarily illustrate a set of user interfaces for implementing a quick sharing function of image content in a cutout flow application scenario.

[0126] Taking a picture in a gallery application as an example, referring to FIG1A , the user interface 110 exemplarily shown in FIG1A may be a user interface of the gallery application, and the user interface 110 may include a picture 111 , which may be a picture that the user is browsing.

[0127] In an embodiment of the present application, a user can trigger the electronic device 100 to perform intelligent cutout and display a thumbnail of the entire image by performing a long press operation on the image (for example, a long press operation performed by the user's finger or a touch device). For example, if the interface content at the location of the long press operation or at the location of the long press operation and its vicinity includes the cutout subject, then the electronic device 100 can perform intelligent cutout and display the cutout subject; for another example, if the interface content at the location of the long press operation or at the location of the long press operation and its vicinity does not include the cutout subject, then the electronic device 100 can display a thumbnail of the entire image.

[0128] Continuing to refer to FIG1A , illustratively, the electronic device 100 can detect a user's long press operation on the picture 111. Assuming that the location of the long press operation is the location of the left flower in the picture 111, in response to the long press operation, the electronic device 100 can identify whether the interface content at the location of the long press operation or at the location of the long press operation and its vicinity includes a cutout subject. If, for example, the location of the long press operation includes the left flower in the picture 111, the electronic device 100 can determine that the cutout subject is the left flower in the picture 111. Further, referring to FIG1B , the electronic device 100 can highlight the cutout subject 113 (i.e., the left flower in the picture 111) to facilitate the user to know whether the cutout subject identified by the electronic device 100 is consistent with the user's intended cutout subject. In an embodiment of the present application, the cutout subject identified by the electronic device 100 can be an interactive object selected by the user for subsequent natural language interaction.

[0129] In an embodiment of the present application, the above-mentioned long press operation can also trigger the electronic device 100 to start the natural language interaction function.

[0130] Continuing with FIG1B , after the electronic device 100 activates the natural language interaction function, it may display a natural language interaction portal. The natural language interaction portal may be used to prompt the user that the electronic device 100 has activated the natural language interaction function and may also be used for subsequent natural language interaction. The natural language interaction portal displayed by the electronic device 100 may be the natural language interaction portal 112 exemplarily shown in FIG1B .

[0131] Optionally, a logo 112 a may be displayed on the natural language interaction entrance 112 , and the logo 112 a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0132] Optionally, a prompt 112b may be displayed on the natural language interaction entrance 112, and the prompt 112b may be used to prompt the user to enter a natural language instruction related to the interactive object. For example, as shown in FIG1B , the user selects a flower as the interactive object, and the prompt 112b may be a prompt for prompting the user to enter a natural language instruction related to the flower (for example, you can ask "What kind of flower is this"). It is easy to understand that the above prompt may be generated by the electronic device 100 based on the identified interactive object, that is, the generated prompt may be a prompt related to the interactive object, and the above prompt is a natural language instruction corresponding to the interactive object that can be recognized by the electronic device 100.

[0133] Optionally, the prompt 112b may also be output in the form of voice broadcast.

[0134] In some embodiments, after the electronic device 100 starts the natural language interaction function, it can start to detect the natural language instructions input by the user in real time, regardless of whether the user cancels the long press operation. In other embodiments, after the electronic device 100 starts the natural language interaction function, the electronic device 100 can also start to detect the natural language instructions input by the user after detecting the user's operation (such as a click operation) on the identifier 112a shown in Figure 1B or the user's operation (such as a click operation) on any position on the natural language interaction entrance 112 shown in Figure 1B.

[0135] In some embodiments, the prompt displayed on the natural language interaction entrance or the prompt announced by voice (for example, prompt 112b) may be displayed / announced only when the electronic device 100 starts the natural language interaction function and no user voice command is detected within a preset time period.

[0136] In some embodiments, after the electronic device 100 starts detecting voice commands input by the user, if the electronic device 100 detects that the user has not canceled the long press operation (i.e., has been long pressing the screen of the electronic device 100), then the electronic device 100 can continue to detect the voice commands input by the user; if the electronic device 100 detects that the user has canceled the long press operation (for example, the user's finger or touch device has left the screen of the electronic device 100), then the electronic device 100 can no longer detect the voice commands input by the user (i.e., end collecting voice input).

[0137] In some embodiments, after the electronic device 100 detects a voice command input by the user, regardless of whether the user cancels the long press operation, if the electronic device 100 does not detect the user continuing to input a voice command within a preset time period, then the electronic device 100 can confirm that the voice command input is complete. In other embodiments, after the electronic device 100 detects a voice command input by the user, if the electronic device 100 detects that the user cancels the long press operation, then the electronic device 100 can confirm that the voice command input is complete.

[0138] In the embodiment of the present application, the natural language instruction input by the user can be a voice instruction or a text instruction. The voice instruction can be input by the user through the microphone on the electronic device 100, and the text instruction can be input by the user through the virtual keyboard on the electronic device 100. It is easy to understand that if the current application supports virtual keyboard input, the natural language instruction input by the user can be a voice instruction or a text instruction; if the current application does not support virtual keyboard input, the natural language instruction input by the user can only be a voice instruction.

[0139] Referring to FIG1C , taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to share the interactive object to a certain application via voice instruction. Assuming that the user wants to share the interactive object, the cutout subject 113 in FIG1B , to a chat group named “Happy Family” in the social application “Changlian”, then the electronic device 100 can detect the relevant voice instruction input by the user (e.g., the voice “Open Changlian Happy Family”) and can display the detected relevant voice instruction input by the user in the form of text (e.g., text 112d “Open Changlian Happy Family”).

[0140] Optionally, after the electronic device 100 starts detecting the voice command input by the user, it can replace the logo 112a shown in Figure 1B with the logo 112c shown in Figure 1C. The logo 112c can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0141] Furthermore, continuing to refer to Figure 1C, after the above-mentioned voice command is input, the electronic device 100 can also analyze the above-mentioned voice command to determine that the operation corresponding to the above-mentioned voice command is to open the "Changlian" application and display the user interface corresponding to the chat group with the group chat name "Happy Family".

[0142] Further, referring to Figure 1D, for example, the electronic device 100 can display the user interface 120 corresponding to the chat group "Happy Family", and the electronic device 100 can support the user to long press and drag the interactive object 113 in Figure 1B in the user interface 120. Further, after the electronic device 100 detects that the user cancels the long press and drag operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), in response to the operation, referring to Figure 1E, the electronic device 100 can send the interactive object 113 in Figure 1B to the chat group named "Happy Family" in the social application "Changlian", thereby realizing the quick sharing function of picture content.

[0143] In the embodiment of the present application, the user may not cancel the long press operation during the entire process from Figure 1A to Figure 1D. In some embodiments, the user may also cancel the long press operation before displaying Figure 1D. In this case, the electronic device 100 may no longer display Figure 1D and directly display Figure 1E, that is, the cutout body 113 is directly sent to the chat group named "Happy Family" in the social application "Changlian".

[0144] It should be noted that the voice instruction "Open Changlian Happy Family" for realizing the quick sharing function of the picture content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the quick sharing function of the picture content needs to meet certain conditions, under which the electronic device 100 can determine where the user wants to share the selected interactive object. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating where the user wants to share the selected interactive object, such as verbs such as "open", "jump to", "share to", and nouns such as the name of the application, the name of the contact / chat group, etc.

[0145] FIG1F-FIG1H exemplarily illustrate a set of user interfaces for implementing quick collection / copying functions of image content in a cutout flow application scenario.

[0146] Taking the interactive object selected by the user as the cutout body 113 shown in Figure 1B as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B, and will not be repeated here.

[0147] Referring to FIG1F , taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to collect / copy the interactive object to a certain application through voice instructions. Assuming that the user wants to collect / copy the interactive object 113 in FIG1B to the "Memo" application, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "Copy this to Memo") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 112e "Copy this to Memo").

[0148] Furthermore, continuing to refer to FIG. 1F , the electronic device 100 may also analyze the voice command, thereby determining that the operation corresponding to the voice command is to open the “Memo” application and display the corresponding user interface.

[0149] Further, referring to FIG1G , illustratively, the electronic device 100 can display a user interface 130, wherein the user interface 130 can be a user interface corresponding to an existing note in the memo (for example, a note named “XX flower”), or a newly created note, and this embodiment of the present application does not limit this. The electronic device 100 can support the user to long press and drag the interactive object of the cutout body 113 in FIG1B in the user interface 130. Further, after the electronic device 100 detects that the user cancels the long press and drag operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), in response to the operation, referring to FIG1H , the electronic device 100 can add the interactive object of the cutout body 113 in FIG1B to the note named “XX flower”, thereby realizing the quick collection / copy function of the picture content.

[0150] In the embodiment of the present application, the user may not cancel the long press operation during the entire process from Figure 1F to Figure 1G. In some embodiments, the user may also cancel the long press operation before Figure 1G is displayed. In this case, the electronic device 100 may no longer display Figure 1G and directly display Figure 1H, that is, the cutout body 113 is directly added to the note named "XX Flower".

[0151] It should be noted that the voice instruction "Copy this to the memo" for realizing the quick collection / copy function of the picture content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the quick collection / copy function of the picture content needs to meet certain conditions, under which the electronic device 100 can determine where the user wants to collect / copy the selected interactive object. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating where the user wants to collect / copy the selected interactive object, such as verbs such as "open", "collect to", "copy to", and nouns such as the name of the application.

[0152] It should be noted that the embodiment of the present application does not limit the display style of the natural language interaction entrance; the embodiment of the present application does not limit the display position of the natural language interaction entrance. For example, it can be the location where the long press operation is located; for another example, it can be a location near the location where the long press operation is located; and so on.

[0153] It should be noted that the meaning of the "natural language interaction portal" has been described in the embodiments of this application, and its name does not constitute any limitation to the embodiments of this application. In other embodiments of this application, the natural language interaction portal may also be referred to as a natural language (or voice / text) recognition box / prompt box / recognition bar / prompt bar / recognition window / prompt window, etc.

[0154] It should be noted that in the interaction method provided in the embodiment of the present application, the operation of the user selecting the interaction object and triggering the electronic device 100 to start the natural language interaction function can be the same operation. In the embodiment of the present application, the operation is only taken as a long press operation as an example, which should not constitute a limitation of the present application. In other embodiments of the present application, the operation can also be other operations.

[0155] It should be noted that in the interaction method provided in the embodiment of the present application, whether the user wants to trigger the electronic device 100 to start the natural language interaction function at the same time after selecting the interaction object can be set by the user independently or by the system default setting of the electronic device 100. The embodiment of the present application does not limit this.

[0156] 2) Entire image transfer (i.e. sending the entire image)

[0157] Referring to FIG1I , a user can trigger electronic device 100 to display a thumbnail of the entire image (e.g., thumbnail 114 shown in FIG1I ) by performing a long press on the image. Alternatively, the user can trigger electronic device 100 to initiate a natural language interaction function, completing the entire image transfer. The specific process of transferring the entire image is similar to the aforementioned transfer of the cutout image, and reference can be made to the aforementioned related content, which will not be repeated here.

[0158] The specific process of the above-mentioned whole-image flow is only based on an example of a certain picture that the user is browsing. In some embodiments, the whole-image flow can also be performed for pictures that the user has browsed before. For example, referring to Figure 1J, the user interface 140 exemplarily shown in Figure 1J can be a user interface in a gallery application that is displayed before browsing a certain picture. The user interface may include thumbnails of one or more pictures. After selecting a thumbnail of a certain picture, the user can also trigger the electronic device 100 to start the natural language interaction function and complete the whole-image flow through the natural language interaction function. The specific process is similar to the above-mentioned whole-image flow for a certain picture that the user is browsing, and will not be repeated here.

[0159] In some embodiments, after the electronic device 100 detects that the user has selected and completed the interactive object and starts the natural language interaction function to display the natural language interaction entrance, if the electronic device 100 detects that the user has long pressed and dragged the interactive object, then in response to this operation, the electronic device 100 can move the natural language interaction entrance along with the interactive object.

[0160] Optionally, if the electronic device 100 detects that the displacement of the interactive object caused by the user long pressing and dragging is greater than a preset displacement threshold, the electronic device 100 may reduce the display of the natural language interaction entrance, or cancel the display of the natural language interaction entrance. If the electronic device 100 does not detect any displacement of the interactive object within a preset time period, the electronic device 100 may resume displaying the natural language interaction entrance.

[0161] For example, referring to Figure 2A, taking the interactive object as thumbnail 114 as an example, the electronic device 100 can detect the user's operation of long pressing and dragging the thumbnail 114. In response to this operation, referring to Figures 2B and 2C, the electronic device 100 can move the natural language interaction entrance 112 along with the thumbnail 114, and can also gradually reduce the display of the natural language interaction entrance 112 as shown in Figures 2B and 2C until it is cancelled as shown in Figure 2D.

[0162] Optionally, while the above-mentioned natural language interaction portal moves and shrinks in size following the interactive object, the electronic device 100 may continue to detect the voice commands input by the user (i.e., continue to receive sound), or no longer detect the voice commands input by the user (i.e., stop receiving sound); when the above-mentioned natural language interaction portal is canceled from display, the electronic device 100 may no longer detect the voice commands input by the user; when the above-mentioned natural language interaction portal is restored to display, the electronic device 100 may continue to detect the voice commands input by the user.

[0163] In some embodiments, referring to Figure 1K, the electronic device 100 can display one or more function options (such as "body segmentation" option, "smart erase" option, "background quick change" option, etc.) while displaying the above-mentioned natural language interaction entrance (such as natural language interaction entrance 112), and display the above-mentioned natural language interaction entrance and one or more function options in a scenario-based menu bar 141 at the same time. The user can trigger the electronic device 100 to perform the operation corresponding to the function option by clicking the corresponding function option, which is convenient and quick.

[0164] (2) Erase the image content

[0165] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to erase a certain part of the picture (which can also be called a certain object in the picture) from the picture, thereby realizing the erasing function of the picture content.

[0166] Taking the interactive object selected by the user as the cutout body 113 shown in Figure 1B as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B, and will not be repeated here.

[0167] For example, if the natural language instruction input by the user is a voice instruction, the electronic device 100 can support the user to erase the interactive object from the picture through a voice instruction. Referring to FIG3A , assuming that the user wants to erase the interactive object 113 of the cutout body from the picture, the electronic device 100 can detect the relevant voice instruction input by the user (e.g., the voice "erase this") and can display the detected relevant voice instruction input by the user in the form of text (e.g., text 112f "erase this").

[0168] Furthermore, referring to FIG3A , after the voice command is input, the electronic device 100 may analyze the voice command to determine that the operation corresponding to the voice command is to erase the interactive object, the cutout body 113 , from the image.

[0169] Furthermore, the electronic device 100 can erase the interactive object cutout subject 113 from the picture through a related algorithm. After erasing, as shown in FIG3B , the electronic device 100 can no longer display the interactive object cutout subject 113 in the picture.

[0170] It should be noted that the voice command "erase this" for implementing the function of erasing the image content is merely exemplary and should not constitute a limitation of this application. It is easy to understand that the voice command for implementing the function of erasing the image content needs to meet certain conditions, under which the electronic device 100 can determine the position of the object in the image that the user wants to erase. Among them, meeting the "certain conditions" can mean that the voice command includes certain keywords for indicating the erasing object and the erasing action.

[0171] In some embodiments, if the electronic device 100 cannot determine the user's specific intention through the voice command input by the user and the location of the long press operation, then the electronic device 100 can further prompt the user to perform the corresponding operation through the natural language interaction portal in the form of voice broadcast or text prompts to determine the user's specific intention. For example, referring to Figure 3C, if the electronic device 100 cannot determine the location of the object in the picture where the user wants to erase, then the electronic device 100 can prompt the user, for example, displaying prompt 112g "Where do you want to erase, please click the location you want to erase", and after detecting that the user performs the corresponding operation according to prompt 112g, the electronic device 100 can then erase the object at the corresponding location in the picture.

[0172] (3) Image content replacement

[0173] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to replace a part of the picture (also referred to as an object in the picture) (i.e., replace the content of a part of the picture with other content), thereby realizing the function of replacing the picture content.

[0174] Taking the interactive object selected by the user as the cutout body 113 shown in Figure 1B as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B, and will not be repeated here.

[0175] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to replace the interactive object through voice instruction. Referring to Figure 4A, assuming that the user wants to replace the interactive object 113 of the cutout body (which can be called the replaced object) with another object (which can be called the replacement object), then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "Replace this with leaves") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 112h "Replace this with leaves").

[0176] Further, referring to Figure 4A, after the above-mentioned voice instruction is input, the electronic device 100 can also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to replace the interactive object of the cutout body 113 with other objects (for example, replace the interactive object of the cutout body 113 with leaves).

[0177] Furthermore, the electronic device 100 can search for replacement objects locally or in the cloud. After the search is complete, the electronic device 100 can select a replacement object by default (or support the user to select a replacement object from multiple searched replacement objects), and replace the interactive object of the cutout subject 113 with the selected replacement object through a relevant algorithm. After the replacement is complete, referring to FIG4B , the electronic device 100 can no longer display the interactive object of the cutout subject 113 in the image, but instead display the selected replacement object (e.g., the replacement object 115 of the leaf) at the location of the original cutout subject 113 in the image. Thus, the function of replacing the image content is realized.

[0178] It should be noted that the voice instruction "Replace this with leaves" for implementing the function of replacing the image content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the function of replacing the image content needs to meet certain conditions, under which the electronic device 100 can determine what the user wants to replace the replaced object with. Among them, the above-mentioned "certain conditions" can mean that the voice instruction includes certain keywords for indicating the replaced object, the replacement object, and the replacement action.

[0179] In some embodiments, the electronic device 100 may also support the user to insert content into the picture. It is easy to understand that if the user wants to insert content into the picture, then, similar to the above-mentioned picture content replacement, the user can enter a voice instruction related to content insertion. After the language instruction is entered, the electronic device 100 can analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to insert content into the picture. Furthermore, the electronic device 100 can search for an insertion object locally or in the cloud. After the search is completed, the electronic device 100 can select an insertion object by default (or support the user to select an insertion object from multiple insertion objects that have been searched), and insert the above-mentioned selected insertion object into the picture through a relevant algorithm (for example, the insertion object can be inserted into the position where the user long presses in the picture). It is easy to understand that the above-mentioned voice instruction related to content insertion needs to meet certain conditions, under which the electronic device 100 can determine what content the user wants to insert into the picture. Among them, the above-mentioned "certain conditions" can refer to the voice instruction including certain keywords for indicating the insertion object, insertion action, and insertion position.

[0180] In some embodiments, the electronic device 100 may also support the user to complete the content when some content in the picture is missing. It is easy to understand that if the user wants to complete the content in the picture, then, similar to the above-mentioned picture content replacement, the user can input a voice instruction related to content completion (for example, a voice instruction "complete the missing content in the picture"). After the voice instruction is input, the electronic device 100 can analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to complete the content in the picture. Furthermore, the electronic device 100 can determine the content that needs to be completed through a relevant algorithm, and add the content to the picture. It is easy to understand that the above-mentioned voice instructions related to content completion need to meet certain conditions, under which the electronic device 100 can determine that the user wants to complete the content in the picture. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the completion action.

[0181] (4) Image content recognition (e.g., identifying objects or faces in images)

[0182] By implementing the interaction method provided in the embodiment of the present application, the electronic device 100 can support the recognition of the interactive object selected by the user in the picture, thereby realizing the recognition function of the picture content.

[0183] Taking the interactive object selected by the user as the cutout body 113 shown in Figure 1B as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B, and will not be repeated here.

[0184] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to use voice instructions to let the electronic device 100 identify the interactive object. Referring to Figure 5A, assuming that the user wants the electronic device 100 to identify the cutout subject 113, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What kind of flower is this" or "What is this", etc.), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 112i "What kind of flower is this" or "What is this", etc.).

[0185] Furthermore, referring to FIG. 5A , after the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is identifying the cutout subject 113 .

[0186] Furthermore, the electronic device 100 can identify the cutout subject 113 through a relevant algorithm. After the identification is completed, referring to FIG5B , the electronic device 100 can display the identification result 116, wherein the identification result 116 may include the name of the cutout subject 113 (for example, "XX flower"), and may also include an introduction to the cutout subject 113. The introduction to the cutout subject 113 can be retrieved by the electronic device 100 locally or in the cloud. It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style, it can also be a normal text style, etc.) and display position of the recognition result 116.

[0187] In some embodiments, the electronic device 100 may also support identification of facial images in pictures. Similar to the above-mentioned identification of the cutout subject 113, referring to FIG5C, assuming that the interactive object selected by the user is the cutout subject 117, the cutout subject 117 may include a facial image, and assuming that the user wants the electronic device 100 to identify the cutout subject 117, then the electronic device 100 can detect the relevant voice instructions input by the user (for example, voice "Is this singer XXX?" or "Who is this?", etc.), and can display the relevant voice instructions input by the user that have been detected in the form of text (for example, text 112j "Is this singer XXX?" or "Who is this?", etc.). Further, referring to FIG5C, after the above-mentioned voice instructions are input, the electronic device 100 can also analyze the above-mentioned voice instructions to determine that the operation corresponding to the above-mentioned voice instructions is to identify the cutout subject 117. Furthermore, the electronic device 100 can identify the cutout subject 117 through a relevant algorithm. After the identification is completed, referring to FIG5D , the electronic device 100 can display the identification result 118, wherein the identification result 118 may include the identity of the cutout subject 117 (for example, "singer XXX"), and may also include an introduction to the cutout subject 117. The introduction to the cutout subject 117 can be retrieved locally or in the cloud by the electronic device 100. It should be noted that the embodiment of the present application does not limit the display style (for example, a card style, a normal text style, etc.) and display position of the recognition result 118.

[0188] It is easy to understand that the above-mentioned voice commands related to image content recognition need to meet certain conditions, under which the electronic device 100 can determine that the user wants the electronic device 100 to perform image content recognition. Among them, the above-mentioned "certain conditions" can mean that the voice command includes certain keywords for indicating the recognition action (such as keywords such as query, recognize, what, and who).

[0189] (5) Picture beautification (e.g., applying makeup to the face of a person in the picture)

[0190] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users in beautifying pictures.

[0191] Taking the interactive object selected by the user as the cutout body 117 shown in Figure 5C as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B, and will not be repeated here.

[0192] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to use voice instructions to let the electronic device 100 beautify the interactive object (for example, beautify the face). Referring to Figure 6, assuming that the user asks the electronic device 100 to beautify the cutout subject 117 (for example, apply lipstick to the cutout subject 117), then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "apply a red bean paste color lipstick") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 112k "apply a red bean paste color lipstick").

[0193] Furthermore, referring to FIG6 , after the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to apply a bean paste-colored lipstick to the cutout body 117 .

[0194] Furthermore, the electronic device 100 can perform intelligent cutout to obtain the cutout body 117, and identify the lip position of the cutout body 117 through a relevant algorithm, and add bean paste color at that position, and display it, thereby achieving the beauty and facial effect desired by the user; or, the electronic device 100 can also identify the facial image at the position where the user long presses through a relevant algorithm, and perform feature analysis on the facial image, determine the position of the lips in the facial image, and add bean paste color at that position, and display it, thereby achieving the beauty and facial effect desired by the user.

[0195] In some embodiments, when the cutout cannot be achieved through a long press operation, the electronic device 100 can also automatically recognize the facial image in the picture and perform corresponding beauty and beautification operations in combination with the voice commands input by the user.

[0196] It is easy to understand that the above-mentioned voice commands related to picture beautification need to meet certain conditions, under which the electronic device 100 can determine how the user wants to beautify the picture. Among them, the above-mentioned "certain conditions" can mean that the voice commands include certain keywords for indicating picture beautification (such as keywords such as applying lipstick, slimming the face, enlarging eyes, smoothing the skin, beautifying the face, beautifying the body, mosaic, graffiti, adding frames / stickers / markers, and blurring).

[0197] (6) Image style transfer

[0198] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to transfer the style of the entire image or a portion of the image. Among them, style transfer can refer to adding a filter effect to the entire image or a portion of the image to change its original style.

[0199] In the scenario of performing style transfer on an image, the specific process of the user selecting an interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 1A and 1B and will not be repeated here.

[0200] Taking the natural language command input by the user as a voice command as an example, the electronic device 100 can support the user to use the voice command to instruct the electronic device 100 to perform style transfer on the interactive object. Referring to FIG7 , taking the interactive object as an entire image as an example, the electronic device 100 can detect the relevant voice command input by the user (e.g., the voice command "a little more retro") and can display the detected relevant voice command input by the user in the form of text (e.g., text 1121 "a little more retro").

[0201] Furthermore, referring to FIG7 , after the above-mentioned voice instruction is input, the electronic device 100 may also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to add a retro filter to the entire picture.

[0202] Furthermore, the electronic device 100 can add a retro filter to the above picture through a relevant algorithm and display it, thereby achieving the retro effect desired by the user.

[0203] It is easy to understand that the above-mentioned voice commands related to image style transfer need to meet certain conditions. Under these conditions, the electronic device 100 can determine how the user wants to transfer the image style. Among them, the above-mentioned "certain conditions" can mean that the voice command includes certain keywords used to indicate image style transfer (such as fresh, beautiful, literary, fashionable, natural, retro, black and white, Chinese style, European and American style, Japanese style, etc.).

[0204] In some embodiments, the electronic device 100 may also support the user to adjust image parameters (e.g., adjusting image brightness, contrast, saturation, sharpness, etc.). The specific process of parameter adjustment is similar to the specific process of style transfer described above and will not be repeated here.

[0205] The above application scenarios for performing corresponding operations on images are all described using images in the Gallery application as an example. However, the application is not limited to images in the Gallery application and can also be used for images in other applications. The following, combined with Figures 8A-8E, uses an image in a social application as an example to describe the process of performing corresponding operations on images.

[0206] In an embodiment of the present application, a user can trigger the electronic device 100 to display a menu bar corresponding to the above-mentioned picture (i.e., the interactive object) by performing a long press operation on the picture (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0207] Referring to Figure 8A, the user interface 120 shown in Figure 8A can be a user interface of a social application (such as the Changlian application) (such as the user interface corresponding to a chat group with the group chat name "Happy Family"), and the user interface may include one or more chat records (such as the chat record in Figure 121).

[0208] Continuing to refer to FIG8A , illustratively, the electronic device 100 may detect a long press operation by the user on the image 121 . In response to the long press operation, the electronic device 100 may display the user interface 150 exemplarily shown in FIG8B and start the natural language interaction function.

[0209] Referring to FIG8B , the user interface 150 may include a natural language interaction entry 151. The natural language interaction entry 151 may be used to notify the user that the electronic device 100 has activated the natural language interaction function and may also be used for subsequent natural language interaction. The user interface 150 may also include a menu bar 152. The user interface 150 may also include the image 121 shown in FIG8A .

[0210] Optionally, a logo 151 a may be displayed on the natural language interaction entrance 151 , and the logo 151 a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0211] Optionally, a prompt 151b may be displayed on the natural language interaction entrance 151, and the prompt 151b may be used to prompt the user to input natural language instructions related to the interactive object. For example, as shown in Figure 8A, the interactive object selected by the user is picture 121, and the electronic device 100 can recognize that the picture 121 includes flowers. The prompt 151b can be a prompt for prompting the user to input natural language instructions related to flowers (for example, you can ask "What kind of flower is this"). Optionally, the electronic device 100 can also automatically recommend functions based on the recognition type and display corresponding function options in the above-mentioned menu bar 152, such as "AI recognition of objects", etc. The user can trigger the electronic device 100 to display the recognition result by clicking on the function option.

[0212] Optionally, the prompt 151b may also be output in the form of voice broadcast.

[0213] Continuing to refer to FIG. 8B , the menu bar 152 may include one or more options (eg, a “copy to album” option, an “edit” option, a “share” option, etc.), and these one or more options may be used to perform corresponding operations on the picture 121 .

[0214] Generally, social applications support virtual keyboard input. Therefore, in the embodiment of the present application, the natural language instructions input by the user can be voice instructions or text instructions. The voice instructions can be input by the user through the microphone on the electronic device 100, and the text instructions can be input by the user through the virtual keyboard on the electronic device 100.

[0215] The embodiment of the present application does not limit the triggering method of the electronic device 100 to detect whether the user inputs a voice command or a text command.

[0216] In some embodiments, the triggering method for the electronic device 100 to detect the user input voice command may be that the electronic device 100 starts the natural language interaction function (that is, starts detecting the user input voice command after starting the natural language interaction function). In other embodiments, the triggering method for the electronic device 100 to detect the user input voice command may also be that the electronic device 100 detects the user's operation on the identifier 151a shown as an example in Figure 8B (for example, a click operation).

[0217] For example, referring to FIG8C , taking the case where the natural language instruction input by the user is a voice instruction, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice “What kind of flower is this”) and can display the detected relevant voice instruction input by the user in the form of text (for example, text 151d “What kind of flower is this”).

[0218] Optionally, after the electronic device 100 starts detecting the voice command input by the user, it can replace the logo 151a shown in Figure 8B with the logo 151c shown in Figure 8C. The logo 151c can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0219] Optionally, referring to FIG8C , the menu bar 152 may be hidden while the user is inputting a voice command.

[0220] Furthermore, referring to FIG8C , after the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to identify the flowers included in the interactive object.

[0221] Furthermore, the electronic device 100 may identify the interactive object using a relevant algorithm. After the identification is complete, as shown in FIG8D , the electronic device 100 may display an identification result 153. The identification result 153 may include the name of the flower included in the interactive object (e.g., "XX Flower") and may also include a brief description of the flower. It should be noted that the present embodiment of the application does not limit the display style (e.g., card style, ordinary text style, etc.) or display location of the identification result 153.

[0222] In some embodiments, the triggering method for the electronic device 100 to detect the user input text instruction may be that the electronic device 100 detects the user's operation on the identifier 151a shown in Figure 8B (for example, a click operation). In other embodiments, the triggering method for the electronic device 100 to detect the user input text instruction may also be that the electronic device 100 detects the user's operation on any position on the natural language interaction entrance 151 shown in Figure 8B (for example, a click operation).

[0223] For example, referring to Figure 8B, if the user wants to enter a text instruction through the virtual keyboard, then the electronic device 100 can detect the user's operation on the identifier 151a shown in Figure 8B (for example, a click operation). In response to the operation, referring to Figure 8E, the electronic device 100 can call up the virtual keyboard, and the user can enter a text instruction through the virtual keyboard. After the text instruction is entered, the electronic device 100 can analyze the text instruction and execute the operation corresponding to the text instruction. The specific process is similar to the analysis and execution process of the aforementioned voice instruction, and will not be repeated here.

[0224] It should be noted that the embodiments of the present application do not limit the display style of the natural language interaction portal; nor do they limit the display location of the natural language interaction portal. For example, it can be displayed above or below the menu bar; for example, it can also be displayed in the menu bar (such as shown in Figure 8F); and so on.

[0225] It should be noted that in the interaction method provided in the embodiment of the present application, the operation of triggering the electronic device 100 to display the menu bar corresponding to the interactive object (e.g., a picture) and the operation of triggering the electronic device 100 to start the natural language interaction function can be the same operation. In the embodiment of the present application, the operation is only taken as a long press operation as an example, which should not constitute a limitation of the present application. In other embodiments of the present application, the operation can also be other operations.

[0226] It should be noted that in the interaction method provided in the embodiment of the present application, whether to trigger the electronic device 100 to display the menu bar corresponding to the interactive object (such as a picture) at the same time as triggering the electronic device 100 to start the natural language interaction function can be set by the user independently or by the system default setting of the electronic device 100. The embodiment of the present application does not limit this.

[0227] It is easy to understand that in the embodiment of the present application, the electronic device 100 can implement various image editing functions (such as image content erasing, image content replacement / insertion / completion, image content recognition, image beautification, image style transfer, image parameter adjustment, etc.) through relevant image processing algorithm models (such as DALL-E2, Midjourny, etc.). Among them, DALL-E2 is an artificial intelligence image generator based on natural language, which can create high-quality images and works of art based on natural language text descriptions, and can implement various image generation / editing functions such as image repair and image expansion; among them, Midjourey is also an artificial intelligence image generator based on natural language, which can generate realistic and diverse images based on natural language text descriptions, and can implement functions such as color adjustment, resolution adjustment, size adjustment, style transfer, etc.

[0228] 2. Perform corresponding operations on the video

[0229] (1) Video content sending

[0230] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to send videos to other electronic devices (such as friends' electronic devices), other applications installed on the electronic device 100, etc., thereby realizing the quick sharing function of video content.

[0231] 9A-9E exemplarily illustrate a set of user interfaces for implementing a quick sharing function for video content.

[0232] Taking a video in a gallery application as an example, referring to FIG9A , the user interface 210 exemplarily shown in FIG9A may be a user interface of the gallery application, and the user interface 210 may include a video 211 , which may be a video viewed by the user before playback.

[0233] In an embodiment of the present application, a user can trigger the electronic device 100 to start the natural language interaction function by performing a long press operation on the video (for example, a long press operation performed by the user's finger or touch device), and can also trigger the video to be selected as an interactive object for subsequent natural language interaction by the user.

[0234] Continuing to refer to Figure 9A, illustratively, the electronic device 100 can detect a user's long press operation on the video 211. In response to the long press operation, referring to Figure 9B, the electronic device 100 can start the natural language interaction function, and the electronic device 100 can also determine that the interaction object selected by the user is the video 211.

[0235] 9B , after the electronic device 100 starts the natural language interaction function, it may display a natural language interaction entry 212 . The natural language interaction entry 212 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0236] Optionally, a logo 212a may be displayed on the natural language interaction entrance 212, and the logo 212a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0237] Optionally, a prompt 212b may be displayed on the natural language interaction portal 212. The prompt 212b may be used to prompt the user to enter a natural language instruction related to the interaction object. For example, as shown in FIG9B , if the user selects a video as the interaction object, the prompt 212b may be used to prompt the user to enter a natural language instruction related to the video (for example, you can ask "What video is this").

[0238] Optionally, the prompt 212b may also be output in the form of voice broadcast.

[0239] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the aforementioned picture content sending, which will not be repeated here.

[0240] Referring to FIG9C , assuming that the natural language instruction input by the user is a voice instruction, the electronic device 100 can support the user to share the interactive object to a certain application via voice instruction. Assuming that the user wants to share the interactive object, video 211 in FIG9A , to a chat group named "Happy Family" in the social application "Changlian", the electronic device 100 can detect the relevant voice instruction input by the user (e.g., the voice "Open Changlian Happy Family") and can display the detected relevant voice instruction input by the user in the form of text (e.g., text 212d "Open Changlian Happy Family").

[0241] Optionally, after the electronic device 100 starts detecting the voice command input by the user, it can replace the logo 212a shown in Figure 9B with the logo 212c shown in Figure 9C. The logo 212c can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0242] Further, referring to Figure 9C, after the above voice command is input, the electronic device 100 can also analyze the above voice command to determine that the operation corresponding to the above voice command is to open the "Changlian" application and display the user interface corresponding to the chat group with the group chat name "Happy Family".

[0243] Further, referring to Figure 9D, exemplarily, the electronic device 100 can display the user interface 120 corresponding to the chat group "Happy Family", and the electronic device 100 can support the user to long press and drag the interactive object video 211 in Figure 9A in the user interface 120. Further, after the electronic device 100 detects that the user cancels the long press and drag operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), in response to the operation, referring to Figure 9E, the electronic device 100 can send the interactive object video 211 in Figure 9A to the chat group named "Happy Family" in the social application "Changlian", thereby realizing the quick sharing function of video content.

[0244] In the embodiment of the present application, the user may not cancel the long press operation during the entire process from Figure 9A to Figure 9D. In some embodiments, the user may also cancel the long press operation before Figure 9D is displayed. In this case, the electronic device 100 may no longer display Figure 9D and directly display Figure 9E, that is, the video 211 is directly sent to the chat group named "Happy Family" in the social application "Changlian".

[0245] It should be noted that the voice instruction "Open Changlian Happy Family" for realizing the quick sharing function of video content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the quick sharing function of video content needs to meet certain conditions, under which the electronic device 100 can determine where the user wants to share the selected interactive object. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating where the user wants to share the selected interactive object, such as verbs such as "open", "jump to", "share to", and nouns such as the name of the application, the name of the contact / chat group, etc.

[0246] It is easy to understand that, similar to the aforementioned sending of picture content, the interactive method provided in the embodiment of the present application can also realize the quick collection / copy function of video content. The specific process can refer to the relevant content of the aforementioned implementation of the quick collection / copy function of picture content, which will not be repeated here.

[0247] (2) Video content (such as specific clips, specific pictures) cropping

[0248] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to crop the video content (such as a specific segment), thereby realizing the cropping function of the video content.

[0249] Also, taking the interactive object selected by the user as the video 211 shown in Figure 9A as an example, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 9A and 9B, and will not be repeated here.

[0250] Taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can support the user to crop the interactive object video 211 through the voice instruction to obtain a certain video segment in the video 211.

[0251] Referring to Figure 10A, assuming that the user wants to obtain a video clip from 0 to 30 seconds in video 211, the electronic device 100 can detect the relevant voice instructions input by the user (for example, the voice "intercept a video clip from 0 to 30 seconds") and can display the detected relevant voice instructions input by the user in the form of text (for example, text 212e "intercept a video clip from 0 to 30 seconds").

[0252] Furthermore, referring to FIG10A , after the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to capture a video segment from 0 to 30 seconds in the video 211 .

[0253] Furthermore, the electronic device 100 can intercept a video segment from 0 to 30 seconds in the video 211 through a related algorithm. After interception, as shown in FIG10B , the electronic device 100 can also save the intercepted video segment.

[0254] It should be noted that the voice instruction "cut a video clip of 0 to 30 seconds" for implementing the cropping function of the video content is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the cropping function of the video content needs to meet certain conditions, under which the electronic device 100 can determine which video clip the user wants to crop / cut. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the cropping action (such as "crop", "cut", etc.), the position and duration of the cropped video clip.

[0255] Figures 10A and 10B above illustrate the process of cropping a video clip from a pre-play video. For a video currently playing, the electronic device 100 can also support cropping a specific video clip from the video. The specific process is similar to cropping a video clip from a pre-play video and will not be described in detail here. For a video currently playing, the electronic device 100 can also support capturing a specific video frame from the video, as described below with reference to Figures 11A and 11B.

[0256] Referring to FIG11A , assuming that the user wants to capture the frame at 00:30 in video 211 , the interactive object selected by the user can be the frame at 00:30 in video 211 . The specific process of the user selecting the interactive object and activating the natural language interaction function of the electronic device 100 can refer to the aforementioned description of FIG9A and FIG9B , which will not be repeated here. The electronic device 100 can detect the relevant voice command input by the user (e.g., the voice "capture this frame") and can display the relevant voice command input by the user in the form of text (e.g., text 212f "capture this frame").

[0257] Furthermore, referring to FIG11A , after the voice command is input, the electronic device 100 may analyze the voice command to determine that the operation corresponding to the voice command is to capture the frame at 00 minutes and 30 seconds in the video 211 .

[0258] Furthermore, the electronic device 100 may capture the frame at 00 minutes and 30 seconds in the video 211 through a relevant algorithm. After the capture is completed, as shown in FIG11B , the electronic device 100 may also save the captured video frame.

[0259] In some embodiments, when a user performs a long press operation on a video being played, the electronic device 100 can pause playing the video when the long press operation is detected, and the video screen displayed during the pause can be the video screen corresponding to the playback progress bar when the user performs the long press operation.

[0260] In other embodiments, for a video being played, the electronic device 100 may also allow the user to capture the video at any time point corresponding to the playback progress bar, rather than the video screen corresponding to the playback progress bar when the user currently performs a long press operation. For example, if the user inputs a voice command to instruct the electronic device 100 to capture a video screen at a certain time point, the electronic device 100 may capture the video screen at that time point based on the voice command.

[0261] In other embodiments, the electronic device 100 may also support the user to capture the video at any time point corresponding to the playback progress bar for a video before playback. For example, after the user performs a long press operation on the video before playback to trigger the electronic device 100 to start the natural language interaction function, the user may enter a voice command to instruct the electronic device 100 to capture the video at a certain time point. Further, after detecting the voice command, the electronic device 100 may capture the video at the corresponding time point based on the voice command.

[0262] It can be understood that the interactive method provided in the embodiments of the present application is not limited to the several corresponding operations performed on pictures and videos described above. More operations can be performed on pictures and videos. For example, as long as the operations on pictures and videos can be implemented through methods in the prior art (for example, by clicking on corresponding options in the menu bar, etc., to implement more operations on pictures and videos), the interactive method provided in the embodiments of the present application can also be implemented.

[0263] 2. Container Application Scenarios

[0264] "Container-type application scenarios" may refer to application scenarios in which the interactive method provided in an embodiment of the present application is used to perform corresponding operations (such as displaying unread message summaries, question-and-answer queries, etc.) on information displayed in list form (such as chat messages, etc.); it may also refer to application scenarios in which the interactive method provided in an embodiment of the present application is used to perform corresponding operations (such as displaying document summaries, displaying notes in documents, full-text translation, question-and-answer queries, format conversion, format beautification, etc.) on information displayed in grid form (such as documents, etc.).

[0265] 1. Perform corresponding operations on chat messages (or chat records)

[0266] (1) Display unread message summary

[0267] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users in viewing unread message summaries. In this way, when there are multiple unread messages, users can quickly obtain the main information to be conveyed by these multiple unread messages by viewing the unread message summaries. The user does not need to view the unread messages one by one, which is convenient and fast, simplifies user operations, and improves user experience.

[0268] Taking a social networking application as an example, referring to FIG12A , the exemplary user interface 310 shown in FIG12A may be a user interface of the social networking application. The user interface 310 may include one or more chat items (e.g., chat item 311, chat item 312, etc.). A user may click on a chat item to view one or more chat messages that may be included in the chat item.

[0269] In an embodiment of the present application, a user can trigger the electronic device 100 to start the natural language interaction function by performing a long press operation on a chat entry (for example, a long press operation performed by the user's finger or a touch device), and can also trigger the chat entry to be selected as an interaction object for subsequent natural language interaction by the user.

[0270] Continuing with FIG12A , taking group chat messages as an example, it can be seen that chat item 311 corresponds to group chat XXX. Chat item 311 indicates that group chat XXX contains multiple unread messages. Assuming the user wishes to quickly access the primary information conveyed by these multiple unread messages, electronic device 100 can detect a long press operation on chat item 311. In response to this long press operation, as shown in FIG12B , electronic device 100 can initiate a natural language interaction function and determine that the user-selected interaction object is chat item 311.

[0271] 12B , after the electronic device 100 starts the natural language interaction function, it may display a natural language interaction entry 313 . The natural language interaction entry 313 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0272] Optionally, a logo 313 a may be displayed on the natural language interaction entrance 313 , and the logo 313 a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0273] Optionally, a prompt 313b may be displayed on the natural language interaction portal 313. The prompt 313b may be used to prompt the user to enter a natural language instruction related to the interaction object. For example, as shown in FIG12B , the user has selected chat item 311 as the interaction object, and the prompt 313b may be used to prompt the user to enter a natural language instruction related to the chat item (for example, you can say "show unread message summary").

[0274] Optionally, the prompt 313b may also be output in the form of voice broadcast.

[0275] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0276] Referring to Figure 12C, taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "show unread message summary"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 313d "show unread message summary").

[0277] Optionally, after the electronic device 100 starts detecting the voice command input by the user, it can replace the logo 313a shown in Figure 12B with the logo 313c shown in Figure 12C. The logo 313c can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0278] Furthermore, referring to FIG. 12C , after the voice instruction is input, the electronic device 100 may further analyze the voice instruction to determine that the operation corresponding to the voice instruction is to display the unread message summary in the chat entry 311 .

[0279] Furthermore, the electronic device 100 may search for unread messages in the chat entry 311 and generate an unread message summary using a related algorithm. After generating the unread message summary, as shown in FIG12D , the electronic device 100 may display the generated unread message summary 314. Thus, the display function of the unread message summary is completed.

[0280] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above-mentioned unread message summary 314.

[0281] It should be noted that the voice instruction "display unread message summary" for realizing the display function of the unread message summary is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the display function of the unread message summary needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the unread message summary. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the viewing of the unread message summary (such as keywords such as "display", "view", "unread message", "new message", "summary", "outline", "brief description", "introduction", "what", etc.).

[0282] (2) Question and answer query

[0283] By implementing the interactive method provided in the embodiments of the present application, the electronic device 100 and the user can conduct question-and-answer queries regarding chat messages (for example, querying which contact in a certain chat entry sent what voice / text message, which contact sent what file, etc.). In this way, the user only needs to go to the chat entry display interface (for example, the user interface 310 shown in Figure 12A) to quickly obtain specific chat information, without the user having to enter the chat interface corresponding to the chat entry to view it. This is convenient and fast, simplifies user operations, and improves user experience.

[0284] Also taking the chat entry 311 as an example in which the interactive object selected by the user is shown as an example in Figure 12A, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 12A and 12B and will not be repeated here.

[0285] Taking the natural language instruction input by the user as a voice instruction as an example, referring to Figure 12E, assuming that the user wants to know what voice / text message Lily, a group member in the group chat XXX corresponding to the chat entry 311, sent in the group chat, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What Lily said") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 313e "What Lily said").

[0286] Further, referring to Figure 12E, after the above-mentioned voice instruction is input, the electronic device 100 can also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to display what voice / text message Lily, a group member in the group chat XXX corresponding to the chat entry 311, has sent in the group chat.

[0287] Furthermore, the electronic device 100 may query the voice / text messages sent by the group member Lily in the group chat XXX corresponding to the chat entry 311. After the query is completed, referring to FIG12F , the electronic device 100 may display the query result 315, which may include the voice / text messages sent by Lily in the group chat XXX that the electronic device 100 has queried.

[0288] It should be noted that the voice / text messages sent by Lily in the group chat XXX queried above may be all voice / text messages sent by Lily in the group chat XXX that have been read and unread by the user, or may be only the voice / text messages sent by Lily in the group chat XXX that have not been read by the user. This embodiment of the present application is not limited to this.

[0289] Optionally, for the voice message found in the above search, the electronic device 100 may convert the voice message into a text message and then display it in the search result, so that it is convenient for the user to view.

[0290] It is easy to understand that if the user wants to know other information sent by a certain contact, for example, what files a certain contact has sent, then the electronic device 100 can display the files sent by the contact in the above query result 315.

[0291] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above-mentioned query result 315.

[0292] It should be noted that the above-mentioned voice instruction "What did Lily say" for question-and-answer queries on chat messages is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the above-mentioned voice instruction for question-and-answer queries on chat messages needs to meet certain conditions, under which the electronic device 100 can determine which specific chat information in the chat message the user wants to view. Among them, the above-mentioned "certain conditions" may mean that the voice instruction includes certain keywords for indicating which specific chat information in the chat message to view (such as the contact's nickname, "say", "ask", "send", "what", "which" and other keywords).

[0293] It should be noted that social applications generally support virtual keyboard input. Therefore, in the embodiments of this application, the natural language instructions input by the user can be voice instructions or text instructions. The above description only uses the example of user inputting voice instructions. The specific process of user inputting text instructions can refer to the relevant content of the aforementioned display application scenario and will not be repeated here.

[0294] It should be noted that the above only uses the example of performing corresponding operations on chat messages to illustrate the interaction method provided in the embodiment of the present application. It is not limited to this. The interaction method provided in the embodiment of the present application can also perform corresponding operations on other types of messages (such as advertising, news, etc. push messages). The specific process is similar to the corresponding operations on chat messages, and will not be repeated here.

[0295] It should be noted that the above-mentioned corresponding operations on chat messages are only explained by taking the two operations of displaying unread message summaries and question-and-answer queries as examples. It is not limited to these two operations and can also be more operations. The embodiments of this application do not limit this.

[0296] 2. Perform corresponding operations on the document

[0297] (1) Display document summary

[0298] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to view document summaries without opening the documents. In this way, when it is necessary to obtain the main information of multiple documents, users can quickly obtain the main information that the documents want to convey without opening each document in turn. This is convenient, fast and efficient, simplifies user operations, and improves user experience.

[0299] For example, referring to FIG. 13A , the user interface 410 may include one or more document icons (eg, icon 411 of document 5 ), and the user may open the corresponding document by clicking on the document icon.

[0300] In an embodiment of the present application, a user can trigger the electronic device 100 to start the natural language interaction function by performing a long press operation on the document icon (for example, a long press operation performed by the user's finger or touch device), and can also trigger the document icon to be selected as an interactive object for subsequent natural language interaction by the user.

[0301] Continuing to refer to Figure 13A, assuming that the user wants to view the summary of document 5, the electronic device 100 can detect the user's long press operation on the icon 411 of document 5. In response to the long press operation, referring to Figure 13B, the electronic device 100 can start the natural language interaction function, and the electronic device 100 can also determine that the interaction object selected by the user is the icon 411 of document 5.

[0302] Continuing to refer to FIG. 13B , after the electronic device 100 starts the natural language interaction function, it may display a natural language interaction entrance 412 . The natural language interaction entrance 412 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0303] Optionally, a logo 412a may be displayed on the natural language interaction entrance 412, and the logo 412a may be used to prompt the user that the electronic device 100 has started the natural language interaction function.

[0304] Optionally, a prompt 412b may be displayed on the natural language interaction portal 412. The prompt 412b may be used to prompt the user to enter a natural language instruction related to the interaction object. For example, as shown in FIG13B , the user selects the icon 411 of document 5 as the interaction object. The prompt 412b may be used to prompt the user to enter a natural language instruction related to the document (for example, you can say "What document is this").

[0305] Optionally, the prompt 412b may also be output in the form of voice broadcast.

[0306] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0307] Referring to Figure 13C, taking the natural language instruction input by the user as a voice instruction as an example, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What document is this" or "Show the summary of the document", etc.), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 412d "What document is this" or "Show the summary of the document", etc.).

[0308] Optionally, after the electronic device 100 starts detecting the voice instructions input by the user, it can replace the logo 412a shown in Figure 13B with the logo 412c shown in Figure 13C. The logo 412c can be used to prompt the user that the electronic device 100 is detecting the voice instructions input by the user.

[0309] Furthermore, referring to FIG13C , after the voice command is input, the electronic device 100 may further analyze the voice command to determine that the operation corresponding to the voice command is to display the summary of the document 5 .

[0310] Furthermore, electronic device 100 may obtain the content of document 5 (e.g., title, body, etc.) and generate a summary of document 5 using a relevant algorithm. After generating the summary of document 5, as shown in FIG13D , electronic device 100 may display generated result 413, which may include the summary of document 5 and, optionally, the title of document 5. Thus, the document summary display function is completed.

[0311] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above result 413.

[0312] It should be noted that the voice instruction "What document is this" for realizing the display function of the document summary is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the display function of the document summary needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the document summary. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating the viewing of the document summary (such as keywords such as "display", "view", "what document", "summary", "outline", "brief description", and "introduction").

[0313] (2) Display notes in the document

[0314] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to view notes (such as marks, annotations, etc.) in a document without opening the document. In this way, when there are multiple documents, users can quickly view the notes in the document without opening each document in turn, which is convenient, fast and efficient, simplifies user operations and improves user experience.

[0315] Taking the example of the interactive object selected by the user as the icon 411 of the document 5 shown in Figure 13A, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 13A and 13B, and will not be repeated here.

[0316] Taking the natural language instruction input by the user as a voice instruction as an example, referring to Figure 13E, assuming that the user wants to view the notes in document 5, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "What is marked in the document") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 412e "What is marked in the document").

[0317] Furthermore, referring to FIG13E , after the voice instruction is input, the electronic device 100 may also analyze the voice instruction to determine that the operation corresponding to the voice instruction is to display the notes in the document 5 .

[0318] Furthermore, electronic device 100 can retrieve document 5 and search for notes within document 5. After the search is complete, as shown in FIG13F , electronic device 100 can display search results 414, which may include the searched notes within document 5. FIG13F shows that the notes within document 5 include highlights and annotations. This completes the display of notes within a document.

[0319] In some embodiments, the electronic device 100 may also save the query result 414, for example, the query result 414 may be saved in a memo, so that the user can view the memo in real time.

[0320] In some embodiments, the electronic device 100 may also support the user in adding notes to the document. For example, the user may input voice instructions related to adding notes, and the electronic device 100 may add corresponding notes to the document based on the voice instructions.

[0321] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above-mentioned query result 414.

[0322] It should be noted that the voice instruction "What is marked in the document" for realizing the function of displaying the notes in the document is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the function of displaying the notes in the document needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to view the notes in the document. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating viewing the notes in the document (such as keywords such as "mark", "mark", "annotate", "what", "note", etc.).

[0323] (3) Document content translation

[0324] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support the user to translate the content of a document and view the translated content without opening the document. In this way, when the user wants to translate the content of a document, the translation of the document content can be completed without the user opening the document and manually performing the translation operation. This is convenient, fast and efficient, simplifies user operations and improves user experience.

[0325] Taking the example of the interactive object selected by the user as the icon 411 of the document 5 shown in Figure 13A, the specific process of the user selecting the interactive object and starting the natural language interaction function of the electronic device 100 can refer to the aforementioned description of Figures 13A and 13B, and will not be repeated here.

[0326] Taking the natural language instruction input by the user as a voice instruction as an example, refer to Figure 13G. Assuming that the user wants to translate the content of document 5 into English, the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "Translate the full text into English") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 412f "Translate the full text into English").

[0327] Furthermore, referring to FIG13G , after the above-mentioned voice instruction is input, the electronic device 100 may also analyze the above-mentioned voice instruction to determine that the operation corresponding to the above-mentioned voice instruction is to translate the full text of the document 5 into English and display the full text of the translated document 5.

[0328] Furthermore, the electronic device 100 can obtain the full text of the document 5 and translate the full text into English. After the translation is completed, referring to FIG13H , the electronic device 100 can display the translation result 415, which can include the full text of the document 5 translated into English. Thus, the document content translation function is implemented.

[0329] In some embodiments, the electronic device 100 may also save the translation result 415 as a new document, so that the user can view it in real time.

[0330] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style or a normal text style, etc.) and display position of the above-mentioned translation result 415.

[0331] It should be noted that the voice instruction "Translate the full text into English" for implementing the document content translation function is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for implementing the document content translation function needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to translate the document content. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating that the document is to be translated / converted into, "translation", "conversion" and other keywords.

[0332] (4) Document format conversion / beautification

[0333] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to convert or beautify the format of a document without opening the document (for example, beautifying the document format into a format that is easy for users to read and has both pictures and text), which is convenient and fast, simplifies user operations, and improves user experience.

[0334] Among them, the specific process of converting or beautifying the format of the document is similar to the specific process of displaying the document summary, displaying the notes in the document, and translating the document content. The only difference is the voice command input by the user, which will not be repeated here.

[0335] It is easy to understand that when a user wants to convert the format or beautify the format of a document, the user can input voice instructions related to format conversion or beautification (for example, "convert the document format to XX format", etc.), and the electronic device 100 can convert the format or beautify the format of the document based on the voice instruction. After completing the format conversion or beautification, the electronic device 100 can display the document after format conversion or beautification for the user to view.

[0336] (5) Question and answer query

[0337] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 and the user can conduct question-and-answer queries on documents (for example, querying whether a certain document includes certain content, etc.). In this way, the user can quickly obtain relevant information about the document without opening the document, which is convenient and fast, simplifies user operations, and improves user experience.

[0338] Among them, the specific process of conducting question-and-answer queries on documents is similar to the specific process of displaying document summaries, displaying notes in documents, and translating document content. The only difference is the voice commands input by the user, which will not be repeated here.

[0339] It is easy to understand that when a user wants to conduct a question-and-answer query on a document, the user can input a voice command related to the question-and-answer query (for example, "Does this document include XXX content?", etc.), and the electronic device 100 can obtain the corresponding document based on the voice command and query the relevant information of the document indicated by the voice command. After the query is completed, the electronic device 100 can display the query results for the user to view.

[0340] It should be noted that, if the application providing user interface 410 supports virtual keyboard input, the natural language instructions input by the user can be not only voice instructions but also text instructions. The above description only uses the example of user inputting voice instructions. The specific process of user inputting text instructions can be referred to the relevant content in the aforementioned display application scenario and will not be repeated here.

[0341] It should be noted that the above-mentioned corresponding operations on documents are only illustrated by five operations: displaying document summaries, displaying notes in documents, translating document content, converting / beautifying document formats, and question-and-answer queries. These operations are not limited to these five operations and can also be more operations. The embodiments of this application do not limit this.

[0342] 3. Operational Application Scenarios

[0343] "Operational application scenarios" may refer to application scenarios in which the interactive method provided in the embodiments of the present application is used to perform corresponding operations (such as querying, collecting, editing, displaying summaries, etc.) on text (such as ordinary text, or special text such as links, names, numbers, email addresses, dates, addresses, etc.).

[0344] 1. Text Collection

[0345] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to quickly collect selected text or selected text and its context, realize the quick text collection function, which is convenient and fast, simplifies user operations, and improves user experience.

[0346] In an embodiment of the present application, a user can trigger the electronic device 100 to select the corresponding text and display the menu bar corresponding to the text by performing a long press operation on the text (for example, a long press operation performed by the user's finger or a touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0347] Taking a reading application as an example, referring to FIG14A , the user interface 510 exemplarily shown in FIG14A may be a user interface of the reading application, and the user interface 510 may include one or more text paragraphs (eg, text paragraph 511 ).

[0348] Continuing to refer to Figure 14A, assuming that the user wants to collect text, the electronic device 100 can detect the user's long press operation on a certain position of the text paragraph 511 (for example, the position shown in Figure 14A). In response to the long press operation, referring to Figure 14B, the electronic device 100 can select the corresponding text (for example, the text "X"). The electronic device 100 can also display a menu bar 513. The menu bar 513 may include one or more options (for example, a "search" option, a "copy" option, a "select all" option, a "translate" option, etc.). These one or more options can be used to perform corresponding operations on the selected text. The electronic device 100 can also start the natural language interaction function.

[0349] It should be noted that the range of text selected by the electronic device 100 through the long press operation can be set according to preset rules. For example, the text selected by the electronic device 100 can be only the text at the location where the long press operation is located, or it can be text in a larger range than the location where the long press operation is located. The embodiments of the present application do not limit this.

[0350] Continuing to refer to FIG. 14B , after the electronic device 100 starts the natural language interaction function, it may display a natural language interaction entrance 512 . The natural language interaction entrance 512 may be used to prompt the user that the electronic device 100 has started the natural language interaction function, and may also be used for subsequent natural language interaction.

[0351] Optionally, a logo 512a may be displayed on the natural language interaction entrance 512, and the logo 512a may be used to prompt the user that the electronic device 100 has activated the natural language interaction function.

[0352] Optionally, a prompt 512b may be displayed on the natural language interaction entrance 512, and the prompt 512b may be used to prompt the user to input natural language instructions related to the interactive object (such as the selected text or the selected text and its context), such as the prompt 512b shown in Figure 14B (for example, you can say "Collect this paragraph", "Collect this sentence", "Collect this page" or "Collect this chapter").

[0353] Optionally, the prompt 512b may also be output in the form of voice broadcast.

[0354] Among them, the timing when the electronic device 100 starts detecting the natural language instructions input by the user and the timing when it ends detecting the natural language instructions input by the user can refer to the relevant content in the display application scenario, which will not be repeated here.

[0355] Referring to Figure 14C, taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to collect text paragraph 511, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "Collect this paragraph") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 512d "Collect this paragraph").

[0356] Optionally, after the electronic device 100 starts detecting the voice command input by the user, it can replace the logo 512a shown in Figure 14B with the logo 512c shown in Figure 14C. The logo 512c can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user.

[0357] Furthermore, referring to FIG14C , after the voice command is input, the electronic device 100 may analyze the voice command to determine that the operation corresponding to the voice command is to collect the text segment (i.e., text segment 511) where the selected text is located.

[0358] Furthermore, the electronic device 100 can obtain the text segment (i.e., text paragraph 511) where the selected text is located through relevant text detection and recognition algorithms. After the acquisition is completed, referring to FIG14D , the electronic device 100 can display a window 513, which can include the acquired text segment to be collected (i.e., text segment 511). Furthermore, the electronic device 100 can collect the acquired text segment to be collected, referring to FIG14E , the electronic device 100 can also display a window 514, which can include information indicating the collection path (e.g., "Collect to Memo") and the acquired text segment to be collected. At this point, the text collection function is completed.

[0359] It is easy to understand that the above-mentioned collection path can be a system preset of the electronic device 100, or can be set by the user independently, and the embodiment of the present application does not limit this.

[0360] In some embodiments, the above-mentioned window 513 and window 514 may not be displayed.

[0361] It is easy to understand that the above-mentioned text detection and recognition algorithm can be one or more, and the embodiment of the present application does not limit the specific type of the algorithm. Exemplarily, the above-mentioned text detection and recognition algorithm can be, for example, an optical character recognition (OCR) algorithm. In the embodiment of the present application, the electronic device 100 can give priority to selecting the text currently selected by the user as the interactive object. Furthermore, the scope of the interactive object can be expanded in combination with the semantics of the user instruction (for example, the text selected by the user is only a part of the text in a certain text paragraph, and the semantics of the user instruction is the entire text paragraph, then the interactive object can be expanded from the above-mentioned part of the text to the entire text paragraph). That is to say, if the electronic device 100 determines that the interactive object is only the selected text based on the user instruction and the text currently selected by the user, then the electronic device 100 can perform the operation corresponding to the above instruction on the selected text; if the electronic device 100 determines that the interactive object includes not only the text currently selected by the user but also its context based on the user instruction and the text currently selected by the user, then the electronic device 100 can detect and identify the context of the selected text. For example, the interactive object is the entire text paragraph where the text currently selected by the user is located. Then, the electronic device 100 can determine the above text paragraph through the paragraph recognition rules in the OCR algorithm (such as blank lines / paragraph identifiers of the text, text alignment, specific identifiers, font styles, etc.). It is easy to understand that the above is only explained by taking the recognition of a paragraph of text as an example. Recognizing a sentence of text, a page of text, a chapter of text, etc. is similar to the above recognition of a paragraph of text, and will not be repeated here.

[0362] It should be noted that the voice instruction "Collect this paragraph" for realizing the text collection function is merely exemplary and should not constitute a limitation of the present application. It is easy to understand that the voice instruction for realizing the text collection function needs to meet certain conditions, under which the electronic device 100 can determine the user's intention to collect text. Among them, the above-mentioned "certain conditions" may refer to the voice instruction including certain keywords for indicating text collection (such as "collect", "this paragraph", "this sentence", "this page", "this chapter", "this article" and other keywords).

[0363] 2. Text query

[0364] By implementing the interactive method provided in the embodiment of the present application, the electronic device 100 can support users to quickly query information related to the selected text, realize the quick text query function, which is convenient and fast, simplifies user operations, and improves user experience.

[0365] Among them, the specific process of the user selecting text and starting the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection. Please refer to the aforementioned description of Figures 14A-14B and will not be repeated here.

[0366] Referring to Figure 14F, assuming that the text selected by the user is "XXX Books", the electronic device 100 can recognize that the text is the name of a book. Therefore, after the electronic device 100 starts the natural language interaction function, the electronic device 100 can optionally display a prompt 512e (for example, "You can ask this for more information") on the natural language interaction entrance 512. The prompt 512e can be used to prompt the user to enter natural language instructions related to the selected text.

[0367] Taking the natural language instruction input by the user as a voice instruction as an example, referring to Figure 14G, assuming that the user wants to know the introduction of the book "XXX Book" that has been selected above, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "show the introduction of this book") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 512f "show the introduction of this book").

[0368] Furthermore, referring to FIG14G , after the voice command is input, the electronic device 100 may analyze the voice command to determine that the operation corresponding to the voice command is to display the introduction of the book corresponding to the text “XXX Books” selected by the user.

[0369] Furthermore, the electronic device 100 can query locally or in the cloud for an introduction to the book corresponding to the text "XXX Books" selected by the user. After the query is completed, referring to FIG14H , the electronic device 100 can display the query result 515, which can include an introduction to the book corresponding to the text "XXX Books" selected by the user. Thus, the text query function is implemented.

[0370] It should be noted that the embodiment of the present application does not limit the display style (for example, it can be a card style, it can also be a normal text style, etc.) and display position of the above-mentioned query result 515.

[0371] It should be noted that the above only uses the example of the text selected by the user being text related to the book title to illustrate the text query function. It is not limited to this, and other texts (such as text related to audio and video names, article names, game names, application names, etc.) can also be used. The embodiments of this application do not limit this.

[0372] It should be noted that the above description only uses a certain reading application as an example to illustrate the text collection and text query functions, and is not limited to this. It can also be other applications that can support text display and text selection. The embodiments of this application do not limit this.

[0373] It should be noted that the above only uses ordinary text as an example to illustrate the process of performing corresponding operations on text using the interactive method provided in the embodiment of the present application. It is not limited to ordinary text. The interactive method provided in the embodiment of the present application is also applicable to performing related operations on special text (such as links, names, numbers, email addresses, dates, addresses, etc.). For example, for a link, by selecting the link and entering the relevant natural language command, a summary of the specific content corresponding to the link can be displayed; for another example, for a name / number (such as a mobile phone number, a personal account number, etc.), by selecting the name / number and entering the relevant natural language command, a communication social (such as adding the name / number to a contact in the address book, etc.) / order tracking can be performed; for another example, for an email address, by selecting the email address and entering the relevant natural language command, an email can be sent; for another example, for a date, by selecting the date and entering the relevant natural language command, a reminder / alarm can be set; for another example, for an address, by entering the address and the relevant natural language command, the navigation / ticketing / weather information corresponding to the address can be queried. It is easy to understand that, similar to the above scenario of performing corresponding operations on ordinary text, in the scenario of performing corresponding operations on special text, the electronic device 100 can also display prompts related to the special text, such as prompts 512b and prompts 512e.

[0374] It should be noted that, if the application providing the user interface 510 supports virtual keyboard input, the natural language instructions input by the user can be not only voice instructions but also text instructions. The above description only uses the example of user inputting voice instructions. The specific process of user inputting text instructions can be referred to the relevant content in the aforementioned display application scenario and will not be repeated here.

[0375] 3. Text Editing

[0376] (1) Text content modification (such as content insertion, deletion, replacement, etc.)

[0377] In order to modify the text content through the natural language interaction function, the user is currently required to input natural language instructions that can accurately express the user's modification intention and the modified text. Usually, the instruction needs to include the text to be modified, the modification method, and the modified target text. For example, the text before modification is "Today I am in Beijing", and the modified text is "Today I am in Shanghai". Then, the natural language instruction input by the user can be, for example, "Change Beijing to Shanghai". It can be seen that the instruction includes "Beijing", the text to be modified, "Change to", a keyword used to describe the modification method, and "Shanghai", the modified target text. That is to say, the above-mentioned relatively complex and cumbersome instructions need to be input to complete the text content modification.

[0378] By implementing the interactive method provided in the embodiments of the present application, the electronic device 100 can support the user inputting instructions that only include the modified target text, and the electronic device 100 can automatically generate one or more semantically coherent modification results based on the instruction and the original text, thereby achieving the purpose of completing the text content modification in a simple and natural manner, which is convenient and fast, and improves the user experience. The following is a detailed description with reference to the user interface schematic diagram.

[0379] For text content modification, common modification requirements include the following three types, which are introduced in detail below.

[0380] 1) Iterative optimization of expression (for example, the text before modification is "I want to make a plan", and the text after modification is "I want to make a systematic fitness plan")

[0381] In an embodiment of the present application, the user can trigger the electronic device 100 to select the corresponding text by performing a long press operation on the text that needs to be modified (for example, a long press operation performed by the user's finger or touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0382] Among them, the specific process of the user selecting the text to be modified and starting the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection. Please refer to the aforementioned description of Figures 14A-14B and will not be repeated here.

[0383] Regarding the modification requirement of "iterative optimization of expression", one or more semantically coherent modification results may be generated during the modification process, which are described in detail below.

[0384] a. Generate multiple semantically coherent modification results

[0385] Taking an application that supports text content modification (such as a memo application) as an example, referring to Figure 15A, the user interface 610 can be a user interface of this application. The user interface 610 can include the text that needs to be modified (for example, the text "The weather is very good today, suitable for outdoor activities."), and can also include a natural language interaction entrance 611 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0386] Among them, the text that needs to be modified selected by the user may include only the text to be modified (for example, the text "The weather is very good today"), or it may include both the text to be modified (for example, the text "The weather is very good today") and the context of the text to be modified (for example, the text "suitable for outdoor activities"). The embodiments of the present application do not limit this.

[0387] Continuing to refer to Figure 15A, taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to add text content related to expressing "Today's weather is better than yesterday's" in the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "better than yesterday"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 611a "better than yesterday").

[0388] Further, continuing to refer to Figure 15A, taking the example of the text selected by the user to be modified as "The weather is very good today", after the above voice instruction is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice instruction (for example, the voice "better than yesterday"), so as to determine that the operation corresponding to the above voice instruction is to add the text "better than yesterday" to the text selected by the user to be modified "The weather is very good today" to generate a semantically coherent modification result.

[0389] Furthermore, the electronic device 100 can generate semantically coherent modification results through relevant algorithms. After the modification results are generated, referring to FIG15B , the electronic device 100 may generate and display multiple semantically coherent modification results, such as modification result 612 “The weather today is better than yesterday” and modification result 613 “The weather today is very good, better than yesterday”. In this case, the electronic device 100 can support the user to select one of the modification results.

[0390] Optionally, the electronic device 100 may prompt the user to select one of the modification results by voice announcement or text prompt (eg, prompt 614 “Which modification result do you want to select”).

[0391] In one possible implementation, the user can select one of the modification results by inputting a voice command. Referring to FIG15C , the user can input a voice-related voice command (e.g., the voice "first"), and the electronic device 100 can display the relevant voice command input by the detected user in the form of text (e.g., text 611b "first"). Further, after the above-mentioned voice command is input, the electronic device 100 can analyze the above-mentioned voice command to determine that the operation corresponding to the above-mentioned voice command is to modify the text before modification according to the first modification result (e.g., modification result 612). Further, referring to FIG15D , the electronic device 100 can modify the text before modification according to the modification result 612 and display the modified text (e.g., the text "Today's weather is better than yesterday's and suitable for outdoor activities.").

[0392] In another possible implementation, referring to Figure 15B, the user can also select one of the modification results by clicking. For example, if the user wants to select modification result 612, the user can click on modification result 612. After the electronic device 100 detects the click operation, it can modify the text before modification according to modification result 612 and display the modified text (for example, the text "Today's weather is better than yesterday's and suitable for outdoor activities.").

[0393] b. Generate a semantically coherent modification result

[0394] Taking an application that supports text content modification (such as a memo application) as an example, referring to Figure 15E, the user interface 610 can be a user interface of this application. The user interface 610 can include text that needs to be modified (such as the text "I want to make a plan"). The user interface 610 can also include a natural language interaction entrance 615 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0395] Continuing to refer to Figure 15E, taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to add text content related to expressing "systematic fitness plan" in the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "systematic fitness"), and can display the detected relevant voice instruction input by the user in the form of text (for example, text 615a "systematic fitness").

[0396] Further, referring to Figure 15E , taking the example of the text selected by the user to be modified as “I want to make a plan”, after the above voice instruction is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice instruction (for example, the voice “systematic fitness”), thereby determining that the operation corresponding to the above voice instruction is to add the text “systematic fitness” to the text selected by the user to be modified “I want to make a plan” to generate a semantically coherent modification result.

[0397] Furthermore, the electronic device 100 can generate a semantically coherent modification result through a relevant algorithm. After the modification result is generated, referring to FIG15F , the electronic device 100 may only generate one semantically coherent modification result. In this case, the electronic device 100 can directly modify the text before the modification according to the generated semantically coherent modification result and display the modified text (for example, “I want to make a systematic fitness plan”).

[0398] Alternatively, referring to FIG15G , electronic device 100 may first display a semantically coherent modification result (e.g., modification result 616 "I want to develop a systematic fitness plan") before displaying the modified text shown in FIG15F . Electronic device 100 then displays the modified text shown in FIG15F after detecting that the user has confirmed the modification through a voice command or a click operation on modification result 616. This gives the user more options. If the user is not satisfied with the modification result, they can re-enter the relevant instructions to generate a new modification result.

[0399] 2) Automatic speech recognition (ASR) errors caused by noise and accents (for example, the text before modification is "Play me a song with high-quality wallpaper", and the text after modification is "Play me a song with rare characters")

[0400] In an embodiment of the present application, the user can trigger the electronic device 100 to select the corresponding text by performing a long press operation on the text that needs to be modified (for example, a long press operation performed by the user's finger or touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function.

[0401] Among them, the specific process of the user selecting the text to be modified and starting the natural language interaction function of the electronic device 100 is similar to the aforementioned text collection. Please refer to the aforementioned description of Figures 14A-14B and will not be repeated here.

[0402] Taking an application that supports text content modification (such as a memo application) as an example, referring to Figure 15H, the user interface 610 can be a user interface of this application. The user interface 610 can include text that needs to be modified (such as the text "A song was played today"). The user interface 610 can also include a natural language interaction entrance 617 to prompt the user that the electronic device 100 has started the natural language interaction function.

[0403] Continuing to refer to Figure 15H, taking the natural language instruction input by the user as a voice instruction as an example, assuming that the user wants to replace "upgrade wallpaper" with "uncommon characters" in the text that needs to be modified, then the electronic device 100 can detect the relevant voice instruction input by the user (for example, the voice "uncommon characters") and can display the detected relevant voice instruction input by the user in the form of text (for example, text 617a "uncommon characters").

[0404] Further, referring to Figure 15H, taking the example of the text selected by the user to be modified as "A song that promotes wallpaper was played today", after the above voice instruction is input, the electronic device 100 can analyze the text selected by the user to be modified and the above voice instruction (for example, the voice of "uncommon characters"), so as to determine that the operation corresponding to the above voice instruction is to replace "promoting wallpaper" with "uncommon characters" in the text selected by the user to be modified as "A song that promotes wallpaper was played today".

[0405] Furthermore, the electronic device 100 can generate a semantically coherent modification result through a relevant algorithm. After the modification result is generated, referring to FIG15I , the electronic device 100 may only generate one semantically coherent modification result. In this case, the electronic device 100 can directly modify the text before the modification according to the generated semantically coherent modification result and display the modified text (for example, “A song with rare characters was played today”).

[0406] Optionally, when the electronic device 100 generates one or more semantically coherent modification results, the electronic device 100 may first display the generated one or more semantically coherent modification results (e.g., modification result 616 shown in FIG15J ) before displaying the modified text shown in FIG15I . The electronic device 100 may then display the modified text after detecting that the user has confirmed the selected modification result through a voice command or a click operation on a modification result. In this way, the user can be given more options. If the user is not satisfied with the modification result, he or she can re-enter the relevant command to generate a new modification result.

[0407] The above voice instructions only take "uncommon characters" as an example. In some embodiments, for homophones or characters with similar pronunciations, in order to avoid errors, the above voice instructions can be more specific voice instructions. For example, the same instruction is used to indicate the intention of adding "uncommon characters". The above voice instruction "uncommon characters" can be a more specific voice instruction, such as "uncommon characters, the "sheng" in life, the "remote" in remote, and the "zi" in font".

[0408] 3) Slips of the tongue and interjections generated during natural language input (for example, the text before modification is "I heard that it will rain tomorrow", and the text after modification is "I heard that it will rain tomorrow")

[0409] During natural language input, users' spoken expressions often produce interjections and slips of the tongue. Despite these interjections and slips of the tongue, the original meaning of the input sentence is fully preserved. Therefore, the electronic device 100 can identify and correct colloquial errors using technologies such as large language models (LLMs). These large language models are deep learning models trained using large amounts of text data that can generate natural language text or understand its meaning. These large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversation.

[0410] In an embodiment of the present application, the user can trigger the electronic device 100 to select the corresponding text by performing a long press operation on the text that needs to be modified (for example, a long press operation performed by the user's finger or touch device). In addition, the electronic device 100 can also be triggered to start the natural language interaction function, thereby automatically identifying and correcting spoken errors.

[0411] The following describes in detail the specific process of the electronic device 100 automatically identifying and correcting spoken errors, focusing on interjections and slips of the tongue in the text that need to be modified.

[0412] a. Tone words in the text that need to be modified

[0413] Similarly, taking an application program that supports text content modification (such as a memo application program) as an example, referring to FIG. 15K, the user interface 610 can be a user interface of the application program. The text to be modified (such as the text "Today, um, the weather is very good") can be included in the user interface 610. It can be seen that the text contains the filler word "um".

[0414] Continuing to refer to FIG. 15K, assuming that the user wants to delete the filler word (such as "um") in the text to be modified, then, the electronic device 100 can detect the operation of the user selecting the text to be modified (such as a long press operation). In response to this operation, the electronic device 100 can select the corresponding text and can automatically identify and delete the filler word in the text through a related algorithm, thereby generating the modification result 615 "Today the weather is very good". Here, the display form of the modification result 615 in the embodiments of the present application is not limited. For example, it can be displayed in the natural language interaction entry shown in FIG. 15K.

[0415] It can be seen that in the embodiments of the present application, without the user inputting a natural language instruction, the electronic device 100 can automatically identify and delete the filler word in the text selected by the user, which is convenient and fast, and can provide the user with an intelligent experience.

[0416] In a possible implementation manner, referring to FIG. 15L, after the electronic device 100 generates the above modification result 615, it can directly modify the text before modification according to the modification result and display the modified text (such as "Today the weather is very good").

[0417] In another possible implementation manner, the electronic device 100 can also display the modified text shown in FIG. 15L after detecting that the user confirms the modification through a voice instruction or a click operation on the modification result 615.

[0418] b. A slip of the tongue in the text to be modified

[0419] Similarly, taking an application program that supports text content modification (such as a memo application program) as an example, referring to FIG. 15M, the user interface 610 can be a user interface of the application program. The text to be modified (such as the text "I'm on a business trip to Beijing today. No, it's Shanghai. I'm having some discussions with several university teachers.") can be included in the user interface 610. It can be seen that there is a slip of the tongue in the text, that is, the city where I'm on a business trip today is Shanghai, not Beijing.

[0420] Continuing with FIG15M , assuming that a user wants to correct a slip of the tongue in a text that needs to be modified, electronic device 100 can detect the user's selection of the text to be modified (e.g., a long press). In response to this operation, electronic device 100 can select the corresponding text and automatically identify and correct the slip of the tongue in the text using a relevant algorithm, thereby generating a modification result 616, "I am going on a business trip to Shanghai today." The embodiment of the present application does not limit the display format of modification result 616; for example, it can be displayed in the natural language interaction portal shown in FIG15M .

[0421] It can be seen that in the embodiment of the present application, the electronic device 100 can automatically identify and correct verbal errors in the text selected by the user without the user having to input natural language instructions. This is convenient and fast, and can provide the user with an intelligent experience.

[0422] In one possible implementation, referring to FIG. 15N , after generating the above-mentioned modification result 616 , the electronic device 100 can directly modify the text before modification according to the modification result and display the modified text (for example, “I am on a business trip to Shanghai today, and I am doing some discussions with several university teachers.”).

[0423] In another possible implementation, the electronic device 100 may also display the modified text shown in FIG15N after detecting that the user confirms the modification through a voice instruction or a click operation on the modification result 616 .

[0424] In some embodiments, during the natural language input process, the electronic device 100 can automatically identify and correct spoken errors. That is, the text generated by natural language input is the text after the spoken errors are corrected. In this way, there is no need for subsequent users to actively select text and modify the text content, which simplifies user operations and improves text input efficiency.

[0425] It is easy to understand that the above text content modification through the natural language interaction function can be implemented by the electronic device 100 based on an input method application (such as the Xiaoyi input method application).

[0426] In some embodiments, the interactive method provided in the embodiments of the present application is not limited to the above-mentioned text content modification. It can also realize text format modification through the natural language interaction function. It is easy to understand that it is similar to the above-mentioned text content modification process, except that the natural language instructions input by the user are different. In the text format modification scenario, the user can input a natural language instruction for indicating text format modification, wherein the above-mentioned text format may include but is not limited to one or more of the following: font, font size, color, alignment, style (title / text), bold, italic, and underline.

[0427] (2) Add comments, display comments, and edit comments in the document

[0428] By implementing the interaction method provided in the embodiments of the present application, when the electronic device 100 opens a certain document and enters the annotation mode, the electronic device 100 can support the user to trigger the electronic device 100 to start the natural language interaction function through a long press operation (for example, a long press operation performed by the user's finger or touch device), and can add annotations to the document based on the natural language interaction function, edit annotations, etc., which is convenient and fast, simplifies user operations, and improves user experience.

[0429] The specific process of adding annotations to a document will be first exemplarily introduced below with reference to FIG. 16A to FIG. 16G .

[0430] 16A , the user interface 710 may be a user interface when the electronic device 100 opens a certain document (eg, a document 711 regarding an annual work summary) and enters an annotation mode.

[0431] In an embodiment of the present application, assuming that the user wants to add a comment to the current page of the document, the electronic device 100 can detect the user's long press operation on the current page of the document 711. In response to the long press operation, the electronic device 100 can start the natural language interaction function and can identify whether there is an interactive object / subject (such as an image, text, etc.) at the location of the long press operation in the current page of the document 711. If not, the electronic device 100 can generate an annotation based on the voice instruction input by the user, and can display the generated annotation at the location of the long press operation; if so, the electronic device 100 can not only generate an annotation based on the voice instruction input by the user, but also highlight the interactive object and display the generated annotation at other locations other than the location of the long press operation (such as the center position of the interactive object). The two cases are introduced below.

[0432] Exemplarily, continue to refer to FIG. 16A. Assume that there is no interaction object at the location where the user performs a long-press operation. For example, the location where the long-press operation is performed as shown in FIG. 16A is blank, that is, there is no interaction object. Then, after the electronic device 100 activates the natural language interaction function, referring to FIG. 16B, the electronic device 100 can display an annotation box 712 at the location where the above long-press operation is performed, and can also display an identifier 712b. The identifier 712b can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user. Assume that the annotation that the user wants to add is "It would be good to add a subtitle here". Then, the electronic device 100 can detect the relevant voice command input by the user (such as the voice "It would be good to add a subtitle here"), and can display the detected relevant voice command input by the user in text form (such as the text 712a "It would be good to add a subtitle here"). Further, after the above voice command is input, if the electronic device 100 detects that the user cancels the long-press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then the electronic device 100 can generate an annotation based on the above voice command. After generating the annotation, referring to FIG. 16C, the electronic device 100 can display the generated annotation (such as annotation 713) at the location where the above long-press operation is performed.

[0433] Exemplarily, refer to FIG. 16D. Assume that there is an interaction object at the location where the user performs a long-press operation. For example, there is the character "year" at the location where the long-press operation is performed as shown in FIG. 16D, that is, there is an interaction object. Then, after the electronic device 100 activates the natural language interaction function, referring to FIG. 16E, the electronic device 100 can display an annotation box 714 at the location where the above long-press operation is performed, and can also display an identifier 714b. The identifier 714b can be used to prompt the user that the electronic device 100 is detecting the voice command input by the user. Assume that the annotation that the user wants to add is "Enlarge the font size of the title". Then, the electronic device 100 can detect the relevant voice command input by the user (such as the voice "Enlarge the font size of the title"), and can display the detected relevant voice command input by the user in text form (such as the text 714a "Enlarge the font size of the title"). Further, after the above voice command is input, if the electronic device 100 detects that the user cancels the long-press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then the electronic device 100 can generate an annotation based on the above voice command. After generating the annotation, referring to FIG. 16F, it can be seen that the electronic device 100 can not only display the generated annotation (such as displaying annotation 715 at the center position of the character "Annual Work Summary"), but can also highlight the interaction object corresponding to the annotation (such as the character "Annual Work Summary").

[0434] It should be noted that the interactive object can be the object at the location of the long press operation, or it can be the object expanded from the object at the location of the long press operation, and this embodiment of the application does not limit this. Referring to Figure 16D, for example, if the object at the location of the long press operation is the text "year", the expanded object can be the text "Annual Work Summary". It can be seen that the embodiments shown in Figures 16D-16F above use the expanded object "Annual Work Summary" as the interactive object.

[0435] In an embodiment of the present application, after the electronic device 100 starts the natural language interaction function in response to the user's long press operation, if the electronic device 100 does not detect the user inputting a voice command within a period of time, and detects that the long press operation is canceled (for example, the user's finger or touch device leaves the screen of the electronic device 100), then, referring to Figure 16G, the electronic device 100 can call up the virtual keyboard and display the comment box 712, and the user can enter text instructions through the virtual keyboard to add comments in the comment box 712.

[0436] In some embodiments, the generated annotation can be displayed in a pierceable floating layer above the application, so as not to affect the user's operation on the application page content.

[0437] In an embodiment of the present application, after an annotation is added to a document, if the document's page content changes (e.g., due to page switching, scrolling, moving, zooming in, or reducing operations), the electronic device 100, after determining that the page content is stably displayed (e.g., after a preset period of time during which the page switching, scrolling, moving, zooming in, or reducing operations have ceased), if the currently displayed page content corresponds to the added annotation, the electronic device 100 may further determine whether to display the annotation based on preset rules. This is described in detail below.

[0438] In the aforementioned process of adding annotations, the electronic device 100 can record the annotation screenshot image (which may include the page content and the annotated image) and the coordinate information of the annotation display position. After the annotation is added and the page content changes, the electronic device 100 can obtain the historical annotation screenshot image and analyze the historical annotation screenshot image. If the currently displayed page content is similar to the page content in a historical annotation screenshot image, then the electronic device 100 can find the annotation corresponding to the page content, and can determine the coordinate information of the updated display position of the annotation based on the display position and display ratio information of the page content and its corresponding annotation in the historical annotation screenshot image, and the display position and display ratio information of the currently displayed page content. Afterwards, the electronic device 100 can display the annotation at the corresponding position on the screen based on the coordinate information of the updated display position of the annotation; if the currently displayed page content is not similar to the page content in one or more historical annotation screenshot images, or the currently displayed page content is similar to the page content in multiple historical annotation screenshot images, then the electronic device 100 may not display the annotation.

[0439] It is easy to understand that whether the currently displayed page content is similar to the page content in the historical annotation screenshot image can be measured using statistical knowledge in mathematics. For example, it can be measured using the "confidence" indicator. If the confidence is higher than a certain preset threshold, it can be said that the currently displayed page content is similar to the page content in a historical annotation screenshot image. If the confidence is lower than a certain preset threshold, it can be said that the currently displayed page content is not similar to the page content in a historical annotation screenshot image.

[0440] For example, referring to FIG16H , taking the currently displayed page content as “Summary” as an example, assuming that the electronic device 100 determines that the currently displayed page content is similar to the page content in a certain historical annotation screenshot image (for example, the page content “Annual Work Summary” shown in FIG16C ), and taking the above-mentioned historical annotation screenshot image as the annotation screenshot image shown in FIG16C as an example, then the electronic device 100 can display the annotation 713 shown in FIG16C in FIG16H , “It is better to add a subtitle here.”

[0441] It is easy to see that the page content shown in Figure 16H may be the page content after the enlargement of Figure 16C (for example, the page content in Figure 16H only enlarges and displays the "Summary" in the "Annual Work Summary" shown in Figure 16C).

[0442] For example, continuing to refer to Figure 16H, taking the currently displayed page content as "Summary" as an example, it is easy to understand that if the page content of one or more pages in the document other than the page shown in Figure 16C also includes the content of "Summary" and corresponding annotations are added, then the electronic device 100 can determine that the currently displayed page content is similar to the page content in multiple historical annotation screenshots, then the electronic device 100 can no longer display the annotations in Figure 16H.

[0443] It is easy to understand that if the electronic device 100 determines that the page content currently displayed in Figure 16H is not similar to the page content in one or more historical annotation screenshots, then the electronic device 100 may no longer display the annotation in Figure 16H.

[0444] In an embodiment of the present application, in the two cases where annotations are not displayed, the electronic device 100 can also support the user to view the annotations. For example, the electronic device 100 can display option 716a in the window 716 shown in Figure 16H. After the electronic device 100 detects the user's operation (such as a click operation) on option 716a, it can display part or all of the annotations added to the document in the form of a window or card for the user to view. In some embodiments, if the currently displayed page content has a corresponding annotation added, the user can also click on the annotation displayed in the form of a window or card to trigger the electronic device 100 to display the annotation as shown in Figure 16H.

[0445] In the embodiment of the present application, after adding an annotation to a document, the electronic device 100 can also support the user to perform editing operations on the annotation (such as deleting the annotation, moving the annotation, modifying the annotation content, etc.), which will be described in detail below.

[0446] In an embodiment of the present application, the electronic device 100 may support the user to trigger the electronic device 100 to delete or move an annotation through a single-click operation.

[0447] For example, referring to FIG16I , taking annotation 713 as an example, electronic device 100 may detect a user's single-click operation on annotation 713. In response to this single-click operation, referring to FIG16J , electronic device 100 may display option 717, which may be used to delete annotation 713. In this case, annotation 713 is in an editable state. If the user wishes to delete annotation 713, electronic device 100 may detect a user's single-click operation on option 717. In response to this single-click operation, electronic device 100 may delete and undisplay annotation 713. Continuing to refer to FIG16J , if the user wishes to move annotation 713, electronic device 100 may detect a user's long press and drag operation on annotation 713 to a certain location. In response to this operation, electronic device 100 may display annotation 713 at that location, and no longer display annotation 713 at the location before the move.

[0448] In some embodiments, the user may trigger the electronic device 100 to cancel displaying the above-mentioned option 717 by clicking on a location other than the annotation 713, so that the annotation 713 may no longer be in the above-mentioned editable state.

[0449] It should be noted that if the annotation is displayed on a pierceable floating layer above the application, in order to prevent the editing operation on the annotation from affecting the application page content below the layer, the above-mentioned single-click operation can be a specific single-click operation, such as a two-finger single-click operation.

[0450] In an embodiment of the present application, the electronic device 100 can support the user to trigger the electronic device 100 to modify the annotation content or insert a new annotation through a long press operation and input of natural language instructions (such as voice instructions, text instructions).

[0451] Exemplarily, referring to FIG. 16K, the currently displayed page of the document 711 shown in FIG. 16K includes only one annotation 713. After the electronic device 100 detects a long - press operation by the user on the currently displayed page of the document 711, in response to this long - press operation, the electronic device 100 can activate the natural - language interaction function, and can receive the natural - language instruction input by the user through the activated natural - language interaction function. The electronic device 100 can also obtain the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713, and can determine whether the distance d between the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713 is less than or equal to a preset distance threshold d1. If so (i.e., d <= d1), the electronic device 100 can replace the annotation content of the annotation 713 based on the above - input natural - language instruction, thus completing the modification of the annotation content; if not (i.e., d > d1), the electronic device 100 can further determine whether the distance d between the coordinates of the position where the long - press operation is located and the coordinates of the annotation 713 is less than or equal to a preset distance threshold d2. If so (i.e., d1 < d <= d2), the electronic device 100 can prompt the user, by means of voice broadcast or interface display, whether to insert a new annotation or replace the annotation content of the annotation 713 based on the above - input natural - language instruction, and can perform the corresponding operation based on the user's selection; if not (i.e., d > d2), the electronic device 100 can insert a new annotation based on the above - input natural - language instruction.

[0452] In some embodiments, in the case where the currently displayed page includes multiple annotations, the electronic device 100 can obtain the annotation closest to the position where the long - press operation is located for comparison.

[0453] The following exemplarily introduces the specific process of the electronic device 100 modifying the annotation content or inserting a new annotation in combination with the user interface.

[0454] Continuing to refer to FIG. 16K, assuming that the distance d between the coordinates of the position where the user long - presses and the coordinates of the annotation 713 is less than or equal to a preset distance threshold d1 (i.e., d <= d1), then, referring to FIG. 16L, the electronic device 100 can detect the relevant voice command input by the user (such as the voice "Add a cover"), and can display the detected relevant voice command input by the user in text form (such as the text 718 "Add a cover"). Further, after the above - mentioned voice command is input, if the electronic device 100 detects that the user cancels the long - press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then, the electronic device 100 can replace the annotation content of the annotation 713 based on the above - mentioned voice command. After the replacement is completed, referring to FIG. 16M, the electronic device 100 can display the replaced annotation. It can be seen that the annotation content of the annotation 713 is replaced from the original "Add a subtitle here" to "Add a cover". Thus, the modification of the annotation content is completed.

[0455] Continuing to refer to FIG. 16K, assuming that the distance d between the coordinates of the position where the user long - presses and the coordinates of the annotation 713 is greater than a preset distance threshold d1 and less than or equal to a preset distance threshold d2 (i.e., d1 < d <= d2), then, referring to FIG. 16N, the electronic device 100 can detect the relevant voice command input by the user (such as the voice "Add a cover"), and can display the detected relevant voice command input by the user in text form (such as the text 718 "Add a cover"). Further, after the above - mentioned voice command is input, if the electronic device 100 detects that the user cancels the long - press operation (such as the operation of the user's finger or touch device leaving the screen of the electronic device 100), then, the electronic device 100 can also display option 719 and option 720. Among them, option 719 can be used to prompt the user to insert a new annotation, and option 720 can be used to prompt the user to replace the annotation content of the annotation 713. If the electronic device 100 detects that the user selects to insert a new annotation through a voice command or a touch operation (such as a click operation on option 719), then, referring to FIG. 16O, the electronic device 100 can insert a new annotation (such as annotation 721) at the corresponding position (such as the position where the above - mentioned long - press operation is located). If the electronic device 100 detects that the user selects to replace the annotation through a voice command or a touch operation (such as a click operation on option 720), then, referring to FIG. 16M, the electronic device 100 can replace the annotation content of the annotation 713 from the original "Add a subtitle here" to "Add a cover" and display the replaced annotation.

[0456] In the above-mentioned annotation editing scenario, after the electronic device 100 starts the natural language interaction function in response to the user's long press operation, if the electronic device 100 does not detect the user inputting a voice instruction within a period of time, and detects that the user cancels the long press operation (for example, the user's finger or touch device leaves the screen of the electronic device 100), then, as shown in Figure 16G in the above-mentioned annotation adding scenario, the electronic device 100 can also call up the virtual keyboard to support the user to enter text instructions through the virtual keyboard. After the input is completed, the electronic device 100 can complete the annotation editing based on the input text instruction. It is easy to understand that completing the annotation editing based on the input text instruction is just a different way of inputting the instruction. The other processes are the same as the above-mentioned process of completing the annotation editing based on the voice instruction, which will not be repeated here.

[0457] FIG17 exemplarily shows a specific process of an interaction method provided in an embodiment of the present application.

[0458] As shown in FIG17 , the method can be applied to the electronic device 100. The specific steps of the method are described in detail below:

[0459] S1701. The electronic device 100 detects a user operation (eg, a long press operation). In response to the user operation, the electronic device 100 activates a natural language interaction function.

[0460] Among them, the above-mentioned user operation can be a touch operation of the user on the screen of the electronic device 100. In the embodiment of the present application, the touch operation can trigger the electronic device 100 to start the natural language interaction function and display the natural language interaction entrance, so that the electronic device 100 can subsequently detect the natural language instructions (such as voice instructions or text instructions) input by the user through the natural language interaction function, and can display the natural language instructions at the natural language interaction entrance. The touch operation can also trigger the electronic device 100 to select an interactive object, so that the corresponding operation can be performed on the interactive object based on the above-mentioned input natural language instructions.

[0461] In the embodiments of the present application, the above user operation is merely an example of a long press operation (e.g., a long press operation on a picture, video, text, chat item, document icon, document page, etc.), and should not constitute a limitation of the present application. In other embodiments of the present application, the above user operation may also be other operations.

[0462] Among them, the specific process of triggering the electronic device 100 to start the natural language interaction function and display the natural language interaction entrance and select the interaction object through the above-mentioned user operation (such as a long press operation) can refer to the relevant text description in the aforementioned user interface embodiment and will not be repeated here.

[0463] S1702. The electronic device 100 detects that the user inputs a natural language instruction, and determines an interaction object based on the natural language instruction and the user operation.

[0464] Specifically, after starting the natural language interaction function, the electronic device 100 can use the natural language interaction function to detect the natural language instructions input by the user. After the natural language instructions are input, the electronic device 100 can determine whether there is a default object in the natural language instruction (that is, the natural language instruction does not include the interactive object) or a pronoun (that is, the natural language instruction includes words such as this, it, etc. used to refer to the interactive object). If so (that is, the natural language instruction does not include keywords used to indicate the interactive object), the electronic device 100 can extract and identify the interactive object at the location of the above-mentioned user operation (such as a long press operation) or the interface content at and near the location, thereby determining the interactive object. If not (that is, the natural language instruction includes keywords used to indicate the interactive object), the electronic device 100 can execute the following step S1703.

[0465] In some embodiments, when there is no default object in the natural language instruction (ie, the natural language instruction includes an interactive object), the electronic device 100 may perform corresponding operations on the interactive object according to the interactive object included in the natural language instruction.

[0466] For example, assuming that the natural language instruction is "brighten the picture", it can be seen that the interactive object included in the natural language instruction is "picture", then the electronic device 100 can increase the brightness of the picture through a relevant algorithm.

[0467] In other embodiments, when the interactive object included in the natural language instruction is different from the interactive object selected by the electronic device 100 triggered by the above-mentioned user operation (such as a long press operation), the electronic device 100 can perform corresponding operations on the interactive object according to the interactive object included in the natural language instruction.

[0468] For example, assuming that the natural language instruction is "brighten the picture", it can be seen that the interactive object included in the natural language instruction is "picture", and the interactive object selected by the electronic device 100 triggered by the above user operation is the cutout subject in the picture (such as a flower). In this case, the electronic device 100 can use a relevant algorithm to increase the overall brightness of the picture, rather than just increasing the brightness of the cutout subject in the picture. That is to say, the interactive object determined by the electronic device 100 is the above-mentioned entire picture, rather than the cutout subject in the picture.

[0469] In other embodiments, when there is a default object or pronoun in the natural language instruction, the electronic device 100 may prioritize the smallest unit object (such as a control, a cutout body, a text segmentation, etc.) that can be identified at the location of the above-mentioned user operation (such as a long press operation) as the interactive object, and then the electronic device 100 may determine whether to expand the scope of the interactive object based on the above-mentioned natural language instruction. If so, the scope of the interactive object may be expanded to determine a new interactive object, and the operation corresponding to the above-mentioned natural language instruction may be performed on the new interactive object.

[0470] For example, assuming that the natural language instruction is "erase", it can be seen that there is a default object in the natural language instruction. Then, the electronic device 100 can use the smallest unit object that can be identified at the location of the above-mentioned user operation as the interactive object, such as using the cutout subject in a certain picture as the interactive object. Further, the electronic device 100 can erase the cutout subject in the picture through relevant algorithms.

[0471] For example, assuming that the natural language instruction is "erase this", it can be seen that the natural language instruction includes the pronoun "this", then the electronic device 100 can also use the smallest unit object that can be identified at the location of the above user operation as the interactive object, such as using the cutout subject in a certain picture as the interactive object. Furthermore, the electronic device 100 can erase the cutout subject in the picture through relevant algorithms.

[0472] For example, assuming that the natural language instruction is "copy this paragraph to the memo", it can be seen that the natural language instruction includes the pronoun "this paragraph". Assuming that the interactive object selected by the electronic device 100 triggered by the above user operation is a part of the text in a certain text paragraph (that is, the selected text), then the electronic device 100 can determine the scope of the interactive object to be expanded based on the "this paragraph" in the above natural language instruction. Furthermore, the electronic device 100 can identify the context of the above selected text through relevant algorithms (such as text detection and recognition algorithms such as OCR), thereby obtaining the entire text paragraph where the above selected text is located. Further, the electronic device 100 can determine that the new interactive object is the entire text paragraph where the above selected text is located, and can copy the entire text paragraph where the above selected text is located to the memo.

[0473] S1703. The electronic device 100 responds to the natural language instruction based on the interactive object.

[0474] Specifically, after determining the interactive object, the electronic device 100 can analyze and understand the natural language instructions input by the user in combination with the interactive object. After analysis and understanding, the electronic device 100 can respond to the natural language instructions, that is, perform the operation corresponding to the natural language instruction for the interactive object. For example, corresponding operations (such as sending, editing, etc.) are performed on display information such as pictures and videos; for another example, corresponding operations are performed on information displayed in list form (such as chat messages, etc.) (such as displaying unread message summaries, question and answer queries, etc.), and corresponding operations are performed on information displayed in grid form (such as documents, etc.) (such as displaying document summaries, displaying notes in documents, full text translation, question and answer queries, format conversion, format beautification, etc.); for another example, corresponding operations are performed on text (such as ordinary text, or special text such as links, names, numbers, email addresses, dates, addresses, etc.) (such as query, collection, editing, display summary, etc.); for another example, operations such as adding annotations to documents, displaying annotations, editing annotations, etc.; and so on.

[0475] Among them, the specific process of the electronic device 100 responding to the natural language instructions input by the user can refer to the relevant text description in the aforementioned user interface embodiment, which will not be repeated here.

[0476] The following describes a software structure of an electronic device 100 provided in an embodiment of the present application.

[0477] FIG18 exemplarily shows a software structure of an electronic device 100 provided in an embodiment of the present application.

[0478] As shown in Figure 18, the software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0479] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: application layer, application framework layer, system library layer, and kernel layer, from top to bottom.

[0480] The application layer can include a series of application packages.

[0481] As shown in Figure 18, the application package may include camera, gallery, calendar, call, map, voice assistant, WLAN, Bluetooth, music, video, short message, input method (not shown in the figure), memo (not shown in the figure), mailbox (not shown in the figure), Changlian (not shown in the figure, i.e. a social application) and other applications.

[0482] Among them, the voice assistant can be an application with voice / text command recognition function. For example, after detecting the user's voice input, voice recognition technology, natural language processing technology, etc. can be used to identify the user's intention and respond in a targeted manner (such as voice answering, starting an application, changing device configuration, etc.). Among them, the electronic device 100 can always turn on the voice assistant, or start the voice assistant after detecting a user operation (such as long pressing an interface element / button, etc.), or start the voice assistant after detecting a wake-up word, which is not limited here.

[0483] The above-mentioned voice / text instructions may refer to voice / text that controls the electronic device 100 to perform one or more operations.

[0484] It should be noted that the name of the voice assistant application is only a term used in the embodiment of this application. The meaning it represents has been recorded in the embodiment of this application, and its name does not constitute any limitation to the embodiment of this application.

[0485] In an embodiment of the present application, a user's touch operation on the screen of the electronic device 100 (e.g., a long press operation on a picture, video, text, chat entry, document icon, document page, etc. displayed on the screen of the electronic device 100) can trigger the electronic device 100 to start a natural language interaction function, so that the electronic device 100 can subsequently detect the natural language instruction (e.g., voice instruction or text instruction) input by the user through the natural language interaction function. In addition, the above-mentioned touch operation can also trigger the electronic device 100 to select an interactive object, so that the corresponding operation can be performed on the interactive object based on the natural language instruction input by the user. For details, please refer to the relevant text descriptions in the aforementioned embodiments, which will not be repeated here.

[0486] In an embodiment of the present application, the natural language interaction function initiated by the electronic device 100 may be a system capability provided by the electronic device 100. In some embodiments, the system capability may be implemented based on a voice assistant application or an input method application.

[0487] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0488] As shown in FIG18 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0489] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0490] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0491] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0492] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0493] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0494] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0495] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0496] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0497] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0498] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0499] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0500] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0501] A 2D graphics engine is a drawing engine for 2D drawings.

[0502] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, Bluetooth driver, and sensor driver.

[0503] In addition to the example shown in Figure 18, the solution provided in this application can also be applied to other types of operating systems, such as Hongmeng operating system, Apple operating system, Windows operating system, etc.

[0504] The following is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0505] FIG19 exemplarily shows the structure of an electronic device 100 provided in an embodiment of the present application.

[0506] As shown in Figure 19, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0507] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0508] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0509] In the embodiments of the present application, the processor 110 may be configured to respond to user operations (e.g., a long press) to select an interactive object, initiate a natural language interaction function, and analyze, understand, and execute natural language commands input by the user, etc. For details, please refer to the relevant descriptions of the aforementioned embodiments and will not be repeated here.

[0510] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0511] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0512] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0513] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.

[0514] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0515] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0516] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0517] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0518] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0519] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 and to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect to other terminal devices, such as AR devices.

[0520] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0521] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device 100 via the power management module 141.

[0522] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0523] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0524] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0525] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0526] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0527] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0528] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0529] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0530] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0531] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0532] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and other factors. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0533] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0534] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0535] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0536] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0537] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0538] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0539] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0540] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0541] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0542] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0543] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0544] In the embodiment of the present application, the electronic device 100 can convert the sound signal into an electrical signal through the microphone 170C. For example, when the electronic device 100 receives a voice command input by the user, the user can make a sound by putting his mouth close to the microphone 170C so that the sound is input into the microphone 170C.

[0545] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0546] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0547] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0548] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0549] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0550] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0551] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the electronic device 100 by inserting or removing it from the SIM card interface 195. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0552] It should be understood that the electronic device 100 shown in FIG19 is merely an example, and the electronic device 100 may have more or fewer components than shown in FIG19, may combine two or more components, or may have a different component configuration. The various components shown in FIG19 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0553] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0554] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0555] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interaction method, applied to an electronic device, wherein the electronic device includes a screen, characterized in that: The method comprises: The electronic device displays a first user interface of a first application; The electronic device detects a first operation of the user on the first user interface, where a position of the first operation on the screen is a first position; In response to the first operation, the electronic device starts a natural language interaction function, where the natural language interaction function is used for the electronic device to interact with the user through natural language instructions; The electronic device detects that a user inputs a first natural language instruction, and determines, based on the first natural language instruction and the first position, that the natural language interaction object is a first interaction object, wherein the first interaction object is included in the first user interface; The electronic device executes an operation corresponding to the first natural language instruction on the first interactive object.

2. The method according to claim 1, characterized in that The first operation is a long press operation.

3. The method according to claim 1 or 2, characterized in that: The electronic device determines, based on the first natural language instruction and the first position, that the natural language interaction object is a first interaction object, specifically including: In a case where the first natural language instruction does not include a first keyword, the first keyword is a keyword used to indicate the natural language interaction object, and the electronic device determines that the first interaction object is an object that can be identified at the first position on the first user interface; or, In a case where the first natural language instruction includes the first keyword, the electronic device determines that the natural language interaction object indicated by the first keyword is the first interaction object.

4. The method according to any one of claims 1 to 3, characterized in that: After the electronic device starts the natural language interaction function, the method further includes: The electronic device displays a natural language interaction portal, and the natural language interaction portal is used to display the first natural language instruction.

5. The method according to claim 4, characterized in that The natural language interaction entrance is located at the first position or near the first position.

6. The method according to any one of claims 1 to 5, characterized in that: After the electronic device starts the natural language interaction function, the method further includes: If the electronic device does not detect that the user inputs a natural language instruction within a first preset time period, the electronic device outputs a first prompt, where the first prompt is used to prompt the user to input a recognizable natural language instruction corresponding to the first interactive object.

7. The method according to any one of claims 4 to 6, characterized in that: The first natural language instruction includes any one of a first voice instruction or a first text instruction, and the method further includes: The electronic device detects a second operation of the user on the natural language interaction portal; The electronic device detects that a user inputs a first natural language instruction, specifically comprising: In response to the second operation, the electronic device detects the first voice instruction input by the user through the microphone, or the electronic device calls up and displays a virtual keyboard and detects the first text instruction input by the user through the virtual keyboard.

8. The method according to any one of claims 1 to 6, characterized in that: After the electronic device detects that the user inputs a first natural language instruction, the method further includes: If the electronic device detects that the user cancels the first operation, the electronic device no longer detects the natural language instructions input by the user.

9. The method according to any one of claims 4 to 8, characterized in that: The method further comprises: When the electronic device detects that the user long presses and drags the first interactive object, the electronic device reduces the display of the natural language interactive object. entrance, or cancel displaying the natural language interaction entrance; When the electronic device detects that the first interactive object does not move again within a second preset time period, the electronic device resumes displaying the natural language interaction entrance.

10. The method according to any one of claims 1 to 9, characterized in that: The first natural language instruction does not include keywords that directly indicate the location or name of the interactive object.

11. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first picture or a first content in the first picture, the first content is a portion of the content in the first picture, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to send the first interactive object to a second application, the electronic device sends the first interactive object from the first application to the second application; or, When the first natural language instruction is used to indicate an erasing intention, the electronic device erases the first content from the first picture; or, In a case where the first natural language instruction is used to indicate an intention to replace the first content with a second content, the electronic device replaces the first content with the second content; or, When the first natural language instruction is used to indicate an intention to insert a third content, the electronic device inserts the third content into the first picture; or, In a case where the first natural language instruction is used to indicate a recognition intention, the electronic device recognizes the first interactive object and displays a recognition result; or, In the case where the first natural language instruction is used to indicate a beautification intention, the electronic device beautifies the first interactive object, wherein the beautification intention includes one or more of the following intentions: beautification, body beautification, blurring, mosaic, graffiti, and adding a frame / sticker / marker; or, In a case where the first natural language instruction is used to indicate an intention to adjust parameters, the electronic device performs parameter adjustment on the first interactive object, wherein the parameters include one or more of the following: filter style, brightness, saturation, and sharpness.

12. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first video, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate a sending intention, the electronic device sends the first video from the first application to a second application; or, In case that the first natural language instruction is used to indicate a cropping intention, the electronic device crops the first video and saves the cropped first video segment or first video frame.

13. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is the first chat item, the first chat item includes a chat message, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to display a summary of an unread message included in the chat message, the electronic device searches for the unread message, and generates and displays a summary of the unread message based on the unread message; or, In a case where the first natural language instruction is used to indicate an intention to query relevant information of the chat message, the electronic device queries and displays the relevant information, wherein the relevant information includes any one of the following: content of the chat message, sender of the chat message.

14. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is an icon of a first document, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to display a content summary of the first document, the electronic device obtains the content of the first document, and generates and displays a summary of the content based on the content; or, In the case where the first natural language instruction is used to indicate an intention to display the notes in the first document, the electronic device queries and displays the notes in the first document, wherein the notes include one or more of the following: marks, annotations; or, In a case where the first natural language instruction is used to indicate an intention to translate the content of the first document, the electronic device translates the content of the first document and displays a translation result; or, In a case where the first natural language instruction is used to indicate an intention to perform format conversion / beautification on the first document, the electronic device performs format conversion / beautification on the first document, and generates and displays the first document after format conversion / beautification.

15. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a first text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to collect the first text, the electronic device collects the first text; or, In a case where the first natural language instruction is used to indicate an intention to query the first text, the electronic device queries the first text and displays a query result, wherein the query result includes relevant information of the first text.

16. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a second text, the first natural language instruction includes a third text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: The electronic device combines the second text and the third text into one or more fourth texts; The electronic device replaces the second text with the fourth text and displays the fourth text.

17. The method according to claim 16, characterized in that In the case that there are multiple fourth texts, before the electronic device replaces the second text with the fourth text, the method further includes: The electronic device outputs a second prompt, where the second prompt is used to prompt the user to select any one of the multiple fourth texts.

18. The method according to any one of claims 1 to 10, characterized in that: The first interactive object is a fifth text, and the electronic device performs an operation corresponding to the first natural language instruction on the first interactive object, specifically including: In a case where the first natural language instruction is used to indicate an intention to modify a text format of the fifth text, the electronic device modifies the text format of the fifth text; The text format includes one or more of the following: font, font size, color, alignment, title style, body style, bold, italic, and underline.

19. An interactive method, applied to an electronic device, characterized in that: The method comprises: The electronic device displays a first user interface, wherein the first user interface includes a first page of a first document; The electronic device detects a first operation of a user on the first page; In response to the first operation, the electronic device starts a natural language interaction function, where the natural language interaction function is used for the electronic device to interact with the user through natural language instructions; The electronic device detects that a user inputs a first natural language instruction; The electronic device generates a first batch of annotations based on the first natural language instruction, and displays the first batch of annotations on the first user interface.

20. The method according to claim 19, characterized in that The first operation is a long press operation.

21. The method according to claim 19 or 20, characterized in that The position of the first operation on the screen of the electronic device is a first position, and the method further includes: The electronic device identifies whether there is an interactive object at the first position. If not, the electronic device displays the first batch of annotations at the first position. If so, the electronic device displays the first batch of annotations at a second position, which is the same as or different from the first position.

22. The method according to claim 21, characterized in that The second position is a center position of the interactive object.

23. The method according to claim 21 or 22, characterized in that The method further comprises: The electronic device adds a mark to the interactive object and displays the interactive object after the mark is added, wherein the mark includes: a highlight mark.

24. The method according to any one of claims 19 to 23, characterized in that: The method further comprises: The electronic device detects a second operation of the user on the first page; In response to the second operation, the electronic device starts the natural language interaction function and determines the distance between the location of the second operation and the location of the first batch of annotations; The electronic device detects that a user inputs a second natural language instruction, and generates a second annotation based on the second natural language instruction; When the distance is less than or equal to a first preset distance threshold, the electronic device replaces the first annotation with the second annotation and displays the second annotation on the first user interface; or, When the distance is greater than a second preset distance threshold, the electronic device adds and displays the second annotation on the first user interface; or, When the distance is greater than the first preset distance threshold and less than or equal to the second preset distance threshold, the electronic device replaces the first annotation with the second annotation based on the user's selection operation and displays the second annotation on the first user interface, or adds and displays the second annotation on the first user interface.

25. The method according to any one of claims 19 to 24, characterized in that: The method further comprises: The electronic device detects a third operation of the user on the first batch of bets; In response to the third operation, the electronic device displays a first option, where the first option is used to delete the first batch of bets; The electronic device detects a fourth operation of the user on the first option, and in response to the fourth operation, the electronic device deletes and cancels display of the first batch of annotations; or, The electronic device detects a fifth operation of a user long pressing and dragging the first batch of annotations to a third position, and in response to the fifth operation, the electronic device displays the first batch of annotations at the third position.

26. The method according to claim 24 or 25, characterized in that The second operation is a long press operation, and the third operation is a single click operation.

27. The method according to any one of claims 19 to 26, characterized in that: The method further comprises: The electronic device detects a sixth operation of the user, where the sixth operation includes any one of the following: switching pages, scrolling pages, zooming in on a page, and zooming out on a page; In response to the sixth operation, the electronic device displays first content on the first user interface, where the first content is content in the first document, and displays annotations related to the first content.

28. The method according to any one of claims 19 to 27, characterized in that: The first natural language instruction includes any one of a first voice instruction or a first text instruction, and the electronic device detects that a user inputs the first natural language instruction, specifically including: The electronic device detects the first voice instruction input by the user through a microphone; or, The electronic device calls up and displays a virtual keyboard, and detects the first text instruction input by a user through the virtual keyboard.

29. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in any one of claims 1-18 or 19-28.

30. A computer storage medium, characterized in that The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 18 or 19 to 28.

Images