Interaction method and apparatus
By implementing the interactive method of automatic search and cursor style transformation on electronic devices, the cumbersome problem of users searching for multimedia information when using electronic devices is solved, and the simplicity of operation and experience quality are improved.
Patent Information
- Application Number
- PCT/CN2024/079098
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2024-02-28
- Publication Date
- 2024-12-05
AI Technical Summary
When users use electronic devices, when calling multimedia information such as pictures and videos, they need to search manually, which is cumbersome and has poor user experience.
An interactive method and device are provided to automatically search for matching media information through text input by the user, simplify user operations and improve user experience. The method includes receiving user operations, obtaining text, searching for media information matching the text semantics, and indicating search results through cursor style transformation, allowing the user to intuitively understand and select the required media information.
No user input search keywords and trigger search commands, simplifying operational processes, improving user experience, and enhancing intuitive understanding of search results and convenience of selection.
Smart Images

Figure CN2024079098_05122024_PF_FP_ABST
Abstract
Description
Interaction method and device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 2, 2023, with application number 202310654308.7 and application name “A Method and Device for Interaction”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of terminal technology, and in particular to an interaction method and device. Background Art
[0004] With the rapid development of mobile internet technology, users are increasingly accessing multimedia information such as images and videos stored on electronic devices (such as mobile phones and tablets). For example, when editing a note in a memo app, if a user needs to insert a related image into the text they are entering, they typically need to open the gallery app, manually search for the target image, and then insert the target image into the text through user operations. This is cumbersome and results in a poor user experience.
[0005] Summary of the Invention
[0006] The embodiments of the present application provide an interactive method and apparatus that can be applied to electronic devices to automatically search for matching media information based on content input by a user, thereby simplifying user operations and improving user experience.
[0007] In a first aspect, an interaction method is provided, which can be applied to electronic devices, such as mobile phones, tablet computers, wearable devices, etc. The method includes: the electronic device receives a first user operation, and obtains a first text in response to the first user operation; wherein the first user operation is used to input text, and the first text includes the text input through the first user operation; the electronic device searches for media information that matches the semantics of the first text based on the first text; the electronic device transforms the first style cursor into a second style cursor, and the second style cursor indicates that the media information has been searched; the electronic device receives a second user operation on the second style cursor, and displays the searched media information in response to the second user operation.
[0008] Optionally, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0009] In this implementation, matching media information can be automatically searched based on the text entered by the user, eliminating the need for the user to enter search keywords or trigger a function button to execute the search command, thereby simplifying user operations and improving the user experience. Furthermore, by indicating that matching media information has been found by changing the cursor, the user can understand the search status and can then perform subsequent operations based on the search results.
[0010] In a possible implementation, the second-style cursor indicates the type and / or content of the searched media information, which allows the user to intuitively understand the type and / or content of the searched media information and improves the user experience.
[0011] In one possible implementation, the second style cursor includes one or more of the following: a thumbnail of the content of the media information; information indicating the media type, the information indicating the media type including an icon for indicating the type of media information, and / or text for indicating the type of media information.
[0012] In one possible implementation, the media information includes pictures, and the second-style cursor displays thumbnails of the searched pictures, thereby more intuitively indicating the search status to the user. Optionally, the thumbnail is a thumbnail of the picture with the highest degree of match with the first text among the searched pictures. In another possible implementation, the media information includes videos, and the second-style cursor displays thumbnails of video frames in the searched videos, thereby more intuitively indicating the search status to the user. Optionally, the thumbnail is a thumbnail of the video frame in the video with the highest degree of match with the first text among the searched videos.
[0013] In a possible implementation, the second-style cursor indicates that there are multiple types of searched media information.
[0014] In one possible implementation, when the second-style cursor indicates multiple types of searched media information, receiving a second user operation on the second-style cursor and displaying the media information includes: receiving a third user operation, and in response to the third user operation, displaying a first menu, wherein the first menu includes multiple function options, each of the multiple function options corresponding to one of the multiple media types; and receiving a user selection operation of at least one function option from the multiple function options, and in response to the selection operation, displaying the media information corresponding to the at least one function option. Optionally, the third user operation includes clicking or long pressing the second-style cursor.
[0015] Through the above implementation, when the second-style cursor indicates that there are multiple types of searched media information, the user can view all of the multiple types based on user operations, thereby allowing the user to select one or more types of media information to view as needed.
[0016] Optionally, the second user operation is used to trigger one of the multiple function options, and in response to the second user operation, media information of a type corresponding to the function option among the multiple types of media information searched is displayed.
[0017] Optionally, the second user operation is used to trigger at least two of the multiple function options, and in response to the second user operation, at least two types of media information corresponding to the function option among the multiple types of media information searched are displayed.
[0018] In one possible implementation, the first user operation acts on a text input box of a first application; after displaying media information that semantically matches the first text, the method further includes: receiving a fourth user operation, and in response to the fourth user operation, selecting the first media information from the displayed media information, and applying the first media information to the first application; wherein the fourth user operation is used to select the first media information from the displayed media information, and apply the first media information to the first application.
[0019] Optionally, applying the first media information to the first application includes: adding the first media information to a record edited using the first application (such as a memo application); or adding the first media information as an attachment to an email edited using the first application (such as a mail application); or sending the first media information to a target user who is conducting a conversation using the first application (such as a social application).
[0020] In a possible implementation, after the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the method further includes: highlighting the first text so that the user can understand the text content that matches the current search result.
[0021] Optionally, after transforming the first style cursor into the second style cursor, or after receiving the second user operation, it also includes: receiving a fifth user operation, the fifth user operation being used to move the cursor to a first position, the first position being within the range of the first text; and in response to the fifth user operation, moving the second style cursor to the first position.
[0022] Optionally, after converting the first-style cursor to the second-style cursor and before receiving the second user operation, the method further includes: receiving a sixth user operation, unhighlighting a portion of the highlighted first text in response to the sixth user operation; and searching for media information that semantically matches the second text based on the second text, where the second text is the text highlighted in response to the sixth user operation. This operation can reduce search keywords based on user needs, thereby obtaining search results that meet the user's needs.
[0023] Optionally, after converting the first-style cursor to the second-style cursor and before receiving the second user operation, the method further includes: receiving a seventh user operation, and in response to the seventh user operation, highlighting a third text other than the first text; and searching for media information that semantically matches the fourth text based on a fourth text, the fourth text including the first text and the third text. Through this operation, search keywords can be added as needed by the user, thereby obtaining search results that meet the user's needs.
[0024] In a possible implementation, after searching for media information matching the first text or receiving the second user operation, the method further includes: displaying search keywords.
[0025] Optionally, the method further includes: receiving an eighth user operation for updating the search keyword and requesting a search for matching media information based on the updated search keyword; in response to the eighth user operation, searching for matching media information based on the updated search keyword; and displaying the media information found based on the updated search keyword. This implementation allows for media information to be searched based on user-specified keywords, ensuring that search results meet user needs.
[0026] In a possible implementation, obtaining the first text includes obtaining text input through the first user operation and a context associated with the text.
[0027] Optionally, the first text may include text input through the first user operation, and may also include previously input text. In the context of an interactive application, the text input through the first user operation is chat content (text) sent to a target user, and the context associated with the text includes the chat content (text) contained in a chat message from the target user.
[0028] In one possible implementation, the searching for matching media information based on the first text includes one or more of the following: searching for media information matching the first text in the electronic device; searching for media information matching the first text in other electronic devices interconnected with the electronic device; searching for media information matching the first text in the cloud space of the electronic device.
[0029] In a second aspect, an interaction method is provided, which can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, etc. The method includes: receiving a first user operation, where the first user operation is used to trigger a first function option in a user interface, and the first function option corresponds to a first media information type; obtaining a context associated with the first user operation and / or the first function option; searching for matching media information based on the semantics of the context and the first media information type corresponding to the first function option; and displaying the searched media information.
[0030] Optionally, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0031] In the above implementation, matching media information can be automatically searched according to the first function option triggered by the user, without the user having to input search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience.
[0032] In one possible implementation, obtaining the context associated with the first user operation includes: obtaining text input by the user before the first user operation and / or text contained in a message received by the electronic device, and using the text as the context.
[0033] In a possible implementation, obtaining the context associated with the first function option includes: obtaining first information associated with the first function option, where the first information is used to describe media information corresponding to the first function option, and using the first information as the context.
[0034] In one possible implementation, the search for matching media information includes one or more of the following: searching for matching media information in the electronic device; searching for matching media information in other electronic devices interconnected with the electronic device; and searching for matching media information in the cloud space of the electronic device.
[0035] According to a third aspect, a device is provided, comprising a unit or module for executing the method as described in any one of the first aspects above.
[0036] In one possible implementation, the device includes a processing unit and an interaction unit. The processing unit is located in a processor, and the interaction unit may include a touch unit or an audio collector, etc., which have the function of sensing user operations. The interaction unit is used to receive a first user operation, and the processing unit is used to obtain a first text in response to the first user operation, search for media information that matches the semantics of the first text based on the first text, and transform the first style cursor into a second style cursor, where the second style cursor indicates that the media information has been searched, wherein the first user operation is used to input text, and the first text includes the text input through the first user operation; the interaction unit is also used to receive a second user operation on the second style cursor, and the processing unit is also used to display the media information in response to the second user operation.
[0037] In a possible implementation, the second-style cursor indicates the type and / or content of the searched media information.
[0038] In one possible implementation, the second style cursor includes one or more of the following: a thumbnail of the content of the media information; information indicating the media type, the information indicating the media type including an icon for indicating the type of media information, and / or text for indicating the type of media information.
[0039] Optionally, the media information includes pictures, and the second-style cursor displays thumbnails of the searched pictures; or, the media information includes videos, and the second-style cursor displays thumbnails of video frames in the searched videos.
[0040] In one possible implementation, the second-style cursor indicates that media information of multiple media types has been searched. Optionally, the interaction unit is specifically configured to receive a third user operation, and the processing unit is specifically configured to display a first menu in response to the third user operation, wherein the first menu includes multiple function options, each of the multiple function options corresponding to one of the multiple media types; the interaction unit is further configured to receive a user selection operation of at least one function option from the multiple function options, and the processing unit is further configured to display the media information corresponding to the at least one function option in response to the selection operation.
[0041] In one possible implementation, the first user operation acts on the text input box of the first application; after the media information is displayed, the interaction unit is also used to receive a fourth user operation, and the processing unit is also used to respond to the fourth user operation, select the first media information from the displayed media information, and apply the first media information to the first application.
[0042] In a possible implementation, after the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the processing unit is further configured to highlight the first text.
[0043] In one possible implementation, after the first style cursor is transformed into the second style cursor and before receiving the second user operation, the interaction unit is also used to receive a fifth user operation, and the fifth user operation is used to move the cursor to a first position, and the first position is within the range of the first text; the processing unit is also used to move the second style cursor to the first position in response to the fifth user operation.
[0044] In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, the interaction unit is also used to receive a sixth user operation, and the processing unit is also used to unhighlight part of the highlighted text in the first text in response to the sixth user operation, and search for media information that semantically matches the second text based on the second text, where the second text is the text highlighted in response to the sixth user operation.
[0045] In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, the interaction unit is also used to receive a seventh user operation, and the processing unit is also used to highlight a third text in response to the seventh user operation, where the third text is text other than the first text, and search for media information that semantically matches the fourth text based on the fourth text, where the fourth text includes the first text and the third text.
[0046] In one possible implementation, after searching for media information that semantically matches the first text, or after receiving the second user operation, the processing unit is further configured to display the search keyword. Optionally, the interaction unit is further configured to receive an eighth user operation that updates the search keyword and requests a search for matching media information based on the updated search keyword; the processing unit is further configured to, in response to the eighth user operation, search for matching media information based on the updated search keyword; and display the media information found based on the updated search keyword.
[0047] In a possible implementation, the processing unit is specifically configured to obtain text input through the first user operation and a context associated with the text input through the first user operation.
[0048] In one possible implementation, when the processing unit searches for media information that semantically matches the first text based on the first text, it performs one or more of the following: searching for media information that semantically matches the first text in the electronic device; searching for media information that semantically matches the first text in other electronic devices interconnected with the electronic device; searching for media information that semantically matches the first text in the cloud space of the electronic device.
[0049] In a possible implementation, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0050] In a fourth aspect, a device is provided, comprising a unit or module for executing the method as described in any one of the second aspects above.
[0051] In one possible implementation, the device includes a processing unit and an interaction unit. The processing unit is located in a processor, and the interaction unit may include a unit having the function of sensing user operations, such as a touch unit or an audio collector. The interaction unit is used to receive a first user operation, where the first user operation is used to trigger a first function option in a user interface, where the first function option corresponds to a first media information type; the processing unit is used to obtain a context associated with the first user operation and / or the first function option, search for matching media information based on the semantics of the context and the first media information type corresponding to the first function option, and display the searched media information.
[0052] In a possible implementation, the processing unit is specifically configured to obtain text input by a user before the first user operation and / or text contained in a message received by the electronic device, and use the text as the context.
[0053] In a possible implementation, the processing unit is specifically configured to: obtain first information associated with the first function option, where the first information is used to describe media information corresponding to the first function option, and use the first information as the context.
[0054] In one possible implementation, when the processing unit searches for matching media information, it is specifically used to perform one or more of the following: searching for matching media information in the electronic device; searching for matching media information in other electronic devices interconnected with the electronic device; searching for matching media information in the cloud space of the electronic device.
[0055] In a possible implementation, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0056] In a fifth aspect, a device is provided, comprising: one or more processors and one or more memories; wherein the one or more memories store one or more programs, and when the programs are executed by the one or more processors, the device executes a method as described in any one of the first aspects above, or executes a method as described in any one of the second aspects above.
[0057] In the sixth aspect, a chip system is provided, which includes at least one chip and a memory, and the at least one chip is used to read and execute the program stored in the memory to implement the method as described in any one of the first aspects above, or to implement the method as described in any one of the second aspects above.
[0058] In the seventh aspect, a computer-readable storage medium is provided, which includes a program. When the program is run on a device, the device executes any method as described in the first aspect above, or executes any method as described in the second aspect above.
[0059] In an eighth aspect, a computer program product is provided. When the computer program product is run on a device, the device executes the method as described in any one of the first aspects above, or executes the method as described in any one of the second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] FIG1 is a schematic diagram of the hardware structure of an electronic device in an embodiment of the present application;
[0061] FIG2 is a block diagram of a software system architecture of an electronic device in an embodiment of the present application;
[0062] FIG3 is a schematic diagram of a cursor style transformation provided by an embodiment of the present application;
[0063] FIG4 is a schematic diagram of another cursor style transformation provided by an embodiment of the present application;
[0064] FIG5 is a schematic diagram of another cursor style transformation provided by an embodiment of the present application;
[0065] FIG6 is a schematic diagram of another cursor style transformation provided by an embodiment of the present application;
[0066] FIG7 a is a schematic diagram of a user interface for automatically searching for matching media information and performing cursor changes according to user input in a memo application scenario, provided by an embodiment of the present application;
[0067] FIG7 b is a schematic diagram of a user interface for displaying search results by triggering a transformed cursor in a memo application scenario, provided by an embodiment of the present application;
[0068] FIG8 is a schematic diagram of a user interface for highlighting text content matching search results in a memo application scenario provided by an embodiment of the present application;
[0069] FIG9 is a schematic diagram of a user interface for increasing or decreasing highlighted text content to trigger related processing in a memo application scenario provided by an embodiment of the present application;
[0070] FIG10 is a user interface diagram of a related processing process of changing a search keyword trigger in a memo application scenario provided by an embodiment of the present application;
[0071] FIG11 is a schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in a social application scenario provided by an embodiment of the present application;
[0072] FIG12 is a schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in another social application scenario provided by an embodiment of the present application;
[0073] FIG13 is a schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in an email application scenario provided by an embodiment of the present application;
[0074] FIG14 is a flow chart of an interaction method provided in an embodiment of the present application;
[0075] FIG15 is a schematic diagram of a user interface for searching for media information of a corresponding type according to a media information type selected by a user and displaying search results in a memo application scenario provided by an embodiment of the present application;
[0076] FIG16 is a schematic diagram of a user interface for searching for media information of a corresponding type according to a media information type selected by a user and displaying search results in a social application scenario provided by an embodiment of the present application;
[0077] FIG17 is a schematic diagram of a user interface change process based on user operations in a scenario for uploading an ID card photo provided by an embodiment of the present application;
[0078] FIG18 is a schematic diagram of a user interface change process based on user operations in a scenario for uploading a resume provided by an embodiment of the present application;
[0079] FIG19 is a flow chart of another interactive method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0080] The embodiment of the present application provides a human-computer interaction method that can be applied to electronic devices (such as mobile terminal devices). It can automatically search and prompt media information that matches the input content based on user input, so that the user can select it.
[0081] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.
[0082] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0083] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.
[0084] The method provided in the embodiment of the present application can be applied to electronic devices. The electronic device can realize human-computer interaction. For example, the electronic device can be an electronic device such as a mobile phone, a tablet computer, a wearable device (for example, a watch, a bracelet, etc.), an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart home device (for example, a smart TV, a smart speaker, etc.), etc., which can have a human-computer interaction function. It is understandable that the embodiment of the present application does not impose any restrictions on the specific type of electronic device.
[0085] The electronic devices to which the embodiments of the present application can be applied include, but are not limited to, Or electronic devices with other operating systems. The electronic device may also be other electronic devices, such as a laptop computer (Laptop) with a touch-sensitive surface (eg, a touch panel).
[0086] Figure 1 shows a schematic diagram of the hardware structure of a possible electronic device. As shown in Figure 1, electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0087] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may serve as the nerve center and command center of the electronic device 100. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor 110 may also include memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. This memory may store instructions or data that have just been used or are being recycled by the processor 110. When the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces the processor 110's waiting time, and thus improves system efficiency.
[0088] The display screen 194 is used to display the display interface of the application, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N (N is a positive integer greater than 1) display screens 194. For example, the electronic device 100 may be a mobile phone with a foldable display screen having at least two display screens. In an embodiment of the present application, the display screen 194 can be used to display a page (or interface) including at least one application component, such as interface 31 or interface 32 shown in Figure 3.
[0089] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, and the software code of at least one application, etc. The data storage area can store data (such as images, videos, etc.) generated during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0090] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194. In an embodiment of the present application, the touch sensor 180K can detect user operations and can pass the information of the detected user operations to the upper layer, so that the electronic device can respond to the user operation to achieve the addition, deletion or update of application components.
[0091] It is understood that the components shown in FIG1 do not constitute a specific limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine or separate some components, or arrange the components differently. In the following embodiments, the electronic device 100 shown in FIG1 is used as an example for description.
[0092] Figure 2 is a block diagram of the software system architecture of an electronic device provided in an embodiment of the present application. As shown in Figure 2, the software system architecture of the electronic device can be a layered architecture. For example, the software can be divided into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime and system library, the kernel layer, and the hardware layer.
[0093] The application layer can include a series of application packages. As shown in Figure 2, the application layer can include camera, settings, skin modules, user interface (UI), third-party applications, etc. Among them, third-party applications can include wireless local area network (WLAN), music, calls, Bluetooth, video, memo, notes, etc.
[0094] In one possible implementation, applications can be developed using Java by calling the application programming interface (API) provided by the application framework layer. Developers can use the application framework layer to interact with the underlying operating system layers (e.g., the hardware layer, kernel layer, etc.) to develop their own applications. The application framework layer primarily provides a series of services and management systems for the operating system.
[0095] The application framework layer provides an application programming interface and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 2, the application framework layer can include an activity manager, a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and so on.
[0096] The activity manager is used to manage the life cycle of each application and provide common navigation back functions, providing an interactive interface for all program windows.
[0097] The window manager manages windowed applications. It can determine the display size, determine whether a status bar is present, lock the screen, take screenshots, and more. Content providers store and retrieve data and make it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book.
[0098] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images. In some embodiments of the present application, a control that receives and displays text (or text) can, while displaying text in response to user input, also provide the user-entered text to an application or service for implementing semantic understanding, and the application or service can further obtain media information that matches the semantics. Optionally, the application can be set at the application layer, for example, the application can be a third-party application; the service can be set at the application framework layer, for example, the service can be a service provided by the operating system. Optionally, the application or service can be a natural language processing tool driven by technologies such as artificial intelligence and neural networks, which can perform semantic analysis and understanding of natural language, such as semantic understanding of user-entered text or user conversations to extract keywords. The application or service can also further search for matching media information based on keywords, or may hand over the search for matching media information based on keywords to other applications or services. The embodiments of the present application do not limit the semantic understanding method or technology used by the application or service, nor the media information search method.
[0099] Optionally, from the perspective of content type, the media information may include one or more of the following: photos, audio, video, documents, contact directory, contact information, electronic receipts, electronic cards, service cards, etc.; from the perspective of data type, the media information may include one or more of the following: image files, audio files, video files, text files, PDF files, files generated by office software (such as Macromedia Office documents or WPS documents), contact data, ticket data, etc. The media information may also include link address information, such as the web page link address in the browser bookmarks. The embodiments of the present application do not limit the content type or data type of the media information.
[0100] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).
[0101] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.
[0102] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.
[0103] The runtime consists of a core library and a virtual machine. The runtime is responsible for operating system scheduling and management. The core library consists of two parts: one for the Java language's callable functions and the other for the operating system's core libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.
[0104] The system library may include multiple functional modules. For example: surface manager, media framework, three-dimensional graphics processing library (for example: OpenGL ES), two-dimensional graphics engine (for example: SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional and three-dimensional layers for multiple applications. The media framework supports a variety of commonly used audio and video format playback and recording, as well as static image files, etc. The media framework can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc. The two-dimensional graphics engine is a drawing engine for two-dimensional drawing. In some embodiments, the three-dimensional graphics processing library can be used to draw three-dimensional motion trajectory images, and the two-dimensional graphics engine can be used to draw two-dimensional motion trajectory images.
[0105] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.
[0106] The hardware layer can include various sensors, such as accelerometers, gravity sensors, touch sensors, etc.
[0107] It should be understood that the hardware structure of the electronic device can be as shown in Figure 1, and the software system architecture can be as shown in Figure 2, wherein the software programs and / or modules corresponding to the software system architecture in the electronic device can be stored in the internal memory 121, and the processor 110 can run the software programs and applications stored in the internal memory 121 to execute the process of the display method provided in the embodiment of the present application.
[0108] In an embodiment of the present application, after the electronic device searches for media information that matches the semantics of the text input by the user, it can change the cursor style to prompt the user that the matching media information has been found.
[0109] A cursor is also called a cursor and can be used to indicate the position of text. When a user enters text in a text box or an application, the cursor can be used to display the position where the user is going to enter the text. The cursor position can also be moved forward or backward based on user operations. Currently, in the user interface of a mobile phone or tablet computer, the cursor is usually displayed as a small vertical line and can flash regularly. In some embodiments of the present application, more diverse cursor styles are provided. As the user enters text (or text), when matching media information is searched based on the text (or text) entered by the user, the first style cursor (such as a small vertical line) can be transformed into a second style cursor provided in an embodiment of the present application. The second style cursor can prompt the user to search for media information that matches the entered text (or text).
[0110] In the following description of this specification, the cursor provided in the embodiments of the present application for prompting a user that matching media information has been found is referred to as the second-style cursor. Correspondingly, the cursor displayed in a normal state or when no matching media information has been found is referred to as the first-style cursor. For example, the first-style cursor may be a small, regularly blinking vertical line, or another cursor style set by the user through system settings.
[0111] In one possible implementation, the second-style cursor may have a different shape, color, and / or size than the first-style cursor, to prompt the user that matching media information has been searched. Generally, changing the cursor's shape is more intuitive and obvious to the user than changing its size and / or color, allowing the user to more intuitively and clearly see the change in cursor style and thus decide on the next action. The following embodiments of this application are described using the example of a second-style cursor having a different shape than the first-style cursor.
[0112] An example can be shown in Figure 3. In Figure 3 (a), a first-style cursor 301 in a normal state is displayed in the memo text input box. The first-style cursor 301 is displayed at the end of the text, specifically displayed as a vertical line shape; in Figure 3 (b), after the electronic device searches for matching media information based on the text input by the user, a second-style cursor 302 is displayed in the memo text input box. The second-style cursor 302 is displayed at the end of the text. In one possible embodiment, the second-style cursor can be displayed in a magnifying glass shape to prompt the user to search for matching media information based on the text content before this cursor. In order to better prompt the user of the searched media information, in a possible implementation method, the type of media information searched can be prompted to the user by displaying a text prompt in the second-style cursor 302. For example, the text "picture" is displayed in the second-style cursor 302 to prompt that matching picture-type media information has been searched.
[0113] In another possible implementation, an identifier for representing a picture can also be displayed in the second-style cursor to prompt that media information of a matching picture type has been searched. An example can be shown in Figure 4. In Figure 4 (a), a first-style cursor 401 in a normal state is displayed in the memo text input box. The first-style cursor 401 is displayed at the end of the text, specifically displayed as a vertical line shape; in Figure 4 (b), when the electronic device searches for a matching photo based on the text entered by the user, a second-style cursor 402 is displayed in the memo text input box. The second-style cursor 402 is displayed at the end of the text, specifically displayed in the shape of a magnifying glass, and a picture type identifier 403 is displayed within the range of the magnifying glass lens to prompt that media information of the picture type has been searched.
[0114] In order to more intuitively prompt the user of the searched media information, in another possible implementation, for the searched media information of the picture or video type, a thumbnail of the picture or video frame can be displayed in the second style cursor. Optionally, a thumbnail of the picture with the highest matching degree or a thumbnail of a video frame (such as the first frame) in the video with the highest matching degree can be displayed. An example can be shown in Figure 5. In Figure 5 (a), a first style cursor 501 in a normal state in the memo text input box is displayed. The first style cursor 501 is displayed at the end of the currently input text and is specifically displayed as a vertical line shape; in Figure 5 (b), when the electronic device searches for matching pictures and / or videos based on the text entered by the user, a second style cursor 502 is displayed in the memo text input box. The type of media information searched and the content of the searched media information can be prompted to the user by displaying a picture thumbnail or a video frame thumbnail in the second style cursor 502. FIG5( b ) takes the searched picture as an example. In the second style cursor 502 in the shape of a magnifying glass, a thumbnail 503 of the picture with the highest matching degree is displayed within the range of the magnifying glass lens to prompt the content of the searched media information.
[0115] Based on the above examples, it can be understood that the second-style cursor can indicate the type and / or content of the searched media information. For example, the second-style cursor includes one or more of the following: a thumbnail or partial image of the content of the media information; information indicating the media type, wherein the information indicating the media type includes an icon indicating the type of the media information; or text indicating the type of the media information.
[0116] It will be understood that the above merely illustrates several possible forms of the second-style cursor, and the embodiments of the present application do not limit this. For example, the second-style cursor may also include local content in the searched image.
[0117] When multiple types of media information are searched, in order to prompt the user that multiple types of media information have been searched, in one possible implementation, a text prompt (for prompting that multiple types of media information have been searched) or some kind of identifier (such as an icon for indicating that multiple types of media information have been searched) can be displayed in the second-style cursor to prompt the user that multiple types of media information have been searched, or through the cursor shape, or through other forms. An example can be shown in Figure 6. In Figure 6 (a), a first-style cursor 601 is displayed in a normal state in the memo text input box. The first-style cursor 601 is displayed at the end of the currently entered text and is specifically displayed as a vertical line. In Figure 6 (b), when the electronic device searches for multiple types of media information that match the text entered by the user, a second-style cursor 602 is displayed in the memo text input box. Taking the search for three types of media information, namely pictures, videos, and documents (such as Word files), as an example, the second-style cursor 602 can be displayed as three magnifying glasses stacked together to prompt the user that three types of media information have been searched. Optionally, the top-level magnifying glass may display relevant information (such as thumbnails, text prompts, icons, etc.) of the media information that has the highest semantic match with the text. Optionally, the top-level magnifying glass may also display relevant information of the media information that matches the user's behavior habits based on the user's behavior habits. For example, as shown in Figure 6 (b), since the user views photos or browses the photo library more frequently than they view videos and word files, thumbnails of the searched images may be displayed in the top-level magnifying glass.
[0118] To better inform the user of the type of media information searched, as shown in FIG6(c), when the user triggers the second-style cursor 602 via a click or other user operation (such as a long press or double-click), a menu 603 is displayed in response. For the three types of media information searched, menu 603 includes three corresponding function options (as shown in the figure, function options 604, 605, and 606), each corresponding to the search results of a type of media information. The user can select a function option in menu 603 (e.g., click a function option) to display the searched media information of the corresponding type. For example, after the user clicks function option 604 in menu 603, the searched image can be displayed in a pop-up window. As shown in FIG6(c), function options 604, 605, and 606 can indicate the corresponding media information type and / or content through thumbnails, text prompts, etc. For example, referring to the embodiments shown in Figures 3-5, the function options may include one or more of the following: a thumbnail or partial image of the media information content; information indicating the media type, including an icon indicating the type of the media information; and text indicating the type of the media information. For example, function option 604, when triggered by an image thumbnail, may display searched images, while function option 605, when triggered by a text prompt, may display searched Word files.
[0119] Optionally, multiple (including two or more) media information types can be selected based on menu 603, allowing the user to view media information of the corresponding types. Taking Figure 6 (c) as an example, when a user long-presses function option 604, check boxes can be displayed before, after, or in the upper right or upper left corners of function options 604, 605, and 606, respectively, allowing the user to select the media information type corresponding to one or more of function options 604, 605, and 606. Figure 6 (d) takes the display of a check box in the upper right corner as an example. When a user long-presses function option 604, check boxes can be displayed in the upper right corners of function options 604, 605, and 606, with the check box in the upper right corner of function option 604 being selected and the check boxes for the other two function options being unselected. After the user selects the media information type to view using the check boxes and clicks the "Confirm" button, the electronic device displays a list of the searched media information of the corresponding types. It is understood that other methods can also be used to implement the multiple selection function, and this embodiment of the application is not limited to this.
[0120] Optionally, menu 603 may further include a function option for displaying all types of media information. When this function option is triggered (e.g., when the user clicks this function option), all types of media information may be displayed. Optionally, different types of media information may be displayed according to the type of media information, such as a list of pictures, a list of word files, and a list of videos.
[0121] In another possible implementation method for the scenario shown in Figure 6, unlike the user operation of long pressing the display menu 603 to view all types of searched media information, when the user clicks or double-clicks the second style cursor 602, all types of searched media information can be directly displayed according to the type of media information.
[0122] It will be understood that the above merely exemplifies several implementation methods of browsing searched media information by performing user operations on the second-style cursor, and the embodiments of the present application do not limit this.
[0123] Optionally, after searching for multiple types of media information, as an alternative to the scenario shown in FIG6 , the first style cursor 601 may be directly transformed into the cursor form shown in FIG6 (c). That is, all types of searched media information may be directly prompted in the transformed cursor, thereby further reducing user operations.
[0124] It is understood that the above second-style cursors are only a few possible examples, and the present embodiment does not limit the shape, size, color, and other attributes of the second-style cursor. For example, using FIG. 5 as an example, the second-style cursor can also be displayed as the second-style cursor 502 rotated 180 degrees, which can minimize the second-style cursor from obscuring the entered text.
[0125] The following describes the user operations and user interface changes during the interaction process implemented in the embodiments of the present application, taking the scenarios of a memo application, a social application, and an email application as examples.
[0126] First, taking the memo application as an example, the user interface change process based on user operations during the implementation of the embodiment of the present application is described.
[0127] 7a and 7b , an embodiment of the present application takes a mobile phone as an example and provides a user interface diagram for automatically searching for matching media information following user input and displaying search results in a memo application scenario.
[0128] As shown in FIG7 a , the interface of the memo application includes a text input box 71 and a soft keyboard 72 . The user can enter text in the text input box 71 using the soft keyboard 72 . Referring to user interfaces 701 to 703 in FIG7 a , the user uses the soft keyboard 72 to continuously enter the text "During the National Day holiday, we took our daughter Yaya to the Animation Theme Park. There is a large, cute cat statue at the entrance of the park" into the text input box 71 . Based on the above text content, the mobile phone can obtain the keywords "National Day," "Ya Ya," and "Animation Theme Park" through semantic understanding. Based on these keywords, the mobile phone searches for multiple photos that match these keywords. For example, the mobile phone can search for photos taken during the "National Day" holiday using the keyword "National Day," search for photos containing Yaya's portrait using the keyword "Ya Ya," and search for photos taken at the Animation Theme Park or photos containing animation-themed elements (such as animation characters and / or animation buildings, etc.) using the keyword "Animation Theme Park."
[0129] Referring to user interface 703 to user interface 705 in FIG7a, since the mobile phone has searched for photos that match the input text, the first style cursor (the vertical line shown in FIG7a) is transformed into the second style cursor (the cursor in the shape of a magnifying glass shown in FIG7a). In order to improve the user experience, dynamic effects can be used for display during the cursor style transformation process. The dynamic effect of the cursor style transformation shown in FIG7a is as follows: in user interface 703, the cursor transforms from a vertical line to a magnifying glass shape; in user interface 704 to user interface 705, in the lens area of the magnifying glass, the thumbnail of the photo that best matches the keyword gradually changes from blurry to clear.
[0130] Referring to user interface 706 in Figure 7b, after a user clicks the second-style cursor in user interface 705, a pop-up window 73 may appear. At this point, the second-style cursor may revert to the first-style cursor, indicating the completion of semantic understanding and related media information search. Window 73 may overlay the soft keyboard window, so that the text input box 71 is not covered or not completely covered. Window 73 may display the searched images. If multiple photos are found, they may be displayed in a stacked format, allowing the user to select a photo by swiping left or right. Optionally, the default photo displayed may be the photo that best matches the semantics of the text. Optionally, window 73 may also display the total number of searched photos to facilitate user understanding of the search results and selection. In another possible implementation, the searched photos may be arranged in a list or grid format, in descending order of matching. Furthermore, if a large number of photos are found, a slider may be provided to allow the user to scroll up and down to browse the searched photos.
[0131] Referring to user interface 707 to user interface 708 in Figure 7b, after the user selects a photo in window 73, he or she can drag it to the text input box 71 by dragging it. Optionally, during the user operation, prompt information can also be displayed on the user interface to guide the user operation, such as the prompt information "Slide to this area to insert the picture" shown in user interface 707 and the prompt information "Release to insert" shown in user interface 708. Referring to user interface 709, when the selected photo is dragged to the text input box 71 and lifted, the photo can be automatically added to the position of the first style cursor. Through the above-mentioned dragging operation, the selected photo can be inserted to the position of the cursor. Optionally, after a searched photo is added to the text by dragging, window 73 can display the next photo with the second highest matching degree. Similar to the above-mentioned operation of adding photos to the text, the mobile phone can add multiple photos to the text in sequence based on the user's dragging operation.
[0132] In some embodiments of the present application, the text content corresponding to the search results can also be highlighted to let the user know which part of the text content matches the currently searched media information. In one possible implementation, after the matching media information is searched based on the text (or text) input by the user, the corresponding text (or text) content can also be highlighted while changing the cursor style. In another possible implementation, after the user triggers the second style cursor by clicking or other operations, the corresponding text content can also be highlighted while displaying the search results. Optionally, the implementation method of highlighting the corresponding text content may include but is not limited to: displaying the corresponding text content in different font colors, or displaying the corresponding text content in different font styles (or font shapes), or displaying the corresponding text content in a highlighted manner, or displaying the corresponding text content in an emphasized display manner (such as making the font bold), or combining the above multiple methods to highlight the corresponding text content.
[0133] An example can be shown in Figure 8. In the user interface 801, a matching video is searched based on the text content displayed in emphasis in the text input box (the paragraph starting with "3:00 p.m." as shown in the figure), so the first style cursor is transformed into a second style cursor. The second style cursor is displayed with a video type identifier, indicating that a matching video is searched based on the text content displayed in emphasis. When the user clicks on other locations in the paragraph, see user interface 802, the second style cursor moves to the location clicked by the user. Since the video search result is searched based on the text content, moving the cursor within the text range will not trigger a new search process, and the current cursor style will not change. Further, the paragraph can remain emphasized. The user can add the searched video at the current cursor location. Optionally, the operation of adding a video can be similar to the operation of adding a picture shown in Figure 7b.
[0134] In some embodiments of the present application, after highlighting the text content corresponding to the search results, the user can reduce or increase the highlighted text content by double-clicking or other operations. Since the highlighted text content changes, a new search process may be triggered, which in turn may cause the search results to change, and accordingly, the cursor style may also change accordingly.
[0135] An example can be shown in Figure 9. In one scenario, in user interface 901, the mobile phone searches for a matching video based on the highlighted text content, and the second-style cursor contains a video type identifier, indicating that a matching video has been found. When the user double-clicks at position 1 in user interface 901, referring to user interface 902, the sentence at position 1 (which can be separated by punctuation marks, such as "The float tour begins at 3:00 p.m.") is no longer highlighted. The mobile phone searches again for the remaining highlighted text content, and the search results remain unchanged, so the cursor style also remains unchanged. It will be understood that this example describes the case where a search is performed again based on the updated highlighted text and the search results remain unchanged. In other scenarios, due to the reduction of the highlighted text content through user operation, when a new search is performed based on the changed text content, the search results obtained by the new search may change due to a change in semantics or a reduction in search keywords. If the type of media information obtained in the new search changes, such as a change in type, or a new type is added, or an existing type is reduced, the second-style cursor will change accordingly to match the new search results.
[0136] In another scenario, when the user double-clicks at position 2 of user interface 901, see user interface 903, the sentence at position 2 (which can be divided by punctuation marks, such as "Ya Ya took more than a dozen photos with the cute cat") is highlighted. See user interface 904. The mobile phone searches again for the text content currently highlighted. The search results include matching videos and pictures. Compared with the text content highlighted in user interface 901, the semantics have changed, and search keywords (such as cute cat, photo, etc.) have been added, resulting in changes in search results. Accordingly, the cursor style also changes accordingly. As shown in user interface 904 in Figure 9, the transformed cursor includes a video type identifier and a picture type identifier. The transformed cursor can prompt that two types of media information have been searched. The user can browse the search results of the corresponding type through the method of the embodiment shown in Figure 6 (for example, clicking on one type of identifier).
[0137] In the above process, the implementation method of using punctuation marks as a separator to select the sentence where the user operation position is located is only an example. The embodiment of the present application does not limit the implementation method of selecting text content based on user operation. For example, in another example, based on the "press-slide-lift" operation of the user's finger, the text content between the "press" action position and the "lift" action position can be selected; in another example, association can be made based on the text arbitrarily selected by the user to select the text content.
[0138] In some embodiments of the present application, after searching for media information that semantically matches the text input by the user, the search keyword can also be displayed. In other embodiments, the search keyword can be displayed when the searched media information is displayed after the second style cursor is triggered. An example is shown in Figure 10. In the user interface 1001, when the second style cursor is clicked, the searched picture is displayed, and the search box 1010 is displayed. The search box 1010 displays the search keywords corresponding to the current search results. For example, the currently searched picture is searched based on the keywords "National Day", "Ya Ya" and "Cute Cat", so the above search keywords are displayed by default in the search box.
[0139] In some embodiments of the present application, the user can also change the search keywords so that the search can be re-performed according to the user's needs. An example is shown in Figure 10. As the user enters text in the text input box of the memo, the mobile phone searches for a matching picture based on the text entered by the user and changes the first style cursor to a second style cursor. When the user triggers the second style cursor, see user interface 1001. A window for displaying search results can be displayed on the upper layer of the soft keyboard window, in which the searched picture is displayed, and a search box 1010 is displayed at the same time. The search box 1010 displays the search keywords corresponding to the search results by default (such as "National Day", "Ya Ya" and "Lovely Cat" as shown in the figure). The search box also includes a command button for triggering the search operation (such as the "Search" button as shown in the figure). When the user clicks on the text input part in the search box 1010, see user interface 1002. Optionally, the full screen can be pulled up to display the search results corresponding to the search box 1010. Further, a soft keyboard can be displayed so that the user can change the search keywords in the search box 1010 through the soft keyboard.
[0140] The user can update the search keywords displayed by default in the search box 1010, for example, by entering new search keywords, deleting some of the search keywords, deleting some of the search keywords and then entering new keywords, or deleting all the search keywords displayed by default in the search box 1010 and then entering new keywords. Referring to user interface 1002, this example takes deleting "National Day" and "Cute Cat" in the search box 1010 as an example, and the updated search keyword is "Ya Ya". Referring to user interface 1003, when the user clicks the "Search" command button, the mobile phone searches for matching media information based on the updated search keywords. Optionally, after searching for media information that matches the updated search keywords, you can exit the full-screen search interface and return to the memo input interface, referring to user interface 1004, where media information that matches the updated search keywords can be displayed. Optionally, the user can select the searched media information and add the selected media information to the current cursor position in the memo text input box.
[0141] Next, a social application is taken as an example to describe the changes in the user interface during the interaction process implemented in the embodiments of the present application.
[0142] 11 , an embodiment of the present application takes a mobile phone as an example and provides a user interface diagram for automatically searching for matching media information according to user input and displaying search results in a social application scenario.
[0143] As shown in Figure 11, in the user interface 1101 of a social application, as the user enters text in the input box 1111, the mobile phone can perform semantic understanding on the text entered by the user, or further perform semantic understanding on the text entered by the user in context, and search for media information based on the results of the semantic understanding. The context may include chat content sent by the other user received by the mobile phone, such as the chat history sent by the other user in Figure 11. When a matching document is found, the cursor is transformed into a second-style cursor 1112, which displays a document type identifier to indicate that a matching document has been found. When the user clicks the second-style cursor 1112, a window 1115 pops up, as shown in user interface 1102. The size of this window can be smaller than the full screen size, for example, it can be overlaid on the top of the soft keyboard window. Window 1115 can display multiple searched documents, which can be sorted from high to low matching degree and / or file type. The user can select one of the documents and send it to the chat window. Optionally, if the user is allowed to send media information by dragging it into the chat window, the text input box and related buttons (such as additional function button 1113) may not be displayed in user interface 1102. Referring to user interface 1103, when a selected document is dragged to a target location and released, the document is sent, or the document and the text entered by the user in text input box 1111 are sent simultaneously.
[0144] In one possible implementation, in the embodiment shown in FIG11 , the user does not need to enter anything into the input box. When the input focus is on the input box (i.e., the cursor is displayed in the input box), the mobile phone can perform semantic understanding of the context and search for media information based on the results of the semantic understanding. The context can be the user's previous chat history. It is understandable that nothing may be entered into the input box at this time. When a matching document is found, the cursor can also be transformed into a second-style cursor 1112, and the user can obtain the searched media information through the second-style cursor 1112.
[0145] 12 , which is a schematic diagram of a user interface for automatically searching for matching media information and displaying search results in accordance with user input in another social application scenario, taking a mobile phone as an example, according to an embodiment of the present application.
[0146] As shown in Figure 12, in user interface 1201 of a social networking application, the mobile phone receives user speech, converts the speech into text, and displays it in a text input box. The mobile phone performs semantic understanding on the text in context and searches for media information based on the results of the semantic understanding. When a matching image is found, the cursor is transformed into a second-style cursor, which may display an image type identifier to indicate that a matching image has been found. When the user clicks the second-style cursor, a pop-up window 1211 (see user interface 1202) appears. This window can be smaller than the full screen size, for example, it can overlay the soft keyboard window. The searched image is displayed in window 1211. (See user interface 1203) The user can drag the image into the chat window. (See user interface 1204) When the image is dragged into the chat window and released, the image is sent. Optionally, the text entered by the user in the text input box can remain in the text box, or the text in the text input box can be sent simultaneously with the image.
[0147] Next, the changes in the user interface during the interaction process implemented in the embodiment of the present application will be described by taking the mail application as an example.
[0148] 13 , an embodiment of the present application takes a mobile phone as an example and provides a user interface diagram for automatically searching for matching media information according to user input and displaying search results in an email application scenario.
[0149] As shown in Figure 13, in the user interface 1301 of the email application, as the user enters text in the email body, the mobile phone displays the text entered by the user in the text input box, performs semantic understanding on the text, and searches for media information based on the results of the semantic understanding. After searching for a matching voice file, the cursor is transformed into a second style cursor, in which a voice type identifier can be displayed to prompt that a matching voice file has been searched. When the user clicks the second style cursor, see user interface 1302, a pop-up window 1311 appears. The size of the window can be smaller than the full screen size, for example, it can be covered on the upper layer of the soft keyboard window. The searched voice file is displayed in window 1311. See user interface 1303. The user can drag the voice file into the email body. When the voice file is dragged into the email body and released, the voice file can be added to the email as an attachment.
[0150] It can be understood that some of the above examples are described as an example of selecting one media information (such as a picture, a video, or a document) from multiple searched media information, but the embodiments of the present application are not limited to this. In some other embodiments of the present application, the user is allowed to select two or more media information from the multiple searched media information. For example, a check box can be provided for each searched media information, so that the user can select multiple media information through the check box.
[0151] The specific implementation of the interaction method provided in the embodiment of the present application is described below in conjunction with Figure 14.
[0152] Referring to Figure 14, there is shown a flowchart of an interactive method provided in an embodiment of the present application, which is used to automatically search for matching media information based on text input by a user, and can further indicate the search status by changing the cursor. The method can be executed by an electronic device, and an example of an electronic device can be shown in Figure 1. The method can also be executed by a software function module within the electronic device, and an example of a software system architecture within the electronic device is shown in Figure 2. The method can also be implemented by a combination of software and hardware.
[0153] As shown in FIG14 , the method may include the following steps:
[0154] Step 1401: The electronic device receives a first user operation, and obtains a first text in response to the first user operation.
[0155] The first user operation can be used to enter text, such as entering text into a text input box within an application. For example, the first user operation is the user entering text into the text box. For example, in the memo application scenario shown in Figure 7a, the first user operation can be the user entering the memo text into the text input box using the soft keyboard. For another example, in the social application scenario shown in Figure 11, the first user operation can be the user entering the conversation text into the text input box using the soft keyboard. For another example, in the email application scenario shown in Figure 13, the first user operation can be the user entering the email body. It is understood that in addition to using the soft keyboard to enter text, text can also be entered using methods such as handwriting. Alternatively, the first user operation can be the user capturing and converting user speech into text, which can then be displayed in the text input box. Alternatively, the first user operation can be a "paste" operation. For example, by long-pressing the text input box and selecting "Paste" from the pop-up menu, text from the clipboard can be entered into the text input box.
[0156] The text input process is usually a process of multiple continuous inputs. For the sake of convenience of description, in the embodiment of the present application, each operation of entering a piece of text content (which may contain only one character or one word, or may contain multiple characters or multiple words, or contain one or more sentences) in the text input box is called a first user operation.
[0157] After entering text in the text input box of the application, in response, a first text containing at least the text content currently entered can be obtained for semantic analysis and understanding. In this way, following the user's text input operation, semantic analysis and understanding can be performed in real time based on the text entered by the user. Optionally, the first text may only include text entered by the first user operation, such as when no text has been entered before the text is entered by the first user operation. Optionally, the first text may include text entered by the first user operation and text entered before that, such as the examples shown in Figures 7a and 7b above. Optionally, the first text includes text entered by the first user operation and a context, and the context has an association relationship with the text entered by the first user operation. For example, in the social application scenario shown in Figure 11, the first user operation is a user operation of entering text in the text input box provided by the social application, and the text is the chat content to be sent to the target user. The context associated with the chat content includes the text content in the chat message previously sent by the target user. It can be understood that in a one-to-one chat (or private chat) scenario, the number of target users is one; in a group chat scenario, the number of target users can be one or more, that is, the context can include the text content in the chat messages of multiple users received by the application.
[0158] In one possible implementation, when a text input box receives a text segment (which may include one or more words or characters) at time n, if no new text is received within a time duration t, semantic understanding of the previously input text content may be performed at time n+t. The value of the time duration t can be preset.
[0159] In one possible implementation, text that has been input and semantically understood and for which matching media information is searched based on the semantic understanding results can be marked as having been semantically understood, thereby avoiding subsequent repeated semantic understanding of the text.
[0160] Step 1402: The electronic device searches for media information that matches the semantics of the first text based on the first text.
[0161] In one possible implementation, the electronic device may search for matching media information based on the semantics of the first text. Optionally, in an embodiment of the present application, a natural language processing tool driven by artificial intelligence, neural networks, or other technologies may be used to implement semantic analysis and understanding.
[0162] In some embodiments of the present application, the functionality of the text input box control can be expanded so that it can respond to specified events (such as user input operation events, or events in which the pause time during the user input operation exceeds a set threshold) to request or call a natural language processing tool to perform semantic understanding of the first text.
[0163] In one possible implementation, the electronic device can search for media information that matches the first text from the media information stored in the electronic device, for example, it can search for pictures and videos that match the first text from a local picture library (such as a photo album) of the electronic device, and search for documents that match the first text from a local folder of the electronic device for storing user documents. In another possible implementation, the electronic device can obtain media information that matches the first text from a cloud space used by the electronic device, wherein the cloud space can be shared by the electronic device with other electronic devices. In another possible implementation, if the electronic device is interconnected with other electronic devices (for example, a far-field connection or a near-field connection), media information that matches the first text can be obtained from the other electronic devices through connection with other electronic devices. It can be understood that the above-mentioned multiple acquisition methods can also be used in combination with each other. The embodiment of the present application does not limit the search scope or source of media information.
[0164] In one possible implementation, the electronic device may prioritize searching for matching media information locally on the electronic device to minimize network overhead. If no matching media information is found, the cloud space is searched for matching media information, and so on. This embodiment of the application does not limit the search order or priority.
[0165] Step 1403: The electronic device transforms the first-style cursor into a second-style cursor, where the second-style cursor can indicate that media information has been searched.
[0166] In the embodiment of the present application, by changing the first-style cursor to the second-style cursor, it is possible to indicate to the user that matching media information has been searched, so that the user can proceed to the next step based on the searched media information. For examples of the second-style cursor, reference can be made to the description of the aforementioned embodiments, such as the several examples shown in Figures 3 to 6.
[0167] Optionally, the type of the searched media information may be indicated by a second-style cursor, for example, by displaying a text prompt, a type identifier, etc. in the second-style cursor. For example, in the example shown in FIG8 , a video type identifier is displayed in the second-style cursor to indicate that video type media information has been searched; for another example, in the example shown in FIG11 , a document type identifier is displayed in the second-style cursor to indicate that document type media information has been searched.
[0168] Optionally, if the searched media information includes images, a thumbnail of the searched image may be displayed in the second-style cursor. For example, in the example shown in FIG7 a , a thumbnail of an image is displayed in the second-style cursor, thereby more intuitively indicating the search result to the user. The thumbnail may be the thumbnail of the image that has the highest degree of matching with the first text among the searched images.
[0169] Optionally, if the searched media information includes a video type, a thumbnail of a video frame from the searched video may be displayed in the second-style cursor, thereby more intuitively indicating the search result to the user. The thumbnail may be a thumbnail of a video frame from the searched video that has the highest degree of match with the first text.
[0170] Optionally, in the case where multiple types of media information are searched, a second-style cursor can be used to indicate that there are multiple types of media information searched. For example, in the example shown in FIG6 , a stacked shape can be used to indicate that multiple types of media information are searched. Furthermore, the indication information of each type of multiple types can be displayed through a third user operation (such as a click operation in the example shown in FIG6 ). In the example shown in FIG6 , the indication information of each type can be displayed in the form of a menu. Specifically, a first menu can be displayed based on the third user operation. The first menu includes multiple function options, and each of the multiple function options corresponds to one of the multiple media information types searched. It can be understood that the embodiment of the present application does not limit the type of the third user operation.
[0171] Step 1404: The electronic device receives a second user operation on a second-style cursor, and displays the searched media information in response to the second user operation.
[0172] Optionally, the second user operation is used to trigger a second style cursor. For example, the second user operation may be an operation of clicking the second style cursor or a double-clicking the second style cursor, etc. The embodiment of the present application does not limit the type of the second user operation.
[0173] In one possible implementation, the electronic device displays the type of media information requested by the second user operation based on the received second user operation. For example, if the second-style cursor indicates only one type of media information, such as only indicating that media information of the image type is searched, the electronic device displays the searched image after receiving the second user operation (such as clicking the second-style cursor). An example may be the scenario shown in Figures 7a and 7b. For another example, if the second-style cursor indicates that multiple types of media information are searched, the electronic device may display the first menu based on the above-mentioned third user operation, and previously, after receiving the second user operation, display the corresponding type of media information according to the type indicated by the second user operation. If the second user operation is used to trigger one of multiple function options, then in response to the second user operation, the electronic device displays the type of media information corresponding to the function option among the multiple types of media information searched; if the second user operation is used to trigger at least two of the multiple function options, then in response to the second user operation, the electronic device displays at least two types of media information corresponding to the function option among the multiple types of media information searched. An example may be the scenario shown in Figure 6.
[0174] In a possible implementation, the searched media information may be displayed in descending order of matching degree.
[0175] In the process shown in FIG. 14 , the electronic device can automatically search for matching media information based on the text entered by the user, eliminating the need for the user to enter search keywords or trigger a function button to execute the search command. This simplifies user operations and improves the user experience. Furthermore, by changing the cursor to indicate that matching media information has been found, the user can understand the search status and perform subsequent operations based on the search results.
[0176] Furthermore, in the process shown in FIG14 , after displaying the media information matching the first text, the process may further include the following steps: receiving a fourth user operation, selecting the first media information from the displayed media information in response to the fourth user operation, and applying the first media information to the first application. The first application is the application that receives the first user operation and obtains the text content entered by the user through the first user operation. In other words, the first user operation is used to enter the text content in the text input box of the first application, thereby causing the first application to receive the text entered by the user.
[0177] The fourth user operation is used to select one or more media information from the displayed media information (for ease of description, the one or more media information selected by the fourth user operation will be referred to as first media information) and apply the first media information to the first application. Depending on the function of the first application, the specific method of applying the first media information to the first application may vary. For example, for a memo application, the first media information can be added to the memo content. For relevant examples, see Figures 7a and 7b. For another example, for a social application, the first media information can be sent to a target user. For relevant examples, see Figures 11 or 12. For another example, for an email application, the first information can be added as an attachment to an email. For relevant examples, see Figure 13.
[0178] In some embodiments of the present application, the text corresponding to the search result can also be highlighted so that the user can understand which part of the text is used to search for matching media information. In one possible implementation, the first text can be highlighted during the process of transforming the first-style cursor into the second-style cursor, or after the first-style cursor is transformed into the second-style cursor, or after receiving the second user operation. An example can be shown in Figure 8.
[0179] In some embodiments of the present application, after highlighting the first text, the user may be allowed to move the cursor position through user operation so that media information can be added to the position later. In one possible implementation, after the first-style cursor is transformed into the second-style cursor and before the second user operation is received, a fifth user operation may be received, the fifth user operation being used to move the cursor to a first position, the first position being within the range of the first text; in response, the electronic device may move the second-style cursor to the first position. An example may be shown in Figure 8.
[0180] In some embodiments of the present application, after highlighting the first text, the highlighting of part of the text can be canceled through a user operation, triggering a new search process. In one possible implementation, after the first-style cursor is transformed into the second-style cursor and before the second user operation is received, a sixth user operation may be received. In response to the sixth user operation, the highlighting of part of the highlighted first text is canceled, and matching media information is searched based on the second text, where the second text is the text highlighted in response to the sixth user operation. An example can be shown in Figure 9.
[0181] In some embodiments of the present application, before highlighting the first text, the highlighted text content can be increased through user operation, thereby triggering a new search process. In one possible implementation, after the first-style cursor is transformed into the second-style cursor and before the second user operation is received, a seventh user operation may be received. In response to the seventh user operation, a third text is highlighted, the third text being text other than the first text; and matching media information is searched based on a fourth text, the fourth text including the first text and the third text. An example can be shown in Figure 9.
[0182] In some embodiments of the present application, after searching for media information matching the first text, or after receiving a second user operation, a search keyword may be displayed. Furthermore, the user may be allowed to update the search keyword in order to search for the required media information. In one possible implementation, after displaying the search keyword, the electronic device may receive an eighth user operation, wherein the eighth user operation is used to update the search keyword and request a search for matching media information based on the updated search keyword; in response to the eighth user operation, the electronic device may search for matching media information based on the updated search keyword and display the media information searched based on the updated search keyword. An example may be shown in Figure 10.
[0183] An embodiment of the present application further provides an interaction method applied to an electronic device, which can be used to search for matching media information based on the text and / or context input by the user according to the media information type selected by the user.
[0184] The following describes the user operations and user interface changes in the interaction process implemented in the embodiments of the present application, taking the memo application scenario, the social application scenario, and the scenario for uploading information for authentication or registration as examples.
[0185] First, taking the memo application as an example, the user interface change process based on user operations during the implementation of the embodiment of the present application is described.
[0186] 15 , which is a user interface diagram for searching for corresponding types of media information and displaying search results based on a user-selected media information type in a memo application scenario, taking a mobile phone as an example, according to an embodiment of the present application.
[0187] As shown in Figure 15, the interface of the memo application includes a text input box 1511, a soft keyboard 1512 and a function option 1513. After being triggered, the function option 1513 can search for matching pictures based on the text input by the user. The function option 1513 can be implemented by a picture control. Referring to user interface 1501, the user uses the soft keyboard 1511 to enter text in the text input box 1512. Referring to user interface 1502, the user clicks function option 1513. In response, referring to user interface 1503, the electronic device searches for matching pictures based on the text entered in the text box, and displays the searched pictures in a pop-up window 1504. Regarding the display method of the searched pictures, please refer to the relevant content in Figure 7b, which will not be repeated here.
[0188] Optionally, the user interface 1501 may further include other functional options, such as a functional option for searching for video-type media information (e.g., a video control), a functional option for searching for document-type media information (e.g., a file control), etc., which are not limited in this embodiment of the present application. It is understood that different types of functional options or different types of controls can be used to search for different types of media information.
[0189] After the searched media information is displayed, the selected media information can be added to the memo content based on user operation. For the description of selecting media information and adding media information to the memo content, please refer to the relevant content in Figure 7b, which will not be repeated here.
[0190] The following describes the process of user interface changes based on user operations during the implementation of the embodiments of the present application, taking a social application scenario as an example.
[0191] 16 , which is a schematic diagram of a user interface for searching for corresponding types of media information and displaying search results in a social application scenario using a mobile phone as an example according to an embodiment of the present application.
[0192] As shown in Figure 16, user interface 1601 of a social networking application includes a text input box, a soft keyboard, and function options 1611 and 1612. When triggered, function option 1611 can search for matching documents based on the semantics of the associated text, while function option 1612 can search for matching images based on the semantics of the associated text. Function option 1611 can be implemented using a file control, while function option 1612 can be implemented using an image control. Referring to user interface 1601, a user enters text into the text input box using the soft keyboard and then clicks function option 1611. In response, referring to user interface 1602, the mobile phone performs semantic understanding based on the text in the text input box and the context, and searches for matching documents based on the results of the semantic understanding. The context may include chat content received by the mobile phone from a corresponding user, such as the chat log sent by the corresponding user in Figure 16. When a matching document is found, a pop-up window 1613 appears. This window can be smaller than the full screen size, for example, it can overlay the soft keyboard window. Window 1613 displays the searched documents, which can be sorted by matching degree from high to low and / or file type. The user can select one of the documents and drag it to the text input box or chat window. Referring to user interface 1603, after dragging the selected document to the target location and releasing it, the document can be sent. Optionally, the text entered by the user in the text input box can also be sent at the same time.
[0193] The following describes the process of user interface changes based on user operations during the implementation of the embodiment of the present application, taking the scenario of uploading information for authentication or registration as an example.
[0194] Referring to Figure 17 , authentication user interface 1701 is used to upload photos of the front and back of a user's ID card to the network. This user interface includes function options 1711 and 1712. When triggered, function options 1711 and 1712 automatically search for ID card images. Taking function option 1711 as an example, function option 1711 can be an image control. Furthermore, function option 1711 displays the text prompt "Upload front." Furthermore, when a user clicks function option 1711, referring to user interface 1702, the phone responds by searching for ID card images, further searching for ID card front images, and displaying the searched images in pop-up window 1713. The user can then upload the searched images.
[0195] Referring to Figure 18 , resume upload user interface 1801 is used to upload a user's resume file to the network. This user interface includes function option 1811. Function option 1811, when triggered, automatically searches for the user's resume. When the user clicks function option 1811, as shown in user interface 1802, the phone searches for the user's resume and displays the searched resume file in a pop-up window 1813. The user can then upload the searched file.
[0196] The specific implementation of the interaction method provided in the embodiment of the present application is described below in conjunction with Figure 19.
[0197] See Figure 19, which is a flow chart of an interactive method provided in an embodiment of the present application, for automatically searching for matching media information based on a user-triggered function option. This method can be executed by an electronic device, an example of which is shown in Figure 1. This method can also be executed by a software function module within the electronic device, an example of which is shown in Figure 2 as a software system architecture within the electronic device. This method can also be implemented using a combination of software and hardware.
[0198] As shown in FIG19 , the method may include the following steps:
[0199] Step 1901: The electronic device receives a first user operation, where the first user operation is used to trigger a first function option in a user interface, and the first function option corresponds to a first media information type.
[0200] Optionally, the first function option can be implemented by a control. The first user operation that triggers the first function option can be a click operation, a double-click operation, or other types of user operations, which are not limited in this embodiment of the present application.
[0201] Optionally, the first function option may correspond to one media information type, such as only a picture type, or only a document type, or only a video type. Optionally, the first function option may correspond to multiple media information types, such as both pictures and videos, or both pictures and documents. This can be implemented through programming of the first function option, and the present embodiment does not impose any limitation on this.
[0202] For the relevant description of the media information and the types of media information involved in the embodiments of the present application, please refer to the aforementioned embodiments and will not be repeated here.
[0203] Step 1902: The electronic device obtains a context associated with the first user operation and / or the first function option.
[0204] In one possible implementation, the process of obtaining the context associated with the first user operation may include obtaining text input by the user before the first user operation and using the text as the context for searching for related media information. For an example, see the scenario shown in FIG15 .
[0205] In one possible implementation, the process of obtaining the context associated with the first user operation may include obtaining text input by the user before the first user operation and content contained in messages sent by other users received by the electronic device. For an example, see the scenario shown in FIG16 .
[0206] In one possible implementation, the process of obtaining the context associated with the first function option may include: obtaining first information associated with the first function option, the first information being used to describe the media information that the first function option is used to obtain, and using the first information as the context. For an example, reference may be made to the scenario shown in Figure 17 . In this scenario, the first information associated with function option 1711 is "Upload a front-facing ID card photo," which describes the function of the function option and includes media information corresponding to the first function option. For example, in this example, "front-facing ID card photo" is used to describe the media information corresponding to the function option.
[0207] Step 1903: The electronic device searches for matching media information according to the semantics of the context and the first media information type corresponding to the first function option.
[0208] In one possible implementation, the electronic device can search for matching media information from the media information stored in the electronic device, such as searching for pictures and videos from the local picture library (such as an album) of the electronic device, and searching for documents from the local folder of the electronic device for storing user documents. In another possible implementation, the electronic device can search for matching media information from the cloud space used by the electronic device, wherein the cloud space can be shared by the electronic device with other electronic devices. In another possible implementation, if the electronic device is interconnected with other electronic devices, the matching media information can be obtained from the other electronic devices through the connection with other electronic devices. It can be understood that the above-mentioned multiple acquisition methods can also be used in combination with each other. The embodiment of the present application does not limit the search scope or source of media information.
[0209] In one possible implementation, the electronic device may prioritize searching for matching media information locally on the electronic device to minimize network overhead. If no matching media information is found, the cloud space is searched for matching media information, and so on. This embodiment of the application does not limit the search order or priority.
[0210] Step 1904: The electronic device displays the searched media information.
[0211] In a possible implementation, the searched media information may be displayed in descending order of matching degree.
[0212] In the process shown in FIG. 19 above, the electronic device can automatically search for matching media information based on the first function option triggered by the user, without the user having to enter search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience.
[0213] Based on the above embodiments, the present application also provides a device, which can be an electronic device or an internal module thereof, and the device includes multiple functional modules; the multiple functional modules interact with each other to implement the functions performed by the electronic device in each method described in the embodiments of the present application. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on the specific implementation. For example, the method steps performed by the electronic device in the above embodiments of the present application are executed.
[0214] Based on the above embodiments, the present application further provides a device, which may be an electronic device or an internal module thereof, comprising at least one processor and at least one memory, wherein the at least one memory stores computer program instructions. When the electronic device is running, the at least one processor executes the functions performed by the electronic device in each method described in the embodiments of the present application. For example, the method steps performed by the electronic device in the above embodiments of the present application are executed.
[0215] Based on the above embodiments, the present application also provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which enables the various methods described in the embodiments of the present application to be executed when the computer program is run.
[0216] Based on the above embodiments, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program (also referred to as code or instructions), which, when executed by a computer, causes the computer to perform the methods described in the embodiments of the present application.
[0217] Based on the above embodiments, the present application further provides a chip, which is used to read a computer program stored in a memory and implement the methods described in the embodiments of the present application.
[0218] Based on the above embodiments, the present application provides a chip system, which includes a processor for supporting a computer device to implement the various methods described in the embodiments of the present application. In a possible embodiment, the chip system also includes a memory, which is used to store the necessary programs and data for the computer device. The chip system can be composed of a chip, or it can include chips and other discrete devices. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0219] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more flow charts and / or one or more boxes in the block diagram.
[0220] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0222] Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of protection of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
Claims
1. An interactive method, characterized in that: Applied to electronic equipment, the method comprises: receiving a first user operation, and acquiring a first text in response to the first user operation; wherein the first user operation is used to input text, and the first text includes text input by the first user operation; searching, according to the first text, for media information that semantically matches the first text; transforming a first-style cursor into a second-style cursor, wherein the second-style cursor indicates that media information has been searched; A second user operation on the second-style cursor is received, and in response to the second user operation, the media information is displayed.
2. The method according to claim 1, characterized in that The second-style cursor indicates the type and / or content of the searched media information.
3. The method according to claim 1 or 2, characterized in that The second cursor style includes one or more of the following: a thumbnail of the content of the media information; The indication information of the media type includes an icon for indicating the type of the media information and / or text for indicating the type of the media information.
4. The method according to claim 3, characterized in that The media information includes pictures, and the second-style cursor displays a thumbnail of the searched picture; or The media information includes a video, and the second style cursor displays a thumbnail of a video frame in the searched video.
5. The method according to any one of claims 2 to 4, characterized in that: The second-style cursor indicates that media information of multiple media types has been searched.
6. The method according to claim 5, characterized in that The receiving a second user operation on the second-style cursor and displaying the media information includes: receiving a third user operation; In response to the third user operation, displaying a first menu, wherein the first menu includes a plurality of function options, each of the plurality of function options corresponding to one of the plurality of media types; Receive a selection operation of a user selecting at least one function option among the multiple function options; In response to the selection operation, media information corresponding to the at least one function option is displayed.
7. The method according to any one of claims 1 to 6, characterized in that: The first user operation acts on a text input box of the first application; After displaying the media information, the method further includes: A fourth user operation is received, and in response to the fourth user operation, the first media information is selected from the displayed media information, and the first media information is applied to the first application.
8. The method according to any one of claims 1 to 7, characterized in that: After the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the method further includes: Highlight the first text.
9. The method according to claim 8, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a fifth user operation, wherein the fifth user operation is used to move a cursor to a first position, wherein the first position is within the range of the first text; In response to the fifth user operation, the second-style cursor is moved to the first position.
10. The method according to claim 9, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a sixth user operation, and in response to the sixth user operation, unhighlighting a portion of the highlighted first text; The media information that matches the semantics of the second text is searched for according to the second text, where the second text is the text highlighted in response to the sixth user operation.
11. The method according to claim 8, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a seventh user operation, and highlighting a third text in response to the seventh user operation, wherein the third text is text other than the first text; The fourth text is searched for media information that matches the semantics of the fourth text, wherein the fourth text includes the first text and the third text.
12. The method according to any one of claims 1 to 11, characterized in that: After searching for media information that semantically matches the first text, or after receiving the second user operation, the method further includes: Displays search keywords.
13. The method according to claim 12, characterized in that Also includes: receiving an eighth user operation, wherein the eighth user operation is used to update the search keyword and request to search for matching media information according to the updated search keyword; In response to the eighth user operation, searching for matching media information according to the updated search keyword; Displays media information searched based on the updated search keyword.
14. The method according to any one of claims 1 to 13, characterized in that: The obtaining of the first text comprises: The text input through the first user operation and the context associated with the text input through the first user operation are acquired.
15. The method according to any one of claims 1 to 14, characterized in that: The step of searching, according to the first text, for media information that semantically matches the first text includes one or more of the following: Searching the electronic device for media information that semantically matches the first text; Searching other electronic devices interconnected with the electronic device for media information that semantically matches the first text; Search the cloud space of the electronic device for media information that semantically matches the first text.
16. The method according to any one of claims 1 to 15, characterized in that: The media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
17. An interactive method, characterized in that: Applied to electronic equipment, the method comprises: Receiving a first user operation, where the first user operation is used to trigger a first function option in a user interface, where the first function option corresponds to a first media information type; Acquire a context associated with the first user operation and / or the first function option; Searching for matching media information according to the semantics of the context and the first media information type corresponding to the first function option; Displays the searched media information.
18. The method according to claim 17, characterized in that The acquiring the context associated with the first user operation includes: The text input by the user before the first user operation and / or the text contained in the message received by the electronic device is obtained, and the text is used as the context.
19. The method according to claim 17, characterized in that The acquiring of the context associated with the first function option includes: First information associated with the first function option is obtained, where the first information is used to describe media information corresponding to the first function option, and the first information is used as the context.
20. The method according to any one of claims 17 to 19, characterized in that: The searched matching media information includes one or more of the following: Searching the electronic device for matching media information; Searching for matching media information in other electronic devices interconnected with the electronic device; Searching for matching media information in the cloud space of the electronic device.
21. The method according to any one of claims 17 to 20, characterized in that: The media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
22. A device, characterized in that: The method comprises a unit or a module for executing the method according to any one of claims 1 to 16, or comprises a unit or a module for executing the method according to any one of claims 17 to 21.
23. A device, characterized in that: include: One or more processors and one or more memories; wherein the one or more memories store one or more programs, and when the programs are executed by the one or more processors, the device executes the method as described in any one of claims 1-16, or executes the method as described in any one of claims 17-21.
24. A chip system, characterized in that: The chip system includes at least one chip and a memory, and the at least one chip is used to read and execute a program stored in the memory to implement the method as described in any one of claims 1 to 16, or to implement the method as described in any one of claims 17 to 21.
25. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program, and when the program is executed on a device, the device executes the method according to any one of claims 1 to 16, or executes the method according to any one of claims 17 to 21.
26. A computer program product, characterized in that When the computer program product is executed on a device, the device is caused to execute the method according to any one of claims 1 to 16, or execute the method according to any one of claims 17 to 21.
Citation Information
Patent Citations
Display apparatus and control method for displaying an operational state of a user's input
CN103813209A
Method for content search and electronic device therefor
CN111512299A
File sending method and device and electronic equipment
CN112333084A
Data transmission method and equipment
CN114077411A
Information processing method and device, computer readable storage medium and electronic device
CN115113773A
