Interaction method and device
By implementing the interactive method of automatic search and cursor style transformation on electronic devices, the cumbersome problem of users manually searching multimedia information when text input is solved, improving operation ease and user experience.
Patent Information
- Application Number
- CN202310654308.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
When users use electronic devices, when they need to insert multimedia information such as pictures or videos into text content, they need to open the gallery application to search manually, which is complicated and has poor user experience.
It provides an interactive method to automatically search for matching media information through the text input by the user, and prompt search results through cursor style transformation, simplifying user operations and improving user experience. The method includes receiving user operations, obtaining text, searching for matching media information, and displaying the searched media information in the text input box. The user can view and select the required media information through the cursor operation.
There is no need for users to manually enter search keywords or trigger search commands, which simplifies the operation process and improves the user experience. It clearly prompts search results through cursor style transformation, which facilitates users to perform subsequent operations.
Smart Images

Figure CN119065535A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to an interaction method and device. Background Art
[0002] With the rapid development of mobile Internet technology, there are more and more scenarios in which users use electronic devices (such as mobile phones or tablets) to call multimedia information such as pictures and videos in electronic devices. For example, when a user uses a memo application to edit a note, if he needs to insert a related picture into the input text content, he usually needs to open the gallery application, search for the target picture manually, and then insert the target picture into the text through user operation. The user operation is cumbersome and the user experience is poor. Summary of the invention
[0003] The embodiments of the present application provide an interactive method and device, which can be applied to electronic devices to automatically search for matching media information based on content input by a user, thereby simplifying user operations and improving user experience.
[0004] In a first aspect, an interaction method is provided, which can be applied to electronic devices, such as mobile phones, tablet computers, wearable devices, etc. The method includes: the electronic device receives a first user operation, and obtains a first text in response to the first user operation; wherein the first user operation is used to input text, and the first text includes the text input by the first user operation; the electronic device searches for media information that semantically matches the first text based on the first text; the electronic device transforms a first style cursor into a second style cursor, and the second style cursor indicates that the media information has been searched; the electronic device receives a second user operation on the second style cursor, and displays the searched media information in response to the second user operation.
[0005] Optionally, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0006] In the above implementation, matching media information can be automatically searched according to the text input by the user, without the need for the user to input search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience. On the other hand, by changing the cursor to indicate that matching media information has been searched, the user can understand the search situation and then perform subsequent operations based on the search situation.
[0007] In a possible implementation, the second-style cursor indicates the type and / or content of the searched media information, which allows the user to intuitively understand the type and / or content of the searched media information, thereby improving the user experience.
[0008] In one possible implementation, the second style cursor includes one or more of the following: a thumbnail of the content of the media information; indication information of the media type, the indication information of the media type including an icon for indicating the type of the media information, and / or text for indicating the type of the media information.
[0009] In one possible implementation, the media information includes pictures, and the second-style cursor displays thumbnails of the searched pictures, so that the search status can be more intuitively indicated to the user. Optionally, the thumbnail is a thumbnail of the picture with the highest matching degree with the first text among the searched pictures. In another possible implementation, the media information includes videos, and the second-style cursor displays thumbnails of video frames in the searched videos, so that the search status can be more intuitively indicated to the user. Optionally, the thumbnail is a thumbnail of the video frame in the video with the highest matching degree with the first text among the searched videos.
[0010] In a possible implementation, the second-style cursor indicates that there are multiple types of searched media information.
[0011] In a possible implementation, when the second-style cursor indicates that there are multiple types of searched media information, receiving a second user operation on the second-style cursor and displaying the media information includes: receiving a third user operation, and in response to the third user operation, displaying a first menu, wherein the first menu includes multiple function options, each of the multiple function options corresponds to one of the multiple media types; receiving a selection operation of a user selecting at least one function option from the multiple function options, and in response to the selection operation, displaying the media information corresponding to the at least one function option. Optionally, the third user operation includes clicking or long pressing the second-style cursor.
[0012] Through the above implementation, when the second style cursor indicates that there are multiple types of searched media information, the user can view all of the multiple types based on user operations, thereby allowing the user to select one or more types of media information to view as needed.
[0013] Optionally, the second user operation is used to trigger one of the multiple function options, and in response to the second user operation, media information of a type corresponding to the function option among the multiple types of media information searched is displayed.
[0014] Optionally, the second user operation is used to trigger at least two of the multiple function options, and in response to the second user operation, at least two types of media information corresponding to the function option among the multiple types of media information searched are displayed.
[0015] In one possible implementation, the first user operation acts on a text input box of a first application; after displaying media information that semantically matches the first text, the method further includes: receiving a fourth user operation, and in response to the fourth user operation, selecting the first media information from the displayed media information, and applying the first media information to the first application; wherein the fourth user operation is used to select the first media information from the displayed media information, and apply the first media information to the first application.
[0016] Optionally, applying the first media information to the first application includes: adding the first media information to a record edited using the first application (such as a memo application); or adding the first media information as an attachment to an email edited using the first application (such as a mail application); or sending the first media information to a target user who is having a conversation using the first application (such as a social application).
[0017] In a possible implementation, after the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the method further includes: highlighting the first text so that the user can understand the text content matching the current search result.
[0018] Optionally, after transforming the first style cursor into the second style cursor, or after receiving the second user operation, it also includes: receiving a fifth user operation, wherein the fifth user operation is used to move the cursor to a first position, wherein the first position is within the range of the first text; and in response to the fifth user operation, moving the second style cursor to the first position.
[0019] Optionally, after the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a sixth user operation, and in response to the sixth user operation, canceling the highlighting of part of the highlighted first text; and searching for media information that semantically matches the second text according to the second text, the second text being the text highlighted in response to the sixth user operation. Through this operation, the search keywords can be reduced according to the user's needs, thereby obtaining search results that meet the user's needs.
[0020] Optionally, after the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a seventh user operation, and in response to the seventh user operation, highlighting a third text, the third text being text other than the first text; and searching for media information that semantically matches the fourth text according to a fourth text, the fourth text including the first text and the third text. Through this operation, search keywords can be added according to user needs, thereby obtaining search results that meet user needs.
[0021] In a possible implementation, after searching for media information matching the first text or receiving the second user operation, the method further includes: displaying search keywords.
[0022] Optionally, the method further includes: receiving an eighth user operation, the eighth user operation being used to update the search keyword and requesting to search for matching media information according to the updated search keyword; in response to the eighth user operation, searching for matching media information according to the updated search keyword; and displaying the media information searched according to the updated search keyword. Through this implementation, media information can be searched according to the keyword specified by the user, so that the search results meet the user's needs.
[0023] In a possible implementation, the acquiring the first text includes: acquiring text input through the first user operation, and a context associated with the text.
[0024] Optionally, the first text may include text input by the first user operation, and on this basis, may also include previously input text. In the scenario of an interactive application, the text input by the first user operation is a chat content (text) for sending to a target user, and the context associated with the text includes: the chat content (text) contained in a chat message from the target user.
[0025] In one possible implementation, the searching for matching media information based on the first text includes one or more of the following: searching for media information matching the first text in the electronic device; searching for media information matching the first text in other electronic devices interconnected with the electronic device; searching for media information matching the first text in the cloud space of the electronic device.
[0026] In a second aspect, an interaction method is provided, which can be applied to electronic devices, such as mobile phones, tablet computers, wearable devices, etc. The method includes: receiving a first user operation, where the first user operation is used to trigger a first function option in a user interface, and the first function option corresponds to a first media information type; obtaining a context associated with the first user operation and / or the first function option; searching for matching media information based on the semantics of the context and the first media information type corresponding to the first function option; and displaying the searched media information.
[0027] Optionally, the media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
[0028] In the above implementation, matching media information can be automatically searched according to the first function option triggered by the user, without the user having to input search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience.
[0029] In a possible implementation, obtaining the context associated with the first user operation includes: obtaining text input by a user before the first user operation and / or text contained in a message received by the electronic device, and using the text as the context.
[0030] In a possible implementation, obtaining the context associated with the first function option includes: obtaining first information associated with the first function option, the first information being used to describe media information corresponding to the first function option, and using the first information as the context.
[0031] In one possible implementation, the search for matching media information includes one or more of the following: searching for matching media information in the electronic device; searching for matching media information in other electronic devices interconnected with the electronic device; searching for matching media information in the cloud space of the electronic device.
[0032] According to a third aspect, a device is provided, comprising a unit or module for executing the method as described in any one of the first aspects above.
[0033] In a possible implementation, the device includes a processing unit and an interaction unit. The processing unit is located in a processor, and the interaction unit may include a unit having a function of sensing user operations, such as a touch unit or an audio collector. The interaction unit is used to receive a first user operation, and the processing unit is used to obtain a first text in response to the first user operation, search for media information that semantically matches the first text based on the first text, and transform the first style cursor into a second style cursor, wherein the second style cursor indicates that the media information has been searched, wherein the first user operation is used to input text, and the first text includes the text input by the first user operation; the interaction unit is also used to receive a second user operation on the second style cursor, and the processing unit is also used to display the media information in response to the second user operation.
[0034] In a possible implementation, the second-style cursor indicates the type and / or content of the searched media information.
[0035] In one possible implementation, the second style cursor includes one or more of the following: a thumbnail of the content of the media information; indication information of the media type, the indication information of the media type including an icon for indicating the type of the media information, and / or text for indicating the type of the media information.
[0036] Optionally, the media information includes pictures, and the second style cursor displays thumbnails of the searched pictures; or, the media information includes videos, and the second style cursor displays thumbnails of video frames in the searched videos.
[0037] In a possible implementation, the second-style cursor indicates that media information of multiple media types has been searched. Optionally, the interaction unit is specifically configured to receive a third user operation, and the processing unit is specifically configured to display a first menu in response to the third user operation, wherein the first menu includes multiple function options, and each function option in the multiple function options corresponds to a media type in the multiple media types; the interaction unit is also configured to receive a selection operation of a user selecting at least one function option in the multiple function options, and the processing unit is also configured to display media information corresponding to the at least one function option in response to the selection operation.
[0038] In one possible implementation, the first user operation acts on a text input box of a first application; after the media information is displayed, the interaction unit is further used to receive a fourth user operation, and the processing unit is further used to select the first media information from the displayed media information in response to the fourth user operation, and apply the first media information to the first application.
[0039] In a possible implementation, after the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the processing unit is further configured to highlight the first text.
[0040] In one possible implementation, after the first style cursor is transformed into the second style cursor and before receiving the second user operation, the interaction unit is also used to receive a fifth user operation, and the fifth user operation is used to move the cursor to a first position, and the first position is within the range of the first text; the processing unit is also used to move the second style cursor to the first position in response to the fifth user operation.
[0041] In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, the interaction unit is also used to receive a sixth user operation, and the processing unit is also used to unhighlight part of the highlighted first text in response to the sixth user operation, and search for media information that semantically matches the second text based on the second text, the second text being the text highlighted in response to the sixth user operation.
[0042] In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, the interaction unit is also used to receive a seventh user operation, and the processing unit is also used to highlight a third text in response to the seventh user operation, the third text being text other than the first text, and searching for media information that semantically matches the fourth text based on a fourth text, the fourth text including the first text and the third text.
[0043] In a possible implementation, after searching for media information that semantically matches the first text, or after receiving the second user operation, the processing unit is further configured to display the search keyword. Optionally, the interaction unit is further configured to receive an eighth user operation, the eighth user operation being used to update the search keyword and request a search for matching media information based on the updated search keyword; the processing unit is further configured to search for matching media information based on the updated search keyword in response to the eighth user operation; and display the media information searched based on the updated search keyword.
[0044] In a possible implementation manner, the processing unit is specifically configured to: obtain text input through the first user operation, and a context associated with the text input through the first user operation.
[0045] In one possible implementation, when the processing unit searches for media information that semantically matches the first text based on the first text, it performs one or more of the following: searching for media information that semantically matches the first text in the electronic device; searching for media information that semantically matches the first text in other electronic devices interconnected with the electronic device; searching for media information that semantically matches the first text in the cloud space of the electronic device.
[0046] In a possible implementation, the media information includes one or more of the following: pictures, videos, audios, documents, electronic tickets, electronic cards, address books, contact information, service cards, and link addresses.
[0047] According to a fourth aspect, a device is provided, comprising a unit or module for executing the method as described in any one of the second aspects above.
[0048] In a possible implementation, the device includes a processing unit and an interaction unit. The processing unit is located in a processor, and the interaction unit may include a unit having a user operation perception function such as a touch unit or an audio collector. The interaction unit is used to receive a first user operation, and the first user operation is used to trigger a first function option in a user interface, and the first function option corresponds to a first media information type; the processing unit is used to obtain a context associated with the first user operation and / or the first function option, and search for matching media information according to the semantics of the context and the first media information type corresponding to the first function option, and display the searched media information.
[0049] In a possible implementation, the processing unit is specifically configured to: obtain text input by a user before the first user operation and / or text contained in a message received by the electronic device, and use the text as the context.
[0050] In a possible implementation, the processing unit is specifically used to: obtain first information associated with the first function option, where the first information is used to describe media information corresponding to the first function option, and use the first information as the context.
[0051] In one possible implementation, when the processing unit searches for matching media information, it is specifically used to perform one or more of the following: searching for matching media information in the electronic device; searching for matching media information in other electronic devices interconnected with the electronic device; searching for matching media information in the cloud space of the electronic device.
[0052] In a possible implementation, the media information includes one or more of the following: pictures, videos, audios, documents, electronic tickets, electronic cards, address books, contact information, service cards, and link addresses.
[0053] In a fifth aspect, a device is provided, comprising: one or more processors and one or more memories; wherein the one or more memories store one or more programs, and when the programs are executed by the one or more processors, the device executes a method as described in any one of the first aspect above, or executes a method as described in any one of the second aspect above.
[0054] In a sixth aspect, a chip system is provided, comprising at least one chip and a memory, wherein the at least one chip is used to read and execute a program stored in the memory to implement a method as described in any one of the first aspects above, or to implement a method as described in any one of the second aspects above.
[0055] In the seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a program, and when the program is run on a device, the device executes any method as described in the first aspect above, or executes any method as described in the second aspect above.
[0056] In an eighth aspect, a computer program product is provided. When the computer program product is run on a device, the device executes a method as described in any one of the first aspect above, or executes a method as described in any one of the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application;
[0058] Figure 2 A software system architecture block diagram of an electronic device in an embodiment of the present application;
[0059] Figure 3 A cursor style change schematic diagram provided in an embodiment of the present application;
[0060] Figure 4 Another cursor style change schematic diagram provided in an embodiment of the present application;
[0061] Figure 5 Another cursor style change schematic diagram provided in an embodiment of the present application;
[0062] Figure 6 Another cursor style change schematic diagram provided in an embodiment of the present application;
[0063] Figure 7aA schematic diagram of a user interface for automatically searching for matching media information and performing cursor changes in a memo application scenario according to a user input provided by an embodiment of the present application;
[0064] Figure 7b A schematic diagram of a user interface for displaying search results by triggering a transformed cursor in a memo application scenario provided by an embodiment of the present application;
[0065] Figure 8 A schematic diagram of a user interface for highlighting text content matching a search result in a memo application scenario provided by an embodiment of the present application;
[0066] Fig. 9 A schematic diagram of a user interface for increasing or decreasing highlighted text content in a memo application scenario to trigger a related processing process provided by an embodiment of the present application;
[0067] Fig.10 A user interface diagram of a related processing process of changing a search keyword trigger in a memo application scenario provided by an embodiment of the present application;
[0068] Fig.11 A schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in a social application scenario provided in an embodiment of the present application;
[0069] Fig.12 A schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in another social application scenario provided in an embodiment of the present application;
[0070] Fig.13 A schematic diagram of a user interface for automatically searching for matching media information following user input and displaying search results in an email application scenario provided in an embodiment of the present application;
[0071] Fig.14 A flowchart of an interactive method provided in an embodiment of the present application;
[0072] Fig.15 A schematic diagram of a user interface for searching for media information of a corresponding type according to a media information type selected by a user and displaying search results in a memo application scenario provided in an embodiment of the present application;
[0073] Fig.16 A schematic diagram of a user interface for searching for media information of a corresponding type according to a media information type selected by a user and displaying search results in a social application scenario provided in an embodiment of the present application;
[0074] Fig.17A schematic diagram of a user interface change process based on user operations in a scenario for uploading an ID card photo provided in an embodiment of the present application;
[0075] Fig.18 A schematic diagram of a user interface change process based on user operations in a scenario for uploading a resume provided in an embodiment of the present application;
[0076] Fig.19 A flowchart of another interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0077] The embodiment of the present application provides a human-computer interaction method that can be applied to electronic devices (such as mobile terminal devices). According to user input, the method can automatically search and prompt media information that matches the input content for user selection.
[0078] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0079] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0080] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.
[0081] The method provided in the embodiment of the present application can be applied to electronic devices. The electronic device can realize human-computer interaction. For example, the electronic device can be an electronic device such as a mobile phone, a tablet computer, a wearable device (e.g., a watch, a bracelet, etc.), a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart home device (e.g., a smart TV, a smart speaker, etc.), etc., which can have a human-computer interaction function. It is understandable that the embodiment of the present application does not impose any restrictions on the specific type of electronic device.
[0082] The electronic devices to which the embodiments of the present application can be applied include, but are not limited to, Or electronic devices with other operating systems. The electronic device may also be other electronic devices, such as a laptop computer (Laptop) with a touch-sensitive surface (eg, a touch panel).
[0083] Figure 1 FIG. 1 shows a possible hardware structure diagram of an electronic device. Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0084] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. When the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
[0085] The display screen 194 is used to display the display interface of the application, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N (N is a positive integer greater than 1) display screens 194. For example, the electronic device 100 may be a mobile phone with a foldable display screen, and the mobile phone has at least two display screens. In an embodiment of the present application, the display screen 194 can be used to display a page (or interface) including at least one application component, for example Figure 3 The interface 31 or the interface 32 shown in .
[0086] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, and the software code of at least one application, etc. The data storage area can store data (such as images, videos, etc.) generated during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0087] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194. In an embodiment of the present application, the touch sensor 180K can detect user operations, and can pass the information of the detected user operations to the upper layer, so that the electronic device can respond to the user operation to implement the addition, deletion or update of application components.
[0088] Understandably, Figure 1 The components shown do not constitute specific limitations on the electronic device, and the electronic device may also include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. Figure 1 The electronic device 100 shown is taken as an example for introduction.
[0089] Figure 2 This is a software system architecture block diagram of an electronic device provided in an embodiment of the present application. Figure 2 As shown, the software system architecture of the electronic device can be a layered architecture, for example, the software can be divided into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (framework, FWK), the runtime and system library, the kernel layer, and the hardware layer.
[0090] The application layer can include a series of application packages. Figure 2 As shown, the application layer may include camera, settings, skin module, user interface (UI), third-party applications, etc. Among them, the third-party applications may include wireless local area network (WLAN), music, calls, Bluetooth, video, memo, notes, etc.
[0091] In one possible implementation, applications can be developed using the Java language by calling the application programming interface (API) provided by the application framework layer. Developers can interact with the underlying layers of the operating system (such as the hardware layer, kernel layer, etc.) through the application framework layer to develop their own applications. The application framework layer is mainly a series of services and management systems for the operating system.
[0092] The application framework layer provides application programming interfaces and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include an activity manager, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0093] The activity manager is used to manage the life cycle of each application and provide common navigation back functions, providing an interactive interface for all program windows.
[0094] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make this data accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0095] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying pictures. In some embodiments of the present application, a control that receives and displays text (or text) can provide the text input by the user to an application or service for implementing semantic understanding while displaying the text in response to a user input operation, and the application or service can further obtain media information matching the semantics. Optionally, the application can be set at the application layer, such as the application can be a third-party application; the service can be set at the application framework layer, such as the service can be a service provided by an operating system. Optionally, the application or service can be a natural language processing tool driven by technologies such as artificial intelligence and neural networks, which can realize semantic analysis and understanding of natural language, such as semantic understanding of the text input by the user or the user dialogue, and extracting keywords. The application or service can also further search for matching media information based on keywords, or it may be handed over to other applications or services to search for matching media information based on keywords. The embodiments of the present application do not limit the semantic understanding method or technology used by the application or service, or the media information search method.
[0096] Optionally, from the perspective of content type, the media information may include one or more of the following: photos, audio, video, documents, contact address books, contact information, electronic tickets, electronic cards, service cards, etc.; from the perspective of data type, the media information may include one or more of the following: image files, audio files, video files, text files, PDF files, files generated by office software (such as macrosoft office documents or WPS documents), contact data, ticket data, etc. The media information may also include link address information, such as a web page link address in a browser favorites. The embodiment of the present application does not limit the content type or data type of the media information.
[0097] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).
[0098] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.
[0099] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0100] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the operating system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of the operating system. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.
[0101] The system library may include multiple functional modules. For example: surface manager, media framework, 3D graphics processing library (e.g. OpenGL ES), 2D graphics engine (e.g. SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media framework can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing. In some embodiments, the 3D graphics processing library can be used to draw 3D motion trajectory images, and the 2D graphics engine can be used to draw 2D motion trajectory images.
[0102] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.
[0103] The hardware layer can include various types of sensors, such as accelerometers, gravity sensors, touch sensors, etc.
[0104] It should be understood that the hardware structure of the electronic device can be as follows Figure 1 As shown, the software system architecture can be Figure 2As shown, the software programs and / or modules corresponding to the software system architecture in the electronic device can be stored in the internal memory 121, and the processor 110 can run the software programs and applications stored in the internal memory 121 to execute the process of the display method provided in the embodiment of the present application.
[0105] In an embodiment of the present application, after the electronic device searches for media information that semantically matches the text input by the user, the electronic device can change the style of the cursor to prompt the user that the matching media information has been searched.
[0106] A cursor is also called a cursor and can be used to indicate the position of text. When a user enters text in a text box or an application, the cursor can be used to display the position where the user is going to enter the text. The cursor position can also be moved forward or backward based on user operations. At present, in the user interface of a mobile phone or tablet computer, the cursor is usually displayed as a small vertical line and can flash regularly. In some embodiments of the present application, more diverse cursor styles are provided. As the user enters text (or text), when matching media information is searched based on the text (or text) entered by the user, the first style cursor (such as a small vertical line) can be transformed into a second style cursor provided in an embodiment of the present application. The second style cursor can prompt the user to search for media information matching the input text (or text).
[0107] In the following description of this specification, the cursor provided in the embodiment of the present application for prompting the user to search for matching media information is referred to as a second-style cursor, and correspondingly, the cursor displayed in a normal state or when matching media information has not been searched is referred to as a first-style cursor. For example, the first-style cursor can be a small vertical line that flashes regularly, or a cursor of other styles set by the user through system settings.
[0108] In a possible implementation, the second style cursor may have a different shape, color, and / or size than the first style cursor, so as to prompt the user to search for matching media information. Generally, changing the shape of the cursor is more intuitive and obvious to the user than changing the size and / or color of the cursor, so that the user can see the change in cursor style more intuitively and obviously, thereby deciding the next operation. The following embodiments of the present application are described by taking the second style cursor having a different shape from the first style cursor as an example.
[0109] An example could be Figure 3 As shown. Figure 3 FIG. 10( a ) shows a first-style cursor 301 in a normal state in a memo text input box. The first-style cursor 301 is displayed at the end of the text and is specifically displayed as a vertical line. Figure 3In (b), when the electronic device searches for matching media information based on the text input by the user, a second style cursor 302 is displayed in the memo text input box. The second style cursor 302 is displayed at the end of the text. In one possible manner, the second style cursor can be displayed in a magnifying glass shape to prompt the user to search for matching media information based on the text content before the cursor. In order to better prompt the user of the searched media information, in a possible implementation, the type of the searched media information can be prompted to the user by displaying a text prompt in the second style cursor 302, such as displaying the text "picture" in the second style cursor 302 to prompt that matching picture type media information has been searched.
[0110] In another possible implementation, an identifier for representing a picture may also be displayed in the second style cursor to prompt that media information of a matching picture type has been searched. Figure 4 As shown, in Figure 4 FIG. 4 shows a first-style cursor 401 in a normal state in a memo text input box. The first-style cursor 401 is displayed at the end of the text and is specifically displayed as a vertical line. Figure 4 In (b), when the electronic device searches for a matching photo according to the text input by the user, a second-style cursor 402 is displayed in the memo text input box. The second-style cursor 402 is displayed at the end of the text, specifically in the shape of a magnifying glass, and a picture type identifier 403 is displayed within the range of the magnifying glass lens to prompt that media information of the picture type has been searched.
[0111] In order to more intuitively prompt the user with the searched media information, in another possible implementation, for the searched media information of the image or video type, a thumbnail of the image or video frame can be displayed in the second style cursor. Optionally, a thumbnail of the image with the highest matching degree or a thumbnail of a video frame (such as the first frame) in the video with the highest matching degree can be displayed. An example can be as follows Figure 5 As shown, in Figure 5 FIG. 5 shows a first-style cursor 501 in a normal state in a memo text input box. The first-style cursor 501 is displayed at the end of the currently input text and is specifically displayed as a vertical line. Figure 5 In (b), when the electronic device searches for matching pictures and / or videos according to the text input by the user, a second style cursor 502 is displayed in the memo text input box. The type of the searched media information and the content of the searched media information can be prompted to the user by displaying a picture thumbnail or a video frame thumbnail in the second style cursor 502. Figure 5In (b) of the figure, taking the searched picture as an example, in the second style cursor 502 in the shape of a magnifying glass, a thumbnail 503 of the picture with the highest matching degree is displayed within the range of the magnifying glass lens to prompt the content of the searched media information.
[0112] Based on the above examples, it can be understood that the second style cursor can indicate the type and / or content of the searched media information. For example, the second style cursor includes one or more of the following: a thumbnail or a partial image of the content of the media information; information indicating the media type, the information indicating the media type includes an icon for indicating the type of the media information; text for indicating the type of the media information.
[0113] It can be understood that the above only exemplifies several possible forms of the second style cursor, and the embodiments of the present application do not limit this. For example, the second style cursor may also include local content in the searched image.
[0114] When multiple types of media information are searched, in order to prompt the user that multiple types of media information are searched, in a possible implementation, a text prompt (for prompting that multiple types of media information are searched) or a certain logo (such as an icon for indicating that multiple types of media information are searched) can be displayed in the second style cursor, or the cursor shape, or other forms can be used to prompt the user that multiple types of media information are searched. An example can be as follows Figure 6 As shown, in Figure 6 FIG. 10( a ) shows a first-style cursor 601 in a normal state in a memo text input box. The first-style cursor 601 is displayed at the end of the currently input text and is specifically displayed as a vertical line. Figure 6 In Figure (b), after the electronic device searches for multiple types of matching media information based on the text input by the user, a second style cursor 602 is displayed in the memo text input box. Taking the search for three types of media information, namely pictures, videos and documents (such as word files) as an example, the second style cursor 602 can be displayed in the form of three magnifying glasses stacked together, prompting the user that three types of media information have been searched. Optionally, the top-level magnifying glass can display relevant information of the media information with the highest semantic match to the text (such as thumbnails, text prompts, icons, etc.). Optionally, the top-level magnifying glass can also display relevant information of the media information that matches the user's behavior habits based on the user's behavior habits. For example, Figure 6 As shown in Figure (b), since the frequency of users viewing photos or browsing the image library is higher than the frequency of viewing videos and word files, thumbnails of the searched pictures can be displayed in the top magnifying glass.
[0115] In order to better prompt the user the type of media information searched, such as Figure 6 As shown in FIG. (c), when the user triggers the second style cursor 602 by clicking or other user operations (such as long pressing or double clicking), a menu 603 is displayed in response. For the three types of media information searched, the menu 603 includes three corresponding function options (such as function options 604, 605 and 606 shown in the figure), each function option corresponds to the search result of a type of media information, and the user can select a function option in the menu 603 (such as clicking a function option), so that the searched media information of the type corresponding to the function option can be displayed. For example, after the user clicks the function option 604 in the menu 603, the searched picture can be displayed in a pop-up window. Figure 6 As shown in FIG. 6( c ), the function options 604 , 605 and 606 may indicate the type and / or content of the media information corresponding to the corresponding function options by means of thumbnails, text prompts, etc. Figure 3-Figure 5 In the illustrated embodiment, the function options may include one or more of the following: thumbnails or partial images of the content of the media information; information indicating the media type, the information indicating the media type including icons for indicating the type of the media information; text for indicating the type of the media information. For example, function option 604 may be triggered by a picture thumbnail indication to display the searched pictures, and function option 605 may be triggered by a text prompt indication to display the searched word files.
[0116] Optionally, multiple (including two or more) media information types can also be selected based on menu 603, so that corresponding types of media information can be viewed. Figure 6 Taking Figure (c) in the figure as an example, when the user long presses function option 604, a check box can be displayed in front of or behind or in the upper right corner or upper left corner of function options 604, 605 and 606, respectively, for the user to select the media information type corresponding to one or more function options among function options 604, 605 and 606. Figure 6 In (d) of the figure, a check box is displayed in the upper right corner as an example. When the user long presses function option 604, a check box can be displayed in the upper right corner of function options 604, 605 and 606, wherein the check box in the upper right corner of function option 604 is selected, and the check boxes of the other two function options are unselected. When the user selects the type of media information to be viewed through the check box and clicks the "Confirm" button, in response, the electronic device will display a list of media information of the corresponding type that has been searched. It can be understood that other methods can also be used to implement the check function of multiple options, and the embodiments of the present application are not limited to this.
[0117] Optionally, the menu 603 may also include a function option for prompting to view all types of media information. When the function option is triggered (for example, after the user clicks the function option), all types of media information may be displayed. Optionally, different types of media information may be displayed according to the type of media information, for example, a picture list, a word file list, and a video list may be displayed.
[0118] against Figure 6 In another possible implementation method under the illustrated scenario, unlike the user operation of long pressing the display menu 603 to view all types of searched media information, when the user clicks or double-clicks the second style cursor 602, all types of searched media information can be directly displayed according to the type of media information.
[0119] It can be understood that the above only exemplifies several implementation methods of browsing the searched media information by performing user operations on the second style cursor, and the embodiments of the present application do not limit this.
[0120] Optionally, when multiple types of media information are searched, Figure 6 As an alternative to the scenario shown, the first style cursor 601 can also be directly transformed into Figure 6 The cursor shape shown in Figure (c) in the figure, that is, all types of searched media information can be directly prompted in the transformed cursor, thereby further reducing the user's operations.
[0121] It can be understood that the above second-style cursors are only several possible examples, and the present application embodiment does not limit the shape, size, color and other attributes of the second-style cursor. Figure 5 For example, the second style cursor may also be displayed as the second style cursor 502 rotated 180 degrees, so as to avoid the second style cursor covering the input text as much as possible.
[0122] The following describes the user operations and user interface changes in the interaction process implemented in the embodiments of the present application, taking the scenarios of a memo application, a social application, and an email application as examples.
[0123] First, taking the memo application as an example, the user interface change process based on user operations during the implementation of the embodiment of the present application is described.
[0124] See also Figure 7a and Figure 7b In the embodiment of the present application, a mobile phone is taken as an example to provide a user interface schematic diagram for automatically searching for matching media information and displaying search results in a memo application scenario following user input.
[0125] like Figure 7aAs shown, in the interface of the memo application, a text input box 71 and a soft keyboard 72 are included, and the user can input text in the text input box 71 through the soft keyboard 72. Figure 7a From user interface 701 to user interface 703 in the figure, the user uses the soft keyboard 72 to continuously input a text into the text input box 71: "During the National Day, we took our daughter Yaya to the animation theme park. There is a big cute cat statue at the entrance of the park." According to the above text content, the mobile phone can obtain the keywords "National Day", "Ya Ya" and "Animation Theme Park" through semantic understanding. Based on the keywords, the mobile phone searches for multiple photos that match this. For example, the mobile phone can search for photos taken during the "National Day" holiday based on the keyword "National Day", search for photos containing Yaya's head portrait based on the keyword "Ya Ya", and search for photos taken in the animation theme park or photos containing animation theme elements (such as animation characters and / or animation buildings, etc.) based on the keyword "Animation Theme Park".
[0126] See also Figure 7a From the user interface 703 to the user interface 705, since the mobile phone has searched for photos matching the input text, the first style cursor (such as Figure 7a ) is transformed into a second-style cursor (such as Figure 7a In order to improve the user experience, dynamic effects can be used to display the cursor style changes. Figure 7a The cursor style change effect shown is: in user interface 703, the cursor changes from a vertical line to a magnifying glass shape; in user interface 704 to user interface 705, in the lens area of the magnifying glass, the thumbnail of the photo that best matches the keyword gradually changes from blurry to clear.
[0127] See also Figure 7bIn the user interface 706 in the text input window, after the user clicks the second style cursor in the user interface 705, a window 73 may pop up, at which point the second style cursor may be restored to the first style cursor, indicating that a semantic understanding and related media information search is completed. Window 73 may be overlaid on the upper layer of the soft keyboard window so that the text input box 71 is not covered or not completely covered. The searched pictures may be displayed in window 73. If there are multiple photos searched, the multiple photos searched may be displayed in a stacked manner, and the user may select a photo by sliding left and right. Optionally, the default displayed photo may be the photo that best matches the semantics of the text. Optionally, the total number of searched photos may also be displayed in window 73 to facilitate the user to understand the search results and make a selection. In another possible implementation, the searched photos may be arranged in a list or grid in order of high to low matching. Further, if the number of searched photos is large, a sliding bar may be provided to allow the user to slide up and down through the sliding bar to browse the searched photos.
[0128] See also Figure 7b From user interface 707 to user interface 708 in window 73, after the user selects a photo in window 73, the user can drag it to the text input box 71 by dragging. Optionally, during the user operation, prompt information can also be displayed on the user interface to guide the user operation, such as the prompt information "Slide to this area to insert the picture" shown in user interface 707 and the prompt information "Release to insert" shown in user interface 708. Referring to user interface 709, when the selected photo is dragged to the text input box 71 and lifted, the photo can be automatically added to the position of the first style cursor. Through the above-mentioned dragging operation, the selected photo can be inserted into the position of the cursor. Optionally, after a searched photo is added to the text by dragging, window 73 can display the next photo with the second highest matching degree. Similar to the above-mentioned operation of adding photos to the text, the mobile phone can add multiple photos to the text in sequence based on the user's dragging operation.
[0129] In some embodiments of the present application, the text content corresponding to the search results can also be highlighted to let the user know which part of the text content matches the currently searched media information. In one possible implementation, after the matching media information is searched based on the text (or text) input by the user, the corresponding text (or text) content can also be highlighted while changing the cursor style. In another possible implementation, after the user triggers the second style cursor by clicking or other operations, the corresponding text content can also be highlighted while displaying the search results. Optionally, the implementation method of highlighting the corresponding text content may include but is not limited to: displaying the corresponding text content in different font colors, or displaying the corresponding text content in different font styles (or font shapes), or displaying the corresponding text content in a highlighted manner, or displaying the corresponding text content in an emphasized display manner (such as making the font bold), or combining the above multiple methods to highlight the corresponding text content.
[0130] An example could be Figure 8 As shown, in user interface 801, a matching video is searched based on the highlighted text content in the text input box (such as the paragraph starting with "3:00 p.m." as shown in the figure), and therefore the first style cursor is transformed into a second style cursor, in which a video type identifier is displayed, indicating that a matching video is searched based on the highlighted text content. When the user clicks elsewhere in the paragraph, see user interface 802, the second style cursor moves to the position where the user clicks. Since the video search result is searched based on the text content, moving the cursor within the text range will not trigger a new search process, and the current cursor style will not change. Furthermore, the paragraph can remain highlighted. The user can add the searched video at the current cursor location. Optionally, the operation of adding a video can be performed with Figure 7b The operation of adding pictures is similar.
[0131] In some embodiments of the present application, based on highlighting the text content corresponding to the search results, the user can reduce the highlighted text content or increase the highlighted text content by double-clicking or other operations. Since the highlighted text content changes, a new search process may be triggered, which may lead to changes in the search results, and accordingly, the cursor style may also change accordingly.
[0132] An example could be Fig. 9As shown. In one scenario, in user interface 901, the mobile phone searches for a matching video based on the highlighted text content, and the second style cursor contains a video type identifier, indicating that a matching video has been searched. When the user double-clicks at position 1 of user interface 901, referring to user interface 902, the sentence at position 1 (which can be divided by punctuation marks, such as "the float tour at 3:00 p.m. begins") is no longer highlighted. For the remaining highlighted text content, the mobile phone searches again, and the search results remain unchanged, so the cursor style also remains unchanged. It can be understood that this example is described by taking the case where the search is performed again according to the updated highlighted text and the search results remain unchanged as an example. In other scenarios, due to the reduction of the highlighted text content through user operation, when searching again according to the changed text content, it is possible that the search results obtained by the re-search may change due to semantic changes or reduction of search keywords. If the type of media information obtained by the re-search changes, such as the type changes, or a new type is added, or the original type is reduced, the second style cursor will change accordingly to match the new search results.
[0133] In another scenario, when the user double-clicks at position 2 in user interface 901, see user interface 903, the sentence at position 2 (which can be divided by punctuation marks, such as "Ya Ya took more than a dozen photos with the cute cat") is highlighted. See user interface 904. The mobile phone searches again for the text content currently highlighted. The search results include matching videos and pictures. Compared with the text content highlighted in user interface 901, the semantics have changed, and search keywords (such as cute cats, group photos, etc.) have been added, resulting in changes in search results. Correspondingly, the cursor style also changes. Fig. 9 As shown in the user interface 904 in FIG. 1 , the transformed cursor includes a video type identifier and a picture type identifier, and the transformed cursor can prompt the search for two types of media information. The user can Figure 6 The method of the illustrated embodiment (eg, clicking on a logo of one type) browses search results of the corresponding type.
[0134] In the above process, the implementation method of using punctuation marks as separators to select the sentence where the user operation position is located is only an example. The embodiment of the present application does not limit the implementation method of selecting text content based on user operation. For example, in another example, based on the user's finger "press-slide-lift" operation, the text content between the "press" action position and the "lift" action position can be selected; in another example, association can be made based on the text selected by the user to select the text content.
[0135] In some embodiments of the present application, after searching for media information that semantically matches the text input by the user, the search keyword may be displayed. In other embodiments, when the searched media information is displayed after the second style cursor is triggered, the search keyword may be displayed. Fig.10 As shown, in user interface 1001, when the second style cursor is clicked, the searched pictures are displayed, and a search box 1010 is displayed. The search box 1010 displays search keywords corresponding to the current search results. For example, the currently searched pictures are searched based on the keywords "National Day", "Ya Ya" and "cute cat", so the above search keywords are displayed in the search box by default.
[0136] In some embodiments of the present application, the user may also be allowed to modify the search keywords so that the search can be performed again according to the user's needs. Fig.10 As shown, as the user enters text in the text input box of the memo, the mobile phone searches for a matching picture based on the text entered by the user, and then changes the first style cursor to the second style cursor. When the user triggers the second style cursor, see user interface 1001, a window for displaying search results can be displayed on the upper layer of the soft keyboard window, in which the searched picture is displayed, and a search box 1010 is displayed at the same time. The search box 1010 displays the search keywords corresponding to the search results by default (as shown in the figure, "National Day", "Ya Ya" and "Lovely Cat"), and the search box also includes a command button for triggering a search operation (as shown in the figure, the "Search" button). When the user clicks on the text input part in the search box 1010, see user interface 1002, optionally, the full screen can be pulled up to display the search results corresponding to the search box 1010, and further, a soft keyboard can be displayed so that the user can change the search keywords in the search box 1010 through the soft keyboard.
[0137] The user can update the search keywords displayed by default in the search box 1010, for example, enter a new search keyword, delete some of the search keywords, enter new keywords after deleting some of the search keywords, or delete all the search keywords displayed by default in the search box 1010 and enter new keywords. Referring to user interface 1002, this example takes the deletion of "National Day" and "Cute Cat" in the search box 1010 as an example, and the updated search keyword is "Ya Ya". Referring to user interface 1003, when the user clicks the "Search" command button, the mobile phone searches for matching media information based on the updated search keywords. Optionally, after searching for media information matching the updated search keywords, the full-screen search interface can be exited and returned to the memo input interface, referring to user interface 1004, which can display media information matching the updated search keywords. Optionally, the user can select the searched media information and add the selected media information to the current cursor position in the memo text input box.
[0138] The following describes the changes in the user interface during the interaction process implemented in the embodiments of the present application by taking a social application as an example.
[0139] See also Fig.11 In an embodiment of the present application, a mobile phone is used as an example to provide a user interface schematic diagram for automatically searching for matching media information according to user input and displaying search results in a social application scenario.
[0140] like Fig.11 As shown, in the user interface 1101 of the social application, as the user enters text in the input box 1111, the mobile phone can perform semantic understanding on the text entered by the user, or further perform semantic understanding on the text entered by the user in combination with the context, and search for media information based on the result of semantic understanding. The context may include the chat content sent by the other user received by the mobile phone, such as Fig.11The chat record sent by the other user in the chat window. When a matching document is searched, the cursor is changed to a second style cursor 1112, in which a document type identifier is displayed to prompt that a matching document has been searched. When the user clicks the second style cursor 1112, referring to the user interface 1102, a pop-up window 1115 is popped up, and the size of the window can be smaller than the full screen size, for example, it can be covered on the upper layer of the soft keyboard window. Multiple documents searched can be displayed in the window 1115, and the multiple documents can be sorted from high to low according to the matching degree and / or file type. The user can select one of the documents and send it to the chat window. Optionally, if the user is allowed to send the media information by dragging the media information to the chat window, the text input box and related buttons (such as the additional function button 1113) may not be displayed in the user interface 1102. Referring to the user interface 1103, when the selected document is dragged to the target location and released, the document is sent, or the document and the text entered by the user in the text input box 1111 are sent at the same time.
[0141] In one possible implementation, Fig.11 In the illustrated embodiment, the user may not need to input anything into the input box. When the input focus is in the input box (i.e., the cursor is displayed in the input box), the mobile phone may semantically understand the context and search for media information based on the result of the semantic understanding. The context may be a previous chat record of the user. It is understandable that nothing may be entered into the input box at this time. When a matching document is searched, the cursor may also be transformed into a second style cursor 1112, and the user may obtain the searched media information through the second style cursor 1112.
[0142] See also Fig.12 , taking a mobile phone as an example, an embodiment of the present application provides a user interface schematic diagram for automatically searching for matching media information and displaying search results in another social application scenario following user input.
[0143] like Fig.12As shown, in the user interface 1201 of the social application, the mobile phone receives the user's voice, converts the voice into text and displays it in the text input box. The mobile phone performs semantic understanding on the text and combines it with the context, and searches for media information based on the result of the semantic understanding. When a matching picture is searched, the cursor is changed to a second style cursor, and the cursor may display an identification of the picture type to prompt that a matching picture has been searched. When the user clicks the second style cursor, see user interface 1202, and a pop-up window 1211 is displayed. The size of the window may be smaller than the full screen size, for example, it may be covered on the upper layer of the soft keyboard window. The searched picture is displayed in window 1211. See user interface 1203, the user can drag the picture into the chat window. See user interface 1204, when the picture is dragged into the chat window and released, the picture is sent out. Optionally, the text entered by the user in the text input box can be retained in the text box, or the text in the text input box can be sent out while sending the picture.
[0144] The following describes the changes in the user interface during the interaction process implemented in the embodiment of the present application by taking the mail application as an example.
[0145] See also Fig.13 In the embodiment of the present application, a mobile phone is taken as an example to provide a user interface schematic diagram for automatically searching for matching media information according to user input and displaying search results in an email application scenario.
[0146] like Fig.13 As shown, in the user interface 1301 of the mail application, as the user inputs text in the body of the mail, the mobile phone displays the text input by the user in the text input box, and performs semantic understanding on the text, and searches for media information based on the results of the semantic understanding. After searching for a matching voice file, the cursor is transformed into a second style cursor, in which a voice type logo may be displayed to prompt that a matching voice file has been searched. When the user clicks the second style cursor, see user interface 1302, a pop-up window 1311, the size of which may be smaller than the full screen size, for example, may be covered on the upper layer of the soft keyboard window. The searched voice file is displayed in window 1311. See user interface 1303, the user can drag the voice file into the body of the mail, and when the voice file is dragged into the body of the mail and released, the voice file can be added to the mail as an attachment.
[0147] It can be understood that some of the above examples are described by taking the example of selecting one media information (such as a picture or a video or a document) from multiple media information searched, but the embodiments of the present application are not limited to this. In some other embodiments of the present application, the user is allowed to select two or more media information from the multiple media information searched. For example, a check box can be provided for each searched media information, so that the user can select multiple media information through the check box.
[0148] Combine the following Fig.14 , the specific implementation method of the interaction method provided in the embodiment of the present application is described.
[0149] See also Fig.14 , is a flow chart of an interactive method provided in an embodiment of the present application, which is used to automatically search for matching media information according to the text input by the user, and can further indicate the search status by changing the cursor. The method can be executed by an electronic device, and an example of an electronic device can be as follows Figure 1 The method may also be executed by a software function module in an electronic device. An example of a software system architecture in an electronic device is as follows: Figure 2 The method can also be implemented by combining software and hardware.
[0150] like Fig.14 As shown, the method may include the following steps:
[0151] Step 1401: The electronic device receives a first user operation, and acquires a first text in response to the first user operation.
[0152] The first user operation may be used to input text, for example, the first user operation may be used to input text into a text input box of an application. For example, the first user operation is a user operation of inputting text into a text box. Figure 7a In the memo application scenario shown, the first user operation may be an operation in which the user enters the memo text in the text input box through the soft keyboard; for example, in Fig.11 In the scenario of the social application shown in FIG. 1 , the first user operation may be an operation in which the user inputs a conversation text in a text input box using a soft keyboard; for example, in Fig.13In the email application scenario shown, the first user operation can be an operation in which the user enters the body of the email. It is understandable that in addition to using the soft keyboard to enter text, text can also be entered using handwriting input and other methods. Optionally, the first user operation can also be an operation of obtaining the user's voice and converting the user's voice into text, and the converted text can be displayed in the text input box. Optionally, the first user operation can also be a "paste" operation. For example, you can long press the text input box and select the "Paste" option in the pop-up menu, so that the text in the clipboard can be entered into the text input box.
[0153] The text input process is usually a process of multiple continuous inputs. For the sake of ease of description, in the embodiment of the present application, each operation of entering a piece of text content (which may contain only one character or one word, or may contain multiple characters or multiple words, or contain one or more sentences) in the text input box is called a first user operation.
[0154] After entering text in the text input box of the application, in response, a first text containing at least the text content currently entered can be obtained for semantic analysis and understanding. In this way, following the user's text input operation, semantic analysis and understanding can be performed in real time based on the text entered by the user. Optionally, the first text may only include text entered through the first user operation, such as when no text has been entered before the text is entered through the first user operation. Optionally, the first text may include text entered through the first user operation and text entered before that, such as the above Figure 7a and Figure 7b Optionally, the first text includes text input by a first user operation and a context, and the context is associated with the text input by the first user operation, such as Fig.11 In the social application scenario shown, the first user operation is a user operation of inputting text into the text input box provided by the social application. The text is the chat content to be sent to the target user, and the context associated with the chat content includes the text content in the chat message previously sent by the target user. It can be understood that in the one-to-one chat (or private chat) scenario, the number of the target user is one; in the group chat scenario, the number of the target user can be one or more, that is, the context can include the text content in the chat messages of multiple users received by the application.
[0155] In a possible implementation, when the text input box receives a text (which may include one or more words or characters) at time n, if no new text is received within time t, semantic understanding of the previously input text content can be performed at time n+t. The value of time t can be preset.
[0156] In a possible implementation, text that has been input and semantically understood and for which matching media information has been searched based on the semantic understanding result can be marked as having been semantically understood, so that subsequent repeated semantic understanding of the text can be avoided.
[0157] Step 1402: The electronic device searches for media information that semantically matches the first text based on the first text.
[0158] In a possible implementation, the electronic device may search for matching media information based on the semantics of the first text. Optionally, in an embodiment of the present application, a natural language processing tool driven by artificial intelligence, neural network or other technologies may be used to implement semantic analysis and understanding.
[0159] In some embodiments of the present application, the functionality of the text input box control can be expanded so that it can respond to specified events (such as user input operation events, or events in which the pause time during the user input operation exceeds a set threshold) to request or call a natural language processing tool to perform semantic understanding of the first text.
[0160] In one possible implementation, the electronic device can search for media information matching the first text from the media information stored in the electronic device, such as searching for pictures and videos matching the first text from a local picture library (such as an album) of the electronic device, and searching for documents matching the first text from a folder of the electronic device for storing user documents. In another possible implementation, the electronic device can obtain media information matching the first text from a cloud space used by the electronic device, wherein the cloud space can be shared by the electronic device and other electronic devices. In another possible implementation, if the electronic device is interconnected with other electronic devices (such as a far-field connection or a near-field connection), media information matching the first text can be obtained from the other electronic devices through connection with other electronic devices. It can be understood that the above-mentioned multiple acquisition methods can also be used in combination with each other. The embodiment of the present application does not limit the search scope or source of media information.
[0161] In one possible implementation, the electronic device may preferentially search for matching media information locally on the electronic device to minimize network overhead. If no matching media information is found, the matching media information is searched from the cloud space, and so on. The present application embodiment does not limit the search order or priority.
[0162] Step 1403: The electronic device transforms the first-style cursor into a second-style cursor, and the second-style cursor can indicate that media information has been searched.
[0163] In the embodiment of the present application, by changing the first style cursor to the second style cursor, it is possible to indicate to the user that matching media information has been searched, so that the user can proceed to the next step according to the searched media information. For examples of the second style cursor, reference can be made to the description in the foregoing embodiment, for example, reference can be made to Figures 3 to 6 Several examples are given.
[0164] Optionally, the type of the searched media information may be indicated by a second-style cursor, for example, by displaying a text prompt, a type identifier, etc. in the second-style cursor. Figure 8 In the example shown, the second style cursor displays a video type identifier to indicate that media information of the video type has been searched. For another example, Fig.11 In the example shown, the second style cursor displays an identifier of the document type to indicate that media information of the document type has been searched.
[0165] Optionally, when the searched media information includes a picture type, a thumbnail of the searched picture may be displayed in the second style cursor, such as Figure 7a In the example shown, a thumbnail of a picture is displayed in the second style cursor, so that the search result can be more intuitively indicated to the user. The thumbnail can be a thumbnail of a picture with the highest matching degree with the first text among the searched pictures.
[0166] Optionally, when the searched media information includes a video type, a thumbnail of a video frame in the searched video may be displayed in the second style cursor, so as to more intuitively indicate the search result to the user. The thumbnail may be a thumbnail of a video frame in the searched video that has the highest matching degree with the first text.
[0167] Optionally, when multiple types of media information are searched, the second style cursor may be used to indicate that the searched media information has multiple types, such as Figure 6 In the example shown, multiple types of media information can be searched through the stacked shape indication. Further, the third user operation (such as Figure 6 Clicking in the example shown) displays instructions for each of the multiple types. Figure 6 In the example shown, each type of indication information can be displayed in a menu manner. Specifically, a first menu can be displayed based on a third user operation, wherein the first menu includes multiple function options, each of which corresponds to one of the multiple media information types searched. It can be understood that the embodiment of the present application does not limit the type of the third user operation.
[0168] Step 1404: The electronic device receives a second user operation on a second-style cursor, and displays the searched media information in response to the second user operation.
[0169] Optionally, the second user operation is used to trigger a second style cursor. For example, the second user operation may be an operation of clicking the second style cursor or an operation of double-clicking the second style cursor. The embodiment of the present application does not limit the type of the second user operation.
[0170] In a possible implementation, the electronic device displays the type of media information requested by the second user operation based on the received second user operation. For example, if the second style cursor only indicates one type of media information, such as only indicating that media information of the type of picture is searched, then after the electronic device receives the second user operation (such as clicking the second style cursor), it displays the searched picture. An example may be as follows: Figure 7a and Figure 7b The scenario shown. For another example, if the second style cursor indicates that multiple types of media information are searched, the electronic device can display the first menu based on the third user operation mentioned above, and after receiving the second user operation, display the corresponding type of media information according to the type indicated by the second user operation. Among them, if the second user operation is used to trigger one of multiple function options, then in response to the second user operation, the electronic device displays the type of media information corresponding to the function option among the multiple types of media information searched; if the second user operation is used to trigger at least two of the multiple function options, then in response to the second user operation, at least two types of media information corresponding to the function option among the multiple types of media information searched are displayed. An example can be as follows Figure 6 The scene shown.
[0171] In a possible implementation, the searched media information may be displayed in descending order of matching degree.
[0172] Above Fig.14 In the process shown, the electronic device can automatically search for matching media information based on the text input by the user, without the need for the user to input search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience. On the other hand, by changing the cursor to indicate that matching media information has been searched, the user can understand the search situation and then perform subsequent operations based on the search situation.
[0173] Further, Fig.14In the process shown, after displaying the media information matching the first text, the following steps may also be included: receiving a fourth user operation, selecting the first media information from the displayed media information in response to the fourth user operation, and applying the first media information to the first application. The first application is an application that receives the first user operation and obtains the text content input by the user through the first user operation, that is, the first user operation is used in the text input box of the first application, so that the first application receives the text input by the user.
[0174] The fourth user operation is used to select one or more media information from the displayed media information (for the convenience of description, the one or more media information selected by the fourth user operation is referred to as the first media information), and apply the first media information to the first application. Depending on the function of the first application, the specific method of applying the first media information to the first application may be different. For example, for a memo application, the first media information can be added to the memo content. For related examples, please refer to Figure 7a and Figure 7b For another example, for a social application, the first media information may be sent to a target user. For related examples, see Fig.11 or Fig.12 For another example, for an email application, the first information can be added to the email as an attachment. For related examples, see Fig.13 .
[0175] In some embodiments of the present application, the text corresponding to the search result can also be highlighted so that the user can understand based on which part of the text the matching media information is searched. In a possible implementation, the first text can be highlighted during the process of transforming the first style cursor into the second style cursor, or after the first style cursor is transformed into the second style cursor, or after receiving the second user operation. An example can be as follows Figure 8 shown.
[0176] In some embodiments of the present application, after highlighting the first text, the user may be allowed to move the cursor position through user operation so that media information can be added to the position later. In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, a fifth user operation may be received, and the fifth user operation is used to move the cursor to the first position, and the first position is within the range of the first text; in response, the electronic device may move the second style cursor to the first position. An example may be as follows Figure 8 shown.
[0177] In some embodiments of the present application, after highlighting the first text, the highlighting of part of the text can be canceled through user operation, and a new search process can be triggered. In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, a sixth user operation may be received, and in response to the sixth user operation, part of the highlighted first text is canceled, and matching media information is searched based on the second text, where the second text is the text highlighted in response to the sixth user operation. An example can be as follows Fig. 9 shown.
[0178] In some embodiments of the present application, before highlighting the first text, the highlighted text content can be increased through user operation, and a new search process can be triggered. In one possible implementation, after the first style cursor is transformed into the second style cursor and before the second user operation is received, a seventh user operation may be received, and in response to the seventh user operation, a third text is highlighted, and the third text is text other than the first text; and matching media information is searched based on a fourth text, and the fourth text includes the first text and the third text. An example can be as follows Fig. 9 shown.
[0179] In some embodiments of the present application, after searching for media information matching the first text, or after receiving a second user operation, a search keyword may be displayed. Furthermore, the user may be allowed to update the search keyword in order to search for the required media information. In one possible implementation, after displaying the search keyword, the electronic device may receive an eighth user operation, wherein the eighth user operation is used to update the search keyword and request to search for matching media information based on the updated search keyword; in response to the eighth user operation, the electronic device may search for matching media information based on the updated search keyword and display the media information searched based on the updated search keyword. An example may be as follows: Fig.10 shown.
[0180] The embodiment of the present application also provides an interaction method applied to an electronic device, which can be used to search for matching media information based on the type of media information selected by the user and based on the text and / or context input by the user.
[0181] The following describes the user operations and user interface changes in the interaction process implemented in the embodiments of the present application, taking a memo application scenario, a social application scenario, and a scenario for uploading information for authentication or registration as examples.
[0182] First, taking the memo application as an example, the user interface change process based on user operations during the implementation of the embodiment of the present application is described.
[0183] See also Fig.15 , taking a mobile phone as an example, an embodiment of the present application provides a user interface schematic diagram for searching for corresponding types of media information according to the media information type selected by the user and displaying the search results in a memo application scenario.
[0184] like Fig.15 As shown, in the interface of the memo application, there are a text input box 1511, a soft keyboard 1512 and a function option 1513. After being triggered, the function option 1513 can search for matching pictures based on the text input by the user. The function option 1513 can be implemented by a picture control. Referring to user interface 1501, the user uses the soft keyboard 1511 to input text in the text input box 1512. Referring to user interface 1502, the user clicks on function option 1513. In response, referring to user interface 1503, the electronic device searches for matching pictures based on the text input in the text box, and displays the searched pictures in a pop-up window 1504. For the method of displaying the searched pictures, please refer to Figure 7b The relevant content in will not be repeated here.
[0185] Optionally, the user interface 1501 may also include other functional options, such as a functional option for searching video type media information (e.g., a video control), a functional option for searching document type media information (e.g., a file control), etc., which is not limited in the embodiment of the present application. It is understandable that different types of functional options or different types of controls can be used to search for different types of media information.
[0186] After the searched media information is displayed, the selected media information can be added to the memo content based on the user operation. For the description of selecting media information and adding media information to the memo content, please refer to Figure 7b The relevant content in will not be repeated here.
[0187] The following describes the process of user interface changes based on user operations during the implementation of the embodiments of the present application by taking a social application scenario as an example.
[0188] See also Fig.16 , taking a mobile phone as an example, an embodiment of the present application provides a user interface schematic diagram for searching for corresponding types of media information and displaying search results according to the media information type selected by a user in a social application scenario.
[0189] like Fig.16As shown, the user interface 1601 of the social application includes a text input box, a soft keyboard, and function options 1611 and 1612. When triggered, function option 1611 can search for matching documents based on the semantics of the relevant text, and when triggered, function option 1612 can search for matching pictures based on the semantics of the relevant text. The function option 1611 can be implemented by a file control, and the function option 1612 can be implemented by a picture control. Referring to user interface 1601, a user uses a soft keyboard to input text in the text input box, and then the user clicks function option 1611. In response, referring to user interface 1602, the mobile phone performs semantic understanding based on the text in the text input box and further combines the context, and searches for matching documents based on the results of the semantic understanding. The context may include chat content sent by the other user received by the mobile phone, such as Fig.16 The chat history sent by the other user in the chat window. When a matching document is searched, a pop-up window 1613 is displayed. The size of the window can be smaller than the full screen size, for example, it can be covered on the upper layer of the soft keyboard window. Window 1613 displays multiple searched documents, and the multiple documents can be sorted from high to low according to the matching degree and / or file type. The user can select one of the documents and drag it to the text input box or chat window. Referring to user interface 1603, when the selected document is dragged to the target location and released, the document can be sent. Optionally, the text entered by the user in the text input box can also be sent at the same time.
[0190] The following describes the process of user interface changes based on user operations during the implementation of the embodiment of the present application, taking the scenario of uploading information for authentication or registration as an example.
[0191] See also Fig.17 , the authentication user interface 1701 is used to upload the front and back photos of the user's ID card to the network side, and the user interface includes function options 1711 and function options 1712. After being triggered, function options 1711 and function options 1712 can automatically search for ID card pictures. Taking function option 1711 as an example, function option 1711 can be a picture control. In addition, a text prompt "Upload front" is displayed on function option 1711. Furthermore, when the user clicks function option 1711, referring to user interface 1702, in response, the mobile phone searches for ID card pictures, and further searches for ID card front pictures, and displays the searched pictures in pop-up window 1713. The user can upload the searched pictures.
[0192] See also Fig.18, the resume upload user interface 1801 is used to upload the user's resume file to the network side, and the user interface includes a function option 1811. After the function option 1811 is triggered, the user's resume can be automatically searched. When the user clicks the function option 1811, referring to the user interface 1802, in response, the mobile phone searches for the user's resume and displays the searched resume file in a pop-up window 1813. The user can upload the searched file.
[0193] Combine the following Fig.19 , the specific implementation method of the interaction method provided in the embodiment of the present application is described.
[0194] See also Fig.19 , is a flow chart of an interactive method provided in an embodiment of the present application, which is used to automatically search for matching media information according to a function option triggered by a user. The method can be executed by an electronic device, and an example of an electronic device can be as follows Figure 1 The method may also be executed by a software function module in an electronic device. An example of a software system architecture in an electronic device is as follows: Figure 2 The method can also be implemented by combining software and hardware.
[0195] like Fig.19 As shown, the method may include the following steps:
[0196] Step 1901: the electronic device receives a first user operation, where the first user operation is used to trigger a first function option in a user interface, and the first function option corresponds to a first media information type.
[0197] Optionally, the first function option can be implemented by a control. The first user operation that triggers the first function option can be a click operation, a double-click operation, or other types of user operations, which are not limited in the embodiments of the present application.
[0198] Optionally, the first function option may correspond to one type of media information, such as only a picture type, or only a document type, or only a video type. Optionally, the first function option may also correspond to multiple types of media information, such as both pictures and videos, or both pictures and documents. This may be implemented by programming the first function option, and the present application embodiment does not limit this.
[0199] For the relevant description of the media information and the types of media information involved in the embodiments of the present application, please refer to the aforementioned embodiments and will not be repeated here.
[0200] Step 1902: The electronic device obtains a context associated with the first user operation and / or the first function option.
[0201] In a possible implementation, the process of obtaining the context associated with the first user operation may include: obtaining text input by the user before the first user operation, and using the text as the context for searching for related media information. Fig.15 The scene shown.
[0202] In a possible implementation, the process of obtaining the context associated with the first user operation may include: obtaining the text input by the user before the first user operation and the content contained in the message sent by other users received by the electronic device. Fig.16 The scene shown.
[0203] In a possible implementation, the process of obtaining the context associated with the first function option may include: obtaining first information associated with the first function option, the first information being used to describe the media information that the first function option is used to obtain, and using the first information as the context. An example may refer to the scenario shown in 17, in which the first information associated with function option 1711 is "upload a front-facing photo of your ID card", which is used to describe the function of the function option, including the media information corresponding to the first function option, such as in this example, "front-facing photo of your ID card" is used to describe the media information corresponding to the function option.
[0204] Step 1903: The electronic device searches for matching media information according to the semantics of the context and the first media information type corresponding to the first function option.
[0205] In one possible implementation, the electronic device may search for matching media information from the media information stored in the electronic device, such as searching for pictures and videos from a local picture library (such as an album) of the electronic device, and searching for documents from a local folder of the electronic device for storing user documents. In another possible implementation, the electronic device may search for matching media information from a cloud space used by the electronic device, wherein the cloud space may be shared by the electronic device and other electronic devices. In another possible implementation, if the electronic device is interconnected with other electronic devices, matching media information may be obtained from the other electronic devices through connection with the other electronic devices. It will be understood that the above-mentioned multiple acquisition methods may also be used in combination with each other. The embodiments of the present application do not limit the search scope or source of media information.
[0206] In one possible implementation, the electronic device may preferentially search for matching media information locally on the electronic device to minimize network overhead. If no matching media information is found, the matching media information is searched from the cloud space, and so on. The present application embodiment does not limit the search order or priority.
[0207] Step 1904: The electronic device displays the searched media information.
[0208] In a possible implementation, the searched media information may be displayed in descending order of matching degree.
[0209] Above Fig.19 In the process shown, the electronic device can automatically search for matching media information according to the first function option triggered by the user, without the user having to input search keywords and trigger a function button to execute the search command, thereby simplifying user operations and improving user experience.
[0210] Based on the above embodiments, the present application also provides a device, which may be an electronic device or its internal module, and the device includes multiple functional modules; the multiple functional modules interact with each other to implement the functions performed by the electronic device in each method described in the embodiments of the present application. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on the specific implementation. For example, the method steps performed by the electronic device in the above embodiments of the present application are executed.
[0211] Based on the above embodiments, the present application also provides a device, which may be an electronic device or its internal module, and the device includes at least one processor and at least one memory, wherein at least one memory stores computer program instructions, and when the electronic device is running, at least one processor executes the functions executed by the electronic device in each method described in the embodiments of the present application. For example, the method steps executed by the electronic device in the above embodiments of the present application are executed.
[0212] Based on the above embodiments, the present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), when the computer program is run, the methods described in the embodiments of the present application are executed.
[0213] Based on the above embodiments, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program (also referred to as code or instruction), which, when executed by a computer, enables the computer to execute the methods described in the embodiments of the present application.
[0214] Based on the above embodiments, the present application further provides a chip, which is used to read a computer program stored in a memory to implement the methods described in the embodiments of the present application.
[0215] Based on the above embodiments, the present application provides a chip system, which includes a processor for supporting a computer device to implement the methods described in the embodiments of the present application. In a possible embodiment, the chip system also includes a memory, which is used to store the necessary programs and data of the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. It should be understood by those skilled in the art that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including a computer-usable program code.
[0216] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0217] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0219] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the protection scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An interactive method, characterized in that: Applied to electronic equipment, the method comprises: receiving a first user operation, and acquiring a first text in response to the first user operation; wherein the first user operation is used to input text, and the first text includes text input by the first user operation; searching, according to the first text, for media information that semantically matches the first text; transforming a first-style cursor into a second-style cursor, wherein the second-style cursor indicates that media information has been searched; A second user operation on the second-style cursor is received, and in response to the second user operation, the media information is displayed.
2. The method according to claim 1, characterized in that The second-style cursor indicates the type and / or content of the searched media information.
3. The method according to claim 1 or 2, characterized in that The second cursor style includes one or more of the following: A thumbnail of the content of the media information; The indication information of the media type includes an icon for indicating the type of the media information and / or text for indicating the type of the media information.
4. The method according to claim 3, characterized in that The media information includes pictures, and the second-style cursor displays a thumbnail of the searched picture; or The media information includes a video, and the second style cursor displays a thumbnail of a video frame in the searched video.
5. The method according to any one of claims 2 to 4, characterized in that: The second-style cursor indicates that media information of multiple media types has been searched.
6. The method according to claim 5, characterized in that The receiving a second user operation on the second-style cursor and displaying the media information includes: receiving a third user operation; In response to the third user operation, displaying a first menu, wherein the first menu includes a plurality of function options, each of the plurality of function options corresponding to one of the plurality of media types; Receive a selection operation of a user selecting at least one function option among the multiple function options; In response to the selection operation, media information corresponding to the at least one function option is displayed.
7. The method according to any one of claims 1 to 6, characterized in that: The first user operation acts on a text input box of the first application; After displaying the media information, the method further includes: A fourth user operation is received, and in response to the fourth user operation, the first media information is selected from the displayed media information, and the first media information is applied to the first application.
8. The method according to any one of claims 1 to 7, characterized in that: After the first-style cursor is transformed into the second-style cursor, or after the second user operation is received, the method further includes: Highlight the first text.
9. The method according to claim 8, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a fifth user operation, wherein the fifth user operation is used to move a cursor to a first position, wherein the first position is within the range of the first text; In response to the fifth user operation, the second-style cursor is moved to the first position.
10. The method according to claim 9, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a sixth user operation, and in response to the sixth user operation, unhighlighting a portion of the highlighted first text; The media information that matches the semantics of the second text is searched for according to the second text, where the second text is the text highlighted in response to the sixth user operation.
11. The method according to claim 8, characterized in that After the first-style cursor is transformed into the second-style cursor and before the second user operation is received, the method further includes: receiving a seventh user operation, and highlighting a third text in response to the seventh user operation, wherein the third text is text other than the first text; The fourth text is searched for media information that matches the semantics of the fourth text, wherein the fourth text includes the first text and the third text.
12. The method according to any one of claims 1 to 11, characterized in that: After searching for media information that semantically matches the first text, or after receiving the second user operation, the method further includes: Displays search keywords.
13. The method according to claim 12, characterized in that Also includes: receiving an eighth user operation, wherein the eighth user operation is used to update the search keyword and request to search for matching media information according to the updated search keyword; In response to the eighth user operation, searching for matching media information according to the updated search keyword; Displays media information searched based on the updated search keyword.
14. The method according to any one of claims 1 to 13, characterized in that: The obtaining of the first text comprises: The text input through the first user operation and the context associated with the text input through the first user operation are acquired.
15. The method according to any one of claims 1 to 14, characterized in that: The step of searching, according to the first text, for media information that semantically matches the first text includes one or more of the following: Searching the electronic device for media information that semantically matches the first text; Searching other electronic devices interconnected with the electronic device for media information that semantically matches the first text; Search the cloud space of the electronic device for media information that semantically matches the first text.
16. The method according to any one of claims 1 to 15, characterized in that: The media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
17. An interactive method, characterized in that: Applied to electronic equipment, the method comprises: Receiving a first user operation, where the first user operation is used to trigger a first function option in a user interface, where the first function option corresponds to a first media information type; Acquire a context associated with the first user operation and / or the first function option; Searching for matching media information according to the semantics of the context and the first media information type corresponding to the first function option; Displays the searched media information.
18. The method according to claim 17, characterized in that The acquiring the context associated with the first user operation includes: The text input by the user before the first user operation and / or the text contained in the message received by the electronic device is obtained, and the text is used as the context.
19. The method according to claim 17, characterized in that The acquiring of the context associated with the first function option includes: First information associated with the first function option is obtained, where the first information is used to describe media information corresponding to the first function option, and the first information is used as the context.
20. The method according to any one of claims 17 to 19, characterized in that: The searched matching media information includes one or more of the following: Searching the electronic device for matching media information; Searching for matching media information in other electronic devices interconnected with the electronic device; Searching for matching media information in the cloud space of the electronic device.
21. The method according to any one of claims 17 to 20, characterized in that: The media information includes one or more of the following: pictures, videos, audios, documents, electronic receipts, electronic cards, address books, contact information, service cards, and link addresses.
22. A device, characterized in that: The method comprises a unit or a module for executing the method according to any one of claims 1 to 16, or comprises a unit or a module for executing the method according to any one of claims 17 to 21.
23. A device, characterized in that: include: One or more processors and one or more memories; wherein the one or more memories store one or more programs, and when the programs are executed by the one or more processors, the device executes the method as described in any one of claims 1-16, or executes the method as described in any one of claims 17-21.
24. A chip system, characterized in that: The chip system includes at least one chip and a memory, and the at least one chip is used to read and execute a program stored in the memory to implement the method as described in any one of claims 1 to 16, or to implement the method as described in any one of claims 17 to 21.
25. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program, and when the program is executed on a device, the device executes the method according to any one of claims 1 to 16, or executes the method according to any one of claims 17 to 21.
26. A computer program product, characterized in that When the computer program product is executed on a device, the device is caused to execute the method according to any one of claims 1 to 16, or execute the method according to any one of claims 17 to 21.
Citation Information
Cited By
Interaction method and apparatus
EP4657221A1
