Display method and device of interactive control, electronic equipment and storage medium

Abstract

The invention is applicable to the technical field of equipment control, and provides an interactive control display method and device, electronic equipment and a storage medium, and the method comprises the following steps: receiving a first operation for any display object in a first interface; the first interface is an interface corresponding to a first application program running in a foreground; executing the first operation, and generating at least one interaction control corresponding to the display object through a second application program; displaying at least one interaction control in a second interface; the second interface is the same as or different from the first interface. Compared with the prior art, the interactive control in the embodiment of the invention is not fixed, but the interactive control matched with the display object can be generated according to the display object operated by the user, so that the personalized interactive control can be generated according to the operated display object, the interactive operation efficiency is improved, and the user experience is improved. The interaction path can be shortened; and the operation difficulty is reduced.

Description

A method, apparatus, electronic device, and storage medium for displaying an interactive control.

[0001] This application claims priority to Chinese Patent Application No. 202410808560.3, filed on June 20, 2024, entitled "A Display Method, Apparatus, Electronic Device and Storage Medium for Interactive Controls", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of equipment control technology, and in particular relates to a method, device, electronic device and storage medium for displaying interactive controls. Background Technology

[0003] To meet diverse user needs, electronic devices can have various types of applications installed. Users can launch the corresponding applications and access services through the interactive controls configured within them, such as chatting online with other users in a chat application or navigating routes in a map application.

[0004] However, with existing interactive technologies, the number and functions of interactive controls configured within the interface of each application are fixed. They cannot generate service controls that match the different objects the user is interacting with. Users need to switch to a specific interface to access the corresponding service, which increases the difficulty of operation for users and prolongs the interaction path required to access the specified service, thereby reducing the efficiency of user operation. Summary of the Invention

[0005] This application provides a method, device, electronic device, and storage medium for displaying interactive controls, which can solve the problem that existing interactive technologies cannot generate service controls that match different user operation objects, requiring users to switch to a specific interface to realize the corresponding service, resulting in difficult interactive operations and long interaction paths.

[0006] In a first aspect, embodiments of this application provide a method for displaying an interactive control, including:

[0007] Receive a first operation on any displayed object within a first interface; the first interface is the interface corresponding to a first application running in the foreground.

[0008] Perform the first operation and generate at least one interactive control corresponding to the display object through the second application;

[0009] At least one of the interactive controls is displayed in the second interface; the second interface may be the same as or different from the first interface.

[0010] The second application can be a smart assistant (such as a voice assistant) or other types of artificial intelligence (AI) applications.

[0011] Implementing the embodiments of this application has the following beneficial effects: When receiving a first operation initiated by a user on a displayed object in the first interface, the electronic device can not only execute the first operation on the displayed object, but also generate an interactive control corresponding to the displayed object through a second application, and display the generated interactive control in the corresponding interface, thus achieving the purpose of automatically matching the associated interactive control according to the displayed object. Compared with existing interactive technologies, the interactive controls in the embodiments of this application are not fixed, but can be generated according to the displayed object operated by the user, thereby enabling the generation of personalized interactive controls according to the operated displayed object, thereby improving the efficiency of interactive operations, shortening the interaction path, and reducing the difficulty of operation.

[0012] In one possible implementation of the first aspect, performing the first operation and generating at least one interactive control corresponding to the display object via a second application includes:

[0013] Based on the content characteristics corresponding to the display object, at least one interactive control corresponding to the content characteristics is generated.

[0014] In one possible implementation of the first aspect, generating at least one interactive control corresponding to the content features of the display object includes:

[0015] At least one interactive control is generated based on the operation type of the first operation and the content characteristics.

[0016] In one possible implementation of the first aspect, generating at least one interactive control corresponding to the content features of the display object includes:

[0017] The content features corresponding to the display object are generated using a feature recognition algorithm that corresponds to the data type of the display object.

[0018] In one possible implementation of the first aspect, the second interface is the interface corresponding to the second application.

[0019] In one possible implementation of the first aspect, after displaying at least one of the interactive controls in the second interface, the method further includes:

[0020] In response to a second operation on the interactive control, an interaction result generated based on the display object is displayed.

[0021] In one possible implementation of the first aspect, displaying an interaction result generated based on the display object in response to a second operation on the interactive control includes:

[0022] In response to a second operation on the interactive control, a third application is launched;

[0023] The interaction results are displayed in the third interface corresponding to the third application.

[0024] In one possible implementation of the first aspect, displaying an interaction result generated based on the display object in response to a second operation on the interactive control includes:

[0025] In response to a second operation on the interactive control, the interaction result is displayed in the second interface; the second interface is the interface corresponding to the second application.

[0026] Secondly, a display device for interactive controls, comprising:

[0027] The first operation receiving unit is used to receive a first operation on any displayed object within the first interface; the first interface is the interface corresponding to the first application running in the foreground.

[0028] A control generation unit is used to perform the first operation and generate at least one interactive control corresponding to the display object through a second application.

[0029] A control display unit is used to display at least one of the interactive controls in a second interface; the second interface may be the same as or different from the first interface.

[0030] In one possible implementation of the second aspect, the control generation unit is specifically used to generate at least one interactive control corresponding to the content features based on the content features corresponding to the display object.

[0031] In one possible implementation of the second aspect, the control generation unit is specifically used to generate at least one of the interactive controls based on the operation type of the first operation and the content characteristics.

[0032] In one possible implementation of the second aspect, the interactive control display unit is specifically used to generate the content features corresponding to the display object through a feature recognition algorithm corresponding to the data type of the display object.

[0033] In one possible implementation of the second aspect, the second interface is the interface corresponding to the second application.

[0034] In one possible implementation of the second aspect, the display device further includes:

[0035] An interactive response unit is used to respond to a second operation on the interactive control and display an interactive result generated based on the display object.

[0036] In one possible implementation of the second aspect, the interactive response includes: a third application launch unit, configured to launch a third application in response to a second operation on the interactive control;

[0037] The third interface display unit is used to display the interaction results in the third interface corresponding to the third application.

[0038] In one possible implementation of the second aspect, the interactive response unit is specifically used to: display the interaction result in the second interface in response to a second operation on the interactive control; the second interface is the interface corresponding to the second application.

[0039] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the steps of the display method described in any of the other aspects above.

[0040] Fourthly, embodiments of this application provide a readable storage medium storing a program that, when executed by a processor, implements the steps of the display method described in any of the other aspects above.

[0041] Fifthly, embodiments of this application provide a program product that, when run on a device, causes the device to perform the steps of the display method described in any of the other aspects above.

[0042] Sixthly, embodiments of this application provide a display method applied to an electronic device, the electronic device including a smart assistant program, the method comprising:

[0043] The first interface of the first application is displayed, and the first interface displays display objects. The first application is a different program from the smart assistant program.

[0044] In response to a user's operation on the display object, the electronic device displays at least one interactive control associated with the display object via the smart assistant program;

[0045] In response to a user's operation on the at least one interactive control, the function corresponding to the at least one interactive control is executed.

[0046] In one possible implementation of the sixth aspect, in response to a user's operation on the display object, the electronic device displays at least one interactive control associated with the display object via the smart assistant program, including:

[0047] Based on the content characteristics corresponding to the display object, at least one interactive control corresponding to the content characteristics is generated.

[0048] In one possible implementation of the sixth aspect, in response to a user's operation on the display object, the electronic device displays at least one interactive control associated with the display object via the smart assistant program, including:

[0049] At least one interactive control is generated based on the operation type of the first operation and the content characteristics.

[0050] In one possible implementation of the sixth aspect, generating at least one interactive control corresponding to the content features based on the content features corresponding to the display object includes:

[0051] The content features corresponding to the display object are generated using a feature recognition algorithm that corresponds to the data type of the display object.

[0052] In one possible implementation of the sixth aspect, the step of executing the function corresponding to the at least one interactive control in response to a user's operation on the at least one interactive control includes:

[0053] In response to a second operation on the interactive control, a third application is launched;

[0054] The interaction results are displayed in the third interface corresponding to the third application.

[0055] In one possible implementation of the sixth aspect, the step of executing the function corresponding to the at least one interactive control in response to a user's operation on the at least one interactive control includes:

[0056] In response to a second operation on the interactive control, the interaction result is displayed in the second interface; the second interface is the interface corresponding to the smart assistant program.

[0057] Other implementations can refer to the aforementioned aspects and will not be elaborated further.

[0058] The beneficial effects of the above aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0059] Figure 1 is a schematic diagram of user operation in an existing scenario;

[0060] Figure 2 is a flowchart illustrating a method for displaying interactive controls according to an embodiment of this application;

[0061] Figure 3 is a schematic diagram of a first interface provided in an embodiment of this application;

[0062] Figure 4 is a schematic diagram of the data acquisition operation logic chain provided in an embodiment of this application;

[0063] Figure 5 is a schematic diagram of generating interactive controls based on displayed content according to an embodiment of this application;

[0064] Figure 6 is a schematic diagram of generating interactive controls based on displayed content according to another embodiment of this application;

[0065] Figure 7 is a schematic diagram of the display of an interactive control provided in an embodiment of this application;

[0066] Figure 8 is a schematic diagram of the display of an interactive control provided in another embodiment of this application;

[0067] Figure 9 is a schematic diagram of the display of an interactive control provided in another embodiment of this application;

[0068] Figure 10 is a schematic diagram of the display of an interactive control provided in another embodiment of this application;

[0069] Figure 11 is a structural block diagram of an interactive control display device provided in an embodiment of this application;

[0070] Figure 12 is a structural block diagram of a display device provided in an embodiment of this application;

[0071] Figure 13 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0072] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0073] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0074] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0075] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0076] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0078] With the continuous development of digitalization, more and more services can be completed through applications. Electronic devices can have different types of applications installed to meet users' needs in different scenarios. Generally, each application can provide users with one type of service. For example, chat applications can provide online chatting functions, while map applications can provide location search and navigation functions. Therefore, if users need to use multiple services with different functions, they often need to switch between different applications.

[0079] For example, Figure 1 illustrates a user operation diagram in an existing scenario. Referring to Figure 1(a), a chat application is running in the foreground of the smartphone. User A and User B are chatting online, discussing a meeting place for tomorrow. The meeting place is specifically "No. xx, xx Street, xx District, xxx City," as shown in chat record 11. Since User A is unaware of the exact location of the meeting place, they need to search for it on a map. This requires the following steps to find the address:

[0080] 1. By long-pressing the chat history 11 above, the corresponding operation menu will pop up, as shown in menu 12 in Figure 1. Click the copy button in menu 12 above to copy the text content corresponding to the meeting point above.

[0081] 2. Exit the chat application and switch to the main interface, as shown in Figure 1(b). Then, find the icon control corresponding to the map application in the main interface, namely icon control 13, and click the icon control 13 to start the map application.

[0082] 3. After the map application is launched, the corresponding map application interface will be displayed, as shown in Figure 1(c). Users can paste the copied text content into the input area 14 of the map application interface, and finally click the search control 15 to complete the location search.

[0083] This demonstrates that the interaction paths for users when using different services are lengthy, increasing the difficulty of operation and reducing user efficiency. This is because the interfaces generated by applications generally only provide a single service. For example, the interface of the chat application mentioned above only provides online chat services, while the interface of the map application only provides location search, navigation, and other map-related services. The controls configured within each interface are also configured around the different functional requirements of that service. That is, the interactive controls within each interface have relatively simple functions and a fixed number of controls. It is impossible to configure associated interactive controls according to the object being operated by the user, nor can it understand the user's operational intent. As a result, in scenarios with multiple types of needs, it increases the number of steps for the user and affects the user experience.

[0084] Example 1:

[0085] To address the problems existing in interactive technologies, this application provides a method for displaying interactive controls. This method generates and displays associated interactive controls based on the display object operated by the user, thereby enabling personalized configuration of associated interactive controls according to the user's operational intent, improving interaction efficiency and reducing interaction paths. In this embodiment, the method for displaying interactive controls can be applied to an electronic device. The electronic device can be a computer, server, laptop, smartphone, or other device capable of interacting with the user. The electronic device can be configured with an interaction module and a display module. The interaction module can be a touchscreen, mouse, keyboard, stylus, etc., to acquire and respond to user-initiated interactive operations. The display module can be a display screen, projector, etc., to display an interface containing interactive controls, on which the user can initiate interactive operations. The interaction module and display module can be built into the electronic device or external devices; the connection relationship between the interaction module / display module and the electronic device is not limited here.

[0086] Specifically, Figure 2 shows a flowchart illustrating the implementation of an interactive control display method according to an embodiment of this application. Referring to Figure 2, the interactive control display method provided in this embodiment includes:

[0087] In S201, a first operation on any displayed object is received within a first interface; the first interface is the interface corresponding to a first application running in the foreground.

[0088] The first operation can be a touch operation, an air operation, or a voice operation; this application does not limit this.

[0089] In this embodiment, the electronic device can have multiple different applications installed, such as chat applications, electronic document applications, map applications, etc. The user can select the icon of the desired application by clicking or double-clicking. When the electronic device receives a launch operation initiated by the user, it can run the application selected by the user (i.e., the first application) and display the operation interface corresponding to that application, i.e., the first interface mentioned above.

[0090] For example, Figure 3 shows a schematic diagram of a first interface provided in an embodiment of this application. Referring to Figure 3(a), the electronic device is a smartphone. The main interface of the smartphone includes icon controls for multiple installed applications, such as control 31 for a chat application, control 32 for a map application, and control 33 for a document editing program. When the electronic device detects that the user clicks on the aforementioned control 31, it recognizes that the user needs to launch the chat application and runs the chat application in the foreground, generating a corresponding chat interface, as shown in Figure 3(b). The user can select the target user for online chat from multiple chat objects. For example, if user B is selected as the chat object, the user will enter the corresponding chat interface, as shown in Figure 3(c).

[0091] In this embodiment, the first interface includes various types of display objects, including text objects of type text, image objects of type image, and control objects for initiating specified operations. The specific types of display objects included in the first interface can be set according to the actual situation and are not limited here.

[0092] In this embodiment, the first interface includes multiple display objects. The user can operate on any one of the display objects within the first interface, i.e., initiate a first operation. For example, continuing with Figure 3, as shown in Figure 3(c), the chat interface includes an avatar control 34 for the chat object. The user can view the user's information by clicking the avatar control 34. The chat interface also includes chat history 35 corresponding to the chat content. The user can long-press the chat history 35 to bring up the corresponding operable menu 36, as shown in Figure 3(d). The operable menu 36 includes options for forwarding, copying, quoting, and deleting text information. The user can perform corresponding operations on the text information in the chat history 35 by clicking the corresponding control in the menu bar.

[0093] In some possible implementations, the user can initiate the first operation by associating it with the interactive module configured on the electronic device. For example, if the electronic device is equipped with a touch screen, the user can initiate the first operation via touch, which includes various operations such as clicking, long-pressing to select, and dragging. If the electronic device is equipped with a keyboard and mouse, the user can initiate the first operation by controlling the keyboard and mouse, such as controlling the cursor on the screen to select the corresponding display object, selecting a specified area of ​​the display object on the screen with the cursor, or inputting text using the keyboard. The specific method for initiating the first operation can be selected according to the actual situation.

[0094] In S202, the first operation is performed, and at least one interactive control corresponding to the display object is generated through the second application.

[0095] In this embodiment, the electronic device can respond to a first operation initiated by the user within the first interface. For example, if the first operation is copying text content within the first interface, the electronic device can respond to the user's copy operation by adding the copied text content to the clipboard. Subsequently, upon receiving a paste instruction from the user, it can copy the relevant text content stored in the clipboard to a specified area. Alternatively, if the first operation is selecting a portion of text content in an electronic document, the electronic device can respond to the user's selection operation by highlighting the selected text content with a preset color. The specific method of executing the first operation can be determined based on the operation type of the first operation and is not limited here.

[0096] Unlike existing technologies, in this embodiment, the electronic device not only executes the first operation initiated by the user within the first interface, but also determines the display object corresponding to the first operation through the second application when the user initiates the first operation, and generates at least one interactive control associated with the display object. This enables the device to understand the user's operation intention, and the user can directly execute the corresponding interactive operation through the aforementioned interactive control, thereby reducing the length of the interaction path and improving the efficiency of human-computer interaction.

[0097] The second application of this application can be a smart assistant program. This smart assistant program can sense various types of operations initiated by the user in the interface of other applications (such as the first interface of the first application), and understand the user's operational intent, that is, understand the intent of the display object specified by the operation, and then generate interactive controls associated with the display object, thereby achieving the purpose of providing personalized service recommendations to the user and improving the user's operational efficiency. For example, this smart assistant application can be a voice assistant application, such as Xiaoyi, Xiao Ai, Siri, etc.

[0098] In one possible implementation, the second application is configured with a big data model, which can be trained using a large number of training samples. These training samples can be operation logic chains collected during device use, including start and end operations. The start operation is the user's historical operation on any displayed object within any interface, and the end operation can be the operation of applying the displayed object from the first interface to other interfaces.

[0099] For example, Figure 4 shows a schematic diagram of a data acquisition operation logic chain provided in an embodiment of this application. Referring to Figure 4, the process of a user copying an address from a dialog box to a map application for address lookup specifically includes the following steps:

[0100] S41. Copy the dialog box containing address information in the chat interface of the chat application.

[0101] S42. Exit the chat application

[0102] S43. Click the map application on the main interface.

[0103] S44. Paste the copied address information into the map application interface and perform an address search.

[0104] Therefore, an electronic device can generate an operation logic chain based on the above multiple steps, which can be represented as follows:<S41→S42→S43→S44> Since S41 is an operation initiated in the chat application interface and S42 is an operation initiated in the map application, the application interfaces corresponding to the two operations are different. Therefore, training samples can be generated based on S41 and S44 in the operation logic chain to train the above-mentioned big data model. Subsequently, when it is detected that the user initiates an operation matching S41 (such as copying another dialog box containing address information in the chat interface of another chat application), an interactive control matching S44 can be generated. This allows the user to directly execute the operation of S44 after completing S41, without the need for intermediate operations, thereby achieving the goal of quickly executing the specified interactive operation, simplifying the interaction path, and improving the user's operation efficiency.

[0105] For example, the interactive controls corresponding to the generated display objects may specifically include the following methods:

[0106] Method 1: Generate associated interactive controls based on the content characteristics of the displayed object.

[0107] In this embodiment, the electronic device can generate at least one interactive control corresponding to the content features of the displayed object. The content features can be used to determine the user's operational intent, i.e., the purpose of initiating the first operation on the displayed object, and what subsequent operations might be performed. This enables the generation of interactive controls corresponding to the user's potential operations, thereby reducing the number of steps required for the user to perform subsequent operations.

[0108] For example, Figure 5 shows a schematic diagram of generating interactive controls based on displayed content according to an embodiment of this application. Referring to Figure 5, the electronic device is a smartphone, and the application currently running in the foreground of the smartphone is a document reading application. The user can generate a corresponding reading interface through the document reading application. The reading interface displays the electronic document opened by the user, i.e., interface 51. The interface 51 includes text paragraphs 52 of text type and table images 53 stored in image format.

[0109] In one scenario, a user selects a text paragraph 52 from the interface 51. Content recognition is performed on the text paragraph 52 to obtain its corresponding content features. These content features can include feature information about the displayed content across multiple dimensions. For example, these content features can be used to determine the data type corresponding to the text paragraph 52, and based on the data type, at least one corresponding dimensional feature can be determined. For instance, for text types, the language type used by the text paragraph 52 and the semantic meaning of the text paragraph 52 can be determined.

[0110] Taking text paragraph 52 as an example, the above content features can be represented as: <text type; English; "weather", "travel">, where "text type" determines the data type of the displayed object, "English" determines the language type used by the displayed object, and "weather" and "travel" represent the semantic content of text paragraph 52. The language type and semantic content dimensions are determined based on the text type data type; if other data types are used, the subsequent dimension features may differ.

[0111] In this embodiment, the electronic device can obtain the aforementioned content features through a second application and determine the corresponding interactive controls based on these content features. Continuing with the example of obtaining the content feature <text type; English; "weather", "travel"> in this scenario, the user initiates a first operation by selecting the aforementioned text paragraph 52. Since this text paragraph 52 is in English, which is inconsistent with the default language set by the electronic device, the electronic device can predict through the second application that the user may need to translate this text paragraph 52. Therefore, an interactive control for directly translating the text content can be generated, resulting in the translation control 53. Furthermore, based on the semantic meaning of content such as "weather" and "travel," the device can predict whether the user needs to query the weather or search for tourist attractions, resulting in the corresponding weather query control 54 and tourist attraction recommendation control 55.

[0112] In some possible implementation scenarios, if the text paragraph 52 is not stored in an editable text format in the electronic document, for example, if an electronic document is stored as an image, meaning that the text content and other content are encapsulated within an image file, and the user selects the aforementioned text paragraph 52, which is a region within the image file, then the data type in the content characteristics obtained above will be "image type." Since the image region contains text data rather than graphic data, it can be predicted that the user may need to extract text data from the image region. Therefore, a text extraction control 56 is obtained. Furthermore, since the language type of the text data is also English, a text translation control 57 can also be generated. For text paragraphs 52 with the same content but different data formats, different interactive controls can be generated. The specific number of interactive controls and their content can be determined based on the content characteristics.

[0113] In one scenario, a user selects a table image 58 from interface 51. Content recognition is performed on table image 58 to obtain its corresponding content features. These content features can be represented as <image type; table; data>, where "image type" determines the data type of the displayed object (i.e., table image 53), and "table" and "data" determine the content features of table image 53. The electronic device can identify the user's operational intent based on the content features of table image 58. Through a big data model within the second application, it can predict the user's possible subsequent operations: generating a table and extracting text. "Extracting text" is determined based on the "image type" dimension feature of the content features, and "generating a table" is determined based on the "table" dimension feature of the content features. Based on these two operations, a control 59 for "extracting text" and a control 510 for "generating a table" can be obtained. In some implementations, each interactive control can be determined based on two or more dimension features of the content features; the specific dimension features required to generate the interactive module can be determined according to the actual situation.

[0114] In some implementations, the electronic device can import content features into the big data model corresponding to the second application, calculate the interaction confidence level of each candidate control through the big data model, and then select the candidate control whose interaction confidence level is greater than a preset confidence level threshold as the interaction control.

[0115] For example, Figure 6 shows a schematic diagram of generating interactive controls based on display content according to another embodiment of this application. Referring to Figure 6, the electronic device is a smartphone, and the application currently running in the foreground is a chat application. Users can communicate online with other users through the chat interface corresponding to the chat application. The chat interface contains multiple chat records, such as chat record 61. Chat record 61 includes address information, specifically "Tomorrow's meeting place is xx street xx number, xx district, xxx city". The user can long-press the chat record to bring up the corresponding menu bar 62 and click the copy control 63 in the menu bar 62 to initiate a copy operation for chat record 61. The second application performs content feature recognition on chat record 61 to obtain the corresponding content features, which can be represented as <text type; Chinese; "address", "location">. The "text type" is used to determine the data type of the display object, the "Chinese" is used to determine the language type used by the display object, and the "address" and "location" are used to represent the semantic content of chat record 61. By predicting user actions based on the above content features, it is possible to predict that users may need to perform address-related interactive operations, such as navigating to the address, taking a taxi to the address, and viewing the address on the map. Therefore, interactive controls corresponding to the above three interactive operations can be generated, namely, interactive control 64 for "navigating to the address", interactive control 65 for "taking a taxi to the address", and interactive control 66 for "viewing on the map".

[0116] In one scenario, the chat interface also includes chat history 67, which specifically records a phone number, "xxx-xxx-xx". Similarly, the user can long-press to bring up the corresponding menu bar and click the copy control within it. The second application performs content feature recognition on chat history 67 to obtain the corresponding content features, which can be represented as <text type; number; "telephone">. The electronic device can then use the second application to generate interactive controls corresponding to these content features, such as an interactive control 68 for "save as contact" and an interactive control 69 for "make a call".

[0117] In some implementations, different data types employ different algorithms to determine content features. Therefore, electronic devices can generate the content features corresponding to the display object using a feature recognition algorithm that matches the data type of the display object. For example, if the display object's data type is text, the feature recognition algorithm can be a text semantic recognition algorithm. Importing the text data corresponding to the display object into this algorithm yields the corresponding text semantic features. If the display object's data is an image, the feature recognition algorithm can be an image recognition algorithm. Importing the image data corresponding to the display object into this algorithm yields the corresponding image semantic features. Other data types can use corresponding feature recognition algorithms to obtain their corresponding content features, thus enabling the use of appropriate algorithms based on data type for content feature recognition, improving the accuracy of content feature recognition.

[0118] Method 2: Generate associated interactive controls based on the content characteristics of the displayed content and the operation type of the first operation.

[0119] In this embodiment, in order to further improve the accuracy of interactive control generation, in addition to obtaining the content features corresponding to the display object, the electronic device can also obtain the operation type corresponding to the first operation initiated by the user. Combining the above two features, an interactive control with logical association with the first operation is generated to improve the accuracy of the generated interactive control.

[0120] In this embodiment, generating interactive controls specifically involves predicting subsequent user actions. Therefore, even when operating on the same display object, different user actions will lead to different predictions of subsequent actions, resulting in different generated interactive controls. Continuing with the example in Figure 6, if the user's first action on chat history 61 is a copy operation, the subsequent possible action is a "paste" operation, which involves pasting the address information into another application interface. Therefore, the resulting interactive controls are three interactive controls related to the paste operation: interactive controls 64 to 66.

[0121] If the user's first action in chat history 61 is a translation operation, then subsequent actions may include opening translation software and selecting the corresponding translation language for text translation. Therefore, the resulting interactive controls could be interactive controls for "translate to English" or "translate to Japanese," etc. The specific language to be translated can be determined based on the user's commonly used translation languages, and is not limited here.

[0122] The above examples demonstrate that the generated interactive control is related not only to the display object corresponding to the first operation, but also to the operation type of the first operation. Therefore, the electronic device can obtain the above two parameters and generate the corresponding interactive control based on these two parameters to improve the accuracy of interactive control generation.

[0123] In S203, at least one of the interactive controls is displayed in the second interface; the second interface may be the same as or different from the first interface.

[0124] In this embodiment, after generating interactive controls related to the display object, the electronic device can display these interactive controls on the second interface. The user can determine whether to select one of the interactive controls based on actual operational needs to quickly execute the corresponding interactive operation.

[0125] In this embodiment, depending on the second interface displaying the interactive controls, the following situations can be identified:

[0126] Scenario 1: The first interface is the same as the second interface.

[0127] In this embodiment, the interactive controls can be displayed within the interface where the user initiates the first operation, meaning the first interface and the second interface are the same, thereby improving the convenience of operating the interactive controls. For example, Figure 7 shows a schematic diagram of the display of interactive controls provided in an embodiment of this application. Referring to Figure 7(a), the user initiates a copy operation on chat record 71 (a chat record containing address information) in the chat interface. In this scenario, the second application can be an application running in the background. Optionally, the second application is a smart assistant program that can detect various operations initiated by the user in the chat interface in the background. When the second application detects that the user has initiated a copy operation on chat record 71, it can generate interactive controls for "navigate to this address," "take a taxi to this address," and "view on map," and display these three controls in the chat interface, as shown in interactive controls 72, 73, and 74 in Figure 7(b).

[0128] Scenario 2: The first interface and the second interface are different interfaces on the same device.

[0129] In this embodiment, the interactive controls can be displayed on a different interface than the first interface, namely a second interface. Specifically, this second interface is the interface corresponding to the second application. In this embodiment, both the first and second applications are foreground applications, and the two applications can be simultaneously displayed on the electronic device's screen via split-screen or window display.

[0130] For example, Figure 8 shows a schematic diagram of the interactive control provided in another embodiment of this application. Referring to Figure 8(a), the electronic device has a split-screen display function, such as a smartphone with a foldable screen. The left and right screens of the foldable screen can be used to display the interfaces of different applications. That is, the entire display area can be divided into area 81 corresponding to the left screen and area 82 corresponding to the right screen. Area 81 can display the interface of the application for reading electronic documents, through which the electronic document opened by the user can be displayed. Area 82 can display the interface of the intelligent assistant program, which can be a voice assistant application, such as Xiaoyi, Xiao Ai, Siri, etc. The intelligent assistant program can be used to coordinate the operation of other interfaces. The user can operate on the electronic document in area 81. For example, the user selects text paragraph 83 in area 81. The display screen is a touch screen, and the user can initiate the selection operation of text paragraph 83 by long-pressing the area corresponding to text paragraph 83. The second application can sense the user's actions in area 81 in real time and simultaneously display the user's current actions in area 82. For example, when displaying the text paragraph 83 selected by the user, a corresponding prompt box 84 can be displayed. In this case, the second application can generate corresponding interactive controls based on the user's selection action on the text paragraph 83, resulting in the "Translate Text" interactive control 85 and the "Extract Text" interactive control 86 as shown in Figure 8(b).

[0131] In some scenarios, if the user selects image region 87 containing a data table within region 81, the second application can also detect the user's image selection operation in real time and display the currently selected image region 87 in region 82, resulting in a corresponding prompt box. In this case, the second application can generate corresponding interactive controls based on the user's selection operation of image region 87, such as interactive controls for "generating a table" and "extracting text".

[0132] For example, Figure 9 shows a schematic diagram of the interactive control provided in another embodiment of this application. Referring to Figure 9(a), the electronic device is a computer running two applications. Different applications can display their corresponding interfaces through different windows. For example, window 91 displays the interface of the application for reading electronic documents, and window 92 displays the interface of the smart assistant program. Similarly, the smart assistant program can be used to assist the user in operating other windows. Similar to the embodiment in Figure 8, the user can select the image area 93 in window 91 with the cursor. The window 92 can display the operation initiated by the user in window 91 in real time, such as displaying the corresponding prompt box 94. The smart assistant program can generate corresponding interactive controls based on the user's selection operation on the image area 93, such as the interactive control 95 for "generating a table" and the interactive control 96 for "extracting text" shown in Figure 9(b).

[0133] Scenario 3: The first interface and the second interface are different interfaces on different devices.

[0134] In this embodiment, the electronic device can operate collaboratively with other electronic devices. For example, Figure 10 shows a schematic diagram of an interactive control provided in another embodiment of this application. Referring to Figure 10(a), the smartphone 101 and the computer 102 are in a collaborative operating state, and a data link is established between the two devices. A chat application runs in the foreground of the smartphone 101, and the user can copy chat history 103 from the chat interface of the chat application. A smart assistant program runs on the computer 102. This smart assistant program can obtain the user's operations on the smartphone 101 through the established data link. For example, if the user copies chat history 103 on the smartphone 101, a corresponding copy operation will be displayed on the computer 102, resulting in a corresponding prompt box 104. This prompt box 104 can display: "Tomorrow's meeting place is xx street xx in xx district of xxx city" has been copied to the clipboard. The intelligent assistant program running on computer 102 can generate corresponding interactive controls based on the user's copy operation of the above chat history 103, resulting in the interactive control 105 for "Navigate to this address", the interactive control 106 for "Take a taxi to this address", and the interactive control 107 for "View on map" as shown in Figure 10(b).

[0135] Furthermore, as another embodiment of this application, after S203, it further includes: in response to a second operation on the interactive control, displaying an interaction result generated based on the display object.

[0136] In this embodiment, interactive controls for quick operation are displayed on the second interface. In this case, if the user needs to perform the interactive operation corresponding to the interactive control, he / she can select one of the interactive controls, that is, initiate a second operation on any interactive control. The second operation can be a click operation or a double-click operation on the interactive control. In some implementation scenarios, the above-mentioned second operation can also be a voice operation, such as by saying the name of the control on the interactive control, such as by saying "view in map" in Figure 7, a voice operation on the above-mentioned interactive control 72 has been initiated. The specific operation required when selecting the interactive control can be determined according to the actual situation, and is not limited here.

[0137] In this embodiment, after receiving the user's second operation on the interactive control, the electronic device can execute the function corresponding to the interactive control selected by the user, such as executing the preset interactive operation in the interactive control, and generating the interactive result based on the above interactive operation. Continuing with Figure 8 as an example, if the user clicks the interactive control 85, it indicates that the text paragraph 83 needs to be translated, converting English to Chinese. Therefore, the corresponding translation result can be displayed in the interface of the smart assistant program, as shown in Figure 8(c). The corresponding translation result (i.e., the interactive result) can be displayed through the prompt box 88.

[0138] In this embodiment, responding to the user's second operation on the interactive control can be divided into the following two cases based on the different interactive content:

[0139] Scenario 1: Display the interaction results on the second interface corresponding to the second application.

[0140] In this embodiment, after the user selects one of the interactive controls on the second interface, the interaction result obtained based on the interactive control can be displayed on the second interface. As shown in Figure 8(c), the above translation result can be displayed on the interface of the intelligent assistant program, that is, the above-mentioned prompt box 88 is obtained.

[0141] For example, taking Figure 9 as an example, if a user clicks the "Generate Table" interactive control 95 with the cursor, the electronic device can generate a corresponding table file based on the content of the image area 93 mentioned above, and display the corresponding table file, as shown in file 97 in Figure 9(c). The user can save the above file 97 to a specified storage area, or open the generated table file by clicking the above file 97 to view the generated table content. The user can perform subsequent operations on the interaction results according to the actual situation, which are not limited here.

[0142] In this embodiment, users can perform corresponding interactive operations by clicking the corresponding interactive controls, which can improve the efficiency of interactive operations. Users do not need to manually call up other application interfaces to complete the corresponding interactive operations, shortening the interaction path and thus improving the user experience.

[0143] Scenario 2: Display the interaction results on the third interface corresponding to the third application.

[0144] In this embodiment, the interactive control is specifically used to invoke other applications (i.e., a third application) and execute corresponding interactive operations through these other applications, thereby generating corresponding interactive results. These other applications are those other than the application currently being operated by the user and the application used to generate the interactive control. In this case, the electronic device can launch the third application and display the interactive results obtained based on the interactive control within the third application's interactive interface.

[0145] Continuing with Figure 7 as an example, if a user clicks the interactive control 72, they need to find the corresponding address on the map. The user's current application is a chat application, and the smart assistant program used to generate the interactive control may not have integrated map functionality. In this case, after detecting the user's click on the interactive control 72, the electronic device can generate an activation command for the map application, launching the map application installed on the electronic device. The copied address information is then input into the map application to determine the location corresponding to that address, resulting in the map interface shown in Figure 7(c).

[0146] In this embodiment, the electronic device can invoke different third applications based on different interactive operations. For example, if the user clicks the interactive control 74 for "hailing a taxi to this address", the third application launched will be a taxi application. That is, the launched third application is determined based on the user's interactive control.

[0147] In this embodiment, the user can launch a third application through the interactive controls generated by the second application and display the interaction results in the third interface of the third application, thereby realizing the quick launch of the relevant application for interactive operation without the user having to exit the current interface and launch the corresponding application, thus improving the user's operating efficiency.

[0148] As can be seen from the above, in the interactive control display method provided in this application embodiment, when the electronic device receives a first operation initiated by the user on a display object in the first interface, it can not only execute the first operation on the display object, but also generate an interactive control corresponding to the display object through a second application, and display the generated interactive control in the corresponding interface, thus achieving the purpose of automatically matching the associated interactive control according to the display object. Compared with existing interactive technologies, the interactive control in this application embodiment is not fixed, but can generate a matching interactive control according to the display object operated by the user, thereby enabling the generation of personalized interactive controls according to the operated display object, thereby improving the efficiency of interactive operation, shortening the interaction path, and reducing the difficulty of operation.

[0149] Example 2:

[0150] Corresponding to the interactive control display method in Embodiment 1 above, Figure 11 shows a structural block diagram of an interactive control display device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0151] Referring to Figure 11, the display device for the above-mentioned interactive control includes:

[0152] The first operation receiving unit 111 is used to receive a first operation on any displayed object within the first interface; the first interface is the interface corresponding to the first application running in the foreground.

[0153] The control generation unit 112 is used to perform the first operation and generate at least one interactive control corresponding to the display object through the second application.

[0154] The control display unit 113 is used to display at least one of the interactive controls in a second interface; the second interface may be the same as or different from the first interface.

[0155] Optionally, the control generation unit 112 is specifically used to generate at least one interactive control corresponding to the content features of the display object.

[0156] Optionally, the control generation unit 112 is specifically used to generate at least one of the interactive controls based on the operation type of the first operation and the content features.

[0157] Optionally, the control display unit 113 is specifically used to generate the content features corresponding to the display object by means of a feature recognition algorithm corresponding to the data type of the display object.

[0158] Optionally, the second interface is the interface corresponding to the second application.

[0159] Optionally, the display device further includes:

[0160] An interactive response unit is used to respond to a second operation on the interactive control and display an interactive result generated based on the display object.

[0161] Optionally, the interactive response includes: a third application launch unit, configured to launch a third application in response to a second operation on the interactive control;

[0162] The third interface display unit is used to display the interaction results in the third interface corresponding to the third application.

[0163] Optionally, the interactive response unit is specifically used to: in response to a second operation on the interactive control, display the interaction result in the second interface; the second interface is the interface corresponding to the second application.

[0164] Example 3:

[0165] Corresponding to the interactive control display method in Embodiment 1 above, Figure 12 shows a structural block diagram of a display device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0166] Referring to Figure 12, the above-described display device is applied to an electronic device, which includes a smart assistant program. The display device includes:

[0167] The first interface display unit 121 is used to display the first interface of the first application, and the first interface displays display objects. The first application is a different program from the smart assistant program.

[0168] An interactive control display unit 122 is used to respond to a user's operation on the display object, wherein the electronic device displays at least one interactive control associated with the display object through the smart assistant program;

[0169] The function response unit 123 is used to execute the function corresponding to the at least one interactive control in response to the user's operation on the at least one interactive control.

[0170] Optionally, the interactive control display unit 122 is specifically used to generate at least one interactive control corresponding to the content features based on the content features corresponding to the display object.

[0171] Optionally, the interactive control display unit 122 is specifically used to generate at least one of the interactive controls based on the operation type of the first operation and the content characteristics.

[0172] Optionally, the interactive control display unit 122 is specifically used to generate the content features corresponding to the display object through a feature recognition algorithm corresponding to the data type of the display object.

[0173] Optionally, the functional response unit 123 includes:

[0174] The third application launch unit is used to launch a third application in response to a second operation on the interactive control;

[0175] The third interface display unit is used to display the interaction results in the third interface corresponding to the third application.

[0176] Optionally, the function response unit 123 is specifically used to: display the interaction result in the second interface in response to a second operation on the interactive control; the second interface is the interface corresponding to the smart assistant program.

[0177] Figure 13 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 13, the electronic device 13 of this embodiment includes: at least one processor 130 (only one processor is shown in Figure 13, and the number of processors may match the actual number of chips included in the electronic device in the embodiment), a memory 131, and a program 132 stored in the memory 131 and executable on the at least one processor 130. When the processor 130 executes the program 132, it implements the steps in the above-described embodiments of the display methods for any of the interactive controls.

[0178] The electronic device 13 may be a VR display device, a smartphone, etc. This electronic device may include, but is not limited to, a processor 130 and a memory 131. Those skilled in the art will understand that FIG13 is merely an example of the electronic device 13 and does not constitute a limitation on the electronic device 13. It may include more or fewer components than illustrated, or combine certain components, or different components; for example, it may also include input / output electronic devices, network access electronic devices, etc.

[0179] The processor 130 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0180] In some embodiments, the memory 131 may be an internal storage unit of the electronic device 13, such as a hard disk or memory of the electronic device 13. In other embodiments, the memory 131 may be an external storage device of the electronic device 13, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 13. Furthermore, the memory 131 may include both internal storage units and external storage devices of the electronic device 13. The memory 131 is used to store the operating system, applications, bootloader, data, and other programs, such as program code. The memory 131 can also be used to temporarily store data that has been output or will be output.

[0181] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0182] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0183] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0184] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0185] This application provides a program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0187] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0188] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A display method of an interactive control, characterized by, The method comprises: receiving a first operation on any display object in a first interface; the first interface is an interface corresponding to a first application program running in the foreground; executing the first operation and generating at least one interactive control corresponding to the display object through a second application program; displaying at least one interactive control in a second interface; the second interface is the same or different interface as the first interface.

2. The display method according to claim 1, wherein The second application program is a smart assistant application program.

3. The display method according to claim 1 or 2, wherein The execution of the first operation and the generation of at least one interactive control corresponding to the display object through the second application program comprise: generating at least one interactive control corresponding to the content features of the display object according to the content features of the display object.

4. The display method according to claim 3, wherein The generation of at least one interactive control corresponding to the content features of the display object according to the content features of the display object comprises: generating at least one interactive control according to the operation type of the first operation and the content features.

5. The display method according to claim 3, wherein The generation of at least one interactive control corresponding to the content features of the display object according to the content features of the display object comprises: generating the content features of the display object through a feature recognition algorithm corresponding to the data type of the display object.

6. The display method according to any one of claims 1 to 5, wherein The second interface is an interface corresponding to the second application program.

7. The display method according to any one of claims 1 to 6, wherein After the display of at least one interactive control in the second interface, the method further comprises: displaying an interaction result generated based on the display object in response to a second operation on the interactive control.

8. The display method according to claim 7, wherein The display of an interaction result generated based on the display object in response to a second operation on the interactive control comprises: starting a third application program in response to a second operation on the interactive control; displaying the interaction result in a third interface corresponding to the third application program.

9. The display method according to claim 7, wherein The display of an interaction result generated based on the display object in response to a second operation on the interactive control comprises: displaying the interaction result in the second interface in response to a second operation on the interactive control; the second interface is an interface corresponding to the second application program.

10. A display method characterized by comprising: The method comprises: displaying a first interface of a first application program, the first interface displaying a display object, the first application program being different from the smart assistant program; displaying at least one interactive control associated with the display object through the smart assistant program in response to a user operation on the display object; performing a function corresponding to the at least one interactive control in response to a user operation on the at least one interactive control.

11. A display device of an interactive control, characterized by, The method comprises: a first operation receiving unit configured to receive a first operation on any display object in a first interface; the first interface is an interface corresponding to a first application program running in the foreground; a control generating unit configured to execute the first operation and generate at least one interactive control corresponding to the display object through a second application program; a control displaying unit configured to display at least one interactive control in a second interface; the second interface is the same or different interface as the first interface.

12. An electronic device, comprising: The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images