Equipment interaction method, electronic equipment, storage medium and product

Abstract

The invention discloses an equipment interaction method, electronic equipment, a storage medium and a product, and relates to the technical field of computer information, the method comprises the following steps: identifying target interaction task information from user input data; target application programs and target template function instructions matched with the target interaction task information are searched from a preset knowledge base, and the interaction task information, the application programs and the template function instructions in the knowledge base are mapped and stored; and combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application program to respond to the target function instruction. According to the application, the convenience of application function use in the intelligent equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer information technology, and in particular to a device interaction method, electronic device, storage medium, and product. Background Art

[0002] With the popularization of smart devices, the number and functions of applications available on smart devices are increasing, covering various fields such as social networking, shopping, entertainment, and office.

[0003] However, when users want to use a specific function of an application, they usually need to go through a series of tedious manual operations, such as finding the target application among numerous application icons, locating the required function in the complex menu hierarchy within the application, etc. When the user is driving, has a hand injury or is busy, it becomes particularly difficult to manually operate the smart device to find and use application functions, which greatly affects the user experience.

[0004] In summary, how to improve the convenience of using application functions in smart devices has become a technical problem that urgently needs to be solved in this field.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a device interaction method, electronic device, storage medium and product, aiming to improve the convenience of using application functions in smart devices.

[0007] To achieve the above objectives, this application proposes a device interaction method, comprising:

[0008] Identify target interaction task information from user input data;

[0009] Searching for a target application and a target template function instruction that match the target interactive task information from a preset knowledge base, wherein each interactive task information, each application and each template function instruction are mapped and stored in the knowledge base;

[0010] The target interactive task information is combined with the target template function instruction to obtain a target function instruction, and the target application is called to respond to the target function instruction.

[0011] In one embodiment, before the step of searching a preset knowledge base for a target application and a target template function instruction that match the target interactive task information, the method further includes:

[0012] Extracting application function data of each application from a preset in-memory database, wherein the application function data includes application component declaration information and an associated intent filtering parameter set;

[0013] Performing semantic parsing on the application function data to generate structured application function data including function description data and interactive task information template;

[0014] After vectorizing the structured application function data, a mapping relationship among each interactive task information, each application program and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base.

[0015] In one embodiment, the method further comprises:

[0016] Listen for system broadcast events of application status changes;

[0017] When a system broadcast event of an application state change is monitored, the changed application function data of the application that triggered the system broadcast event is extracted and parsed from the memory database to obtain the changed structured application function data;

[0018] The knowledge base is updated based on the changed structured application function data.

[0019] In one embodiment, the step of searching a preset knowledge base for a target application and a target template function instruction that matches the target interactive task information includes:

[0020] determining, from a preset knowledge base, candidate applications that can process the target interactive task information;

[0021] When there is only one candidate application, determining the candidate application as a target application, and determining a target template function instruction that matches the target application and the target interactive task information;

[0022] When there are multiple candidate applications, a target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and a target template function instruction matching the target application and the target interaction task information is determined.

[0023] In one embodiment, before the step of combining the target interaction task information with the target template function instruction to obtain a target function instruction and calling the target application to respond to the target function instruction, the step further includes:

[0024] When there are multiple pieces of target interaction task information, determining an execution data dependency relationship between each piece of target interaction task information;

[0025] The step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction includes:

[0026] For first interactive task information and second interactive task information that have an execution data dependency relationship, combining the first interactive task information with a first template function instruction to obtain a first function instruction, calling a first application to respond to the first function instruction, and obtaining intermediate data generated by the first application, wherein the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is a template function instruction in the target template function instruction corresponding to the first interactive task information, and the first application is an application in the target application corresponding to the first interactive task information;

[0027] The intermediate data, the second interactive task information and the second template function instruction are combined to obtain a second function instruction, and a second application is called to respond to the second function instruction, wherein the second template function instruction is a template function instruction corresponding to the second interactive task information in the target template function instruction, and the second application is an application corresponding to the second interactive task information in the target application.

[0028] In one embodiment, the step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction further includes:

[0029] For third interaction task information and fourth interaction task information that do not have an execution data dependency relationship, combining the third interaction task information with a third template function instruction to obtain a third function instruction, and calling a third application to respond to the third function instruction, wherein the third interaction task information and the fourth interaction task information are any two pieces of interaction task information in each target interaction task information, the third template function instruction is a template function instruction in the target template function instruction corresponding to the third interaction task information, and the third application is an application in the target application corresponding to the third interaction task information;

[0030] The fourth interactive task information is combined with the fourth template function instruction to obtain a fourth function instruction, and a fourth application is called to respond to the fourth function instruction, wherein the fourth template function instruction is a template function instruction in the target template function instruction corresponding to the fourth interactive task information, and the fourth application is an application in the target application corresponding to the fourth interactive task information.

[0031] In one embodiment, the step of identifying target interaction task information from user input data includes:

[0032] Converting user input data into semantic vectors and using a hybrid retrieval strategy to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set;

[0033] Based on the user profile and the historical interaction data recorded in the knowledge base, target interaction task information is determined from the semantic matching candidate set.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the device interaction method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the device interaction method as described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the device interaction method as described above.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] In the present application, target interaction task information is identified from user input data; target applications and target template function instructions that match the target interaction task information are searched from a preset knowledge base, wherein each interaction task information, each application, and each template function instruction in the knowledge base are mapped and stored; the target interaction task information is combined with the target template function instruction to obtain the target function instruction, and the target application is called to respond to the target function instruction. In the present application, by automatically identifying target interaction task information from user input data, and quickly matching the target application and the target template function instruction based on a preset knowledge base, and then generating the target function instruction and calling the target application to respond, the user does not need to perform tedious manual operations, but can quickly and accurately trigger the specific function of the application through natural language input or other convenient interaction methods, thereby improving the convenience of using application functions in smart devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 A flowchart of the first embodiment of the device interaction method of the present application is provided;

[0042] Figure 2 A flowchart of the second embodiment of the device interaction method of the present application is provided;

[0043] Figure 3 A flowchart of the third embodiment of the device interaction method of the present application is provided;

[0044] Figure 4 This is a schematic diagram of the structure of the device interaction apparatus according to an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the device interaction method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] With the popularization of smart devices, the number and functions of applications available on smart devices are increasing, covering various fields such as social networking, shopping, entertainment, and office.

[0050] However, when users want to use a specific function of an application, they usually need to go through a series of tedious manual operations, such as finding the target application among numerous application icons, locating the required function in the complex menu hierarchy within the application, etc. When the user is driving, has a hand injury or is busy, it becomes particularly difficult to manually operate the smart device to find and use application functions, which greatly affects the user experience.

[0051] In summary, how to improve the convenience of using application functions in smart devices has become a technical problem that urgently needs to be solved in this field.

[0052] In order to solve the above technical problems, the embodiments of the present application propose to identify target interactive task information from user input data; search for target applications and target template function instructions that match the target interactive task information from a preset knowledge base, wherein each interactive task information, each application and each template function instruction in the knowledge base are mapped and stored; the target interactive task information is combined with the target template function instruction to obtain the target function instruction, and the target application is called to respond to the target function instruction. In the embodiments of the present application, by automatically identifying the target interactive task information from user input data and quickly matching the target application and the target template function instruction based on the preset knowledge base, and then generating the target function instruction and calling the target application to respond, the user does not need to perform tedious manual operations, but can quickly and accurately trigger the specific functions of the application through natural language input or other convenient interaction methods, thereby improving the convenience of using application functions in smart devices.

[0053] It should be noted that the execution subject of this embodiment can be an intelligent device that supports the use of AI (Artificial Intelligence) tools and has data processing, network communication, and program execution functions, such as a mobile phone, tablet, computer, etc., or an electronic device that can implement the above functions. The following uses an intelligent device as an example to illustrate this embodiment and the following embodiments.

[0054] The following is a first embodiment of the device interaction method of the present application. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the device interaction method of the present application.

[0055] In this embodiment, the device interaction method includes steps S10 to S30:

[0056] Step S10, identifying and obtaining target interactive task information from user input data;

[0057] It should be noted that the smart device of this embodiment may utilize an AI assistant. This AI assistant has natural language processing capabilities and is authorized to interact with the device system. It can receive a variety of user commands in the form of voice, text, etc., and execute operational tasks for applications in the smart device based on the commands. User input data refers to information input by the user to the AI ​​assistant through voice, text, or other means, which is used to express the user's intentions or needs.

[0058] The AI ​​assistant in the smart device receives user input data, parses and understands the user input data to identify the target interaction task information corresponding to the user's intention. The target interaction task information refers to the specific operation or function that the user wants the AI ​​assistant to perform, such as: navigating to West Lake in Hangzhou or sending a WeChat message to Zhang San, etc.

[0059] In one feasible implementation, step S10 may include steps S101 to S102:

[0060] Step S101: converting user input data into semantic vectors, and using a hybrid search strategy to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set;

[0061] The AI ​​assistant first receives user input, which can be in the form of voice commands, text messages, etc. Subsequently, the AI ​​assistant uses natural language processing (NLP) technology to convert the user input data into semantic vectors. The semantic vector conversion process involves processing the user input data through word segmentation, part-of-speech tagging, and syntactic analysis to extract the semantic information and encode it into vector form. This process is not specifically limited in this embodiment.

[0062] After obtaining the semantic vector, the AI ​​assistant uses a hybrid retrieval strategy to filter out irrelevant information. The hybrid retrieval strategy can combine multiple retrieval methods, such as keyword-based retrieval, semantic similarity-based retrieval, etc., to comprehensively evaluate the degree of match between the semantic vector and the interactive task information in the preset knowledge base. In this way, the AI ​​assistant can effectively exclude irrelevant information that does not match the user's intention, thereby obtaining a more accurate semantic matching candidate set.

[0063] Step S102: determining target interaction task information from a semantic matching candidate set based on the user profile and historical interaction data recorded in the knowledge base.

[0064] It should be noted that the knowledge base can specifically be a collection of RAG (Retrieval-Augmented Generation) documents of the AI ​​model adopted by the AI ​​assistant. The knowledge base includes data information on the user's historical use of the AI ​​assistant, that is, historical interaction data. The historical interaction data records the user's past interaction history with the AI ​​assistant, including the user's requests, the AI ​​assistant's responses, and the user's feedback. The AI ​​assistant can generate a user portrait through the analysis of this data information. The user portrait contains multi-dimensional data such as the user's basic information, interest preferences, and usage habits. These data help the AI ​​assistant understand the user's intentions and needs more accurately.

[0065] After obtaining the semantic matching candidate set, the AI ​​assistant will further analyze it in combination with the user portraits and historical interaction data recorded in the knowledge base. By analyzing this data, the AI ​​assistant can understand the user's usage habits and preferences, thereby more accurately matching the user's current needs.

[0066] Based on the analysis results of user portraits and historical interaction data, the AI ​​assistant selects the target interaction task information that best meets the user's intention from the semantic matching candidate set. This target interaction task information is used to guide the AI ​​assistant to further search for target applications and target template function instructions in the knowledge base to meet the user's needs.

[0067] Therefore, by converting semantic vectors, filtering irrelevant information, and determining target interaction task information by combining user portraits and historical interaction data, the accuracy of user demand matching is improved from multiple dimensions, bringing users a higher-quality and more efficient interaction experience.

[0068] It is worth mentioning that the ways to start the AI ​​assistant of a smart device may include: waking up the AI ​​assistant according to the wake-up word; or, controlling the AI ​​startup control; or, pressing the preset physical button. To further prevent information leakage or malicious calls to application functions, voiceprint recognition can be performed when waking up the AI ​​assistant with the wake-up word to prevent others from calling it. At the same time, through permission management, when the application receives and sends sensitive Intents, such as dialing, sending SMS, paying, activating services, and providing private information, it verifies whether the source of the Intent is authorized. For example, if the AI ​​assistant of the smart device is authorized to make calls, other third-party applications will not be authorized to make calls.

[0069] Step S20, searching a preset knowledge base for target applications and target template function instructions that match the target interactive task information, wherein the knowledge base stores mappings of each interactive task information, each application, and each template function instruction;

[0070] It should be noted that the knowledge base is also used to store the mapping relationship between interactive task information, applications, and functional instructions, so that the AI ​​assistant can quickly find and match the corresponding applications and functional instructions after receiving user input data. The target application is the application that is matched in the knowledge base based on the target interactive task information and can perform the function required by the user; the target template functional instruction is a pre-defined instruction format used to describe how to call the target application to perform a specific function. When generating the target functional instruction, the AI ​​assistant will fill the specific parameters in the target interactive task information into the target template functional instruction.

[0071] After identifying the target interaction task information, the AI ​​assistant will access the preset knowledge base for matching and search. Based on the target interaction task information, it will retrieve the matching target application and corresponding target template function instructions in the knowledge base.

[0072] In one feasible implementation, step S20 may include steps S201 to S203:

[0073] Step S201, determining candidate applications that can process target interactive task information from a preset knowledge base;

[0074] The AI ​​assistant first accesses the preset knowledge base and searches the knowledge base based on the specific content of the target interaction task information to determine which applications have the ability to process the target interaction task information. The search results will form a list of alternative applications, all of which are considered to have the potential to meet the user's needs.

[0075] Step S202: When there is only one candidate application, the candidate application is determined as the target application, and a target template function instruction that matches the target application and the target interactive task information is determined;

[0076] If there is only one alternative application matched in the knowledge base, the AI ​​assistant will directly determine that application as the target application, and then further search the knowledge base for target template function instructions that match the target interaction task information. The target template function instructions are pre-defined and are used to describe the instruction format of how to call the target application to perform a specific function.

[0077] For example, the target interactive task information is to navigate to the West Lake in Hangzhou. Only one application that can handle the target interactive task information is retrieved in the knowledge base: AutoNavi Map. The target template function instructions further found include at least Action and Data parameter template parts. Taking the smart device using the Android system as an example, the Action is android.intent.VIEW, and the specific operation class used to define the intent is VIEW (view). The Data parameter template is androidamap: / / navipoiname={location}, which is used to define the specific data carried by the intent as the location.

[0078] Step S203 , when there are multiple candidate applications, a target application is determined from the candidate applications based on historical interaction data recorded in the knowledge base, and a target template function instruction matching the target application and the target interaction task information is determined.

[0079] If the knowledge base matches multiple candidate applications, the AI ​​assistant can refer to the historical interaction data recorded in the knowledge base. By analyzing this data, the AI ​​assistant can understand the user's usage habits and preferences, thereby more accurately matching the user's current needs. Based on the analysis of historical interaction data, the AI ​​assistant will filter out the target application from the candidate applications that best meets the user's intent and determine the target template function instructions within the target application that match the target interaction task information.

[0080] For example, the target interactive task information is navigation to West Lake in Hangzhou. The applications retrieved in the knowledge base that can handle the target interactive task information include AutoNavi Map and Baidu Map. The AI ​​assistant determines that the user prefers to use AutoNavi Map for route navigation by analyzing historical interaction data, and then determines that the target application is AutoNavi Map.

[0081] Step S30: combining the target interactive task information with the target template function instruction to obtain a target function instruction, and calling the target application program to respond to the target function instruction.

[0082] It should be noted that the target function instruction is a complete instruction generated by filling the specific parameters in the target interactive task information into the target template function instruction, and is used to call the target application and perform corresponding operations.

[0083] After finding the target application and the target template function instruction, the AI ​​assistant fills the specific parameters (such as location, contact, etc.) from the target interaction task information into the target template function instruction to generate a complete target function instruction. It then calls the target application and sends the target function instruction to it. After receiving the instruction, the target application will perform the corresponding operation, such as opening a navigation app and setting a destination, or opening the WeChat app and sending a specified message to a specified contact.

[0084] Therefore, in this embodiment, by automatically identifying the target interaction task information from the user input data, and quickly matching the target application and target template function instructions based on the preset knowledge base, and then generating the target function instructions and calling the target application response, the user does not need to perform tedious manual operations, but only needs to use natural language input or other convenient interaction methods to quickly and accurately trigger the specific functions of the application, thereby improving the convenience of using application functions in smart devices.

[0085] Based on the above first embodiment, the second embodiment of the device interaction method of this application is proposed. In this embodiment, the same or similar contents as those of the above first embodiment can be referred to the above introduction and will not be repeated hereafter. Figure 2 , step S40 may be further included before step S30:

[0086] Step S40: when there are multiple pieces of target interactive task information, determining the execution data dependency relationship of each piece of target interactive task information;

[0087] It should be noted that the AI ​​assistant may identify multiple target interaction task information from the user input data, that is, the user expects the AI ​​assistant to complete multiple specific operations or functions. For example, if the user input data is "help me order a coffee and then navigate to pick it up", then two target interaction task information of "order coffee" and "navigate to pick it up" can be identified. The execution data dependency relationship refers to whether the execution of the interaction task to be performed by one target interaction task information depends on the execution result of the interaction task to be performed by another target interaction task information. The AI ​​assistant can determine the execution data dependency relationship between multiple interaction tasks to be performed by multiple target interaction task information by analyzing the semantic structure of the target interaction task information, the user's historical interaction habits, and the logical relationship between the interaction tasks.

[0088] On this basis, step S30 may include steps S301 to S302:

[0089] Step S301: For first interactive task information and second interactive task information having an execution data dependency relationship, the first interactive task information is combined with a first template function instruction to obtain a first function instruction, a first application is called to respond to the first function instruction, and intermediate data generated by the first application is obtained, wherein the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is a template function instruction in the target template function instruction corresponding to the first interactive task information, and the first application is an application in the target application corresponding to the first interactive task information;

[0090] When the AI ​​assistant determines that two interactive task information in the target interactive task information (such as "order coffee and then navigate to get it") have execution data dependencies, it will process the two interactive task information in sequence according to the dependencies.

[0091] Specifically, for the first interactive task information (e.g., "order coffee"), the AI ​​assistant combines this first interactive task information with a first template function instruction (a predefined instruction format for calling a coffee ordering application) to generate a first function instruction. It then calls the first application (e.g., Meituan) to respond to the first function instruction and execute the coffee ordering operation. After executing the coffee ordering operation, the first application generates intermediate data (e.g., the coffee shop address), which serves as the basis for processing the second interactive task information.

[0092] Step S302: Combine the intermediate data, the second interactive task information and the second template function instruction to obtain a second function instruction, and call the second application to respond to the second function instruction, wherein the second template function instruction is the template function instruction corresponding to the second interactive task information in the target template function instruction, and the second application is the application corresponding to the second interactive task information in the target application.

[0093] After obtaining the intermediate data generated by the first application, the AI ​​assistant combines the intermediate data, the second interactive task information (such as "navigation") and the second template function instruction (a predefined instruction format for calling the navigation application) to generate a second function instruction. Then, the AI ​​assistant calls the second application (such as AutoNavi Map) to respond to the second function instruction, and uses the intermediate data (the address of the coffee shop) as the navigation destination to execute the navigation operation to get coffee.

[0094] In addition, when it is detected that the first target application does not return the intermediate data, a preset abnormal prompt message is output.

[0095] In one feasible implementation, step S30 may further include steps S303 to S304:

[0096] Step S303: For the third interaction task information and the fourth interaction task information that do not have an execution data dependency, the third interaction task information is combined with the third template function instruction to obtain a third function instruction, and a third application is called to respond to the third function instruction, wherein the third interaction task information and the fourth interaction task information are any two pieces of interaction task information in each target interaction task information, the third template function instruction is the template function instruction in the target template function instruction corresponding to the third interaction task information, and the third application is the application in the target application corresponding to the third interaction task information;

[0097] When the AI ​​assistant determines that there is no execution data dependency between two interactive task information in the target interactive task information (such as "order coffee" and "play music"), the two interactive task information will be processed separately.

[0098] Specifically, for the third interactive task information (such as "order coffee"), the AI ​​assistant combines the third interactive task information with the third template function instruction (a pre-defined instruction format for calling the coffee ordering application) to generate a third function instruction, and then calls the third application (such as Meituan) to respond to the third function instruction and execute the coffee ordering operation.

[0099] Step S304: combine the fourth interactive task information with the fourth template function instruction to obtain a fourth function instruction, and call the fourth application to respond to the fourth function instruction, wherein the fourth template function instruction is the template function instruction in the target template function instruction corresponding to the fourth interactive task information, and the fourth application is the application in the target application corresponding to the fourth interactive task information.

[0100] Similarly, for the fourth interactive task information (such as "play music"), the AI ​​assistant combines the fourth interactive task information with the fourth template function instruction (a predefined instruction format for calling a music playback application) to generate a fourth function instruction, and then calls the fourth application (such as Kugou Music) to respond to the fourth function instruction and perform the operation of playing music.

[0101] Therefore, according to whether there is an execution data dependency between the interactive task information, different strategies are adopted to call the target application and respond to the target function instructions. For interactive task information that does not have an execution data dependency, the corresponding applications can be called in parallel to improve the interaction efficiency; for interactive task information that has a data dependency, serial calls are made in the dependency order, and intermediate data is used for subsequent interactions to ensure the correctness and consistency of the interaction, so that users can complete complex operations more efficiently and smoothly.

[0102] Based on the above-mentioned first embodiment and / or second embodiment, a third embodiment of the device interaction method of the present application is proposed. In this embodiment, the same or similar contents as those of the above-mentioned first embodiment and second embodiment can be referred to the above introduction and will not be repeated hereafter. Figure 3 , before step S20, steps A10 to A30 may also be included:

[0103] Step A10: extracting application function data of each application from a preset in-memory database, wherein the application function data includes application component declaration information and an associated intent filtering parameter set;

[0104] It should be noted that in this embodiment, when the smart device installs or updates the application package file, it will parse the application manifest file in the file. The application manifest file is used to declare the basic information, components, application capabilities and other information of the application, and store the parsed information (at least including the application component declaration information and the associated intent filtering parameter set) into a preset memory database.

[0105] For example, taking the Android system as an example, when the system installs or updates an APK (Android application package) file, the first operation is to parse the AndroidManifest.xml (Android application manifest file) in the APK file. The system will read component tags such as activity and service in the file, and store the read information in the memory database of PackageManagerService (PMS), that is, store the application function data of each application in the memory database.

[0106] The AI ​​assistant accesses an in-memory database that stores application functionality data for each application installed on the user's device. This data includes, but is not limited to, application component declaration information (such as the labels and attributes of components like Activities and Services) and the associated intent filter parameter sets. These intent filter parameter sets define the specific user intents or request types that an application can respond to and are the foundation for enabling application capabilities to interact with the AI ​​assistant. By extracting this data, the AI ​​assistant can fully understand the functional scope and responsiveness of each application.

[0107] Step A20: semantically analyze the application function data to generate structured application function data including function description data and interactive task information templates;

[0108] After extracting the application function data, the AI ​​assistant performs semantic parsing on it. The purpose of semantic parsing is to convert the original application component declaration information and intent filter parameter set into a form that is easier to understand and process.

[0109] Specifically, the AI ​​assistant will analyze the action, category, data and other fields in the component tags in the application function data, understand the functions and intentions they represent, and combine this information with natural language descriptions to generate structured application function data containing function description data and interaction task information templates. The function description data is used to describe the functional characteristics of the application, and the interaction task information template is the sentence format or example that the user may use when interacting with the application.

[0110] Step A30 : After vectorizing the structured application function data, a mapping relationship between each interactive task information, each application program and each template function instruction is obtained, and the mapping relationship is stored in a knowledge base.

[0111] After completing the semantic analysis, the AI ​​assistant will vectorize the structured application function data. Vectorization processing is to establish a mapping and storage between the capability meaning and the structured intent. The capability meaning is such as "AutoNavi Map (app name) navigates to a certain place (actions)", and the structured intent is such as "action = VIEW, data = androidamap: / / navi...". Finally, the mapping relationship between each interactive task information, each application and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base. The mapping relationship can be stored in the form of a data table or other forms, so that the AI ​​assistant can match information more efficiently.

[0112] For example, the data table may be in the form of the following table 1:

[0113] Table 1

[0114]

[0115] The instruction template is the interactive task information, the target application is the application that can process the interactive task information, and the Action and Data parameter templates can be combined into corresponding template function instructions.

[0116] In one feasible implementation, the device interaction method may further include steps A40 to A60:

[0117] Step A40: monitoring system broadcast events of application status changes;

[0118] In this embodiment, to ensure that the AI ​​assistant can promptly perceive and respond to changes in the application's state, a corresponding monitoring mechanism is configured to listen for system broadcast events indicating application state changes. These system broadcast events are typically issued by the device's operating system to notify other components or services of changes in the application's state (such as installation, update, and uninstallation). By monitoring these events, the AI ​​assistant can dynamically grasp the latest application status.

[0119] For example, under the Android operating system, the AI ​​assistant can use BroadcastReceiver (broadcast receiver) to listen to system broadcast events for installing, updating, and uninstalling applications, and use PackageManager.getPackageInfo() (a package information query interface) to obtain a list of all installed applications.

[0120] Step A50: when a system broadcast event indicating an application state change is monitored, extract and parse the changed application function data of the application that triggered the system broadcast event from the memory database to obtain the changed structured application function data;

[0121] When a system broadcast event indicating an application status change is detected, the AI ​​assistant will locate and extract the changed application function data of the application from the in-memory database based on the application identification information carried in the system broadcast event. It will then parse the changed application function data to generate the changed structured application function data containing the latest function description data and interactive task information template.

[0122] Step A60: updating the database based on the changed structured application function data.

[0123] After obtaining the changed structured application function data, the AI ​​assistant will update the in-memory database based on this data. The update process involves overwriting or merging the changed application function data into the original records of the corresponding application in the in-memory database to ensure that the application function data stored in the knowledge base is always consistent with the actual status of the application.

[0124] Therefore, the system broadcasts events that monitor application status changes, and when events are monitored, it promptly extracts and parses the changed application function data, and then updates the knowledge base, so that the data in the knowledge base is always synchronized with the actual functions of the application, avoiding the problem of invalid interaction instructions or matching errors due to application updates or status changes, and ensuring the reliability and stability of the interaction.

[0125] In summary, this embodiment builds a complete knowledge base by pre-extracting, parsing, and vectorizing application function data. Simultaneously, it monitors application state changes and updates the knowledge base, ensuring real-time synchronization between interactive operations and actual application functions to avoid interaction errors. Consequently, after receiving user input, the AI ​​assistant can quickly match target applications with target template function instructions from the knowledge base. For multi-task interaction scenarios, the execution order is intelligently arranged based on data dependencies to ensure the correctness and coherence of complex interaction logic. This improves the accuracy, efficiency, and stability of device interaction, allowing users to conveniently and smoothly complete diverse operations using natural language, fully meeting user interaction needs in different scenarios and optimizing the user-device interaction experience.

[0126] The present application also provides a device interaction device, please refer to Figure 4 , the device interaction device includes:

[0127] The information recognition module 10 is used to identify target interactive task information from user input data;

[0128] An information matching module 20 is configured to search a preset knowledge base for target applications and target template function instructions that match target interactive task information, wherein the knowledge base stores mappings of interactive task information, application programs, and template function instructions;

[0129] The instruction response module 30 is used to combine the target interactive task information with the target template function instruction to obtain a target function instruction, and call the target application program to respond to the target function instruction.

[0130] Optionally, the device interaction apparatus further includes a data storage module (not shown), which is configured to:

[0131] Extracting application function data of each application from a preset in-memory database, wherein the application function data includes application component declaration information and an associated intent filtering parameter set;

[0132] Perform semantic analysis on application function data to generate structured application function data including function description data and interactive task information template;

[0133] After vectorizing the structured application function data, the mapping relationship between each interactive task information, each application program and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base.

[0134] Optionally, the device interaction apparatus further includes a data updating module (not shown), which is configured to:

[0135] Listen for system broadcast events of application status changes;

[0136] When a system broadcast event of an application state change is monitored, the application function data after the application that triggered the system broadcast event is extracted and parsed from the in-memory database to obtain the changed structured application function data;

[0137] The knowledge base is updated based on the changed structured application function data.

[0138] Optionally, the information matching module 20 is further configured to:

[0139] Determine candidate applications that can process target interactive task information from a pre-set knowledge base;

[0140] When there is only one candidate application, determining the candidate application as the target application, and determining a target template function instruction that matches the target application and the target interaction task information;

[0141] When there are multiple candidate applications, a target application is determined from the candidate applications based on historical interaction data recorded in the knowledge base, and a target template function instruction that matches the target application and target interaction task information is determined.

[0142] Optionally, the instruction response module 30 is further configured to:

[0143] When there are multiple pieces of target interactive task information, determining the execution data dependency relationship of each piece of target interactive task information;

[0144] For first interactive task information and second interactive task information having an execution data dependency relationship, combining the first interactive task information with the first template function instruction to obtain a first function instruction, calling a first application to respond to the first function instruction, and obtaining intermediate data generated by the first application, wherein the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is a template function instruction in the target template function instruction corresponding to the first interactive task information, and the first application is an application in the target application corresponding to the first interactive task information;

[0145] The intermediate data, the second interactive task information and the second template function instruction are combined to obtain a second function instruction, and the second application is called to respond to the second function instruction, wherein the second template function instruction is the template function instruction corresponding to the second interactive task information in the target template function instruction, and the second application is the application corresponding to the second interactive task information in the target application.

[0146] Optionally, the instruction response module 30 is further configured to:

[0147] For third interaction task information and fourth interaction task information that do not have an execution data dependency relationship, combining the third interaction task information with a third template function instruction to obtain a third function instruction, and calling a third application to respond to the third function instruction, wherein the third interaction task information and the fourth interaction task information are any two pieces of interaction task information in each target interaction task information, the third template function instruction is a template function instruction in the target template function instruction corresponding to the third interaction task information, and the third application is an application in the target application corresponding to the third interaction task information;

[0148] The fourth interactive task information is combined with the fourth template function instruction to obtain a fourth function instruction, and the fourth application is called to respond to the fourth function instruction, wherein the fourth template function instruction is the template function instruction in the target template function instruction corresponding to the fourth interactive task information, and the fourth application is the application in the target application corresponding to the fourth interactive task information.

[0149] Optionally, the information identification module 10 is further configured to:

[0150] Convert user input data into semantic vectors, and use a hybrid retrieval strategy to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set;

[0151] Based on user profiles and historical interaction data recorded in the knowledge base, the target interaction task information is determined from the semantic matching candidate set.

[0152] An embodiment of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the device interaction method in the above embodiment.

[0153] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0154] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a DSP processor, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a microphone or accelerometer; an output device 1008 including, for example, a speaker or vibrator; a storage device 1003 including, for example, a magnetic tape or hard disk; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows an electronic device with various systems, it should be understood that not all of the illustrated systems are required to be implemented or present. Greater or fewer systems may alternatively be implemented or provided.

[0155] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0156] Compared with the prior art, the beneficial effects of the electronic device provided in the embodiment of the present application are the same as the beneficial effects of the device interaction method provided in the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0157] It should be understood that the various parts disclosed in the embodiments of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0158] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0159] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the device interaction method in the above-mentioned embodiment.

[0160] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: radio frequency), etc., or any suitable combination thereof.

[0161] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0162] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device performs the functions defined in the method of the embodiment disclosed in this application.

[0163] The computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0165] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0166] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned device interaction method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the device interaction method provided in the above-mentioned embodiment, and will not be repeated here.

[0167] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned device interaction method when executed by a processor.

[0168] Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the device interaction method provided by the above embodiment, and will not be repeated here.

[0169] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A device interaction method, characterized in that: The device interaction method includes: Identify target interaction task information from user input data; Searching for a target application and a target template function instruction that match the target interactive task information from a preset knowledge base, wherein each interactive task information, each application and each template function instruction are mapped and stored in the knowledge base; The target interactive task information is combined with the target template function instruction to obtain a target function instruction, and the target application is called to respond to the target function instruction.

2. The device interaction method according to claim 1, wherein: Before the step of searching a preset knowledge base for a target application and a target template function instruction that matches the target interactive task information, the method further includes: Extracting application function data of each application from a preset in-memory database, wherein the application function data includes application component declaration information and an associated intent filtering parameter set; Performing semantic parsing on the application function data to generate structured application function data including function description data and interactive task information template; After vectorizing the structured application function data, a mapping relationship among each interactive task information, each application program and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base.

3. The device interaction method according to claim 2, wherein: The method further comprises: Listen for system broadcast events of application status changes; When a system broadcast event of an application state change is monitored, the changed application function data of the application that triggered the system broadcast event is extracted and parsed from the memory database to obtain the changed structured application function data; The knowledge base is updated based on the changed structured application function data.

4. The device interaction method according to claim 1, wherein: The step of searching a preset knowledge base for a target application and a target template function instruction that matches the target interactive task information includes: determining, from a preset knowledge base, candidate applications that can process the target interactive task information; When there is only one candidate application, determining the candidate application as a target application, and determining a target template function instruction that matches the target application and the target interactive task information; When there are multiple candidate applications, a target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and a target template function instruction matching the target application and the target interaction task information is determined.

5. The device interaction method according to claim 1, wherein: Before the step of combining the target interaction task information with the target template function instruction to obtain a target function instruction and calling the target application to respond to the target function instruction, the method further includes: When there are multiple pieces of target interaction task information, determining an execution data dependency relationship between each piece of target interaction task information; The step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction includes: For first interactive task information and second interactive task information that have an execution data dependency relationship, combining the first interactive task information with a first template function instruction to obtain a first function instruction, calling a first application to respond to the first function instruction, and obtaining intermediate data generated by the first application, wherein the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is a template function instruction in the target template function instruction corresponding to the first interactive task information, and the first application is an application in the target application corresponding to the first interactive task information; The intermediate data, the second interactive task information and the second template function instruction are combined to obtain a second function instruction, and a second application is called to respond to the second function instruction, wherein the second template function instruction is a template function instruction corresponding to the second interactive task information in the target template function instruction, and the second application is an application corresponding to the second interactive task information in the target application.

6. The device interaction method according to claim 5, wherein: The step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction further includes: For third interaction task information and fourth interaction task information that do not have an execution data dependency relationship, combining the third interaction task information with a third template function instruction to obtain a third function instruction, and calling a third application to respond to the third function instruction, wherein the third interaction task information and the fourth interaction task information are any two pieces of interaction task information in each target interaction task information, the third template function instruction is a template function instruction in the target template function instruction corresponding to the third interaction task information, and the third application is an application in the target application corresponding to the third interaction task information; The fourth interactive task information is combined with the fourth template function instruction to obtain a fourth function instruction, and a fourth application is called to respond to the fourth function instruction, wherein the fourth template function instruction is a template function instruction in the target template function instruction corresponding to the fourth interactive task information, and the fourth application is an application in the target application corresponding to the fourth interactive task information.

7. The device interaction method according to claim 1, wherein: The step of identifying target interactive task information from user input data includes: Converting user input data into semantic vectors and using a hybrid retrieval strategy to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set; Based on the user profile and the historical interaction data recorded in the knowledge base, target interaction task information is determined from the semantic matching candidate set.

8. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the device interaction method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the device interaction method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the device interaction method described in any one of 1 to 7 are implemented.

Images