Device interaction method, electronic device, storage medium and product

Abstract

The application discloses a device interaction method, an electronic device, a storage medium and a product, relates to the technical field of computer information, and the method comprises the following steps: obtaining target interaction task information from user input data; finding a target application program and a target template function instruction matched with the target interaction task information from a preset knowledge base, wherein each interaction task information, each application program and each template function instruction in the knowledge base are stored in a mapping mode; combining the target interaction task information and the target template function instruction to obtain a target function instruction, and calling the target application program to respond to the target function instruction. The application can improve the convenience of application program function use in the intelligent device.

Description

Technical Field

[0001] This application relates to the field of computer information technology, and in particular to a device interaction method, electronic device, storage medium and product. Background Technology

[0002] With the increasing popularity of smart devices, the number and functions of available applications on smart devices are also increasing, covering various fields such as social networking, shopping, entertainment, and office work.

[0003] However, when users want to use a specific function of an application, they usually have to go through a series of tedious manual operations, such as finding the target application among many application icons and locating the required function in the complex menu hierarchy within the application. When users are driving, have injured hands, or are busy, manually operating smart devices to find and use application functions becomes particularly difficult, which greatly affects the user experience.

[0004] In summary, improving the ease of use of applications in smart devices has become a pressing technical problem that needs to be solved in this field.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a device interaction method, electronic device, storage medium and product, which aims to improve the ease of use of application functions in smart devices.

[0007] To achieve the above objectives, this application proposes a device interaction method, comprising:

[0008] Identify the target interaction task information from user input data;

[0009] The target application and target template function instructions that match the target interaction task information are searched from a preset knowledge base, wherein each interaction task information, each application and each template function instruction are mapped and stored in the knowledge base;

[0010] The target interactive task information is combined with the target template function instruction to obtain the target function instruction, and the target application is invoked to respond to the target function instruction.

[0011] In one embodiment, before the step of searching for the target application and target template function instructions that match the target interaction task information from a preset knowledge base, the method further includes:

[0012] The application function data of each application is extracted from a preset memory database, wherein the application function data includes application component declaration information and associated intent filtering parameter set;

[0013] The application function data is semantically parsed to generate structured application function data containing function description data and interactive task information templates;

[0014] After vectorizing the structured application function data, the mapping relationship between each interactive task information, each application and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base.

[0015] In one embodiment, the method further includes:

[0016] Listen for system broadcast events that indicate changes in application state;

[0017] When a system broadcast event for an application state change is detected, the application function data of the application that triggered the system broadcast event is extracted and parsed from the memory database to obtain the changed structured application function data.

[0018] The knowledge base is updated based on the modified structured application function data.

[0019] In one embodiment, the step of searching for target application and target template function instructions that match the target interaction task information from a preset knowledge base includes:

[0020] Select alternative applications capable of processing the target interactive task information from a preset knowledge base;

[0021] When there is only one candidate application, the candidate application is determined as the target application, and a target template function instruction that matches the target application and the target interaction task information is determined.

[0022] When there are multiple candidate applications, the target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and the target template function instruction that matches the target application and the target interaction task information is determined.

[0023] In one embodiment, before the step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction, the method further includes:

[0024] When there are multiple target interaction task information items, determine the execution data dependency relationship of each target interaction task information item;

[0025] The step of combining the target interactive task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction, includes:

[0026] For first and second interactive task information that have execution data dependencies, the first interactive task information is combined with a first template function instruction to obtain a first function instruction. The first application is called to respond to the first function instruction and obtain intermediate data generated by the first application. Here, the first interactive task information and the second interactive task information are any two interactive task information in each of the target interactive task information. The first template function instruction is the template function instruction in the target template function instructions that corresponds to the first interactive task information. The first application is the application in the target application that corresponds to the first interactive task information.

[0027] The intermediate data, the second interactive task information, and the second template function instruction are combined to obtain a second function instruction, and a second application is invoked to respond to the second function instruction. The second template function instruction is the template function instruction in the target template function instruction that corresponds to the second interactive task information, and the second application is the application in the target application that corresponds to the second interactive task information.

[0028] In one embodiment, the step of combining the target interaction task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction, further includes:

[0029] For third and fourth interactive task information that do not have execution data dependencies, the third interactive task information is combined with a third template function instruction to obtain a third function instruction, and a third application is invoked to respond to the third function instruction. Here, the third interactive task information and the fourth interactive task information are any two interactive task information from each of the target interactive task information, the third template function instruction is the template function instruction corresponding to the third interactive task information in the target template function instructions, and the third application is the application in the target application corresponding to the third interactive task information.

[0030] The fourth interactive task information is combined with the fourth template function instruction to obtain the fourth function instruction, and the fourth application is invoked to respond to the fourth function instruction. The fourth template function instruction is the template function instruction in the target template function instruction that corresponds to the fourth interactive task information, and the fourth application is the application in the target application that corresponds to the fourth interactive task information.

[0031] In one embodiment, the step of identifying target interaction task information from user input data includes:

[0032] User input data is converted into semantic vectors, and a hybrid retrieval strategy is used to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set;

[0033] Based on user profiles and historical interaction data recorded in the knowledge base, target interaction task information is determined from the semantic matching candidate set.

[0034] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the device interaction method as described above.

[0035] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the device interaction method described above.

[0036] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the device interaction method described above.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] In this application, target interaction task information is identified from user input data; target applications and target template function instructions that match the target interaction task information are searched from a preset knowledge base, wherein each interaction task information, each application, and each template function instruction is mapped and stored in the knowledge base; the target interaction task information and target template function instructions are combined to obtain target function instructions, and the target application is invoked to respond to the target function instructions. This application automatically identifies target interaction task information from user input data and quickly matches target applications and target template function instructions based on a preset knowledge base, thereby generating target function instructions and invoking the target application to respond. This allows users to quickly and accurately trigger specific functions of applications without cumbersome manual operations, simply through natural language input or other convenient interaction methods, improving the ease of use of application functions in smart devices. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the first embodiment of the device interaction method of this application;

[0042] Figure 2 This is a flowchart illustrating the second embodiment of the device interaction method of this application.

[0043] Figure 3 This is a flowchart illustrating the third embodiment of the device interaction method of this application.

[0044] Figure 4 This is a schematic diagram of the structure of the device interaction device according to an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the device interaction method in the embodiments of this application.

[0046] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0048] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0049] With the increasing popularity of smart devices, the number and functions of available applications on smart devices are also increasing, covering various fields such as social networking, shopping, entertainment, and office work.

[0050] However, when users want to use a specific function of an application, they usually have to go through a series of tedious manual operations, such as finding the target application among many application icons and locating the required function in the complex menu hierarchy within the application. When users are driving, have injured hands, or are busy, manually operating smart devices to find and use application functions becomes particularly difficult, which greatly affects the user experience.

[0051] In summary, improving the ease of use of applications in smart devices has become a pressing technical problem that needs to be solved in this field.

[0052] To address the aforementioned technical problems, this application proposes a method to identify target interaction task information from user input data; search a preset knowledge base for target applications and target template function instructions that match the target interaction task information, wherein each interaction task information, application, and template function instruction is mapped and stored in the knowledge base; and combine the target interaction task information with the target template function instructions to obtain target function instructions, thereby invoking the target application to respond to the target function instructions. In this application embodiment, by automatically identifying target interaction task information from user input data and quickly matching target applications and target template function instructions based on a preset knowledge base, and then generating target function instructions and invoking the target application to respond, users can quickly and accurately trigger specific functions of applications without cumbersome manual operations, simply through natural language input or other convenient interaction methods, thus improving the ease of use of application functions in smart devices.

[0053] It should be noted that the executing entity in this embodiment can be a smart device that supports the use of AI (Artificial Intelligence) tools and has data processing, network communication, and program execution functions, such as a mobile phone, tablet, or computer, or an electronic device capable of performing the above functions. The following description uses a smart device as an example to illustrate this embodiment and the subsequent embodiments.

[0054] The following presents a first embodiment of the device interaction method of this application. (Refer to...) Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the device interaction method of this application.

[0055] In this embodiment, the device interaction method includes steps S10 to S30:

[0056] Step S10: Identify and obtain target interaction task information from user input data;

[0057] It should be noted that the smart device in this embodiment can use an AI assistant. This AI assistant has natural language processing capabilities and device system interaction permissions, and can receive diverse commands from the user in the form of voice, text, etc., and execute operation tasks for applications in the smart device based on the commands. User input data refers to information that the user inputs to the AI ​​assistant through voice, text, or other means. This information is used to express the user's intentions or needs.

[0058] AI assistants in smart devices receive user input data and parse and understand it to identify the target interaction task information corresponding to the user's intent. The target interaction task information refers to the specific operation or function that the user wants the AI ​​assistant to perform, such as navigating to West Lake in Hangzhou or sending a WeChat message to Zhang San.

[0059] In one feasible implementation, step S10 may include steps S101 to S102:

[0060] Step S101: Convert the user input data into a semantic vector, and use a hybrid retrieval strategy to filter irrelevant information in the semantic vector to obtain a semantic matching candidate set;

[0061] The AI ​​assistant first receives user input data, which can be in the form of voice commands, text messages, etc. Then, the AI ​​assistant uses Natural Language Processing (NLP) technology to transform the user input data into semantic vectors. This semantic vector transformation process involves word segmentation, part-of-speech tagging, and syntactic analysis of the user input data to extract semantic information and encode it into vector form. This embodiment does not specifically limit this process.

[0062] After obtaining the semantic vector, the AI ​​assistant uses a hybrid retrieval strategy to filter out irrelevant information. This strategy combines multiple retrieval methods, such as keyword-based retrieval and semantic similarity-based retrieval, to comprehensively evaluate the matching degree between the semantic vector and the interactive task information in the preset knowledge base. As a result, the AI ​​assistant can effectively eliminate irrelevant information that does not match the user's intent, thereby obtaining a more accurate set of semantic matching candidates.

[0063] Step S102: Based on user profiles and historical interaction data recorded in the knowledge base, determine the target interaction task information from the semantic matching candidate set.

[0064] It should be noted that the knowledge base can specifically be a collection of RAG (Retrieval-Augmented Generation) documents of the AI ​​model used by the AI ​​assistant. This knowledge base includes data on the user's historical use of the AI ​​assistant, i.e., historical interaction data. Historical interaction data records the user's past interaction history with the AI ​​assistant, including the user's requests, the AI ​​assistant's responses, and the user's feedback. The AI ​​assistant can analyze this data to generate a user profile. The user profile contains multi-dimensional data such as the user's basic information, interests and preferences, and usage habits. This data helps the AI ​​assistant to more accurately understand the user's intentions and needs.

[0065] After obtaining the semantic matching candidate set, the AI ​​assistant will further analyze the user profiles and historical interaction data recorded in the knowledge base. By analyzing this data, the AI ​​assistant can understand the user's usage habits and preferences, thereby more accurately matching the user's current needs.

[0066] Based on the analysis of user profiles and historical interaction data, the AI ​​assistant selects the target interaction task information that best matches the user's intent from the semantic matching candidate set. This target interaction task information is used to guide the AI ​​assistant to further search for target applications and target template function instructions in the knowledge base to meet the user's needs.

[0067] Thus, by transforming semantic vectors, filtering irrelevant information, and combining user profiles and historical interaction data to determine target interaction task information, the accuracy of matching user needs is improved from multiple dimensions, bringing users a better and more efficient interactive experience.

[0068] It's worth noting that the AI ​​assistant on a smart device can be activated in several ways: by using a wake word; by controlling the AI ​​activation control; or by pressing a preset physical button. To further prevent information leakage or malicious invocation of application functions, voiceprint recognition can be performed when activating the AI ​​assistant with a wake word to prevent unauthorized access. Simultaneously, through permission management, when an application receives and sends sensitive intents, such as dialing, sending SMS messages, making payments, activating services, or providing private information, the source of the intent is verified to ensure authorization. For example, the AI ​​assistant on the smart device may be authorized to make phone calls, while third-party applications may not be authorized to do so.

[0069] Step S20: Search for target applications and target template function instructions that match the target interaction task information from the preset knowledge base. The knowledge base stores each interaction task information, each application, and each template function instruction in a mapping manner.

[0070] It's important to note that the knowledge base also stores the mapping relationships between interaction task information, applications, and function commands. This allows the AI ​​assistant to quickly find and match the corresponding applications and function commands after receiving user input data. The target application is the application matched in the knowledge base based on the target interaction task information, capable of performing the user's required function. The target template function command is a predefined command format used to describe how to call the target application to perform a specific function. When generating the target function command, the AI ​​assistant fills the target template function command with the specific parameters from the target interaction task information.

[0071] After identifying the target interaction task information, the AI ​​assistant will access a preset knowledge base for matching and searching. Based on the target interaction task information, it will retrieve the target application and corresponding target template function instructions that match it from the knowledge base.

[0072] In one feasible implementation, step S20 may include steps S201 to S203:

[0073] Step S201: Determine alternative applications that can process target interactive task information from a preset knowledge base;

[0074] The AI ​​assistant first accesses a pre-set knowledge base. Based on the specific content of the target interaction task information, it searches the knowledge base to determine which applications have the ability to process the target interaction task information. The search results will form a list of candidate applications, all of which are considered to be likely to meet the user's needs.

[0075] Step S202: When there is only one candidate application, the candidate application is determined as the target application, and the target template function instruction that matches the target application and the target interaction task information is determined.

[0076] If only one candidate application is matched in the knowledge base, the AI ​​assistant will directly identify that application as the target application. Then, it will further search the knowledge base for target template function instructions that match the target interaction task information. The target template function instructions are predefined and describe the instruction format for calling the target application to perform a specific function.

[0077] For example, if the target interaction task information is navigation to West Lake in Hangzhou, only one application that can handle this target interaction task information can be found in the knowledge base: Gaode Map. Further searching for the target template function instruction will at least include Action and Data parameter template parts. Taking the smart device using the Android system as an example, the Action is android.intent.VIEW, which is used to define the specific operation class of the intent as VIEW (view), and the Data parameter template is androidamap: / / navipoiname={location}, which is used to define the specific data carried by the intent as the location.

[0078] Step S203: When there are multiple candidate applications, the target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and the target template function instruction that matches the target application and the target interaction task information is determined.

[0079] If multiple candidate applications are matched in the knowledge base, the AI ​​assistant can refer to the historical interaction data recorded in the knowledge base. By analyzing this data, the AI ​​assistant can understand the user's usage habits and preferences, thereby more accurately matching the user's current needs. Based on the analysis results of historical interaction data, the AI ​​assistant will select the target application that best matches the user's intent from the candidate applications and determine the target template function instructions in that target application that match the target interaction task information.

[0080] For example, if the target interaction task information is navigation to West Lake in Hangzhou, the knowledge base finds applications that can handle this target interaction task information, including Amap and Baidu Maps. The AI ​​assistant determines that the user prefers to use Amap for route navigation by analyzing historical interaction data, thus identifying Amap as the target application.

[0081] Step S30: Combine the target interactive task information with the target template function instructions to obtain the target function instructions, and call the target application to respond to the target function instructions.

[0082] It should be noted that the target function instruction is a complete instruction generated by filling the specific parameters in the target interactive task information into the target template function instruction. It is used to call the target application and perform the corresponding operation.

[0083] After locating the target application and the target template function instruction, the AI ​​assistant fills the target interaction task information (such as location, contact, etc.) into the target template function instruction to generate a complete target function instruction. Then, it calls the target application and sends the target function instruction to that application. Upon receiving the instruction, the target application will perform the corresponding operation, such as opening a navigation application and setting a destination, or opening the WeChat application and sending a specified message to a specified contact.

[0084] Therefore, in this embodiment, by automatically identifying target interaction task information from user input data and quickly matching target application and target template function instructions based on a preset knowledge base, target function instructions are generated and the target application is invoked to respond. This allows users to quickly and accurately trigger specific functions of the application without cumbersome manual operations, simply by using natural language input or other convenient interaction methods, thus improving the ease of use of application functions in smart devices.

[0085] Based on the first embodiment described above, a second embodiment of the device interaction method of this application is proposed. In this embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description, and will not be repeated hereafter. In this embodiment, refer to... Figure 2 Step S40 may be included before step S30:

[0086] Step S40: When there are multiple target interaction task information items, determine the execution data dependency relationship of each target interaction task information item;

[0087] It's important to note that the AI ​​assistant can identify multiple target interaction tasks from user input data. This means the user expects the AI ​​assistant to perform multiple specific operations or functions. For example, if the user inputs "Order me a coffee and then navigate to pick it up," the assistant can identify two target interaction tasks: "Order coffee" and "Navigate to pick it up." Execution data dependency refers to whether the execution of one target interaction task depends on the execution result of another target interaction task. The AI ​​assistant can determine the execution data dependencies between multiple interaction tasks by analyzing the semantic structure of the target interaction task information, the user's historical interaction habits, and the logical relationships between the various interaction tasks.

[0088] Based on this, step S30 may include steps S301 to S302:

[0089] Step S301: For the first interactive task information and the second interactive task information that have an execution data dependency relationship, the first interactive task information is combined with the first template function instruction to obtain the first function instruction, the first application is called to respond to the first function instruction, and the intermediate data generated by the first application is obtained. Here, the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is the template function instruction in the target template function instructions that corresponds to the first interactive task information, and the first application is the application in the target application that corresponds to the first interactive task information.

[0090] When the AI ​​assistant determines that there is an execution data dependency between two interactive tasks in the target interactive task information (such as "order coffee and then navigate to pick it up"), it will process the two interactive task information in sequence according to the dependency.

[0091] Specifically, for the first interactive task information (such as "order coffee"), the AI ​​assistant combines this first interactive task information with a first template function instruction (a predefined instruction format used to call the coffee ordering application) to generate a first function instruction. Then, it calls the first application (such as Meituan) to respond to the first function instruction and execute the coffee ordering operation. After the coffee ordering operation is completed, the first application generates intermediate data (such as the coffee shop's address), which will serve as the basis for processing the second interactive task information.

[0092] Step S302: Combine the intermediate data, the second interactive task information, and the second template function instruction to obtain the second function instruction, and call the second application to respond to the second function instruction. The second template function instruction is the template function instruction in the target template function instruction that corresponds to the second interactive task information, and the second application is the application in the target application that corresponds to the second interactive task information.

[0093] After obtaining the intermediate data generated by the first application, the AI ​​assistant combines the intermediate data, the second interactive task information (such as "navigation"), and the second template function instruction (a predefined instruction format for calling the navigation application) to generate a second function instruction. Then, the AI ​​assistant calls the second application (such as Amap) to respond to the second function instruction and uses the intermediate data (the address of the coffee shop) as the destination of the navigation to execute the operation of navigating to get coffee.

[0094] In addition, when the first target application is detected to have failed to return intermediate data, a preset error message is output.

[0095] In one feasible embodiment, step S30 may further include steps S303 to S304:

[0096] Step S303: For the third interactive task information and the fourth interactive task information that do not have an execution data dependency relationship, combine the third interactive task information with the third template function instruction to obtain the third function instruction, and call the third application to respond to the third function instruction. Here, the third interactive task information and the fourth interactive task information are any two interactive task information in each target interactive task information, the third template function instruction is the template function instruction in the target template function instruction that corresponds to the third interactive task information, and the third application is the application in the target application that corresponds to the third interactive task information.

[0097] When the AI ​​assistant determines that there is no execution data dependency between two interactive tasks (such as "order coffee" and "play music") in the target interactive task information, it will process these two interactive task information separately.

[0098] Specifically, for the third interactive task information (such as "order coffee"), the AI ​​assistant combines the third interactive task information with the third template function instruction (a predefined instruction format for calling the coffee ordering application) to generate a third function instruction. Then, it calls the third application (such as Meituan) to respond to the third function instruction and execute the coffee ordering operation.

[0099] Step S304: Combine the fourth interactive task information with the fourth template function instruction to obtain the fourth function instruction, and call the fourth application to respond to the fourth function instruction. The fourth template function instruction is the template function instruction in the target template function instruction that corresponds to the fourth interactive task information, and the fourth application is the application in the target application that corresponds to the fourth interactive task information.

[0100] Similarly, for the fourth interactive task information (such as "play music"), the AI ​​assistant combines the fourth interactive task information with the fourth template function instruction (a predefined instruction format for calling the music playback application) to generate the fourth function instruction. Then, it calls the fourth application (such as Kugou Music) to respond to the fourth function instruction and execute the operation of playing music.

[0101] Therefore, depending on whether there is an execution data dependency between interactive task information, different strategies are adopted to call the target application and respond to the target function instructions. For interactive task information without execution data dependency, the corresponding application can be called in parallel to improve the interaction efficiency. For interactive task information with data dependency, it is called serially according to the dependency order, and intermediate data is used for subsequent interaction to ensure the correctness and coherence of the interaction, so that users can complete complex operations more efficiently and smoothly.

[0102] Based on the first and / or second embodiments described above, a third embodiment of the device interaction method of this application is proposed. In this embodiment, content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. In this embodiment, refer to... Figure 3 Steps A10 to A30 may also be included before step S20:

[0103] Step A10: Extract application function data of each application from the preset memory database. The application function data includes application component declaration information and associated intent filtering parameter set.

[0104] It should be noted that, in this embodiment, when a smart device installs or updates an application package file, it parses the application manifest file in the file. The application manifest file is used to declare the basic information, components and application capabilities of the application, and stores the parsed information (including at least the application component declaration information and the associated intent filtering parameter set) into a preset memory database.

[0105] For example, taking a smart device using the Android system as an example, when the system installs or updates an APK (Android application package) file, the first operation is to parse the AndroidManifest.xml (Android application manifest file) in the APK file. The system will read component tags such as activity and service in the file and store the read information in the in-memory database of PackageManagerService (PMS), that is, store the application function data of each application in the in-memory database.

[0106] The AI ​​assistant accesses an in-memory database that stores application function data for each application installed on the user's device. This application function data includes, but is not limited to, application component declaration information (such as tags and attributes of components like Activity and Service) and associated intent filtering parameter sets. These intent filtering parameter sets define the specific user intents or request types that the application can respond to, forming the foundation for interaction between application capabilities and the AI ​​assistant. By extracting this data, the AI ​​assistant can gain a comprehensive understanding of the functional scope and responsiveness of each application.

[0107] Step A20: Semantic parsing of the application function data to generate structured application function data containing function description data and interactive task information templates;

[0108] After extracting the application function data, the AI ​​assistant performs semantic parsing on this data. The purpose of semantic parsing is to transform the original application component declaration information and intent filtering parameter set into a more easily understood and processed form.

[0109] Specifically, the AI ​​assistant analyzes fields such as action, category, and data in the component tags of the application's functional data to understand the functions and intentions they represent. It then combines this information with natural language descriptions to generate structured application functional data that includes functional description data and interaction task information templates. The functional description data describes the application's functional characteristics, while the interaction task information templates are the sentence formats or examples that users may use when interacting with the application.

[0110] Step A30: After vectorizing the structured application function data, obtain the mapping relationship between each interactive task information, each application and each template function instruction, and store the mapping relationship in the knowledge base.

[0111] After semantic parsing is completed, the AI ​​assistant will vectorize the structured application function data. Vectorization involves establishing a mapping between the meaning of the capability and the structured intent. The meaning of the capability is like "Amap (app name) navigates to a certain place (actions)", and the structured intent is like "action = VIEW, data = androidamap: / / navi...". Finally, the mapping relationship between each interactive task information, each application and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base. The storage format of the mapping relationship can be a data table or other formats, so that the AI ​​assistant can perform information matching more efficiently.

[0112] For example, the data table can be in the form shown in Table 1 below:

[0113] Table 1

[0114]

[0115] The instruction template is the interactive task information, the target application is an application that can process the interactive task information, and the Action and Data parameter templates can be combined to form corresponding template function instructions.

[0116] In one feasible embodiment, the device interaction method may further include steps A40 to A60:

[0117] Step A40: Listen for system broadcast events indicating application state changes;

[0118] In this embodiment, to ensure that the AI ​​assistant can promptly perceive and respond to changes in the application's state, a corresponding listening mechanism is configured to monitor system broadcast events that indicate changes in the application's state. These system broadcast events are typically issued by the device's operating system to notify other components or services of changes in the application's state (such as installation, updates, uninstallation, etc.). By monitoring such events, the AI ​​assistant can dynamically grasp the latest state of the application.

[0119] For example, in the Android operating system, the AI ​​assistant can listen for system broadcast events related to the installation, updating, and uninstallation of applications through BroadcastReceiver, and use PackageManager.getPackageInfo() (a package information query interface) to obtain a list of all installed applications.

[0120] Step A50: When a system broadcast event for an application state change is detected, extract and parse the application function data after the application that triggered the system broadcast event from the memory database to obtain the changed structured application function data.

[0121] When a system broadcast event indicating an application state change is detected, the AI ​​assistant will locate and extract the application's modified application function data from the memory database based on the application identifier information carried in the system broadcast event. Then, it will parse this modified application function data to generate modified structured application function data containing the latest function description data and interactive task information templates.

[0122] Step A60: Update the database based on the changed structured application function data.

[0123] After obtaining the modified structured application function data, the AI ​​assistant will update the in-memory database based on this data. The update process involves overwriting or merging the modified application function data into the original records of the corresponding application in the in-memory database to ensure that the application function data stored in the knowledge base always remains consistent with the actual state of the application.

[0124] Therefore, by listening to system broadcast events for application state changes, and promptly extracting and parsing the changed application function data when an event is detected, the knowledge base is updated. This ensures that the data in the knowledge base is always synchronized with the actual functions of the application, avoiding problems such as invalid or mismatched interactive commands due to application updates or state changes, and guaranteeing the reliability and stability of the interaction.

[0125] In summary, this embodiment establishes a complete knowledge base by pre-extracting, parsing, and vectorizing application function data. Simultaneously, it monitors application state changes and updates the knowledge base accordingly, ensuring real-time synchronization between interactive operations and actual application functions to avoid errors. Therefore, after the AI ​​assistant receives user input data, it can quickly match the target application with the target template function instructions from the knowledge base. For multi-task interaction scenarios, it intelligently arranges the execution order based on data dependencies, ensuring the correctness and coherence of complex interaction logic. This improves the accuracy, efficiency, and stability of device interaction, allowing users to conveniently and fluently complete diverse operations using natural language, fully meeting user interaction needs in different scenarios, and optimizing the user-device interaction experience.

[0126] This application also provides a device interaction apparatus, please refer to... Figure 4 The device interaction device includes:

[0127] Information recognition module 10 is used to identify target interactive task information from user input data;

[0128] Information matching module 20 is used to search for target applications and target template function instructions that match the target interaction task information from a preset knowledge base. The knowledge base stores each interaction task information, each application and each template function instruction in a mapping manner.

[0129] The instruction response module 30 is used to combine the target interactive task information with the target template function instruction to obtain the target function instruction, and call the target application to respond to the target function instruction.

[0130] Optionally, the device interaction apparatus further includes a data storage module (not shown), which is used for:

[0131] Extract application function data of each application from a pre-set memory database. The application function data includes application component declaration information and associated intent filtering parameter sets.

[0132] Semantic parsing of application function data generates structured application function data containing function description data and interactive task information templates;

[0133] After vectorizing the structured application function data, we obtain the mapping relationship between each interactive task information, each application and each template function instruction, and store the mapping relationship in the knowledge base.

[0134] Optionally, the device interaction apparatus further includes a data update module (not shown), which is used for:

[0135] Listen for system broadcast events that indicate changes in application state;

[0136] When a system broadcast event for an application state change is detected, the application function data after the application that triggered the system broadcast event is extracted and parsed from the memory database to obtain the structured application function data after the change.

[0137] The knowledge base is updated based on the modified structured application function data.

[0138] Optionally, the information matching module 20 is also used for:

[0139] Identify candidate applications that can handle target interactive task information from a pre-defined knowledge base;

[0140] When there is only one candidate application, the candidate application is identified as the target application, and the target template function instruction that matches the target application and the target interaction task information is identified.

[0141] When there are multiple candidate applications, the target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and the target template function instruction that matches the target application and the target interaction task information is determined.

[0142] Optionally, the instruction response module 30 is also used for:

[0143] When there are multiple target interaction task information items, determine the execution data dependency relationship of each target interaction task information item;

[0144] For first interactive task information and second interactive task information that have execution data dependencies, the first interactive task information is combined with the first template function instruction to obtain the first function instruction, the first application is called to respond to the first function instruction, and the intermediate data generated by the first application is obtained. Here, the first interactive task information and the second interactive task information are any two interactive task information in each target interactive task information, the first template function instruction is the template function instruction in the target template function instructions that corresponds to the first interactive task information, and the first application is the application in the target application that corresponds to the first interactive task information.

[0145] The intermediate data, the second interactive task information, and the second template function instruction are combined to obtain the second function instruction, and the second application is called to respond to the second function instruction. The second template function instruction is the template function instruction in the target template function instruction that corresponds to the second interactive task information, and the second application is the application in the target application that corresponds to the second interactive task information.

[0146] Optionally, the instruction response module 30 is also used for:

[0147] For the third and fourth interactive task information that do not have an execution data dependency relationship, the third interactive task information is combined with the third template function instruction to obtain the third function instruction, and the third application is called to respond to the third function instruction. Here, the third interactive task information and the fourth interactive task information are any two interactive task information in each target interactive task information, the third template function instruction is the template function instruction in the target template function instruction that corresponds to the third interactive task information, and the third application is the application in the target application that corresponds to the third interactive task information.

[0148] The fourth interactive task information is combined with the fourth template function instruction to obtain the fourth function instruction, and the fourth application is called to respond to the fourth function instruction. The fourth template function instruction is the template function instruction in the target template function instruction that corresponds to the fourth interactive task information, and the fourth application is the application in the target application that corresponds to the fourth interactive task information.

[0149] Optionally, the information recognition module 10 is also used for:

[0150] The user input data is converted into a semantic vector, and a hybrid retrieval strategy is used to filter irrelevant information in the semantic vector to obtain a semantic matching candidate set.

[0151] Based on user profiles and historical interaction data recorded in the knowledge base, target interaction task information is determined from the semantic matching candidate set.

[0152] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the device interaction method described above.

[0153] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0154] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a DSP processor), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a microphone, an accelerometer, etc.; an output device 1008 including, for example, a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 allows the electronic device to exchange data wirelessly or via wired communication with other devices. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. It can be implemented alternatively or with more or fewer systems.

[0155] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0156] Compared with the prior art, the beneficial effects of the electronic device provided in this application embodiment are the same as the beneficial effects of the device interaction method provided in the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0157] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0158] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0159] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the device interaction method described in the above embodiments.

[0160] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0161] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0162] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0163] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0166] The readable storage medium provided in this application embodiment is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-described device interaction method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the device interaction method provided in the above-described embodiments, and will not be repeated here.

[0167] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the device interaction method described above.

[0168] Compared with the prior art, the beneficial effects of the computer program product provided in this application embodiment are the same as the beneficial effects of the device interaction method provided in the above embodiments, and will not be repeated here.

[0169] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A device interaction method, characterized in that, The device interaction method includes: Identify the target interaction task information from user input data; The target application and target template function instructions that match the target interaction task information are searched from a preset knowledge base, wherein each interaction task information, each application and each template function instruction are mapped and stored in the knowledge base; The target interactive task information is combined with the target template function instruction to obtain the target function instruction, and the target application is invoked to respond to the target function instruction. Before the step of searching the preset knowledge base for the target application and target template function instructions that match the target interaction task information, the method further includes: The application function data of each application is extracted from a preset memory database, wherein the application function data includes application component declaration information and associated intent filtering parameter set; The application function data is semantically parsed to generate structured application function data containing function description data and interactive task information templates; After vectorizing the structured application function data, the mapping relationship between each interactive task information, each application and each template function instruction is obtained, and the mapping relationship is stored in the knowledge base.

2. The device interaction method as described in claim 1, characterized in that, The method further includes: Listen for system broadcast events that indicate changes in application state; When a system broadcast event for an application state change is detected, the application function data of the application that triggered the system broadcast event is extracted and parsed from the memory database to obtain the changed structured application function data. The knowledge base is updated based on the modified structured application function data.

3. The device interaction method as described in claim 1, characterized in that, The step of searching for target applications and target template function instructions that match the target interaction task information from a preset knowledge base includes: Select alternative applications capable of processing the target interactive task information from a preset knowledge base; When there is only one candidate application, the candidate application is determined as the target application, and a target template function instruction that matches the target application and the target interaction task information is determined. When there are multiple candidate applications, the target application is determined from the candidate applications based on the historical interaction data recorded in the knowledge base, and the target template function instruction that matches the target application and the target interaction task information is determined.

4. The device interaction method as described in claim 1, characterized in that, Before the step of combining the target interactive task information with the target template function instruction to obtain the target function instruction, and calling the target application to respond to the target function instruction, the method further includes: When there are multiple target interaction task information items, determine the execution data dependency relationship of each target interaction task information item; The step of combining the target interactive task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction, includes: For first and second interactive task information that have execution data dependencies, the first interactive task information is combined with a first template function instruction to obtain a first function instruction. The first application is called to respond to the first function instruction and obtain intermediate data generated by the first application. Here, the first interactive task information and the second interactive task information are any two interactive task information in each of the target interactive task information. The first template function instruction is the template function instruction in the target template function instructions that corresponds to the first interactive task information. The first application is the application in the target application that corresponds to the first interactive task information. The intermediate data, the second interactive task information, and the second template function instruction are combined to obtain a second function instruction, and a second application is invoked to respond to the second function instruction. The second template function instruction is the template function instruction in the target template function instruction that corresponds to the second interactive task information, and the second application is the application in the target application that corresponds to the second interactive task information.

5. The device interaction method as described in claim 4, characterized in that, The step of combining the target interactive task information with the target template function instruction to obtain a target function instruction, and calling the target application to respond to the target function instruction, further includes: For third and fourth interactive task information that do not have execution data dependencies, the third interactive task information is combined with a third template function instruction to obtain a third function instruction, and a third application is invoked to respond to the third function instruction. Here, the third interactive task information and the fourth interactive task information are any two interactive task information from each of the target interactive task information, the third template function instruction is the template function instruction corresponding to the third interactive task information in the target template function instructions, and the third application is the application in the target application corresponding to the third interactive task information. The fourth interactive task information is combined with the fourth template function instruction to obtain the fourth function instruction, and the fourth application is invoked to respond to the fourth function instruction. The fourth template function instruction is the template function instruction in the target template function instruction that corresponds to the fourth interactive task information, and the fourth application is the application in the target application that corresponds to the fourth interactive task information.

6. The device interaction method as described in claim 1, characterized in that, The step of identifying target interactive task information from user input data includes: User input data is converted into semantic vectors, and a hybrid retrieval strategy is used to filter irrelevant information in the semantic vectors to obtain a semantic matching candidate set; Based on user profiles and historical interaction data recorded in the knowledge base, target interaction task information is determined from the semantic matching candidate set.

7. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the device interaction method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the device interaction method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the device interaction method as described in any one of claims 1 to 6.

Images