AI control method and device through generative dialogue, terminal and medium

Through the generative dialogue AI method, AI applications identify user needs and generate operation instructions, simulate user operations, solving the limitations of artificial intelligence control across platforms and cross-applications, and improving the convenience of user experience and information security.

CN119025631APending Publication Date: 2024-11-26胡凝
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310609675.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing technology has limitations in realizing cross-platform and cross-application artificial intelligence control, resulting in inconvenient user experience.

Method used

Through the generative dialogue AI method, AI applications communicate with users, identify user needs, analyze applications within smart devices, generate and execute operation instructions, simulate user operations, and achieve interaction with target applications, solve cross-platform and cross-platform The limitations of applied artificial intelligence control.

Benefits of technology

Without manual operation by users, AI applications can be used to control various applications on smart devices, improving the convenience of user experience, achieving efficient and flexible intelligent device control, and ensuring user information security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025631A_ABST
    Figure CN119025631A_ABST
Patent Text Reader

Abstract

The invention provides a generative dialogue AI control method and device, a terminal and a medium. The method comprises the following steps: an AI application performs dialogue communication with a user and identifies a user demand; the AI application analyzes N application programs in the intelligent equipment according to user requirements, and determines a target application and an implementation way; the AI application generates an operation instruction for the target application according to the user demand and the implementation way; and the AI application executes the operation instruction on the intelligent equipment, simulates user operation to realize interaction with the target application, and feeds back an execution result to a user. According to the invention, the AI application installed on the intelligent device carries out dialogue communication with the user, identifies the user demand, analyzes and determines the target application and the implementation way, generates and executes the operation instruction for the target application, and simulates the user operation so as to realize interaction with the target application, manual operation of the user is not needed, and the user experience is improved. The problem of limitation of cross-platform and cross-application artificial intelligence control is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent device technology, and in particular to a method, device, terminal and medium for controlling AI through generative dialogue. Background Art

[0002] With the popularity of smart devices and the rapid development of artificial intelligence technology, more and more users want to interact with these devices in a simple and intuitive way. The traditional manual operation method limits the convenience of user experience to a certain extent. Therefore, it is becoming increasingly important to use artificial intelligence technology to control smart devices and interact with users. However, existing technologies still have limitations in realizing cross-platform and cross-application artificial intelligence control.

[0003] Therefore, it is necessary to propose a method, device, terminal and medium for solving the above problems through generative dialogue artificial intelligence (AI) control. Summary of the invention

[0004] The purpose of the present invention is to provide a generative dialogue AI control method, device, terminal and medium to improve the existing problems of inconvenient manual operation and limitations of cross-platform and cross-application artificial intelligence control.

[0005] In a first aspect, the present invention provides a method for controlling AI through generative dialogue, which is applied to AI applications, and the method includes: the AI ​​application communicates with a user through dialogue and identifies user needs; the AI ​​application analyzes N application programs in the smart device according to the user needs, determines a target application and an implementation path, wherein N is a positive integer; the AI ​​application generates operation instructions for the target application according to the user needs and the implementation path; the AI ​​application executes the operation instructions on the smart device, simulates user operations, so as to achieve interaction with the target application, and feeds back the execution results to the user.

[0006] The beneficial effects of the method provided by the present invention are: through the AI ​​application installed on the smart device to communicate with the user and identify user needs, analyze and determine the target application and implementation path, generate and execute operation instructions for the target application, simulate user operations to achieve interaction with the target application, without the need for manual operation by the user, the AI ​​application can be used to control various application programs on the smart device, thus solving the problem of limitations of cross-platform and cross-application artificial intelligence control.

[0007] In a possible embodiment, the AI ​​application feeds back the feedback information of the target application to the user, modifies the operation instruction according to the feedback information of the target application and the user instruction, and executes the modified operation instruction on the smart device.

[0008] In a possible embodiment, before the AI ​​application communicates with the user and identifies the user's needs, the method further includes:

[0009] Collect usage data of applications on various smart devices, including but not limited to user operation records, user application operation records and application feedback records;

[0010] Building a learning model using deep learning and reinforcement learning methods, and simulating the interaction process between the user and the application of the smart device based on the usage data to achieve training and learning of the learning model;

[0011] The AI ​​application analyzes the N application programs in the smart device according to the user needs and determines the target application and implementation path, including: inputting the user needs into the learning model to obtain the target application and implementation path.

[0012] In a possible embodiment, the AI ​​application analyzes N application programs in the smart device according to the user needs, and determines a target application and an implementation approach that can meet the user needs, including:

[0013] The AI ​​application analyzes the N application programs in the smart device according to the user needs to obtain M optional applications that can meet the user needs, and determines the target application and implementation path from the M optional applications according to the user's operating habits, where M is a positive integer greater than N.

[0014] In a possible embodiment, the learning model is updated by online learning and transfer learning techniques.

[0015] In a possible embodiment, after the AI ​​application communicates with the user and identifies the user's needs, it also includes: the AI ​​application uses deep learning and natural language processing technology to identify and learn the user's voice features.

[0016] In a possible embodiment, during the interaction with the target application, operations involving sensitive information prompt the user to operate, wherein the sensitive information includes but is not limited to payment, password input, fingerprint, and face scanning.

[0017] In a second aspect, the present invention further provides a device for controlling a generative dialogue AI, the device comprising a module / unit for executing any possible design method of the first aspect. These modules / units can be implemented by hardware, or by executing corresponding software implementations by hardware.

[0018] In a third aspect, an embodiment of the present invention provides a terminal, including a processor and a memory, wherein the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the terminal can implement any possible design method of the first aspect.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, the computer-readable storage medium comprising a computer program, and when the computer program runs on a terminal, the terminal executes any possible design method of the first aspect above.

[0020] In a fifth aspect, an embodiment of the present invention further provides a method comprising a computer program product, which, when executed on a terminal, enables the terminal to execute any possible design of any of the above aspects.

[0021] For the beneficial effects of the second to fifth aspects, reference may be made to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the generative dialogue AI control method of the present invention.

[0023] Figure 2 It is a schematic diagram of the present invention's control device through generative dialogue AI.

[0024] Figure 3 It is a schematic diagram of the structure of the terminal of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0026] In view of the problems existing in the prior art, an embodiment of the present invention provides a generative dialogue AI control method, which is applied to artificial intelligence (AI) applications. Figure 1 This is a flow chart of the generative dialogue AI control method of the present invention, see Figure 1 , the method comprising:

[0027] S101: The AI ​​application communicates with the user and identifies user needs.

[0028] Specifically, the AI ​​application communicates directly with the user, analyzes the user's voice, and identifies the user's needs.

[0029] S102: The AI ​​application analyzes N application programs in the smart device according to the user needs, and determines a target application and an implementation approach, where N is a positive integer.

[0030] Specifically, the implementation approach includes the required operation process, pages, triggering events, and information in pages and pictures that is effective to users.

[0031] S103: The AI ​​application generates an operation instruction for the target application according to the user demand and the implementation approach.

[0032] Preferably, the operation instruction is a command description of a series of operation instruction sequences. The AI ​​application generates a command description of a series of operation instruction sequences according to the user requirements, the implementation approach and the analysis of the target application to achieve the user requirements.

[0033] S104: The AI ​​application executes the operation instruction on the smart device, simulates the user operation to achieve interaction with the target application, and feeds back the execution result to the user.

[0034] Specifically, the AI ​​application simulates command execution on the smart device according to the command description, that is, the AI ​​application executes the command description on the smart device, simulates user operations, and uses screen output as feedback to achieve interaction between the AI ​​application and the target application until an execution result is achieved.

[0035] In a preferred embodiment, the AI ​​application feeds back the feedback information of the target application to the user, modifies the operation instruction according to the feedback information of the target application and the user instruction, and executes the modified operation instruction on the smart device. In this embodiment, during the interaction process, the AI ​​application modifies the operation instruction according to the feedback information of the target application and the user's instruction to better meet the user's needs. For example, the user's demand is to book a flight from location C to location D tomorrow. The AI ​​application obtains feedback information of four flights that meet the user's demand according to the user's demand, and feeds back the feedback information of the four flights to the user. The user replies to select one of the flights. The AI ​​application modifies the operation instruction according to the user's reply and the information of the selected flight, and completes the ticket reservation for the selected flight.

[0036] In a preferred embodiment, before the AI ​​application communicates with the user and identifies the user's needs, it also includes:

[0037] Collect usage data of applications on various smart devices, where the usage data includes but is not limited to user operation records, user application operation records and application feedback records, where the user operation records are user inputs on smart devices, that is, the mapping relationship between user operations and instructions, and the user operation records include but are not limited to click events, text and voice inputs, and the user application operation records are user inputs of smart devices to applications, using the above-mentioned input instructions, and the application feedback records are feedback from smart devices in response to user inputs, and the application feedback records include but are not limited to screen outputs of pictures and texts obtained in the form of screenshots, and voice outputs.

[0038] Feature extraction is performed on the usage data. Specifically, image features are extracted from the screenshots using computer vision technology. The image features include but are not limited to interface elements, text content, and color. Key information is extracted from text and voice input information using natural language processing technology.

[0039] A learning model is constructed using deep learning and reinforcement learning methods, and the interaction process between the user and the application of the smart device is simulated based on the usage data to achieve training and learning of the learning model, so that the learning model learns how to perform specific tasks in specific scenarios. For example, the learning model can learn how to perform search operations or navigate to specific pages in different applications. Specifically, the training and learning of the learning model includes: by identifying the text content in the image and the area that responds to the event, analyzing and understanding the meaning of the text content and the consequences of the regional event triggering, so as to obtain learning results. The reinforcement learning method is used to enable the learning model to understand user needs by simulating the interaction process between the user and the application of the smart device. The learning model tries different operation sequences, observes the feedback results, and continuously optimizes the strategy to achieve the best performance.

[0040] The AI ​​application analyzes the N applications in the smart device according to the user needs and determines the target application and implementation path, including: inputting the user needs into the learning model to obtain the target application and implementation path. Specifically, the learning model obtains the target application and implementation path according to the operating habits obtained during training.

[0041] In this embodiment, the AI ​​application can intelligently learn how to use the application of the smart device to provide users with convenient and efficient services in actual scenarios.

[0042] In a specific embodiment, the AI ​​application automatically learns the user's operating habits and preferences by analyzing the user's operation records, behavioral data, and applications in the smart device.

[0043] In a preferred embodiment, the AI ​​application analyzes N applications in the smart device according to the user needs, and determines a target application and implementation path that can meet the user needs, including:

[0044] The AI ​​application analyzes the N application programs in the smart device according to the user needs to obtain M optional applications that can meet the user needs, and determines the target application and implementation path from the M optional applications according to the user's operating habits, where M is a positive integer greater than N.

[0045] Furthermore, when the number of applications obtained according to the operating habits is greater than or equal to two, a target application and an implementation approach of the target application are selected and determined according to a user instruction.

[0046] In a preferred embodiment, the learning model is updated through online learning and transfer learning technology. Specifically, the learning model is deployed to an actual smart device so that the learning model can continuously learn and optimize during actual use through online learning and transfer learning technology to better adapt to new applications. Online learning technology allows the learning model to continuously update the learning model based on newly collected data, while transfer learning technology allows the learning model to apply knowledge learned in one application to other similar applications.

[0047] In a specific embodiment, after the AI ​​application communicates with the user and identifies the user's needs, it also includes: the AI ​​application uses deep learning and natural language processing technology to identify and learn the user's voice features to achieve more accurate voice recognition and understanding.

[0048] In a preferred embodiment, during the interaction with the target application, operations involving sensitive information prompt the user to operate, wherein the sensitive information includes but is not limited to payment, password input, fingerprint, and face recognition. Specifically, when encountering privacy and high security related behaviors, stop and change to user operation.

[0049] In a specific embodiment, by setting learning restrictions, when the learning model encounters an operation related to sensitive information, the learning model will not be trained and learned to ensure user safety. Specifically, by setting special operation commands, when the learning model encounters a specific operation or a specific keyword, the learning model will not learn the operation.

[0050] In some possible embodiments, the feedback method of the execution result includes but is not limited to presentation through the interface of the target application, the AI ​​application describing the interface of the target application, or the AI ​​application replying to the feedback of the target application through sound.

[0051] In a preferred embodiment, the AI ​​application shares the operation data of the application online. The information shared online includes commands and operation instructions, but does not include the content of the smart device, so as to protect user privacy and assist users to better operate new applications on smart devices under the premise of ensuring user safety. The new application generates the next instruction based on the content feedback of the local machine during the operation according to the instructions. Specifically, after the learning model has learned the operation of a certain application and desensitized the operation habits of the application, the operation model of the desensitized application is shared online with other users.

[0052] In a specific embodiment, when designing a text command format of an instruction sequence, it is necessary to consider various operation events supported by the smart device to ensure that the format is universal and extensible. The following is an example text command format:

[0053] [{

[0054] "event":"Operation type",

[0055] "target":"Target element (optional)",

[0056] "position":{

[0057] "x":"X-axis coordinate (optional)",

[0058] "y":"Y-axis coordinate (optional)"

[0059] },

[0060] "duration":"Duration (optional)",

[0061] "value":"Input value (optional)",

[0062] "delay":"Delay time (optional)"

[0063] },...]

[0064] Specifically, the types of the operation instructions include the following events:

[0065] Operation Type Operational Events tap Click Actions doubleTap Double-click operation longPress Long press operation swipe Slide Operation zoomIn Zoom in operation zoomOut Zoom out operation rotate Rotation Operation textInput Text input operations back Return Operation home Return to the main screen launchApp Open application operation closeApp Close application operation

[0066] Here are some sample commands: Click Action Example: {

[0067] "event":"tap",

[0068] "target":"#loginButton",

[0069] "delay":500

[0070] }

[0071] Slide operation example:

[0072] {

[0073] "event":"swipe",

[0074] "position":{

[0075] "x":300,

[0076] "y":500

[0077] },

[0078] "duration":1000

[0079] }

[0080] Text input operation example:

[0081] {

[0082] "event":"textInput",

[0083] "target":"#searchBar",

[0084] "value":"AI control smart devices",

[0085] "delay":1000

[0086] }

[0087] This text command format can be used to conveniently describe operation events supported by various smart devices, and enable AI applications to control smart devices based on these descriptions.

[0088] Specifically, the smart devices include but are not limited to smart phones, car computers, tablet computers, handheld computers, and wearable devices.

[0089] The specific process of the generative dialogue AI control method of the present invention is described below in conjunction with specific embodiments.

[0090] An AI application APP named "Smart Butler" is installed on the user's smart device. The user opens the AI ​​application APP by clicking or voice triggering. The user issues a command saying "What is the weather today?" The AI ​​application recognizes the user's voice information and determines that the user's need is "check today's weather". According to the user's need, the AI ​​application locates and searches for the APP on the smart device, and determines that the target application is "Moji Weather APP" and the specific way to obtain today's weather conditions on the "Moji Weather APP" based on the user's operating habits. The AI ​​application generates a command description of a series of operation instruction sequences for the "Moji Weather APP". The AI ​​application executes the operation instructions on the smart device, opens the "Moji Weather APP", and obtains today's weather data, for example, today's weather is cloudy, overcast, the temperature is 18°, the east wind is level 1, the humidity is 57%, and clothing suggestions. The AI ​​application feeds back the acquired weather data for today to the user in the form of voice broadcast.

[0091] The present invention proposes a technical solution for managing various applications in smart devices in a generative dialogue manner using an AI application through a generative dialogue method. The AI ​​application dialogues with the user and identifies user needs, and inputs the user needs into the learning model. The learning model uses the operating habits obtained during training and the content of the target application performance of the smart device as feedback to interact with the target application of the smart device, thereby realizing control of the smart device through the AI ​​application without the need for manual operation by the user, and can realize cross-platform and cross-application artificial intelligence control, thereby realizing efficient and flexible control of smart devices, improving the convenience of user experience, and ensuring the user's information security.

[0092] In addition, the present invention also provides a generative dialogue AI control device, Figure 2 This is a schematic diagram of the generative dialogue AI control device of the present invention, see Figure 2 The device includes: a user demand identification unit 201, which is used to communicate with the user and identify the user's needs; a speech recognition unit 202, which is used to perform speech recognition on the speech information to obtain user needs; an application and path determination unit 203, which is used to analyze N applications in the smart device according to the user needs, and determine the target application and the implementation path, where N is a positive integer; an operation instruction generation unit 204, which is used to generate an operation instruction for the target application according to the user needs and the implementation path; an interaction unit 205, which is used to execute the operation instruction on the smart device, simulate user operations, so as to achieve interaction with the target application; a feedback unit 206, which is used to feed back the execution results to the user. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0093] In some other embodiments of the present invention, the embodiments of the present invention disclose a terminal, Figure 3 is a schematic diagram of the structure of the terminal of the present invention, see Figure 3 , the terminal may include: one or more processors 301; memory 302; display 303; one or more applications (not shown); and one or more computer programs 304, and the above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the above memory 302 and configured to be executed by the one or more processors 301, and the one or more computer programs 304 include instructions, which may be used to execute the following: Figure 1 and Figure 2 And each step in the corresponding embodiment.

[0094] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0095] Each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program codes.

[0097] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A generative dialogue AI control method, applied to AI applications, characterized in that: The method comprises: The AI ​​application communicates with the user and identifies the user's needs; The AI ​​application analyzes N application programs in the smart device according to the user needs, and determines a target application and an implementation approach, where N is a positive integer; The AI ​​application generates an operation instruction for the target application according to the user demand and the implementation approach; The AI ​​application executes the operation instruction on the smart device, simulates the user operation to achieve interaction with the target application, and feeds back the execution result to the user.

2. The method according to claim 1, characterized in that During the interaction process, the AI ​​application feeds back the feedback information of the target application to the user, modifies the operation instruction according to the feedback information of the target application and the user instruction, and executes the modified operation instruction on the smart device.

3. The method according to claim 1, characterized in that Before the AI ​​application communicates with the user and identifies the user's needs, it also includes: Collect usage data of applications on various smart devices, including but not limited to user operation records, user application operation records and application feedback records; Building a learning model using deep learning and reinforcement learning methods, and simulating the interaction process between the user and the application of the smart device based on the usage data to achieve training and learning of the learning model; The AI ​​application analyzes the N application programs in the smart device according to the user requirements to determine a target application and an implementation path, including: inputting the user requirements into the learning model to obtain a target application and an implementation path; The AI ​​application analyzes the N application programs in the smart device according to the user needs, and determines a target application and an implementation approach that can meet the user needs, including: The AI ​​application analyzes the N application programs in the smart device according to the user needs to obtain M optional applications that can meet the user needs, and determines the target application and implementation path from the M optional applications according to the user's operating habits, where M is a positive integer greater than N.

4. The method according to claim 1, characterized in that During the interaction with the target application, operations involving sensitive information prompt the user to operate, wherein the sensitive information includes but is not limited to payment, password input, fingerprint, and face recognition.

5. A device controlled by generative dialogue AI, characterized in that: The device comprises: A user demand identification unit, used to communicate with users and identify user needs; A speech recognition unit, used for performing speech recognition on the speech information to obtain user needs; An application and path determination unit, configured to analyze N application programs in the smart device according to the user's needs, and determine a target application and an implementation path, wherein N is a positive integer; An operation instruction generating unit, used for generating an operation instruction for the target application according to the user demand and the implementation approach; An interaction unit, configured to execute the operation instruction on the smart device and simulate user operation to achieve interaction with the target application; The feedback unit is used to feed back the execution result to the user.

6. The device according to claim 5, characterized in that The device also includes an instruction correction unit; The feedback unit is further used to feed back the feedback information of the target application to the user during the interaction process; The instruction correction unit is used to correct the operation instruction according to the feedback information of the target application and the user instruction; The interaction unit is further used to obtain the modified operation instruction during the interaction process, and execute the modified operation instruction on the smart device.

7. The device according to claim 5, characterized in that The device also includes: A data collection unit, used to collect usage data of applications of various smart devices, wherein the usage data includes but is not limited to user operation records, user application operation records and application feedback records; A learning model building unit, used to build a learning model using deep learning and reinforcement learning methods; A training and learning unit, used to simulate the interaction process between the user and the application of the smart device based on the usage data, so as to realize the training and learning of the learning model; The application and approach determination unit analyzes the N applications in the smart device according to the user needs to determine the target application and the implementation approach, and is specifically used to: input the user needs into the learning model to obtain the target application and the implementation approach; the application and approach determination unit analyzes the N applications in the smart device according to the user needs to determine the target application and the implementation approach, and is specifically used to: analyze the N applications in the smart device according to the user needs to obtain M optional applications that can meet the user needs, and determine the target application and the implementation approach from the M optional applications according to the user's operating habits, wherein M is a positive integer greater than N.

8. The device according to claim 5, characterized in that The device also includes: The sensitive operation prompt unit is used to prompt the user to perform operations involving sensitive information, wherein the sensitive information includes but is not limited to payment, password input, fingerprint, and face scanning.

9. A terminal, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program; The processor is configured to execute the computer program stored in the memory, so that the terminal executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.