Method, device, equipment and readable storage medium for executing automated processes

Abstract

The present invention discloses a method, device, equipment and readable storage medium for executing an automated process, and relates to the field of financial technology. The method includes the steps of: obtaining multimodal data corresponding to a terminal, using the multimodal data as input to a deep learning model to obtain an intent analysis model; determining the behavioral intention corresponding to the automated process in the terminal through the intent analysis model; and executing the target operation instruction corresponding to the automated process according to the behavioral intention to execute the automated process. The present invention realizes analyzing the behavioral intention corresponding to the automated process through the intent analysis model, executing the automated process according to the behavioral intention, and avoiding the occurrence of a situation where the automated process fails to execute due to changes in the execution environment corresponding to the automated process, thereby improving the adaptability of the automated process to different execution environments, that is, improving the adaptability of the automated process, and improving the execution success rate of the automated process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology of financial technology (Fintech), and in particular to a method, device, equipment and readable storage medium for executing an automated process. Background Art

[0002] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech), and data processing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] In the current process automation scenario, the automated process can only be executed mechanically according to the pre-set instructions. If the automated process is to be executed successfully, it is necessary to ensure that the execution environment of the automated process is completely consistent with the set environment. For example, in the RPA (Robotic Process Automation) automation process, the corresponding RPA robot can only execute according to the pre-set instruction sequence, but the actual execution environment of the RPA robot to execute the automated process will always be different from the preset execution environment. That is, there will be some dynamic factors in the actual execution environment that cause the RPA robot to fail to successfully execute the automated process, or cause the automated process to fail to execute according to the normal process. For example, there may be dynamic factors such as pop-up ads and automatic updates of background programs in actual scenarios.

[0004] It can be seen from this that in the current process automation scenarios, the applicability and success rate of automated processes are poor. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and readable storage medium for executing an automated process, aiming to solve the technical problems of poor applicability and low success rate of automated processes in existing process automation scenarios.

[0006] To achieve the above object, the present invention provides a method for executing an automated process, the method comprising the steps of:

[0007] Acquire multimodal data corresponding to the terminal, and use the multimodal data as input to a deep learning model to obtain an intent analysis model;

[0008] Determine the behavior intention corresponding to the automation process in the terminal through the intention analysis model;

[0009] Execute the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process.

[0010] Preferably, after the step of acquiring multimodal data corresponding to the terminal and using the multimodal data as input of the deep learning model to obtain the intention analysis model, the method further includes:

[0011] Acquire local usage data of a user corresponding to the terminal, perform federated learning based on the local usage data, and update the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model;

[0012] The step of determining the behavior intention corresponding to the automation process in the terminal through the intention analysis model includes:

[0013] The behavioral intention corresponding to the automated process in the terminal is determined through the updated intent analysis model.

[0014] Preferably, the step of acquiring local usage data of a user corresponding to the terminal, performing federated learning based on the local usage data, updating the intent analysis model according to the learning result of the federated learning, and obtaining the updated intent analysis model comprises:

[0015] Acquire local usage data of a user corresponding to the terminal, perform federated learning based on the local usage data, and obtain a gradient value obtained by the federated learning;

[0016] The model parameters in the intent analysis model are updated according to the gradient value to obtain an updated intent analysis model.

[0017] Preferably, after the step of acquiring multimodal data corresponding to the terminal and using the multimodal data as input of the deep learning model to obtain the intention analysis model, the method further includes:

[0018] Acquire interaction data between the terminal and the server, perform federated learning based on the interaction data, and update the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model;

[0019] The step of determining the behavior intention corresponding to the automation process in the terminal through the intention analysis model includes:

[0020] The behavioral intention corresponding to the automated process in the terminal is determined through the updated intent analysis model.

[0021] Preferably, after the step of determining the behavior intention corresponding to the automation process in the terminal by using the intention analysis model, the method further includes:

[0022] Detecting whether a non-process factor not belonging to the automated process is detected during the execution of the automated process in the terminal;

[0023] If the non-process factor is detected, the step of executing the target operation instruction corresponding to the automation process according to the behavioral intention to execute the automation process includes:

[0024] Execute the target operation instruction corresponding to the automation process according to the behavior intention and the non-process factors to execute the automation process.

[0025] Preferably, the step of acquiring multimodal data corresponding to the terminal and using the multimodal data as input of a deep learning model to obtain an intention analysis model includes:

[0026] Obtain multimodal data corresponding to the terminal, and obtain model sample data corresponding to the terminal;

[0027] The multimodal data and the model sample data are used as inputs of a deep learning model to obtain an intent analysis model.

[0028] Preferably, after the step of executing the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process, the method further includes:

[0029] Detecting the execution success rate of the automated process;

[0030] If the execution success rate is lower than the preset success rate, the intention analysis model is updated.

[0031] In addition, to achieve the above-mentioned purpose, the present invention further provides an execution device of an automated process, the execution device of the automated process comprising:

[0032] An acquisition module, used to acquire multimodal data corresponding to the terminal, and use the multimodal data as input of the deep learning model to obtain an intent analysis model;

[0033] A determination module, used to determine the behavior intention corresponding to the automation process in the terminal through the intention analysis model;

[0034] An execution module is used to execute the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides an execution device for an automated process, wherein the execution device for the automated process includes a memory, a processor, and an execution program for the automated process stored in the memory and executable on the processor, wherein when the execution program for the automated process is executed by the processor, the steps of the execution method for the automated process corresponding to the federated learning server are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an execution program of an automation process is stored. When the execution program of the automation process is executed by a processor, the steps of the execution method of the automation process as described above are implemented.

[0037] The present invention obtains an intention analysis model by using the multimodal data corresponding to the terminal as the input of the deep learning model, determines the behavioral intention corresponding to the automation process in the terminal through the intention analysis model, and executes the target operation instruction corresponding to the automation process according to the behavioral intention to execute the automation process. The intention analysis model is used to analyze the behavioral intention corresponding to the automation process, and the automation process is executed according to the behavioral intention, so as to avoid the situation where the automation process fails to execute due to changes in the execution environment corresponding to the automation process, thereby improving the adaptability of the automation process to different execution environments, that is, improving the adaptability of the automation process, and improving the execution success rate of the automation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a first embodiment of a method for executing an automated process of the present invention;

[0039] Figure 2 is a flow chart of a second embodiment of the method for executing the automated process of the present invention;

[0040] Figure 3 is a flow chart of a third embodiment of the method for executing an automated process of the present invention;

[0041] Figure 4 It is a functional schematic diagram module diagram of a preferred embodiment of the execution device of the automated process of the present invention;

[0042] Figure 5 It is a structural diagram of the hardware operating environment involved in the embodiment of the present invention.

[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0045] The present invention provides a method for executing an automated process, referring to Figure 1 , Figure 1 FIG. 1 is a flow chart of a first embodiment of a method for executing an automated process according to the present invention.

[0046] The embodiments of the present invention provide embodiments of the method for executing an automated process. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0047] The execution method of the automated process is applied to a server or terminal, and the terminal may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, and fixed terminals such as digital TVs and desktop computers. In various embodiments of the execution method of the automated process, for ease of description, the execution subject is omitted to illustrate various embodiments. The execution method of the automated process includes:

[0048] Step S10, obtaining multimodal data corresponding to the terminal, and using the multimodal data as input of a deep learning model to obtain an intent analysis model.

[0049] When an acquisition instruction for acquiring multimodal data corresponding to a terminal is detected, the multimodal data corresponding to the terminal is acquired according to the acquisition instruction. Among them, the acquisition instruction can be set by the user as needed, or it can be triggered by a pre-set timed task. In this embodiment, the multimodal data of one terminal can be acquired, and the multimodal data corresponding to multiple terminals can also be acquired. The multimodal data includes but is not limited to the operation data corresponding to the input operation of inputting text, the mouse clicking operation of clicking the mouse, and the mouse moving operation of moving the mouse triggered by the terminal user in the terminal. Through the multimodal data, it can be determined what function the user's operation instruction is specifically intended to achieve, that is, to determine the behavioral intention corresponding to the operation instruction. In this embodiment, there is no limit on how many terminals correspond to the multimodal data to be acquired, nor is there a limit on the acquisition time of the multimodal data. It is also possible to acquire the multimodal data of the terminal in the last 10 days, or to acquire the multimodal data generated by the terminal in the last 15 days.

[0050] After obtaining the multimodal data, the multimodal data is used as the input of the preset deep learning model, that is, the multimodal data is used as the training sample data of the deep learning model, and the deep learning model is trained by the training sample data to train the intent analysis model. Among them, the deep learning model includes but is not limited to DBN (Deep Belief Network), CNN (Convolution Neural Networks), RNN (Recurrent Neural Network). In the process of obtaining the intent analysis model, the specific deep learning model to be used can be pre-set by the user according to needs. It can be understood that by using multimodal data as training sample data for the deep learning model, the obtained user intent analysis can analyze the user behavior intention of the operation instructions (input operation, mouse click operation and mouse movement operation, etc.) of each terminal.

[0051] Furthermore, the step S10 includes:

[0052] Step a, obtaining multimodal data corresponding to the terminal, and obtaining model sample data corresponding to the terminal.

[0053] Further, in order to improve the accuracy of the obtained intention analysis model in analyzing the behavior intention, after obtaining the multimodal data corresponding to the terminal, the model sample data corresponding to the terminal is obtained. Among them, the model sample data can be the screen capture data corresponding to the terminal, and the screen capture data can be captured by each terminal user in the operation terminal through the screen capture program pre-set in the background. In the screen capture process, the screen capture frequency can be set according to specific needs, such as being set to obtain 1 frame of screen capture image per second, or being set to obtain 1 frame of screen capture image every 2 seconds. The size of the screen capture frequency is not limited in this embodiment. It can be understood that the screen capture data includes at least two screen capture images. If the screen capture data corresponding to multiple terminals is obtained, each screen capture image in the screen capture data can carry the terminal identification and screen capture time of the corresponding terminal. The terminal identification can be used to determine which terminal the screen capture image belongs to, and the screen capture time can be used to determine the generation time of the screen capture image. Further, the model sample data can also be the screen data captured by the camera of the terminal during the operation of the terminal by each terminal user.

[0054] Step b: using the multimodal data and the model sample data as input of a deep learning model to obtain an intent analysis model.

[0055] After obtaining the model sample data and multimodal data, the multimodal data and model sample data are used as the input of the deep learning model, that is, the multimodal data and model sample data are used as training sample data to train the deep learning model to obtain the intent analysis model. It can be understood that the changes in the corresponding terminal screen interface can be recorded through the model sample data. Therefore, the intent analysis model obtained by combining the multimodal data and model sample data training will have a higher accuracy when analyzing the behavioral intention of the operation instruction. It can be understood that when analyzing the user behavior intention corresponding to the operation instruction through the intent analysis model, it is only necessary to use the operation instruction as the input of the intent analysis model. At this time, the output of the intent analysis model is the behavioral intention corresponding to the operation instruction.

[0056] Step S20: determining the behavior intention corresponding to the automation process in the terminal through the intention analysis model.

[0057] After the intention analysis model is obtained, the behavioral intention corresponding to the automated process in the terminal is determined through the intention analysis model. Among them, the automated process is a workflow pre-set in the terminal and automatically executed by the terminal. In the automated process, it is composed of a series of operation instructions. The number of operations of the operation instructions in each automated process can be equal or unequal. The behavioral intention is the function to be achieved by the automated process, such as opening a web page or realizing a search function. For example, an automated process includes three operation instructions, which are to execute operation instruction A first, then execute operation instruction B, and finally execute operation instruction C. The target operation instructions are the various operation instructions pre-set in the automated process.

[0058] Step S30: executing the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process.

[0059] After determining the behavioral intention of the target operation instruction corresponding to the automation process, the target operation instruction corresponding to the automation process is executed according to the determined behavioral intention to execute the automation process. It can be understood that the behavioral intention corresponding to the automation process can be identified through the intention analysis model, which avoids the situation that when executing the automation process, the operation instruction originally set is directly executed without considering the behavioral intention corresponding to the automation process, so that in the process of executing the automation process, if there are factors that were not considered in the process of setting the automation process, the automation process execution fails or the execution error occurs. For example, when a certain automation process is to open a certain web page, at this time, the corresponding operation instruction is an open instruction. If an advertisement pop-up window automatically pops up in the terminal at this time, in the absence of an intention analysis model, the open instruction may open the advertisement page corresponding to the advertisement pop-up window, and does not open the web page in the automation process; and through the intention analysis model, it can be analyzed that the purpose of the current operation is to open the web page. If an advertisement pop-up window appears in the process of automatically opening the web page through the open instruction, the advertisement pop-up window can be ignored, and the web page can be opened directly according to the open instruction, or the advertisement pop-up window can be closed first, and then the web page can be opened according to the open instruction, thereby improving the success rate of automatically opening the web page.

[0060] Furthermore, the method for executing the automated process also includes:

[0061] Step c: detecting whether a non-process factor not belonging to the automated process is detected during the execution of the automated process in the terminal.

[0062] Furthermore, after determining the automated process that currently needs to be executed in the terminal, it is detected whether non-process factors that do not belong to the automated process are detected during the execution of the automated process. It is understandable that non-process factors are factors that affect the execution of the automated process, such as fluctuations in the execution environment corresponding to the automated process, such as the automated retrieval process that automatically implements the retrieval function, and the search box position and retrieval content corresponding to the user's retrieval instruction are pre-set. Under normal circumstances, the retrieval function can be automatically implemented by entering the retrieval content in the search box position, but if the search box position changes due to changes in the terminal execution environment, the change in the search box position is a non-process factor.

[0063] If the non-process factor is detected, the step of executing the target operation instruction corresponding to the automation process according to the behavioral intention to execute the automation process includes:

[0064] Step d: executing the target operation instruction corresponding to the automated process according to the behavioral intention and the non-process factors to execute the automated process.

[0065] If a non-process factor that does not belong to the automated process is detected during the execution of the automated process, the target operation instruction corresponding to the automated process is executed through the behavioral intention and non-process factors determined by the intention analysis model to successfully execute the automated process. It can be understood that when non-process factors appear in the automated process, the target operation instruction of the automated process needs to be adaptively changed according to the non-process factors in order to successfully execute the automated process. Continuing with the above example, since it is determined that the position of the search box has changed, in the process of executing the automated retrieval process, since the behavioral intention is to realize the retrieval function first, in the process of executing the automated retrieval process, the search box position in the terminal will be determined first, and then the search content will be output in the determined search box position to successfully execute the automated retrieval process, instead of directly entering the search content in the original search box position. Further, if a non-process factor that does not belong to the automated process is not detected during the execution of the automated process, the target operation instruction corresponding to the automated process is directly executed. It can be understood that for the same automated process, the corresponding non-process factors are different, and the target operation instructions may also be different.

[0066] This embodiment uses the multimodal data corresponding to the terminal as the input of the deep learning model to obtain the intention analysis model, determines the behavioral intention corresponding to the automation process in the terminal through the intention analysis model, and executes the target operation instruction corresponding to the automation process according to the behavioral intention to execute the automation process. It realizes the analysis of the behavioral intention corresponding to the automation process through the intention analysis model, and the execution of the automation process according to the behavioral intention, avoiding the situation where the automation process fails to execute due to changes in the execution environment corresponding to the automation process, thereby improving the adaptability of the automation process to different execution environments, that is, improving the adaptability of the automation process, and improving the execution success rate of the automation process.

[0067] Furthermore, after determining the behavioral intention corresponding to the automated process, during the execution of the automated process, it can be determined whether the auxiliary software needs to be started according to the behavioral intention. If it is determined that the auxiliary software needs to be started according to the behavioral intention, the auxiliary software is started at the same time during the execution of the automated process to improve the success rate of the automated process. For example, during the execution of the automatic translation process, the translator can be started at the same time to improve the translation efficiency. Whether the specific behavioral intention needs to start the auxiliary software can be set in advance. If it is determined that the auxiliary software does not need to be started according to the behavioral intention, the automated process is directly executed.

[0068] Furthermore, this embodiment does not need to independently develop corresponding solutions for various scenarios of the execution environment, thereby reducing development costs and improving the utilization rate of terminal storage space.

[0069] Furthermore, a second embodiment of the method for executing an automated process of the present invention is proposed. The difference between the second embodiment of the method for executing an automated process and the first embodiment of the method for executing an automated process is that, with reference to Figure 2 , the method for executing the automated process further includes:

[0070] Step S40, obtaining local usage data of the user corresponding to the terminal, performing federated learning based on the local usage data, updating the intent analysis model according to the learning result of the federated learning, and obtaining an updated intent analysis model.

[0071] For ease of understanding, let's first explain federated learning. Federated Learning is an emerging basic technology for artificial intelligence. Its design goal is to ensure information security during big data exchange, protect terminal data and personal privacy data, and carry out efficient machine learning between multiple parties or multiple computing nodes under the premise of legality and compliance. In the system architecture of federated learning, this embodiment introduces the system architecture of federated learning using a scenario involving two data owners (i.e., Enterprise A and Enterprise B) as an example. The architecture can be extended to scenarios involving multiple data owners. Suppose Enterprise A and Enterprise B want to jointly train a machine learning model, and their business systems each have relevant data of their respective users. In addition, Enterprise B also has label data that the model needs to predict. For data privacy protection and security considerations, Enterprise A and Enterprise B cannot directly exchange data, and can use a federated learning system to build a model.

[0072] Building a model using federated learning includes two parts. The first part is encrypted sample alignment. Since the user groups of the two companies do not completely overlap, the system uses encrypted user sample alignment technology to confirm the common users of both parties without exposing non-overlapping users to each other, so as to combine the features of these users for modeling. The second part is encrypted model training. After determining the common user group, the data can be used to train the machine learning model. In order to ensure the confidentiality of the data during the training process, encrypted training is required with the help of a third-party collaborator C. Taking the linear regression model as an example, the training process can be divided into the following 4 steps:

[0073] Step 1: Collaborator C distributes the public key to Enterprise A and Enterprise B to encrypt the data that needs to be exchanged during the training process.

[0074] Step 2: Enterprise A and Enterprise B exchange intermediate results for calculating gradients in encrypted form.

[0075] Step 3: Enterprise A and Enterprise B perform calculations based on the encrypted gradient values ​​respectively, while Enterprise B calculates the loss based on its label data and summarizes the results to Collaborator C. Collaborator C calculates the total gradient value by summarizing the results and decrypts it.

[0076] Step 4: Collaborator C sends the decrypted gradient values ​​back to Enterprise A and Enterprise B respectively. Enterprise A and Enterprise B update the parameters of their respective models based on the gradient values.

[0077] The above steps are iterated until the loss function corresponding to the linear regression model converges. The loss function is a pre-set function, and the entire training process is completed. During the sample alignment and model training process, the data of Enterprise A and Enterprise B are retained locally, and the data interaction during training will not lead to data privacy leakage. Therefore, the two parties are able to achieve cooperative training models with the help of federated learning.

[0078] According to different data sets, federated learning is divided into horizontal federated learning, vertical federated learning and federated transfer learning (FmL).

[0079] Horizontal federated learning is when two data sets have a lot of overlap in user features but a little overlap in users. The data sets are split horizontally (i.e., in the user dimension) and the data with the same user features but not exactly the same users is taken out for training. For example, there are two banks in different regions, and their user groups come from their respective regions, with very little overlap. However, their businesses are very similar, so the recorded user features are the same. At this point, horizontal federated learning can be used to build a joint model.

[0080] Vertical federated learning is to split the dataset vertically (i.e., feature dimension) when there is a large overlap of users in two datasets but a small overlap of user features, and take out the data with the same users but different user features for training. For example, there are two different institutions, one is a bank in a certain place, and the other is an e-commerce company in the same place. Their user groups are likely to include most of the residents in the place, so the user intersection is large. However, since the bank records the user's income and expenditure behavior and credit rating, while the e-commerce company keeps the user's browsing and purchase history, the intersection of their user features is small. Vertical federated learning is to aggregate these different features in an encrypted state to enhance the model's capabilities. At present, many machine learning models such as logistic regression models, tree structure models, and neural network models have gradually been proven to be able to be built on this federated system.

[0081] Federated transfer learning is to overcome the situation of insufficient data or labels by using transfer learning without splitting the data when there is little overlap between the users and user features of the two data sets. For example, there are two different institutions, one is a bank in China and the other is an e-commerce company in the United States. Due to geographical restrictions, the user groups of these two institutions have little overlap. At the same time, due to the different types of institutions, the data features of the two only overlap to a small extent. In this case, in order to conduct effective federated learning, transfer learning must be introduced to solve the problem of small unilateral data size and few labeled samples, thereby improving the effect of the model.

[0082] It should be noted that in the process of training the intent analysis model, the daily behavior data of each terminal user is required. These daily behavior data are stored locally on the terminal held by each user as the local usage data of the terminal corresponding to the user. Among the local usage data, some data belongs to the user's private data. Without the user's authorization, the user's private data cannot be obtained, that is, the user's private data cannot be used as training sample data to train the deep learning model and obtain the intent analysis model. At present, some applications in the terminal usually force the user to agree to the use of private data in advance in the user agreement, or obtain the user's private data in other uninformed ways, which significantly reduces the privacy of the user's private data. By adopting federated learning, the intent analysis model can be updated without touching the user's original local usage data.

[0083] Obtain the local usage data of the user corresponding to the terminal, wherein the local usage data includes but is not limited to multimodal data. Compared with multimodal data, the local usage data also includes some privacy data of the user, as well as some usage data after statistical analysis, such as the frequency of users starting a certain application, the duration of each use of the application, etc. After obtaining the local usage data of the user corresponding to the terminal, perform federated learning based on the local usage data to obtain the learning results of the federated learning, and update the intent analysis model based on the learning results to obtain an updated intent analysis model. It should be noted that the specific use of horizontal federated learning, vertical federated learning, or federated transfer learning is determined by the characteristics of the local usage data.

[0084] Furthermore, the step S40 includes:

[0085] Step e: obtaining local usage data of the user corresponding to the terminal, performing federated learning based on the local usage data, and obtaining a gradient value obtained by the federated learning.

[0086] Step f, updating the model parameters in the intention analysis model according to the gradient value to obtain an updated intention analysis model.

[0087] Furthermore, the local usage data of the user corresponding to the terminal is obtained. It can be understood from the above explanation of federated learning that in order to ensure the privacy of the local usage data of each terminal, the local usage data of the user corresponding to each terminal is encrypted with the public key sent by the federal server (collaborator), that is, in the process of federated learning, the local usage data used are encrypted, and the original local usage data of the user corresponding to each terminal cannot be known. In this embodiment, in order to perform federated learning, this embodiment can be set to obtain the local usage data of the users corresponding to at least two terminals. After the federal server uses the local usage data for federated learning, the learning result of the federated learning will be obtained. The learning result is the gradient value corresponding to the federated learning. At this time, the federal server will return the gradient value to each terminal.

[0088] After obtaining the gradient value obtained by federated learning, the model parameters in the intent analysis model are updated according to the gradient value to obtain an updated intent analysis model. Specifically, the gradient value can be directly used to replace the model parameters in the intent analysis model, or the weights between each model parameter and the gradient value in the intent analysis model can be set, and then the model parameter is multiplied by the corresponding weight to obtain the first parameter, and the gradient value is multiplied by the corresponding weight to obtain the second parameter, and the sum of the first parameter and the second parameter is calculated, and the sum of the first parameter and the second parameter is determined as the updated model parameter of the intent analysis model. If the original parameter value of the A model parameter in the intent analysis model is a, and its corresponding gradient value is b, the weight value of the original parameter value is a1, and the weight value of the A model parameter corresponding to the gradient value is b1, then the updated A model parameter = a×a1+b×b1, where a1+b1=1.

[0089] Step S20 includes:

[0090] Step S21, determining the behavior intention corresponding to the automation process in the terminal through the updated intention analysis model.

[0091] After the updated intent analysis model is obtained, the behavior intention corresponding to the preset automation process in the terminal is determined according to the updated intent analysis model.

[0092] After obtaining the intent analysis model, this embodiment obtains the local usage data of terminal users through federated learning, thereby breaking the limitations of data silos between various terminals. The intent analysis model can be updated in combination with the local usage data of various terminal users. On the basis of protecting the privacy of local usage data of various terminal users and the data ownership of the data owner (terminal), it overcomes the defects caused by the small amount of sample data for training the intent analysis model, and further improves the accuracy of intent analysis of the obtained intent analysis model.

[0093] Furthermore, a third embodiment of a method for executing an automated process of the present invention is proposed.

[0094] The difference between the third embodiment of the method for executing the automation process and the first and / or second embodiments of the method for executing the automation process is that, with reference to Figure 3 , the method for executing the automated process further includes:

[0095] Step S50, obtaining interaction data between the terminal and the server, performing federated learning based on the interaction data, updating the intent analysis model according to the learning result of the federated learning, and obtaining an updated intent analysis model.

[0096] Step S20 also includes:

[0097] Step S21, determining the behavior intention corresponding to the automation process in the terminal through the updated intention analysis model.

[0098] The interaction data between the terminal and the server is obtained, wherein the interaction data includes valid operation data during the interaction between the terminal and the server, such as search engine retrieval operations, web form entry operations, etc. The server in this embodiment may be a local server corresponding to a cloud server and a terminal. After the interaction data is obtained, federated learning is performed based on the interaction data, and the intent analysis model is updated according to the learning results of the federated learning to obtain an updated intent analysis model, and the behavioral intention corresponding to the pre-set automation process in the terminal is determined by the updated intent analysis model. It should be noted that the principle of federated learning based on interaction data is the same as the principle of federated learning based on the local usage data of terminal users, and will not be repeated here.

[0099] It should be noted that in the execution method of the automated process, federated learning can be performed only through local usage data to update the intent analysis model; federated learning can be performed only through interaction data to update the intent analysis model; or federated learning can be performed through both local usage data and interaction data to update the intent analysis model.

[0100] This embodiment updates the intent analysis model by performing federated learning through the interaction data between the terminal and the server, so that the intent analysis model can be applied in various interaction scenarios. The interaction data is used to update the intent analysis model without infringing the privacy of various terminal data, so that the intent analysis model can understand the user's behavioral intention in the interaction scenario, and then execute the automated interaction process through the intent analysis model, thereby improving the flexibility of the intent analysis model, improving the applicability of the automated process, and improving the execution efficiency and success rate of the automated process corresponding to the interaction scenario, so that the execution results of the automated process in the terminal truly meet the expected operation results of the developer.

[0101] Furthermore, a fourth embodiment of a method for executing an automated process of the present invention is proposed.

[0102] The fourth embodiment of the method for executing the automated process is different from the first, second and / or third embodiments of the method for executing the automated process in that the method for executing the automated process further includes:

[0103] Step j, detecting the execution success rate of the automated process.

[0104] Step h: if the execution success rate is less than the preset success rate, update the intention analysis model.

[0105] After the execution of the automated process is completed, the execution success rate of the automated process in the terminal can be detected. Specifically, the execution success rate is equal to the number of successful executions of the automated process within the preset time length divided by the number of all automated processes executed within the preset time length. In this embodiment, the specific duration of the preset time length is not limited. After the execution success rate is calculated, it is determined whether the execution success rate is less than the preset success rate, wherein the size of the preset success rate can be set according to specific needs. This embodiment does not impose specific restrictions on the size of the preset success rate. When it is determined that the execution success rate is less than the preset success rate, the intent analysis model is updated. At this time, the multimodal data corresponding to the terminal can be re-acquired to update the intent analysis model, or the local use data, interaction data, etc. can be used to update the intent analysis model; if it is determined that the execution success rate is greater than or equal to the preset success rate, there is no need to update the intent analysis model.

[0106] This embodiment further improves the accuracy of the intent analysis model in analyzing behavioral intentions by detecting the execution success rate of the automated process and updating the intent analysis model when the execution success rate is lower than a preset success rate.

[0107] In addition, the present invention also provides an execution device for an automated process, referring to Figure 4 , the execution device of the automated process includes:

[0108] An acquisition module 10 is used to acquire multimodal data corresponding to the terminal, and use the multimodal data as input of a deep learning model to obtain an intent analysis model;

[0109] A determination module 20, configured to determine the behavior intention corresponding to the automation process in the terminal through the intention analysis model;

[0110] The execution module 30 is used to execute the target operation instruction corresponding to the automation process according to the behavior intention, so as to execute the automation process.

[0111] Furthermore, the acquisition module 10 is also used to acquire local usage data of the user corresponding to the terminal;

[0112] The execution device of the automated process includes:

[0113] A first federated learning module, configured to perform federated learning based on the local usage data;

[0114] A first updating module, used for updating the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model;

[0115] The determination module 20 is further configured to determine the behavior intention corresponding to the automation process in the terminal through the updated intention analysis model.

[0116] Furthermore, the first federated learning module is further used to perform federated learning based on the local usage data to obtain a gradient value obtained by the federated learning;

[0117] The first updating module is also used to update the model parameters in the intention analysis model according to the gradient value to obtain an updated intention analysis model.

[0118] Furthermore, the execution device of the automated process includes:

[0119] A second federated learning module, configured to perform federated learning based on the interaction data;

[0120] A second updating module is used to update the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model;

[0121] The determination module 20 is further configured to determine the behavior intention corresponding to the automation process in the terminal through the updated intention analysis model.

[0122] Furthermore, the execution device of the automated process includes:

[0123] A first detection module, used to detect whether a non-process factor that does not belong to the automated process is detected during the execution of the automated process in the terminal;

[0124] The execution module 30 is further configured to execute the target operation instruction corresponding to the automation process according to the behavior intention and the non-process factor if the non-process factor is detected, so as to execute the automation process.

[0125] Furthermore, the acquisition module 10 is also used to acquire model sample data corresponding to the terminal; and use the multimodal data and the model sample data as inputs of a deep learning model to obtain an intent analysis model.

[0126] Furthermore, the execution device of the automated process includes:

[0127] A second detection module is used to detect the execution success rate of the automation process;

[0128] The third updating module is used to update the intention analysis model if the execution success rate is less than a preset success rate.

[0129] The specific implementation of the automated process execution device of the present invention is basically the same as the various embodiments of the automated process execution method described above, and will not be described in detail herein.

[0130] In addition, the present invention also provides an execution device for an automated process. Figure 5 As shown, Figure 5 It is a structural diagram of the hardware operating environment involved in the embodiment of the present invention.

[0131] It should be noted that Figure 5 That is, it can be a schematic diagram of the structure of the hardware operating environment of the execution device of the automation process. The execution device of the automation process in the embodiment of the present invention can be a terminal device such as a PC, a portable computer, etc.

[0132] like Figure 5 As shown, the execution device of the automated process may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0133] Those skilled in the art will understand that Figure 5 The structure of the execution device of the automation process shown in the figure does not constitute a limitation on the execution device of the automation process, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0134] like Figure 5 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an execution program of an automated process. The operating system is a program that manages and controls the hardware and software resources of the execution device of the automated process, and supports the execution program of the automated process and the operation of other software or programs.

[0135] exist Figure 5In the execution device of the automation process shown, the user interface 1003 is mainly used to connect to other terminals and communicate data with other terminals; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the execution program of the automation process stored in the memory 1005, and execute the steps of the execution method of the automation process as described above.

[0136] The specific implementation of the automated process execution device of the present invention is basically the same as the above-mentioned automated process execution method embodiments, and will not be described in detail here.

[0137] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which an execution program of an automation process is stored. When the execution program of the automation process is executed by a processor, the steps of the execution method of the automation process as described above are implemented.

[0138] The specific implementation of the computer-readable storage medium of the present invention is basically the same as the various embodiments of the execution method of the above-mentioned automation process, and will not be repeated here.

[0139] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0140] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0142] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for executing an automated process, characterized in that: The method for executing the automated process comprises the following steps: Acquire multimodal data corresponding to the terminal and model sample data corresponding to the terminal, and use the multimodal data and the model sample data as inputs of a deep learning model to obtain an intent analysis model, wherein the multimodal data includes operation data corresponding to an input operation of inputting text, a mouse clicking operation of clicking a mouse, and a mouse moving operation of moving a mouse triggered by a terminal user in the terminal, and the model sample data records changes in a terminal screen interface; Determine the behavior intention corresponding to the automation process in the terminal through the intention analysis model; Execute the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process.

2. The method for executing an automated process according to claim 1, wherein: After the step of acquiring multimodal data corresponding to the terminal and using the multimodal data as input of the deep learning model to obtain the intention analysis model, the method further includes: Acquire local usage data of a user corresponding to the terminal, perform federated learning based on the local usage data, and update the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model; The step of determining the behavior intention corresponding to the automation process in the terminal through the intention analysis model includes: The behavioral intention corresponding to the automated process in the terminal is determined through the updated intent analysis model.

3. The method for executing an automated process according to claim 2, wherein: The steps of acquiring local usage data of a user corresponding to the terminal, performing federated learning based on the local usage data, updating the intent analysis model according to the learning result of the federated learning, and obtaining the updated intent analysis model include: Acquire local usage data of a user corresponding to the terminal, perform federated learning based on the local usage data, and obtain a gradient value obtained by the federated learning; The model parameters in the intent analysis model are updated according to the gradient value to obtain an updated intent analysis model.

4. The method for executing an automated process according to claim 1, wherein: After the step of acquiring multimodal data corresponding to the terminal and using the multimodal data as input of the deep learning model to obtain the intention analysis model, the method further includes: Acquire interaction data between the terminal and the server, perform federated learning based on the interaction data, and update the intent analysis model according to the learning result of the federated learning to obtain an updated intent analysis model; The step of determining the behavior intention corresponding to the automation process in the terminal through the intention analysis model includes: The behavioral intention corresponding to the automated process in the terminal is determined through the updated intent analysis model.

5. The method for executing an automated process according to claim 1, wherein: After the step of determining the behavior intention corresponding to the automation process in the terminal by the intention analysis model, the method further includes: Detecting whether a non-process factor not belonging to the automated process is detected during the execution of the automated process in the terminal; If the non-process factor is detected, the step of executing the target operation instruction corresponding to the automation process according to the behavioral intention to execute the automation process includes: Execute the target operation instruction corresponding to the automation process according to the behavior intention and the non-process factors to execute the automation process.

6. The method for executing an automated process according to any one of claims 1 to 5, characterized in that: After the step of executing the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process, the method further includes: Detecting the execution success rate of the automated process; If the execution success rate is lower than the preset success rate, the intention analysis model is updated.

7. An execution device for an automated process, characterized in that: The execution device of the automated process includes: An acquisition module, used to acquire multimodal data corresponding to the terminal and model sample data corresponding to the terminal, and use the multimodal data and the model sample data as inputs of a deep learning model to obtain an intent analysis model, wherein the multimodal data includes operation data corresponding to an input operation of inputting text, a mouse clicking operation of clicking a mouse, and a mouse moving operation of moving a mouse triggered by a terminal user in the terminal, and the model sample data records changes in a terminal screen interface; A determination module, used to determine the behavior intention corresponding to the automation process in the terminal through the intention analysis model; An execution module is used to execute the target operation instruction corresponding to the automation process according to the behavior intention to execute the automation process.

8. An automated process execution device, characterized in that: The execution device of the automation process includes a memory, a processor, and an execution program of the automation process stored in the memory and executable on the processor. When the execution program of the automation process is executed by the processor, the steps of the execution method of the automation process as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an execution program of an automation process, and when the execution program of the automation process is executed by a processor, the steps of the execution method of the automation process according to any one of claims 1 to 6 are implemented.

Images