Interaction guiding method and device, storage medium and electronic device

Abstract

The application discloses an interactive guidance method and device, a storage medium and an electronic device. The method comprises the following steps: receiving an interactive guidance request, wherein the interactive guidance request carries a task description text; in the case that there is a historical interactive operation record, intercepting a current screenshot, wherein the historical interactive operation record is used to represent the completed interactive operation of the user for the interface; and outputting a single interactive operation described in natural language according to the historical interactive operation record, the current screenshot and the task description text. Therefore, by using the embodiment of the application, the repeated operation or the omission of necessary steps can be prevented, so that the uncertainty and operation burden of the user in the human-computer cooperation process are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an interactive guiding method and device, a storage medium and an electronic device. BACKGROUND

[0002] In a modern office environment, employees often need to handle complex multi-step tasks, such as creating, approving and submitting a purchase order in an enterprise resource planning (ERP) system. These tasks usually involve the operation of multiple interfaces, each of which has specific input requirements and operation steps. The complexity and multi-step nature of these tasks place high demands on the attention and operation accuracy of employees.

[0003] At present, there are various intelligent assistant tools on the market, which aim to help users automate and simplify these complex tasks. For example, some intelligent assistants can recognize the user's input "create a purchase order" instruction and then automatically navigate to the corresponding page of the ERP system.

[0004] However, these intelligent assistants often lack in-depth analysis capabilities for the current screen state, and cannot accurately determine whether the current interface has completed the necessary operations. For example, if the user has manually filled in part of the form information, the intelligent assistant may not be able to recognize these input contents, resulting in repeated operations or missing necessary steps, thereby increasing the uncertainty and operation burden of the user in the human-computer collaboration process. SUMMARY

[0005] The embodiments of the present application provide an interactive guiding method, device, storage medium and electronic device. In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This part is not a general review, nor does it determine the key / important components or delineate the protection scope of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0006] In a first aspect, the embodiments of the present application provide an interactive guiding method, and the method comprises:

[0007] receiving an interactive guiding request, the interactive guiding request carrying a task description text;

[0008] In the case where there is a historical interactive operation record, a current screenshot is intercepted, and the historical interactive operation record is used to represent the interactive operation that the user has completed for the interface;

[0009] According to the historical interactive operation record, the current screenshot and the task description text, a single interactive operation described in natural language is output.

[0010] Optionally, according to the historical interaction operation record, the current screenshot and the task description text, a single interaction operation described in natural language is output, including:

[0011] All the interactive elements existing in the current screenshot are identified as the current state representation;

[0012] According to the current state representation, it is verified whether each historical interaction operation in the historical interaction operation record is interacted successfully, and a verification result is obtained;

[0013] The historical interaction operation interacted successfully in the verification result is taken as an effective interaction operation;

[0014] According to the effective interaction operation, the current state representation and the task description text, a single interaction operation described in natural language is output.

[0015] Optionally, the current state representation includes the current state of each interactive element;

[0016] According to the current state representation, it is verified whether each historical interaction operation in the historical interaction operation record is interacted successfully, including:

[0017] The target interactive element and the operation type corresponding to each historical interaction operation in the historical interaction operation record are obtained;

[0018] According to the operation type, the expected state of the target interactive element is derived;

[0019] From the current state of each interactive element, the actual state of the interactive element same as the target interactive element is obtained;

[0020] In the case that the expected state is consistent with the actual state, it is determined that each historical interaction operation in the historical interaction operation record is interacted successfully.

[0021] Optionally, according to the effective interaction operation, the current state representation and the task description text, a single interaction operation described in natural language is output, including:

[0022] The task target representation is obtained by analyzing the task description text;

[0023] The current state representation is compared with the task target representation to determine the task progress required for interaction;

[0024] According to the effective interaction operation and the task progress, the target interaction operation required for the next interaction is determined;

[0025] In the case that the target interaction operation exists, the target interaction operation is converted into a single interaction operation described in natural language;

[0026] The single interaction operation is output.

[0027] Optionally, the method further comprises:

[0028] generating first prompt information indicating that the user scrolls down the current interaction interface in the absence of the target interaction operation;

[0029] displaying the first prompt information.

[0030] Optionally, the method further comprises:

[0031] in the case where the verification result indicates that all the historical interaction operations have failed, outputting a single interaction operation described in natural language according to the current screenshot and the task description text.

[0032] Optionally, the method further comprises:

[0033] in the case where the single interaction operation is an abnormal interaction operation requiring user intervention, generating second prompt information corresponding to the abnormal interaction operation;

[0034] displaying the second prompt information.

[0035] In a second aspect, an interaction guiding device is provided, and the device comprises:

[0036] a receiving module configured to receive an interaction guiding request, the interaction guiding request carrying a task description text;

[0037] a capturing module configured to capture a current screenshot in the case where there is a historical interaction operation record, the historical interaction operation record being used to represent an interaction operation that has been completed by the user for the interface;

[0038] an outputting module configured to output a single interaction operation described in natural language according to the historical interaction operation record, the current screenshot and the task description text.

[0039] In a third aspect, a computer storage medium is provided, and the computer storage medium stores a plurality of instructions, the instructions being adapted to be loaded by a processor and to execute the method steps described above.

[0040] In a fourth aspect, an electronic device is provided, and the electronic device can comprise a processor and a memory; the memory stores a computer program, and the computer program is adapted to be loaded by the processor and to execute the method steps described above.

[0041] The technical solutions provided in the embodiments of the present application can have the following beneficial effects:

[0042] In the embodiments of the present application, on the one hand, in the case that there is a historical interaction operation record, a current screenshot is intercepted, the historical interaction operation record is used to represent the interaction operation that has been completed by the user for the interface, so that the system can fully understand the past operation of the user and the current interface state, precise planning of the next operation can be realized, and repeated execution of the completed operation due to neglect of the historical operation or interface change is avoided, thereby effectively preventing repeated operation or omission of necessary steps. On the other hand, the system informs the user of the specific operation of the next step in a clear and natural language according to the historical interaction operation record, the current screenshot and the task description text, the user only needs to perform according to the indication, without the need to judge the complex task flow by himself, and the operation failure is reduced due to the precise planning of the system, which makes the user more relaxed in the process of human-computer cooperation, and reduces the uncertainty and operation burden.

[0043] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0045] Figure 1 is a flow diagram of an interaction guidance method provided by an embodiment of the present application;

[0046] Figure 2 is a process schematic block diagram of an interaction guidance process provided by the present application;

[0047] Figure 3 is a structural schematic diagram of an interaction guidance device provided by the present application;

[0048] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following description and drawings sufficiently illustrate specific embodiments of the present application to enable one of ordinary skill in the art to practice them.

[0050] It should be clear that the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0051] The following description refers to the accompanying drawings. Unless otherwise noted, like elements in different drawings have the same or similar notation, and the same or similar elements in different figures have the same or similar numbers. The following exemplary embodiments are described with reference to the accompanying drawings, wherein the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0052] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for descriptive purposes, and cannot be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. The association between the associated objects is described, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0053] The present application provides an interactive guidance method, device, storage medium and electronic equipment to solve the problems in the above related technical problems. In the embodiments of the present application, on the one hand, in the case of existing historical interaction operation records, the current screenshot is intercepted, and the historical interaction operation records are used to represent the interaction operation completed by the user for the interface, so that the system can fully understand the past operation of the user and the current interface state, and the next operation can be accurately planned, avoiding repeated execution of the completed operation due to neglect of historical operation or interface change, thereby effectively preventing repeated operation or missing necessary steps. On the other hand, the system informs the user of the specific operation of the next step in a clear and natural language according to the historical interaction operation records, the current screenshot and the task description text, the user only needs to perform according to the indication, without the need to judge the complex task flow by himself, and the accurate planning of the system reduces the operation error, which makes the user more relaxed in the process of human-computer cooperation, reduces uncertainty and operation burden, which will be described in detail below by exemplary embodiments.

[0054] The following will be described in conjunction with the accompanying drawings Figure 1 -Appendix Figure 2 The interactive guidance method provided by the embodiments of the present application will be described in detail. The method can be realized by relying on a computer program and can run on an interactive guidance device based on the von Neumann system. The computer program can be integrated in an application or run as an independent tool application.

[0055] Please refer to Figure 1 A flowchart of an interactive guidance method is provided for the embodiments of the present application. As shown in Figure 1 The method of the embodiments of the present application can include the following steps:

[0056] S101, Receive interactive guidance request, the interactive guidance request carries task description text;

[0057] An interactive guidance request is a request initiated by the user to the client, requesting the client to provide instructions for the next step. This request includes a description of the task the user wishes to complete. The task description text is a natural language description entered by the user into the client, used to represent the task content that the user wants the client to complete.

[0058] In some embodiments of this application, a user inputs task description text through a user interface (such as a webpage, application, etc.) and submits a request. The system receives the interactive guidance request submitted by the user through a network interface. This request includes the task description text. The system parses the interactive guidance request and extracts the task description text carried in the request.

[0059] For example, a user enters "Buy a red T-shirt" in the smart assistant interface of an e-commerce website. The user clicks the "Submit" button, triggering an interactive guidance request. The system receives the interactive guidance request submitted by the user through an API interface. The request content includes the task description text "Buy a red T-shirt". The system parses the interactive guidance request and extracts the task description text "Buy a red T-shirt".

[0060] S102, if there is a historical interaction operation record, take a screenshot of the current screen. The historical interaction operation record is used to represent the interaction operations that the user has completed on the interface.

[0061] The historical interaction operation record refers to all the operation steps and results that the user has completed, as recorded by the system. This record helps the system understand which operations the user has performed, thus avoiding duplicate operations or missing necessary steps. The current screenshot is an image of the user interface captured by the system at a specific moment. This image provides visual information about the current interface, helping the system understand its state, including which elements are visible and which operations have been completed.

[0062] In some embodiments of this application, the system queries a local database or memory to check for records of executed operations related to the current task. The system confirms the existence of historical interaction operation records. The system then captures an image of the current screen.

[0063] For example, the system query finds that the user has already logged into the e-commerce website and browsed the product category pages. The system confirms that the user has completed the login and category browsing operations. At this point, the system captures the current screen, and the screenshot shows the user currently on the product search page.

[0064] In some embodiments of the present application, in the absence of the historical interaction operation record, the single interaction operation described in natural language can be output according to the current screenshot and the task description text.

[0065] Specifically, the executable operation types include mouse operation, keyboard operation, scroll operation and control operation. The mouse operation includes single click, double click, right click and drag. The keyboard operation includes combination key and input text. The scroll operation includes up, down, left and right scrolling. The control operation includes waiting, task completion, request for manual assistance and navigation to URL.

[0066] S103, output the single interaction operation described in natural language according to the historical interaction operation record, the current screenshot and the task description text.

[0067] In some embodiments of the present application, the specific process of outputting the single interaction operation described in natural language according to the historical interaction operation record, the current screenshot and the task description text includes: identifying all the interactive elements existing in the current screenshot as the current state representation; verifying whether each historical interaction operation in the historical interaction operation record is successful according to the current state representation to obtain a verification result; taking the historical interaction operation that is successful in the verification result as the valid interaction operation; and outputting the single interaction operation described in natural language according to the valid interaction operation, the current state representation and the task description text.

[0068] In some embodiments of the present application, in the case that the verification result indicates that each historical interaction operation fails, the single interaction operation described in natural language is output according to the current screenshot and the task description text.

[0069] In the case that the verification result indicates that each historical interaction operation fails, it means that the user has not operated, and at this time, only the current screenshot and the task description text need to be considered to determine the subsequent interaction operation.

[0070] The current state representation includes the current state of each interactive element.

[0071] Specifically, the specific process of verifying whether each historical interaction operation in the historical interaction operation record is successful according to the current state representation includes: obtaining the target interactive element and the operation type corresponding to each historical interaction operation in the historical interaction operation record; deriving the expected state of the target interactive element according to the operation type; obtaining the actual state of the interactive element same as the target interactive element from the current state of each interactive element; and determining that each historical interaction operation in the historical interaction operation record is successful in the case that the expected state is consistent with the actual state.

[0072] Specifically, according to the valid interaction operation, the current state representation and the task description text, outputting a specific process of a single interaction operation described in natural language includes: analyzing the task description text to obtain a task target representation; comparing the current state representation with the task target representation to determine the task progress of the required interaction; determining the target interaction operation required for the next interaction according to the valid interaction operation and the task progress; in the case that the target interaction operation exists, converting the target interaction operation into a single interaction operation described in natural language; and outputting the single interaction operation.

[0073] Further, in the case that the target interaction operation does not exist, generating first prompt information indicating that the user needs to slide down the current interaction interface; and displaying the first prompt information.

[0074] Wherein, the absence of the target interaction operation indicates that the current interaction interface does not have a target element, and at this time the user needs to slide down the current interaction interface to re-execute the step of capturing the current screenshot to find the target element.

[0075] In some embodiments of the present application, in the case that the single interaction operation is an abnormal interaction operation requiring user intervention, second prompt information corresponding to the abnormal interaction operation is generated; and the second prompt information is displayed.

[0076] For example, the system identifies an abnormal interaction operation on the current screen, such as requiring the user to input a verification code or perform a two-dimensional code scanning. The system generates the prompt information "the current operation requires you to manually input a verification code, please continue after completing the verification". The system displays the prompt information "the current operation requires you to manually input a verification code, please continue after completing the verification" in the intelligent assistant interface.

[0077] For example Figure 2 As shown, Figure 2 is a process schematic block diagram of an interaction guidance process provided by the present application. First, an interaction guidance request is received, and the interaction guidance request carries a task description text. Then, it is determined whether there is a historical interaction operation record. If yes, a current screenshot is captured. Then, all the interactive elements existing in the current screenshot are identified as a current state representation. Next, according to the current state representation, it is verified whether each historical interaction operation in the historical interaction operation record is successfully interacted to obtain a verification result. If yes, the historical interaction operation successfully interacted in the verification result is taken as a valid interaction operation. According to the valid interaction operation, the current state representation and the task description text, a single interaction operation described in natural language is outputted. If there is no historical interaction operation record, according to the current screenshot and the task description text, a single interaction operation described in natural language is outputted. If each historical interaction operation is not successfully interacted, according to the current screenshot and the task description text, a single interaction operation described in natural language is outputted.

[0078] In the embodiments of the present application, the atomicity and clarity of operation are guaranteed in all interaction links, significantly reducing human-computer interaction errors. The task context and interface details can be accurately understood, and the operation plan can be dynamically adjusted, which is more efficient. It has strong scalability and is applicable to various intelligent assistants and complex and variable systems. It has the ability to automatically request human intervention for abnormal scenarios, effectively preventing process blockage. Historical operations combined with interface feedback can effectively avoid redundant and erroneous operations.

[0079] In the embodiments of the present application, on the one hand, in the case that there is a historical interaction operation record, a current screenshot is intercepted, and the historical interaction operation record is used to represent the interaction operation that the user has completed for the interface, so that the system can fully understand the user's past operation and the current interface state, and can accurately plan the next operation, avoiding repeated execution of completed operations due to neglect of historical operations or interface changes, thereby effectively preventing repeated operations or missing necessary steps. On the other hand, the system informs the user of the specific operation of the next step in a clear and natural language according to the historical interaction operation record, the current screenshot and the task description text, and the user only needs to perform according to the instructions without self-judging complex task flow, and the accurate planning of the system reduces operation errors, which makes the user more relaxed in the process of human-computer cooperation, and reduces uncertainty and operation burden.

[0080] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0081] Please refer to Figure 3 which shows a structural schematic diagram of an interaction guiding device provided by an example embodiment of the present application. The interaction guiding device can be realized by software, hardware or a combination of both to become all or part of an electronic device. The device 1 includes a receiving module 10, an intercepting module 20 and an output module 30.

[0082] The receiving module 10 is configured to receive an interaction guiding request, and the interaction guiding request carries a task description text.

[0083] The intercepting module 20 is configured to intercept a current screenshot in the case that there is a historical interaction operation record, and the historical interaction operation record is used to represent the interaction operation that the user has completed for the interface.

[0084] The output module 30 is configured to output a single interaction operation described in a natural language according to the historical interaction operation record, the current screenshot and the task description text.

[0085] It should be noted that the interactive guidance device provided in the above embodiment is only exemplified by the above division of functional modules when the interactive guidance method is executed, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the electronic device is divided into different functional modules to complete all or part of the functions described above. In addition, the interactive guidance device and the interactive guidance method provided in the above embodiment belong to the same concept, and the implementation process is embodied in the method embodiment, which will not be described here.

[0086] The serial numbers of the embodiments of the present application are only for description, not representing the advantages or disadvantages of the embodiments.

[0087] In the embodiments of the present application, on the one hand, in the case that there is a historical interaction operation record, a current screenshot is intercepted, and the historical interaction operation record is used to represent the interaction operation completed by the user for the interface, so that the system can fully understand the past operation of the user and the current interface state, precise planning of the next operation can be realized, repeated execution of the completed operation due to neglect of the historical operation or interface change is avoided, and thus repeated operation or omission of necessary steps is effectively prevented. On the other hand, the system informs the user of the specific operation of the next step in a clear and natural language according to the historical interaction operation record, the current screenshot and the task description text, the user only needs to perform according to the indication, does not need to judge the complex task flow by himself, and the operation failure is reduced due to the precise planning of the system, which makes the user more relaxed in the human-computer cooperation process and reduces the uncertainty and operation burden.

[0088] The present application also provides a computer readable medium having program instructions stored thereon, which, when executed by a processor, implement the interactive guidance method provided by each of the method embodiments.

[0089] The present application also provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the interactive guidance method of each of the method embodiments.

[0090] Please refer to Figure 4 The present application provides a structural schematic diagram of an electronic device. As shown in Figure 4 The electronic device 1000 can include at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0091] The communication bus 1002 is used to realize the connection and communication between the components.

[0092] The user interface 1003 can include a display screen (Display) and a camera (Camera), and the optional user interface 1003 can also include a standard wired interface and a wireless interface.

[0093] The network interface 1004 can optionally include a standard wired interface, a wireless interface (e.g., a WI-FI interface).

[0094] The processor 1001 can include one or more processing cores. The processor 1001 connects various parts within the electronic device 1000 through various interfaces and lines, and performs various functions of the electronic device 1000 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. The processor 1001 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 can be integrated with one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU is mainly responsible for processing an operating device, a user interface, and an application program. The GPU is responsible for rendering and drawing content to be displayed on a display screen. The modem is responsible for processing wireless communication. It can be understood that the modem can also not be integrated into the processor 1001, but can be implemented by a separate chip.

[0095] The memory 1005 can include a random access memory (RAM) and a read-only memory (ROM). The memory 1005 can include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1005 can include a program storage area and a data storage area. The program storage area can store instructions for implementing an operating device, instructions for at least one function (e.g., a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area can store data related to the above-mentioned various method embodiments, etc. The memory 1005 can also be at least one storage device located away from the processor 1001. For example, Figure 4As shown, the memory 1005 as a computer storage medium can include an operation device, a network communication module, a user interface module, and an interactive guidance application.

[0096] In Figure 4 In the electronic device 1000 as shown, the user interface 1003 is mainly used to provide an interface for user input, and obtain data input by the user; and the processor 1001 can be used to call the interactive guidance application stored in the memory 1005, and specifically perform the following operations:

[0097] Receiving an interactive guidance request, the interactive guidance request carrying a task description text;

[0098] In the case where there is a historical interaction operation record, a current screenshot is intercepted, and the historical interaction operation record is used to represent the interaction operation that has been completed by the user for the interface;

[0099] According to the historical interaction operation record, the current screenshot, and the task description text, a single interaction operation described in natural language is output.

[0100] In one embodiment, when the processor 1001 performs the operation of outputting a single interaction operation described in natural language according to the historical interaction operation record, the current screenshot, and the task description text, it specifically performs the following operations:

[0101] Identifying all the interactive elements existing in the current screenshot as a current state representation;

[0102] According to the current state representation, verifying whether each historical interaction operation in the historical interaction operation record is successfully interacted to obtain a verification result;

[0103] The historical interaction operation that is successfully interacted in the verification result is taken as an effective interaction operation;

[0104] According to the effective interaction operation, the current state representation, and the task description text, a single interaction operation described in natural language is output.

[0105] In one embodiment, when the processor 1001 performs the operation of verifying whether each historical interaction operation in the historical interaction operation record is successfully interacted according to the current state representation, it specifically performs the following operations:

[0106] Obtaining the target interaction element and the operation type corresponding to each historical interaction operation in the historical interaction operation record;

[0107] According to the operation type, deducing the expected state of the target interaction element;

[0108] From the current state of each interactive element, the actual state of the interactive element same as the target interaction element is obtained;

[0109] In the case where the expected state is consistent with the actual state, it is determined that each historical interaction operation in the historical interaction operation record is interacted successfully.

[0110] In an embodiment, the processor 1001, when executing the single interaction operation described in natural language according to the valid interaction operation, the current state representation and the task description text, specifically performs the following operations:

[0111] Analyzing the task description text to obtain a task target representation;

[0112] Comparing the current state representation with the task target representation to determine the task progress required for interaction;

[0113] According to the valid interaction operation and the task progress, determining the target interaction operation required for the next interaction;

[0114] In the case where the target interaction operation exists, converting the target interaction operation into a single interaction operation described in natural language;

[0115] Outputting the single interaction operation.

[0116] In an embodiment, the processor 1001 further performs the following operations:

[0117] In the case where the target interaction operation does not exist, generating a first prompt information indicating that the user slides down the current interaction interface;

[0118] Displaying the first prompt information.

[0119] In an embodiment, the processor 1001 further performs the following operations:

[0120] In the case where the verification result indicates that all the historical interaction operations fail to interact, outputting a single interaction operation described in natural language according to the current screenshot and the task description text.

[0121] In an embodiment, the processor 1001 further performs the following operations:

[0122] In the case where the single interaction operation is an abnormal interaction operation requiring user intervention, generating a second prompt information corresponding to the abnormal interaction operation;

[0123] Displaying the second prompt information.

[0124] In the embodiments of the present application, on the one hand, in the case that there is a historical interaction operation record, a current screenshot is intercepted, and the historical interaction operation record is used to represent the interaction operation that has been completed by the user for the interface, so that the system can fully understand the past operation of the user and the current interface state, and the next operation can be accurately planned, and the repeated operation or omission of necessary steps due to neglect of the historical operation or interface change can be avoided, thereby effectively preventing repeated operation or omission of necessary steps. On the other hand, the system informs the user of the specific operation of the next step in a clear and natural language according to the historical interaction operation record, the current screenshot and the task description text, the user only needs to perform according to the indication, and does not need to judge the complex task flow by himself, and the operation failure is reduced due to the accurate planning of the system, which makes the user more relaxed in the process of human-computer cooperation, and reduces the uncertainty and operation burden.

[0125] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the interactive program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium of the interactive program can be a magnetic disc, an optical disc, a read-only memory or a random access memory, etc.

[0126] The above only describes the preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application, so equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. An interactive guidance method, characterized by, The method comprises: receiving an interaction guidance request carrying a task description text; the interaction guidance request is used to require a client to provide operation guidance for the next step; in the case where there is a historical interaction operation record, intercepting a current screenshot, the historical interaction operation record is used to represent the interaction operation that a user has completed for an interface; according to the historical interaction operation record, the current screenshot and the task description text, outputting a single interaction operation described in natural language; wherein the outputting a single interaction operation described in natural language according to the historical interaction operation record, the current screenshot and the task description text comprises: identifying all the interactive elements existing in the current screenshot as a current state representation; the current state representation comprises the current state of each interactive element; verifying whether each historical interaction operation in the historical interaction operation record is successful according to the current state representation to obtain a verification result; taking the historical interaction operation that is successful in the verification result as a valid interaction operation; analyzing the task description text to obtain a task target representation; comparing the current state representation with the task target representation to determine the task progress required for interaction; determining a target interaction operation required for the next step of interaction according to the valid interaction operation and the task progress; in the case where the target interaction operation exists, converting the target interaction operation into a single interaction operation described in natural language; and outputting the single interaction operation; wherein the verifying whether each historical interaction operation in the historical interaction operation record is successful according to the current state representation comprises: obtaining a target interactive element and an operation type corresponding to each historical interaction operation in the historical interaction operation record; deriving an expected state of the target interactive element according to the operation type; obtaining an actual state of an interactive element same as the target interactive element from the current state of each interactive element; and in the case where the expected state is consistent with the actual state, determining that each historical interaction operation in the historical interaction operation record is successful.

2. The method of claim 1, wherein, The method further comprises: in the case where the target interaction operation does not exist, generating first prompt information indicating that a user scrolls down a current interaction interface; displaying the first prompt information.

3. The method of claim 1, wherein, The method further comprises: in the case where the verification result indicates that all the historical interaction operations fail, outputting a single interaction operation described in natural language according to the current screenshot and the task description text.

4. The method of claim 1, wherein, The method further comprises: in the case where the single interaction operation is an abnormal interaction operation requiring user intervention, generating second prompt information corresponding to the abnormal interaction operation; displaying the second prompt information.

5. An interactive guidance device implemented using the method of any one of claims 1-4, characterized by The device comprises: a receiving module configured to receive an interaction guidance request carrying a task description text; an intercepting module configured to intercept a current screenshot in the case where there is a historical interaction operation record, the historical interaction operation record being used to represent the interaction operation that a user has completed for an interface; An output module is configured to output a single interaction operation described in natural language according to the historical interaction operation record, the current screenshot and the task description text.

6. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded and executed by a processor to implement the method according to any one of claims 1-4.

7. An electronic device, comprising: Comprise: A processor and a memory; wherein the memory stores a computer program, which is adapted to be loaded and executed by the processor to implement the method according to any one of claims 1-4. A processor and a memory; wherein the memory stores a computer program, which is adapted to be loaded and executed by the processor to implement the method according to any one of claims 1-4.

Images