Operation assistance method, device, electronic device, storage medium and program product

Abstract

The present application discloses an operation assistance method, device, electronic device, storage medium and program product, which belongs to the field of artificial intelligence technology and is used to improve the personalization and customization of operation assistance. The method includes: when the user selects the first assistance mode, obtaining the task description input by the user; generating a first operation guide according to the task description and the priority queue, wherein each element in the priority queue is a vector corresponding to the operation completed by the user; when the user selects the second assistance mode, determining the second operation guide corresponding to the preset task selected by the user; displaying the first operation guide or the second operation guide to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an operation assistance method, device, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of information technology, computer software-aided systems have been widely used in programming development, office automation, audio and video entertainment, etc. Traditional computer software-aided systems build large databases on the user's computer and provide software usage suggestions to users through pipeline operations such as input deconstruction, database retrieval, and response sorting. However, this method has limitations such as long response time, poor flexibility, and low accuracy. Although the emergence of large model technology has improved the flexibility of system response to a certain extent, it still faces problems such as low accuracy and high computing resource requirements.

[0003] At present, most computer software-assisted systems use a retrieval-based auxiliary information query algorithm based on user input, that is, using user input as a model and querying the database based on literal similarity to return operation recommendations. This method requires the user's local database to store candidate data in advance, and due to limited data, the query flexibility is poor and the accuracy is low, and accurate responses cannot be achieved in most cases. This proposal proposes a generative operation recommendation based on a large language model to improve the flexibility and accuracy of system responses.

[0004] The real-time performance of the related technology is poor. At present, most computer software-assisted systems use a response sorting algorithm based on literal similarity, which only considers the text overlap between user input and query results without considering semantic features. In addition, the pattern matching between character strings is inefficient and time-consuming, which reduces the real-time performance of the system.

[0005] The degree of personalization of related technologies is low. At present, computer software-assisted systems do not consider the user's operating habits when making auxiliary operation recommendations. This approach results in that in many cases, even if the system response is effective, it is not adopted by the user.

[0006] The scalability of related technologies is poor. At present, most computer software-assisted systems run locally on the client. Since client computers usually do not have computing hardware such as GPUs and TPUs installed, this approach limits the use of advanced retrieval and sorting algorithms.

[0007] In summary, most current computer software-aided systems have poor real-time performance, low personalization, and poor scalability, which results in low practicality of computer software-aided systems. Summary of the invention

[0008] The embodiments of the present application provide an operation assistance method, device, electronic device, storage medium and program product, which can solve the problem of low practicality of previous computer software assistance systems.

[0009] In the first aspect, an embodiment of the present application provides an operation assistance method, which includes: when the user selects a first assistance mode, obtaining a task description input by the user; generating a first operation guide based on the task description and a priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; when the user selects a second assistance mode, determining a second operation guide corresponding to a preset task selected by the user; and displaying the first operation guide or the second operation guide to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide.

[0010] In the second aspect, an embodiment of the present application provides an operation process assistance device, which includes: a first selection module, used to obtain a task description input by a user when the user selects a first assistance mode; a processing module, used to generate a first operation guide based on the task description and a priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; a second selection module, used to determine a second operation guide corresponding to a preset task selected by the user when the user selects a second assistance mode; and an operation module, used to display the first operation guide or the second operation guide to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first aspect.

[0014] In an embodiment of the present application, when the user selects a first assistance mode, a task description input by the user is obtained; a first operation guide is generated based on the task description and a priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; when the user selects a second assistance mode, a second operation guide corresponding to a preset task selected by the user is determined; and the first operation guide or the second operation guide is displayed to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide, thereby improving the flexibility and accuracy of the system response, improving the real-time performance of the system, improving the personalization of the recommended response, improving the effectiveness and advancement of the operation assistance, and improving the practicality of the operation assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of an operation assistance method provided in an embodiment of the present application; Figure 2 is a flowchart of another operation assistance method provided in an embodiment of the present application; Figure 3 is a structural diagram of an operation assistance system provided in an embodiment of the present application; Figure 4 is a structural schematic diagram of an operation assisting device provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0018] The following, in conjunction with the accompanying drawings, describes in detail the operation assistance method, device, electronic device, storage medium and program product provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0019] Figure 1 An operation assistance method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: Step 102: When the user selects the first assistance mode, obtain a task description input by the user.

[0020] The user downloads an APP for accessing the office cloud computer on a personal PC or mobile device, then opens the APP to connect to the cloud computer and run the software, while turning on the intelligent software assistance system; the user selects the operating mode of the assistance system, including the first assistance mode: "Assistance mode" and the second assistance mode: "Teaching mode".

[0021] Figure 2 A schematic diagram of a process provided by an embodiment of the present application is shown below. Figure 2 The operation assistance method provided in the embodiment of the present application is described in detail. If the user chooses to run the system in the first assistance mode, namely the "assistance mode", the user needs to input a task description to the system. The background operation generation model first generates the correct operation process according to the task description and feeds back to the user.

[0022] Step 104: Generate a first operation guide according to the task description and the priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user.

[0023] The user inputs a task description into the system, and the background operation generation model first generates the correct first operation guide based on the task description input by the user and the priority queue and feeds it back to the user, where each element in the priority queue is a vector corresponding to the operation completed by the user.

[0024] In one implementation, generating a first operation guide based on the task description and the priority queue includes: generating multiple candidate operation sequences by beam search based on the task description and the priority queue; scoring each of the candidate operation sequences based on user preference and sequence length, and using the candidate operation sequence with the highest score as the first operation guide.

[0025] Specifically, the system maintains a priority queue H={ }, where the elements Map a series of user operations into a dense vector for easy storage. Each element Corresponding to a priority weight ,in Randomly initialized to a common operation combination built into the system, The average initialization method is adopted, that is, .

[0026] When the operation generation model is called for the tth time (t≥2) to generate the subsequent operation flow of a given task based on the user's historical operations, it indicates that there are steps in the user's last operation that are inconsistent with the demonstration operation. The system updates the user preference queue H according to the following rules: = Q[-1] ; =(1-1 / N) +1 / N, k=t; =(1-1 / N) , k≠t; in, For the video embedding function, the system rearranges the elements in the queue H according to the updated weights.

[0027] Whether generating from scratch or continuing the user's existing operations, the system uses a beam search method to sample a cluster of candidate operation sequences from the model output. The evaluation of each candidate sequence is carried out from two aspects: user preference and sequence length. The user preference has a higher priority than the sequence length. The system only feeds back the candidate sequence with the highest score to the user. The specific scoring method is as follows: = ; = * + ; in is the L2 norm, Calculate the length of the sequence, is the kth candidate sequence sampled.

[0028] Step 106: When the user selects the second assistance mode, determine a second operation guide corresponding to the preset task selected by the user.

[0029] Specifically, if the user chooses to run the system in the second auxiliary mode "teaching mode", the user needs to select a preset task from the system preset task list, and then determine the corresponding second operation guide according to the preset task selected by the user.

[0030] Step 108: Display the first operation guide or the second operation guide to the user, so that the user can complete the operation corresponding to the first operation guide or the second operation guide.

[0031] Specifically, if the user selects the first auxiliary mode, the software interface can be recorded (continuously screenshotted) during the operation according to the process of the first operation guide to obtain the operation process and operation results; each time the user completes an operation step, the system sends the user operation status to the background feedback model, and the feedback model determines the consistency between the user operation and the system-generated operation. If the user selects the second auxiliary mode, the corresponding operation process of the second operation guide can be retrieved from the manually pre-entered database according to the user's selection result and fed back to the user in steps. The system records the software interface (continuously screenshotted) during the user's use of the software according to the second operation guide to obtain the operation process and operation results.

[0032] In one implementation, after the first operation guide or the second operation guide is displayed to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide, it also includes: recording each operation step of the user; when the user selects the first auxiliary mode and the user's last operation step is inconsistent with the first operation guide, updating the first operation guide according to the user's last operation step; when the user selects the second auxiliary mode and the user's last operation step is inconsistent with the second operation guide, feeding back the inconsistent difference information to the user.

[0033] Specifically, each time the user completes an operation step, the system records the user's operation and sends it to the background feedback model, which determines the consistency between the user's operation and the operation guide. When the user selects the first auxiliary mode, if the operation completed by the user is consistent with the first operation guide generated by the system, the system reads the next step in the operation flow of the generated first operation guide and feeds back to the user; if the operation completed by the user is inconsistent with the first operation guide generated by the system, the recorded user operation together with the task description is forwarded to the background operation generation model, and the model regenerates the subsequent operation flow based on the user's current operation to update the first operation guide. The background feedback generation model reads the first step in the operation flow of the regenerated updated first operation guide and feeds back to the user as a prompt for the next operation.

[0034] When the user selects the second assistance mode, if the operation completed by the user is consistent with the operation in the second operation guide, the system reads the next operation suggestion from the second operation guide and feeds it back to the user. If the operation completed by the user is inconsistent with the operation in the second guide, the feedback generation model generates a difference information description accordingly and feeds it back to the user.

[0035] In an embodiment of the present application, when the user selects a first assistance mode, a task description input by the user is obtained; a first operation guide is generated based on the task description and a priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; when the user selects a second assistance mode, a second operation guide corresponding to a preset task selected by the user is determined; the first operation guide or the second operation guide is displayed to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide, thereby improving the flexibility and accuracy of the system response, improving the real-time performance of the system, improving the personalization of the recommendation system response, improving the effectiveness and advancement of the recommendation algorithm, and improving the practicality of the operation assistance.

[0036] In one implementation, recording each operation of the user includes: each time the user completes an operation, recording the operation completed by the user into a linear list, and the elements in the linear list are a screenshot sequence of the operation task completed by the user.

[0037] Before the system records the user's operation, it first maintains an empty linear table Q={ }; After the user completes k steps and clicks the "Next" button, k elements Q={ },in is a sequence of screenshots of the software's running status in the i-th operation step, is the jth frame in the screenshot sequence.

[0038] In one implementation, each time the user completes an operation and records the operation in a linear list, the method further includes: converting the last element in the linear list into a text described in a natural language; judging whether the last operation of the user is consistent with the first operation guide by comparing the similarity between the text described in the natural language and the system operation corresponding to the first operation guide, or judging whether the last operation of the user is consistent with the second operation guide by comparing the similarity between the text described in the natural language and the system operation corresponding to the second operation guide.

[0039] Before judging the consistency between the user operation and the system (preset / generated) operation, the feedback generation model first translates the last element Q[-1] in the user operation record table Q into text form, and then judges whether the two are consistent by comparing the similarity between the user's last operation described in natural language and the first operation guide, or judges whether the user's last operation is consistent with the second operation guide by comparing the similarity between the text described in natural language and the system operation corresponding to the second operation guide. In this way, the consistency of the user operation and the operation guide can be checked.

[0040] In one implementation, displaying the first operation guide or the second operation guide to the user includes: displaying the first operation guide or the second operation guide in a description manner combining brief text with a small-resolution graphics interchange format image.

[0041] Specifically, the operation prompt information of the first operation guide or the second operation guide fed back to the user is given in the form of a combination of short text and a small-resolution GIF image, and is displayed floatingly on the client desktop outside the software workspace.

[0042] In addition, the present invention also provides a professional software auxiliary system for cloud computers, the structure of which is as follows: Figure 3 As shown, including: Mode selection module: used to provide users with modes for running the software auxiliary system after they start using professional software, including a first auxiliary mode: "auxiliary mode" and a second auxiliary mode: "teaching mode".

[0043] User status recording module: used to record the running status of the software while the user is using the software, and insert it into the linear table Q in the form of a screenshot sequence. In addition, after the user confirms that a step has been completed, the module also needs to send the last element in the table Q together with the operation description of the corresponding step given by the system to the background feedback generation model.

[0044] Model fine-tuning module: used to perform supervised fine-tuning on the background model. This module trains the background model, including: 1. Training the feedback generation model to translate the user operations described in the video into natural language; 2. Training the feedback generation model to compare the differences between the query operation sequence and the key operation sequence, and describe the detected differences in brief natural language; 3. Training the operation generation model to generate the corresponding operation process and its effects based on the task described in natural language, where the operation process is described in brief natural language, and the visual effect corresponding to an operation step is described by a small-resolution GIF image; 4. Training the operation generation model to generate subsequent operation processes and effects based on the task description, user historical operations, and the current software status.

[0045] Model scheduling module: used to dynamically call the background model according to the operation mode selected by the user during the operation of the software-assisted system, including: calling the feedback generation model to determine the consistency of the user operation with the first operation guide or the second operation guide, and calling the operation generation model in the "assisted mode" to generate a complete operation process from scratch according to the task description, or generate subsequent operation processes according to the user's current operation.

[0046] Feedback explicit module: It is used to read the operation process entered manually in advance or the operation process adaptively generated by the operation generation model step by step, and display such information in real time on the client display. In addition, this module automatically obtains the resolution of the client display and automatically adjusts the font, font size (resolution) and position of the text (image) information that needs to be explicit, so as not to hinder the user's normal use of the software during the auxiliary process.

[0047] In one embodiment, the model fine-tuning module includes: feedback generation model fine-tuning: supervised instruction fine-tuning for the feedback generation model, which uses instructions in the following format to train the feedback generation model to describe the difference between the query sequence and the key sequence in a short natural language: { Instructions: Using the key sequence as a reference, briefly describe the differences between the query sequence and the key sequence; Input: Key operation: [key operation description], Query operation: [query operation description]; Output: [Description of the difference between key operation and query operation]; } Among them, "[key operation description]" is instantiated as a manually preset operation step description in actual applications, and "[query operation description]" is instantiated as a user operation step narrated by the feedback model in the application; or, the feedback generation model is trained using instructions in the following format to narrate the user operation described in the video into natural language: { Instructions: Use short natural language to describe the software operations involved in the following video; Input: [video recording of software interface]; Output: [Software operation described in natural language]; } Operation generation model fine-tuning: It is used for supervised instruction fine-tuning of the operation generation model. This module uses instructions in the following format to train the operation generation model to generate the correct operation process and expected effect according to the task description: { Instructions: [Task Description]; Output: [Guide operation description] [effect gif image]; } Or, use the following format of instruction training operation generation model to generate subsequent operation process and effect according to task description, user historical operation, and current software status: { Instructions: [Task Description]; Input: [Historical operation description] [Software screenshot] Output: [Guide operation description] [effect gif image]; } Furthermore, the model scheduling module includes: Feedback generation model scheduling: In "teaching mode", the feedback generation model is called to determine the consistency of user operations with preset guide operations and generate feedback, or in "assistance mode", the feedback generation model is called to determine the consistency of user operations with system-generated operations and generate feedback.

[0048] Operation generation model scheduling: In "assisted mode", the operation generation model is called to generate a complete operation process guide from scratch based on the task description. When the user confirms that an operation step has been completed, the feedback generation model determines that the user operation is inconsistent with the model-generated operation and calls the operation generation model again to generate subsequent operation processes based on the user's current operation status.

[0049] User preference scheduling: Maintain a priority queue H of size N in the cloud computer. }, used to store the user's software usage preferences. If the user's operation at a certain step is inconsistent with the operation retrieved or generated by the system, the system updates the user preference queue H according to the following rules: = Q[-1] ; =(1-1 / N) +1 / N, k=t; =(1-1 / N) , k≠t; Model output scheduling: In "Assisted Mode", the system uses beam search to sample multiple candidate operation sequences from the output of the operation generation model at a time. Model output scheduling scores the candidate sequences as follows, and only the sequences with the highest scores are sent to the feedback explicit module: = ; = + ; in is the L2 norm, Calculate the length of the sequence, is the kth candidate sequence sampled.

[0050] The operation assistance method provided in the embodiment of the present application has high compatibility and strong portability. The professional software assistance system proposed in the present invention is independent of the software itself, and the model obtained through sufficient training is theoretically compatible with any software without any fine-tuning. In addition, the embodiment of the present application runs on the cloud computer side, does not occupy and rely on any resources of the client, and is truly user-friendly and plug-and-play. It is highly advanced and intelligent. The method of instruction fine-tuning proposed in the present invention accurately adapts the large model to a specific application field, which enables the present invention to more accurately generate software operation guides, provide difference descriptions, and generate operation processes that are more in line with user habits according to user preferences, thereby improving the effectiveness of the auxiliary system and user satisfaction. Personalization and customization of software. The present invention maintains a priority queue for each user to model their software usage habits, and uses it as a scoring and sorting standard for candidate operations when recommending, so that the present invention has a higher degree of personalization and customization than existing recommendation systems and general software.

[0051] It should be noted that the operation assistance method provided in the embodiment of the present application can be executed by an operation assistance device or a control module in the operation assistance device for executing the operation assistance method. In the embodiment of the present application, the operation assistance device provided in the embodiment of the present application is described by taking the operation assistance device executing the operation assistance method as an example.

[0052] Figure 4 is a schematic diagram of the structure of an operation assisting device according to an embodiment of the present application. Figure 4 As shown, the operation assistance device 400 includes: a first selection module 410 , a processing module 420 , a second selection module 43 and an operation module 440 .

[0053] The first selection module 410 is used to obtain the task description input by the user when the user selects the first assistance mode; the processing module 420 is used to generate a first operation guide based on the task description and the priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; the second selection module 430 is used to determine the second operation guide corresponding to the preset task selected by the user when the user selects the second assistance mode; the operation module 440 is used to display the first operation guide or the second operation guide to the user so that the user can complete the operation corresponding to the first operation guide or the second operation guide.

[0054] In one implementation, the processing module 420 is used to generate multiple candidate operation sequences through a beam search method based on the task description and the priority queue; score each of the candidate operation sequences based on user preference and sequence length, and use the candidate operation sequence with the highest score as the first operation guide.

[0055] In one implementation, the processing module 420 is also used to record each operation of the user; when the user selects the first auxiliary mode and the user's last operation is inconsistent with the first operation guide, the first operation guide is updated according to the user's last operation; when the user selects the second auxiliary mode and the user's last operation is inconsistent with the second operation guide, the inconsistent difference information is fed back to the user.

[0056] In one implementation, the processing module 420 is further configured to record the operations completed by the user into a linear list each time the user completes an operation, and the elements in the linear list are a screenshot sequence of the operation tasks completed by the user.

[0057] In one implementation, the processing module 420 is also used to convert the last element in the linear list into a text described in natural language; by comparing the similarity between the text described in natural language and the system operation corresponding to the first operation guide, it is determined whether the user's last operation is consistent with the first operation guide, or by comparing the similarity between the text described in natural language and the system operation corresponding to the second operation guide, it is determined whether the user's last operation is consistent with the second operation guide.

[0058] In one implementation, the processing module 420 is further configured to display the first operation guide or the second operation guide in a description manner of a combination of brief text and a small-resolution graphics interchange format image.

[0059] The operation assisting device in the embodiment of the present application may be a device, or a component, integrated circuit, or chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0060] The operation assistance device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0061] The operation assisting device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0062] like Figure 5 As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores a program or instruction that can be executed on the processor 501, and the program or instruction is implemented when executed by the processor 501: when the user selects the first auxiliary mode, obtaining a task description input by the user; generating a first operation guide based on the task description and a priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; when the user selects the second auxiliary mode, determining a second operation guide corresponding to a preset task selected by the user; and displaying the first operation guide or the second operation guide to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide.

[0063] In one implementation, multiple candidate operation sequences are generated by beam search according to the task description and the priority queue; each candidate operation sequence is scored according to user preference and sequence length, and the candidate operation sequence with the highest score is used as the first operation guide.

[0064] In one implementation, each operation of the user is recorded; when the user selects the first assistance mode and the user's last operation is inconsistent with the first operation guide, the first operation guide is updated according to the user's last operation; when the user selects the second assistance mode and the user's last operation is inconsistent with the second operation guide, the inconsistent difference information is fed back to the user.

[0065] In one implementation, each time a user completes an operation, the operation completed by the user is recorded in a linear table, and the elements in the linear table are a sequence of screenshots of the operation tasks completed by the user.

[0066] In one implementation, the last element in the linear list is converted into a text described in natural language; by comparing the similarity between the text described in natural language and the system operation corresponding to the first operation guide, it is determined whether the user's last operation is consistent with the first operation guide, or by comparing the similarity between the text described in natural language and the system operation corresponding to the second operation guide, it is determined whether the user's last operation is consistent with the second operation guide.

[0067] In one implementation, the first operation guide or the second operation guide is displayed in a description manner of a combination of brief text and a small-resolution graphics interchange format image.

[0068] The specific execution steps can refer to the various steps of the above-mentioned operation assistance method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0069] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0070] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0071] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0072] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0073] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned operation assistance method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0074] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0075] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the various processes of the embodiment of the above-mentioned operation assistance method and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0076] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0078] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An operation assisting method, characterized in that: include: When the user selects the first assistance mode, obtaining a task description input by the user; Generate a first operation guide according to the task description and the priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; In case the user selects the second assistance mode, determining a second operation guide corresponding to the preset task selected by the user; The first operation guide or the second operation guide is displayed to the user, so that the user completes the operation corresponding to the first operation guide or the second operation guide.

2. The method according to claim 1, characterized in that The step of generating a first operation guide according to the task description and the priority queue includes: According to the task description and the priority queue, a plurality of candidate operation sequences are generated by a beam search method; Each of the candidate operation sequences is scored according to user preference and sequence length, and the candidate operation sequence with the highest score is used as the first operation guide.

3. The method according to claim 1, characterized in that After displaying the first operation guide or the second operation guide to the user so that the user completes the operation corresponding to the first operation guide or the second operation guide, the method further includes: Record every step of the user's operation; When the user selects the first assistance mode and the last operation of the user is inconsistent with the first operation guide, updating the first operation guide according to the last operation of the user; When the user selects the second assistance mode and the last operation of the user is inconsistent with the second operation guide, the inconsistent difference information is fed back to the user.

4. The method according to claim 3, characterized in that The recording of each operation of the user includes: Every time the user completes an operation, the operation completed by the user is recorded in a linear table, and the elements in the linear table are a screenshot sequence of the operation task completed by the user.

5. The method according to claim 4, characterized in that After each operation completed by the user, the operation completed by the user is recorded in the linear table, and the method further includes: Convert the last element in the linear list into a text described in natural language; By comparing the similarity between the text described in natural language and the system operation corresponding to the first operation guide, it is determined whether the user's last operation is consistent with the first operation guide, or by comparing the similarity between the text described in natural language and the system operation corresponding to the second operation guide, it is determined whether the user's last operation is consistent with the second operation guide.

6. The method according to claim 1, characterized in that The displaying the first operation guide or the second operation guide to the user includes: The first operation guide or the second operation guide is displayed in a description mode of combining brief text with a small-resolution graphics interchange format image.

7. An operation process auxiliary device, characterized in that: include: A first selection module, used for obtaining a task description input by a user when the user selects a first assistance mode; A processing module, configured to generate a first operation guide according to the task description and the priority queue, wherein each element in the priority queue is a vector corresponding to an operation completed by the user; A second selection module, used for determining a second operation guide corresponding to the preset task selected by the user when the user selects the second assistance mode; The operation module is used to display the first operation guide or the second operation guide to the user, so that the user can complete the operation corresponding to the first operation guide or the second operation guide.

8. An electronic device, characterized in that: It comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the operation assistance method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the operation assistance method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the steps of the operation assistance method according to any one of claims 1 to 6.

Images