Operation guiding method, device, equipment, storage medium and computer program product

Abstract

The application provides an operation guidance method and device, equipment, a storage medium and a computer program product; the method comprises the following steps: in response to an operation guidance request for a target function of software, determining an initial state of the software; encoding processing of a requirement text included in the operation guidance request is performed to obtain text features of the requirement text, and the initial state is encoded processed to obtain state features of the initial state; the text features and the state features are fused to obtain first fusion features, and based on the first fusion features, a vector knowledge base of the software is searched, and a candidate state corresponding to a candidate state feature searched is determined as an expected state of the software; difference analysis is performed on the initial state and the expected state to obtain a difference sequence between the initial state and the expected state; based on the difference sequence, operation sequence prediction processing is performed to obtain a target operation sequence, and operation guidance is performed for the target function based on the target operation sequence.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to an operation guidance method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] As software functions become increasingly complex, users often face challenges such as high operational difficulty and learning costs. Against this backdrop, teaching guidance systems, as a key intelligent information processing application aimed at improving user experience, have been widely introduced into various software products. These systems guide users to complete specific tasks step by step through proactive prompts, step-by-step breakdowns, and real-time interaction, thereby helping users quickly become familiar with core functions and improve operational efficiency. However, teaching guidance systems in related technologies still suffer from insufficient accuracy and real-time performance in personalized guidance. Summary of the Invention

[0003] This application provides an operation guidance method, apparatus, device, storage medium, and computer program product, which realizes personalized operation guidance for target functions.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an operation guidance method, the method comprising:

[0006] In response to an operation guidance request for a target function of the software, determine the initial state of the software;

[0007] The requirement text included in the operation guidance request is encoded to obtain the text features of the requirement text, and the initial state is encoded to obtain the state features of the initial state.

[0008] The text features and the state features are fused to obtain a first fused feature. Based on the first fused feature, the vector knowledge base of the software is retrieved, and the candidate states corresponding to the retrieved candidate state features are determined as the expected states of the software. The vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature that represents each candidate state of the software.

[0009] A difference analysis is performed on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state;

[0010] Based on the difference sequence, operation sequence prediction processing is performed to obtain the target operation sequence, and operation guidance for the target function is performed based on the target operation sequence.

[0011] This application provides an operation guidance device, the device comprising:

[0012] A data acquisition module is used to determine the initial state of the software in response to an operation guidance request for a target function of the software.

[0013] The encoding module is used to encode the requirement text included in the operation guidance request to obtain the text features of the requirement text, and to encode the initial state to obtain the state features of the initial state.

[0014] The retrieval module is used to fuse the text features and the state features to obtain a first fused feature, and to retrieve the vector knowledge base of the software based on the first fused feature, and to determine the candidate state corresponding to the retrieved candidate state feature as the expected state of the software. The vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature that characterizes each candidate state of the software.

[0015] The difference analysis module is used to perform difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state;

[0016] The interaction module is used to perform operation sequence prediction processing based on the difference sequence to obtain the target operation sequence, and to provide operation guidance for the target function based on the target operation sequence.

[0017] This application provides an electronic device, the electronic device comprising:

[0018] Memory is used to store executable instructions or computer programs.

[0019] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the operation guidance method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the operation guidance method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the operation guidance method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] By encoding the requirement text included in the operation guidance request, textual features that accurately express the user's intent can be extracted. Simultaneously, by encoding the initial state of the software, state features reflecting the current software environment can be generated. Then, the textual features and state features are fused into a first fusion feature, which is then used to search in a vector knowledge base. This allows the candidate states corresponding to the candidate state features to be identified as the desired states. Through feature fusion and knowledge base retrieval, a deep analysis of user needs and state matching are achieved. Subsequently, a difference sequence is generated by performing difference analysis on the initial state and the desired state. Based on the operation sequence prediction processing of the difference sequence, the target operation sequence is finally generated. This enables the operation guidance to dynamically adapt to the user's real-time operation environment and personalized needs, thereby improving the execution efficiency of the operation guidance. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of the operation guidance system provided in the embodiments of this application;

[0025] Figure 2 This is a schematic diagram of the structure of an electronic device for operation guidance provided in an embodiment of this application;

[0026] Figure 3A This is a first flowchart illustrating the operation guidance method provided in this application embodiment;

[0027] Figure 3B This is a second flowchart illustrating the operation guidance method provided in the embodiments of this application;

[0028] Figure 3C This is a third flowchart illustrating the operation guidance method provided in the embodiments of this application;

[0029] Figure 3D This is a schematic diagram of the fourth process of the operation guidance method provided in the embodiments of this application;

[0030] Figure 4 This is a schematic diagram of the operation guidance method provided in the embodiments of this application;

[0031] Figure 5 This is a fifth flowchart illustrating the operation guidance method provided in the embodiments of this application;

[0032] Figure 6 This is a schematic diagram of the sixth process of the operation guidance method provided in the embodiments of this application.

[0033] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0039] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0041] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0042] 2) Snapshot: A snapshot is a static record that captures and persists the complete running state, data configuration, and user interface layout of a software system at a specific point in time. Snapshots systematically save all instantaneous information, including memory data, register values, open file descriptors, process execution status, and user interface component tree.

[0043] 3) Human-computer interaction interface: An interface used to provide human-computer interaction functions / display operation guidance information. Examples include graphical user interfaces (GUIs), augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing viewing of stereoscopic images without special glasses), multimodal interfaces (interfaces combining multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-computer interfaces (BMIs).

[0044] 4) Neural Network Model: A neural network model is a computational model inspired by the human brain's nervous system. It consists of a large number of interconnected artificial neurons connected by weights. Through training, a neural network model can learn the complex nonlinear mapping relationship between input data and output target. Neural network models have the ability to automatically extract features from data and perform distributed parallel information processing, thereby achieving the approximation of high-dimensional complex functions such as classification, regression, and pattern recognition. The parameter adjustment process of a neural network model usually relies on optimization methods such as backpropagation algorithm and gradient descent.

[0045] 5) Large Language Model (LLM): These are language models with a huge number of parameters, usually containing hundreds of millions of parameters. They are used to handle complex natural language processing tasks, such as text generation, translation, and question answering. The advantage of large language models is that they can capture the complexity and nuances of language, thereby generating more natural, coherent and in-depth text.

[0046] 6) Knowledge Graph: A knowledge graph is a structured graph model used to represent and store knowledge. It consists of entities and relationships. Entities are nodes in the knowledge graph, representing specific things or concepts (such as people, places, events, etc.), while relationships are edges that connect these entities, representing semantic connections between entities (such as "belongs to", "located in", "author", etc.).

[0047] 7) Retrieval-Augmented Generation (RAG): Retrieval-Augmented Generation is a technical framework that combines information retrieval and text generation. The framework first retrieves information fragments related to the current task or query from external knowledge sources, and then uses these retrieved information as supplementary context, inputting them together with the original input into the language model to generate the final output. It can effectively utilize real-time or domain-specific information in external knowledge bases, thereby significantly improving the factual accuracy, information richness and overall quality of the generated text, while also helping to reduce the risk of the model generating fictitious or outdated content.

[0048] 8) Prompt: This refers to the input text or question used to guide the language model to generate specific output. The design of the prompt helps to clarify the user's needs, enabling the language model to provide output information that is highly relevant to the user's input.

[0049] 9) Application Programming Interface (API): A set of predefined functions, protocols and tools used to build software and applications, allowing different software systems to interact with each other and share data and functions without needing to understand the internal workings.

[0050] This application provides an operation guidance method, apparatus, electronic device, computer-readable storage medium, and computer program product, which realize personalized operation guidance for target functions.

[0051] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the operation guidance system provided in the embodiment of this application. In the operation guidance system 10 provided in the embodiment of this application, in order to support an operation guidance application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0052] Terminal 400 can be used to obtain an operation guidance request carrying the required text.

[0053] In some embodiments, an operation guidance plugin may be embedded in the client running in the terminal 400 to implement the operation guidance method locally on the client. For example, the terminal 400 calls the operation guidance plugin to implement the operation guidance method, which, in response to an operation guidance request for the target function of the software, determines the initial state of the software; encodes the requirement text included in the operation guidance request to obtain the text features of the requirement text, encodes the initial state to obtain the state features of the initial state; fuses the text features and the state features to obtain a first fused feature, and based on the first fused feature, searches the vector knowledge base of the software, and determines the candidate state corresponding to the retrieved candidate state features as the desired state of the software; performs a difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state; performs operation sequence prediction processing based on the difference sequence to obtain a target operation sequence, and performs operation guidance for the target function based on the target operation sequence.

[0054] In some embodiments, after the terminal 400 obtains an operation guidance request carrying a requirement text, it calls the operation guidance interface of the server 200 (which can be provided as a cloud service, i.e., an operation guidance service). The server 200 implements the operation guidance method through an operation guidance plugin, determines the initial state of the software, encodes the requirement text to obtain its text features, encodes the initial state to obtain its state features, fuses the text features and state features to obtain a first fused feature, and searches the software's vector knowledge base based on the first fused feature, determining the candidate state corresponding to the retrieved candidate state features as the desired state of the software; performs difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state; performs operation sequence prediction processing based on the difference sequence to obtain a target operation sequence, and returns the target operation sequence to the terminal 400. The terminal 400 then provides operation guidance for the target function based on the target operation sequence.

[0055] The operation guidance method of this application embodiment can be applied to various application scenarios in various fields, as described in detail below.

[0056] For example, in a software teaching guidance system, when users are learning to use the multi-source data fusion function (i.e., the target function) of data analysis software (i.e., the software), the error rate of user operation increases because the operation manuals in related technologies only provide fixed operation procedures. To address the above-mentioned technical problems in related technologies, in this embodiment, in response to the terminal 400 receiving the user's input request text "merge sales table and inventory table", the request text is encapsulated into an operation guidance request for the multi-source data fusion function of the software, and the operation guidance interface of the server 200 is called. The server 200 determines the initial state of the data analysis software, then encodes the request text to obtain text features, encodes the initial state to obtain state features, and merges the text features and state features to obtain a first fusion feature. Next, based on the first fusion feature, a vector knowledge base is retrieved to obtain the desired state. Subsequently, a difference analysis is performed on the initial state and the desired state to generate a difference sequence. Finally, based on the difference sequence, a target operation sequence is predicted and generated and returned to the terminal 400. This achieves accurate derivation from the dynamic initial state to the target operation path, improves the user's operation accuracy, and effectively reduces the learning threshold of the multi-source data fusion function.

[0057] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0058] Terminal 400 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0059] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device for operation guidance provided in an embodiment of this application. Figure 2 The electronic device 500 shown can be Figure 1The terminal 400 or server 200, and the electronic device 500 include: at least one processor 510, memory 550, and at least one network interface 520. The various components in server 200 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0060] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0062] In some embodiments, when the operation guidance task is independently completed by the terminal 400, the server 200 provided in this application embodiment does not include the user interface 530.

[0063] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0064] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0065] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0066] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0067] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0068] Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530.

[0069] In some embodiments, when the operation guidance task is completed independently by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.

[0070] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532; in some embodiments, when the operation guidance task is independently completed by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.

[0071] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An operation guidance device 555 stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 5551, an encoding module 5552, a retrieval module 5553, a difference analysis module 5554, and an interaction module 5555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0072] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the operation guidance method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0073] As mentioned above, the electronic device implementing the operation guidance method of this application embodiment can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below. See [link to relevant documentation]. Figure 3A , Figure 3A This is a first flowchart illustrating the operation guidance method provided in the embodiments of this application, which will be combined with... Figure 3A The steps shown are explained.

[0074] In step 101, in response to an operation guidance request for the target function of the software, the initial state of the software is determined.

[0075] The operation boot request is a standardized data packet used to request operation boot for the target function of the software. The initial state refers to the state of the software at the time the operation boot request is received, which can be a software snapshot, memory image, or system log, etc., without limitation.

[0076] As an example, before "determining the initial state of the software in response to an operation guidance request for the software's target function", the system receives operation guidance request information for the software's target function input by the user. When the operation guidance request information is text-based, it is determined as a request text and encapsulated into an operation guidance request for the software's target function. When the operation guidance request information is audio-based, it undergoes speech recognition processing to obtain a request text and is then encapsulated into an operation guidance request for the software's target function.

[0077] In response to an operation guidance request for the target function of the software, the system acquires the state of each running session window, the user interface (UI) of each running session window, the global configuration information of the software, the object behavior data of the software, user preference data, etc., and serializes the above data into the initial state of the software. The state of the session window includes the content displayed in the session window, the size and position of the session window, and the values ​​of the various states of the data modules corresponding to the session window. The global configuration information of the software includes the file save path and the configuration information such as the scaling of the software window. The object behavior data of the software includes the usage time of the software and the level of proficiency in using it.

[0078] For example, when configuring a network download task, a user can input the operation guidance requirement information "how to set up multi-threaded high-speed download" through a text input box. This operation guidance requirement information is defined as a requirement text, which is then encapsulated into an operation guidance request for the software's target function. In response to the operation guidance request for the software's target function, if the software only contains a single running download window (i.e., a session window), the software obtains the content displayed in the download window, the size and position of the download window, the maximum download speed of the data module corresponding to the download window, the concurrency of the data module, the proxy settings of the data module, the user interface of the download window (which can be in HTML or DOM tree format), the software's global configuration information, the software's object behavior data, user preference data, etc. Finally, the above data is serialized into the software's initial state (which can be in JSON or XML format).

[0079] In step 102, the requirement text included in the operation guidance request is encoded to obtain the text features of the requirement text, and the initial state is encoded to obtain the state features of the initial state.

[0080] As an example, historical operation guidance requests for any function of the software and object behavior data of the software are obtained. Based on the historical requirement text of the historical operation guidance requests and object behavior data, the requirement text is converted into a second prompt word. The second prompt word is encoded to obtain text features. The initial state can be encoded based on an independent text encoder (e.g., CLIP's text encoder, Sentence-BERT, or E5 model) to obtain the initial state features. Alternatively, the initial state can be encoded based on the text encoding module in a large language model (LLM, such as the GPT series model, ChatGLM model, or LLaMA series model) to obtain the initial state features.

[0081] In some embodiments, see Figure 3B , Figure 3B This is a second flowchart illustrating the operation guidance method provided in the embodiments of this application. Figure 3A The step 102 shown, "encoding the requirement text included in the operation guidance request to obtain the text features of the requirement text", can be achieved through the following steps 1021 to 1024, which are explained in detail below.

[0082] In step 1021, historical operation guidance requests for any function of the software and object behavior data of the software are obtained.

[0083] Among them, historical operation guidance requests refer to operation guidance requests for any function of the software within a preset time range, and object behavior data refers to the quantitative record of the user's use of the software. Object behavior data may include the duration of software use and the user's proficiency in using the software.

[0084] In step 1022, based on the historical requirement text of the historical operation guidance request and the object behavior data, prompt words are constructed from the requirement text to obtain the first prompt word.

[0085] In some embodiments, Figure 3B Step 1022 shown can be achieved through the following processing: encoding the requirement text, historical requirement text, and object behavior data respectively to obtain the first encoding feature of the requirement text, the second encoding feature of the historical requirement text, and the third encoding feature of the object behavior data; integrating the second encoding feature and the third encoding feature into the first encoding feature to obtain the second fusion feature; and decoding the second fusion feature to obtain the first prompt word.

[0086] As an example, the requirement text is encoded to obtain the first encoded feature of the requirement text. The historical requirement text is encoded to obtain the second encoded feature of the historical requirement text. The object behavior data is encoded to obtain the third encoded feature of the object behavior data. The first, second, and third encoded features are fused to obtain the second fused feature. The second fused feature is then decoded based on an independent text decoder (e.g., CLIP's text decoder or Transformer decoder) to obtain the first prompt word. Alternatively, the second fused feature can be decoded based on the text decoding module in a large language model to obtain the first prompt word. This is not limited here. The encoding of the requirement text, historical requirement text, and object behavior data separately can be referred to the implementation of encoding the initial state above, which will not be elaborated here. The first, second, and third encoded features can be fused through linear fusion methods such as matrix concatenation or weighted summation. Alternatively, the first, second, and third encoded features can be fused through multiple cascaded cross-attention layers. This is not limited here.

[0087] In this embodiment, the requirement text, historical requirement text, and object behavior data are encoded separately to obtain a first encoded feature, a second encoded feature, and a third encoded feature, preserving the uniqueness of each data source. Subsequently, the three types of encoded features are fused into a unified second fused feature through linear fusion methods such as matrix concatenation and weighted summation, or through deep feature interaction via multiple cascaded cross-attention layers. The second fused feature simultaneously contains the current requirement, historical behavior patterns, and user habits. Finally, based on an independent text decoder or the text decoding module in a large language model, the second fused feature is decoded to generate the first prompt word, effectively improving the accuracy and contextual relevance of prompt word generation and laying a solid foundation for subsequent prompt word optimization.

[0088] In step 1023, the first prompt word is adjusted based on the preset prompt word template to obtain the second prompt word.

[0089] The prompt template is a set of general and reusable prompt frameworks used to standardize and unify the format and content of prompts.

[0090] As an example, the content of the first prompt is filtered and reconstructed based on the constraints contained in the prompt template to obtain the reconstructed first prompt. The constraints refer to the rules and standards for filtering and reconstructing the first prompt. Based on the role limitation information contained in the prompt template, the reconstructed first prompt is subjected to a role perspective transformation to obtain the first prompt after the role perspective transformation. Based on the language style elements contained in the prompt template, the language style of the first prompt after the role perspective transformation is rendered to obtain the second prompt.

[0091] For example, when the first prompt is "How to reduce the power consumption of a smartphone?", the constraints could include a content focus rule ("retain only core concepts related to the technical solution") and a terminology standardization rule ("convert colloquial or broad descriptions into more specific and professional technical terms"). During the process of filtering and reconstructing the content of the first prompt based on the constraints contained in the prompt template, non-core words such as "how" and "?" are filtered out based on the content focus rule. Based on the terminology standardization rule, "reduce...power consumption" in the first prompt is standardized to "power consumption optimization," and "smartphone" is broken down and specified into subsystem technologies such as "battery technology" and "system scheduling," resulting in the reconstructed first prompt. The prompts are: "Energy consumption optimization," "Battery technology," and "System scheduling." Then, based on the role constraint "Senior Hardware Engineer," the perspective of the reconstructed first prompt is changed, resulting in the prompt phrase: "From a hardware engineering perspective, systematically analyze the key technical paths for smartphone energy consumption optimization, focusing on energy efficiency control schemes for core components such as battery characteristics, system scheduling, and radio frequency modules." Finally, based on the language style element "rigorous academic report style," the language style of the first prompt phrase after the perspective change is refined, resulting in the prompt phrase: "Please, as a senior hardware engineer, determine the path for smartphone energy consumption optimization from three technical levels: lithium-ion battery characteristics, system scheduling algorithms, and radio frequency module control."

[0092] In step 1024, the second prompt word is encoded to obtain text features.

[0093] As an example, the second prompt word can be encoded using a separate text encoder to obtain text features, or it can be encoded using the text encoding module in a large language model to obtain text features; no limitation is made here.

[0094] In this embodiment, by acquiring historical operation guidance requests and object behavior data including software usage time and user proficiency within a preset time range, a comprehensive data foundation is provided for subsequent analysis. Based on the aforementioned historical requirement text and object behavior data, prompt words are constructed for the requirement text to generate the first prompt word, ensuring the accuracy and relevance of the prompt word source. At the same time, the first prompt word is filtered and reconstructed using the constraints contained in the preset prompt word template. Then, the role perspective is switched through role limitation information, and style rendering is performed based on language style elements. Finally, a second prompt word with standardized format and unified content is obtained, effectively improving the quality and applicability of the prompt word.

[0095] In some embodiments, during execution Figure 3ABefore the step 103 shown, "retrieving the software's vector knowledge base based on the first fusion feature", the following processing can also be performed: obtaining the software's product knowledge base; for each descriptive text contained in the product knowledge base, performing state prediction processing based on the descriptive text to obtain the software's candidate state; encoding the candidate state to obtain the candidate state's state feature; creating an initial vector knowledge base, and storing the state feature of the candidate state corresponding to each descriptive text into the initial vector knowledge base to obtain the vector knowledge base.

[0096] The product knowledge base contains descriptive text for each function of the software, and the candidate state is the state of the software after executing the function corresponding to the descriptive text.

[0097] As an example, from at least one trusted data source, obtain descriptive text for each function of the software. The descriptive text describes the purpose, operation process, applicable scenarios, and usage limitations of a specific function. Trusted data sources can include official documentation, Q&A communities, user logs, etc., without limitation. For each descriptive text contained in the product knowledge base, perform the following processing: Based on a preset third prompt word and the descriptive text, call a large language model to perform state prediction processing. The output of the large language model is determined as the candidate state of the software. This can be done using a separate text encoder to encode the candidate state and obtain its state features, or it can be done using the text encoding module in the large language model. Without limitation, after performing the above processing for each descriptive text contained in the product knowledge base, create an initial vector knowledge base and store the state features of the candidate state corresponding to each descriptive text in the initial vector knowledge base, resulting in a vector knowledge base.

[0098] For example, for the software's data backup and recovery function, the corresponding description text could be: "This function allows users to package the project data in the current workspace into a compressed file for local or cloud backup, and can select a specific backup version to perform a full recovery when needed. It is suitable for scenarios such as system reinstallation, device replacement, or accidental data deletion. Before use, ensure that the storage medium has sufficient space, and the recovery operation will overwrite all current unsaved modifications." The preset third prompt could be: "Based on the provided function description, predict the state that the software may enter after executing this function. Please focus on describing the possible state of the session window, the user interface of the session window, the software's global configuration information, and the software's object behavior data." Based on the preset third prompt and description text, the large language model is called to obtain the candidate states of the software, and the state features of the candidate states are stored in the initial vector knowledge base.

[0099] In this embodiment, by obtaining descriptive text from a trusted data source, an accurate and reliable information source is provided for building the product knowledge base, ensuring the standardization and completeness of the functional description. Based on the preset third prompt words and descriptive text, a large language model is called for state prediction processing, which can systematically generate candidate states including dimensions such as session window state, user interface, global configuration information, and object behavior data. The candidate states are encoded by a text encoder or the text encoding module in the large language model to obtain state features with strong representation capabilities, laying a solid foundation for subsequent retrieval operations. By storing the state features corresponding to each descriptive text into an initial vector knowledge base, a structured vector knowledge base is formed, realizing the transformation from unstructured descriptive text to structured state features, ensuring the consistency of state features, and providing reliable data support for software state prediction and operation guidance.

[0100] In step 103, the text features and state features are fused to obtain the first fused feature. Based on the first fused feature, the vector knowledge base of the software is retrieved, and the candidate state corresponding to the retrieved candidate state feature is determined as the expected state of the software.

[0101] The vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature that represents each candidate state of the software.

[0102] As an example, the text features and state features can be fused using the context fusion unit in the large language model to obtain a first fused feature, and the desired state of the software can be retrieved from the software's vector knowledge base based on the first fused feature; alternatively, step 1022 shown above can be referred to to fuse the text features and state features to obtain a first fused feature, and the desired state of the software can be retrieved from the software's vector knowledge base based on the first fused feature, without limitation here.

[0103] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the operation guidance method provided in the embodiments of this application. Figure 3A The step 103 shown, "based on the first fusion feature, retrieve the vector knowledge base of the software and determine the candidate state corresponding to the retrieved candidate state feature as the expected state of the software", can be implemented through the following steps 1031 to 1032, which are explained in detail below.

[0104] In step 1031, at least one candidate state feature is determined from the multiple candidate state features contained in the vector knowledge base, whose first similarity with the first fused feature is greater than a similarity threshold.

[0105] As an example, the first similarity between each candidate state feature in the multiple candidate state features contained in the vector knowledge base and the first fusion feature is determined, and at least one candidate state feature whose first similarity with the first fusion feature is greater than a preset similarity threshold is determined. The first similarity can be cosine similarity, Euclidean distance or Manhattan distance, etc., and is not limited here.

[0106] For example, with a preset similarity threshold of 0.9, candidate state features with a first similarity greater than the similarity threshold of 0.9 are determined from multiple candidate state features contained in the vector knowledge base: candidate state feature 1, candidate state feature 2, and candidate state feature 3. Among them, the first similarity between candidate state feature 1 and the first fusion feature is 0.91, the first similarity between candidate state feature 2 and the first fusion feature is 0.95, and the first similarity between candidate state feature 3 and the first fusion feature is 0.96.

[0107] In step 1032, the desired state of the software is determined based on at least one candidate state feature and the confidence level of each of the at least one candidate state feature.

[0108] Here, confidence level refers to the degree of trustworthiness of the data source corresponding to the candidate state features.

[0109] As an example, when the number of candidate state features in the vector knowledge base whose first similarity to the first fusion feature is greater than the similarity threshold is one, the candidate state corresponding to the candidate state feature is determined as the expected state of the software; when the number of candidate state features in the vector knowledge base whose first similarity to the first fusion feature is greater than one, the expected state of the software is determined based on at least one candidate state feature and the confidence level of each candidate state feature among the at least one candidate state feature.

[0110] Continuing from the previous example, from the multiple candidate state features contained in the vector knowledge base, candidate state features with a first similarity greater than the similarity threshold of 0.9 with the first fused feature are identified: Candidate State Feature 1, Candidate State Feature 2, and Candidate State Feature 3. Candidate State Feature 1 is obtained by performing state prediction processing based on the descriptive text in the data source "Official Documentation," Candidate State Feature 2 is obtained by performing state prediction processing based on the descriptive text in the data source "Question and Answer Community," and Candidate State Feature 3 is obtained by performing state prediction processing based on the descriptive text in the data source "User Logs." When the trust level of the data source "Official Documentation" is 1 (completely trustworthy), the trust level of the data source "Question and Answer Community" is 0.8, and the trust level of the data source "User Logs" is 0.9, the confidence level of Candidate State Feature 1 is 1, the confidence level of Candidate State Feature 2 is 0.8, and the confidence level of Candidate State Feature 3 is 0.9.

[0111] In this embodiment, by calculating the first similarity between the first fusion feature and each candidate state feature in the vector knowledge base, and filtering out candidate state features that exceed the similarity threshold, the precise filtering of massive state features is effectively achieved, ensuring the relevance of subsequent processing data. Based on at least one candidate state feature obtained from the filtering and its corresponding confidence level, the desired state is determined. When there is a single candidate state feature, its corresponding state is directly used as the desired state. When there are multiple candidate state features, the confidence level of each candidate state feature is comprehensively considered. This ensures both the processing efficiency in the case of a single candidate state and the decision reliability in the case of multiple candidate states. Through the dual protection of similarity threshold and confidence level, the accuracy and robustness of the determination of the desired state are significantly improved, providing a solid foundation for subsequent software state prediction and operation guidance.

[0112] In some embodiments, Figure 3C Step 1032 shown can be implemented by the following process: for each candidate state feature in at least one candidate state feature, determine a first similarity between the candidate state feature and the first fusion feature; adjust the first similarity based on the confidence level to obtain a second similarity; decode the candidate state feature corresponding to the maximum second similarity to obtain the desired state of the software.

[0113] As an example, for each candidate state feature among at least one candidate state feature, the following processing is performed: a first similarity is obtained between the candidate state feature and a first fused feature; the product of the first similarity and the confidence level of the candidate state feature is determined as a second similarity. The desired state of the software can be obtained by decoding the candidate state feature corresponding to the maximum second similarity using an independent text decoder, or by decoding the candidate state feature corresponding to the maximum second similarity using the text decoding module in a large language model; neither is limited here.

[0114] Continuing from the previous example, from the multiple candidate state features contained in the vector knowledge base, candidate state features with a first similarity greater than the similarity threshold of 0.9 with the first fused feature are identified: candidate state feature 1, candidate state feature 2, and candidate state feature 3. The product of the first similarity of candidate state feature 1 (0.91) and its confidence level (1), which is 0.91, is determined as the second similarity. The product of the first similarity of candidate state feature 2 (0.95) and its confidence level (0.8), which is 0.76, is determined as the second similarity. The product of the first similarity of candidate state feature 3 (0.96) and its confidence level (0.9), which is 0.864, is determined as the second similarity. The candidate state feature 1 corresponding to the largest second similarity is decoded to obtain the expected state of the software.

[0115] In this embodiment, a second similarity is obtained by calculating the first similarity between the candidate state feature and the first fused feature, and multiplying it by the confidence level of the candidate state feature itself. This effectively integrates the semantic correlation between features and the reliability of each candidate feature, thereby achieving accurate quantitative evaluation of the candidate state feature. By selecting the candidate state feature corresponding to the maximum second similarity and using a text decoder or the text decoding module in a large language model for decoding processing, the desired state of the software is finally obtained. This significantly improves the accuracy and robustness of state selection and ensures that the final output desired state result has higher credibility and representativeness.

[0116] In step 104, a difference analysis is performed on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state.

[0117] As an example, a first knowledge graph is constructed based on the initial state, and a second knowledge graph is constructed based on the desired state. By comparing the differences between the first knowledge graph corresponding to the initial state and the second knowledge graph corresponding to the desired state, a difference sequence is obtained.

[0118] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the operation guidance method provided in the embodiments of this application. Figure 3A Step 104 shown can be implemented through steps 1041 to 1043, which are explained in detail below.

[0119] In step 1041, a first knowledge graph is constructed using the first attribute information contained in the initial state as the first node and the relationship between the first attribute information as the first edge.

[0120] As an example, the initial state contains multiple types of first attribute information. These multiple types include the type corresponding to the state of the session window, the type corresponding to the user interface of the session window, the type corresponding to the global configuration information of the software, and the type corresponding to the object behavior data of the software. Based on the predefined relational mapping rules and the logical dependencies between the data, the relationship between the first attribute information is determined. The first knowledge graph is constructed with the first attribute information as the first node and the relationship between the first attribute information as the first edge.

[0121] For example, the type of the first node A is the type corresponding to the state of the session window, and the type of the first node B is the type corresponding to the user interface of the session window. The first node A can be used to represent any function in the session window, and the first node B can be used to represent a button component in the session window. Then the relationship between the first node A and the first node B can be "implementation", that is, "first node B - implementation → first node A".

[0122] In step 1042, a second knowledge graph is constructed using the second attribute information contained in the desired state as the second node and the relationship between the second attribute information as the second edge.

[0123] As an example, the construction of the second knowledge graph can refer to step 1041 shown above, which will not be repeated here.

[0124] In step 1043, a difference sequence is determined based on the differences between the first knowledge graph and the second knowledge graph.

[0125] As an example, a difference sequence is constructed by comparing the first difference between nodes of the same type in the first knowledge graph and the second knowledge graph, comparing the second difference between corresponding edges in the first knowledge graph and the second knowledge graph, and based on at least one first difference and at least one second difference.

[0126] In this embodiment, a first knowledge graph is systematically constructed by using the first node as the first node corresponding to the session window state, user interface, global configuration information, and object behavior data contained in the initial state, and establishing the first edge according to predefined relation mapping rules and logical dependencies. Correspondingly, a second knowledge graph is constructed in the same way based on the second attribute information in the desired state. By comparing the first difference between nodes of the same type in the two knowledge graphs and the second difference between the corresponding edges, and constructing a difference sequence based on this, the accurate capture and quantitative description of the structured differences between the initial state and the desired state are achieved.

[0127] In some embodiments, Figure 3D Step 1043 shown can be implemented through the following processing: For each first node in the first knowledge graph, a second node of the same type as the first node is determined from the second knowledge graph. If the first attribute information corresponding to the first node is different from the second attribute information corresponding to the second node, the difference between the first attribute information corresponding to the first node and the second attribute information corresponding to the second node is determined as the first difference. For each first edge in the first knowledge graph, a second edge is determined from the second knowledge graph. The two second attribute information corresponding to the two second nodes connected by the second edge is the same as the two first attribute information corresponding to the two first nodes connected by the first edge. If the relationship represented by the first edge is different from the relationship represented by the second edge, the difference between the relationship represented by the first edge and the relationship represented by the second edge is determined as the second difference. In response to the fact that the number of both the first difference and the second difference is not zero, a difference sequence is constructed based on at least one first difference and at least one second difference.

[0128] As an example, for each first node in the first knowledge graph, the following processing is performed: From the second knowledge graph, a second node of the same type as the first node is identified. If the first attribute information corresponding to the first node is different from the second attribute information corresponding to the second node, the difference between the first attribute information corresponding to the first node and the second attribute information corresponding to the second node is determined as the first difference. For example, if the "permission level" attribute value of the first node in the first knowledge graph is "administrator" and the "permission level" attribute value of the second node in the second knowledge graph is "ordinary user", then "permission level: administrator → ordinary user" is recorded as the first difference.

[0129] For each first edge in the first knowledge graph, perform the following processing: Determine the second edge from the second knowledge graph whose two second attribute information corresponding to the two connected second nodes is the same as the two first attribute information corresponding to the two connected first nodes of the first edge. If the relationship represented by the first edge is different from the relationship represented by the second edge, the difference between the relationship represented by the first edge and the relationship represented by the second edge is determined as the second difference. For example, if there is a "subordinate" relationship between the first node "User A" and the first node "Department X" in the first knowledge graph, while the relationship between the first node "User A" and the first node "Department X" in the second knowledge graph is "supervised", then record "User A-Department X: Subordinate → Supervised" as the second difference.

[0130] When the number of both the first and second differences is not zero, a difference sequence is constructed based on at least one first difference and at least one second difference; when the number of the first difference is zero and the number of the second difference is not zero, a difference sequence is constructed based on at least one second difference; when the number of the first difference is not zero and the number of the second difference is zero, a difference sequence is constructed based on at least one first difference.

[0131] In this embodiment, by systematically comparing the first knowledge graph and the second knowledge graph, for each first node, a second node of the same type is located in the second knowledge graph. If the corresponding attribute information is different, the difference in attribute information is identified as the first difference. At the same time, for each first edge, the corresponding second edge connecting nodes with the same attribute information is found in the second knowledge graph. If the relationship represented by the two is inconsistent, the inconsistency in the relationship is identified as the second difference. Subsequently, based on the existence of the first difference and the second difference, a difference sequence is flexibly constructed to ensure that the differences in attribute information at the node level and the differences in relationship logic at the edge level are fully captured and structurally integrated, thereby generating a complete and accurate difference sequence that represents all the key differences between the two knowledge graphs.

[0132] In step 105, operation sequence prediction processing is performed based on the difference sequence to obtain the target operation sequence, and operation guidance for the target function is performed based on the target operation sequence.

[0133] As an example, operation sequence prediction processing is performed based on the difference sequence to obtain the target operation sequence. The target operation sequence refers to a set of ordered operation instructions that need to be executed to transition from the initial state to the desired state. Each operation instruction in the target operation sequence is parsed in turn to obtain the identification information and interaction action of the interface element to be operated corresponding to each operation instruction. Based on the identification information and interaction action of the interface element to be operated corresponding to each operation instruction, the interface element to be operated in the software's user interface is dynamically rendered and prompted. Specifically, the target area where the interface element to be operated is located is highlighted, and the non-target area is darkened. At the same time, a prompt box containing operation instructions is displayed. This process relies on a preset UI configuration file template. By modifying the values ​​of fields such as the highlighted area identifier, shadow intensity, and prompt text in the configuration file, precise control of various interface elements can be achieved to guide the operation of the target function.

[0134] In some embodiments, Figure 3A The step 105 shown, "performing operation sequence prediction based on the difference sequence to obtain the target operation sequence", can be achieved through the following process: calling a neural network model based on the difference sequence, performing operation sequence prediction processing through the neural network model to obtain the target operation sequence.

[0135] As an example, the difference sequence is determined as the input of the neural network model, so as to perform operation sequence prediction processing through the neural network model to obtain the output information of the neural network model, and the output information of the neural network model is determined as the target operation sequence. The neural network model can be a recurrent neural network, a long short-term memory network, or a large language model, etc., without limitation.

[0136] In some embodiments, during execution Figure 3A After step 105 shown, the following processing can also be performed: during the execution of the operation guidance, user operation data for the operation guidance is collected, and a first reward value is determined based on the user operation data; in response to receiving evaluation information for the operation guidance, a second reward value is determined based on the evaluation information; the first reward value and the second reward value are fused to obtain a third reward value, and the parameters of the neural network model are adjusted based on the third reward value.

[0137] As an example, during the execution of the operation guidance, user operation data is collected. This data refers to the records of the user's actual actions during the guidance process, including click count, operation duration, whether certain instructions were skipped, and whether instructions were completed. Based on this data, a first reward value is determined. This first reward value is a quantitative indicator calculated from the user operation data. For example, if the user completes all instructions according to the guidance, they will receive a higher first reward value; conversely, if the user frequently skips certain instructions, the first reward value will be lower. After the operation guidance is completed, in response to the user's evaluation information on the operation guidance, a second reward value is determined based on the evaluation information. The second reward value is used to measure the user's overall satisfaction with the operation guidance. Based on the weight coefficients corresponding to the preset first reward value and the preset weight coefficients corresponding to the second reward value, the first reward value and the second reward value are weighted and summed to obtain a third reward value. The third reward value is used to guide the parameter adjustment of the neural network model. The parameter adjustment of the neural network model refers to backpropagating the error signal according to the third reward value and updating the weight matrix and bias terms inside the neural network model to improve the prediction accuracy and adaptability of the neural network model. The parameter adjustment process usually adopts algorithms such as gradient descent and Adam optimizer, which are not limited here.

[0138] In this embodiment, during the execution of the operation guidance, user operation data, including the number of clicks, operation duration, skipped instructions, and completed instructions, is collected. Based on this data, a first reward value is calculated to quantitatively evaluate the compliance of the operation process. After the operation guidance is completed, in response to the received user evaluation information, a second reward value is further determined to measure overall satisfaction. Subsequently, the first reward value and the second reward value are weighted and summed according to preset weight coefficients to obtain a comprehensive third reward value. The third reward value serves as a key feedback signal in reinforcement learning, guiding the neural network model to adjust its parameters, i.e., updating its internal weight matrix and bias terms. This enables the neural network model to continuously optimize its operation sequence prediction ability based on actual interaction feedback, thereby continuously improving the accuracy of the prediction results and its adaptability to users' personalized needs.

[0139] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0140] As software functions become increasingly complex, users often face challenges such as high operational difficulty and learning costs. Against this backdrop, teaching guidance systems, as a key intelligent information processing application aimed at improving user experience, have been widely incorporated into various software products. These systems guide users step-by-step through proactive prompts, step-by-step breakdowns, and real-time interaction, helping them quickly familiarize themselves with core functions and improve operational efficiency. However, significant bottlenecks still exist in the technical implementation of these systems, primarily manifested in the following aspects:

[0141] 1) Disconnect between instruction and operation: Most instructional guidance systems in related technologies rely on text-based instruction, such as providing manuals or official documents containing a large number of technical terms, resulting in a high barrier to entry for users to understand and use the system. To address the technical problem of the disconnect between instruction and operation in related technologies, the operation guidance method provided in this application embodiment responds to user needs and provides instructional guidance on a human-computer interaction interface in real time, while simultaneously recording the user's operation status, effectively reducing the user's learning cost and ease of use.

[0142] 2) Low efficiency in help document search: While some teaching guidance systems in related technologies offer text search functionality, they only support literal matching and lack fuzzy search and relevance judgment capabilities. If users fail to input precise or comprehensive keywords, the search results will be scattered or have a low degree of relevance to the actual needs. To address the technical problem of low efficiency in help document search in related technologies, the operation guidance method provided in this application introduces a large natural language model to build a bridge between user needs and system operations, enabling the system to accurately understand user intent and thus provide precise help and teaching guidance.

[0143] 3) Lack of state awareness and personalization: Most teaching guidance systems in related technologies have fixed and monotonous teaching processes, provide limited information, and lack scalability. To address the technical problem of lacking state awareness and personalization in related technologies, the operation guidance method provided in this application utilizes large-scale modeling technology to generate solutions and, based on the analysis of user operation habits and historical records, achieves more personalized teaching guidance and services.

[0144] 4) Scattered Q&A and Teaching Information: In related technologies, besides the software's built-in manual, users often need to rely on user communities or third-party platforms provided by the software developer to seek solutions. However, the relevant information in these communities or platforms is scattered, fragmented, and of varying quality, requiring users to spend a significant amount of time searching and integrating information. To address the technical problem of scattered Q&A and teaching information in related technologies, the operation guidance method provided in this application integrates teaching guidance and solutions into a system, significantly reducing user information retrieval time and improving software usage efficiency.

[0145] Meanwhile, the operation guidance method provided in this application embodiment also provides version control function in the production environment, builds a "one-stop" workflow, and further simplifies user operation.

[0146] The product performance of the operation guidance method provided in the embodiments of this application will be described below.

[0147] The user input field can be presented as a search-like text input box, supporting speech conversion to obtain the user's natural language text (i.e., the requirement text mentioned above). This can be combined with the version control system shown below, inputting the version history and natural language text into the teaching guidance model (large model) to generate an operation sequence (i.e., the target operation sequence mentioned above). The generated operation sequence is then parsed to provide operational guidance to the user. For example, in the human-computer interaction interface of the target software to be taught, the target area used for operational guidance can be highlighted, while the remaining areas can be darkened. See [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the operation guidance method provided in this application embodiment. A prompt box 402 (target area) is presented in the human-computer interaction interface 401 to guide the user's operation. The implementation of the teaching guidance function relies on the UI interface provided by the target software, such as highlighting, shadowing, and prompt text.

[0148] The operation guidance method provided in this application stores the required user interface (UI) performance in a configuration file. During each step of the guidance process, the target software dynamically adjusts the UI performance by reading the configuration file. For example, based on the provided configuration file template, the value of any field in the configuration file template can be modified to achieve precise control over the UI performance.

[0149] The operation guidance method provided in this application embodiment can support version recording function. Since most users of the teaching guidance system are non-technical personnel, the version recording system can be used to roll back erroneous operations and migrate environments.

[0150] At the same time, a product community is built to encourage users to share version history to the community platform, so that users no longer need to rely on manuals to learn step by step, but can easily achieve "operation migration" by sharing other users' version history.

[0151] In addition, this application provides a sandbox mode, which is suitable for enterprise training scenarios. By providing a sandbox copy based on the production environment, learners can conduct operational drills in an isolated environment, avoiding direct contact with the real production environment, thereby ensuring system security and stability.

[0152] See Figure 5 , Figure 5This is a schematic diagram of the fifth process of the operation guidance method provided in the embodiments of this application. The following will be combined with... Figure 5 The steps shown illustrate the technical implementation of the operation guidance method provided in the embodiments of this application.

[0153] Step 1: Obtain a snapshot of the target software (i.e., the software mentioned above) at the moment of input operation (i.e., the initial state mentioned above).

[0154] In response to user input, a snapshot of the target software at that moment is generated, and the snapshot is managed uniformly by relying on the version control system.

[0155] The generated snapshots can be stored in file formats such as JSON or XML. The data to be stored mainly includes the following:

[0156] 1) Input the state of all session windows in the target software at the time of operation, including the content, size and position information of each window, the current state value of the data module associated with each session window (for example, the data module corresponding to the download session window should include parameters such as maximum download speed, number of concurrent users and proxy settings), and the user interface structure of each session window. The user interface structure can be stored in the form of a document object model tree (HTML) or DOM tree.

[0157] 2) Input the target software's global configuration at the time of operation, such as file save path, session window scaling settings, and other data-related configurations.

[0158] 3) Input user information at the time of operation, including user ID, the time of use of the target software, and user preference data.

[0159] 4) Other relevant data, such as necessary information like the file being processed at that moment.

[0160] It is worth noting that every user operation or operation sequence must generate a corresponding snapshot and be recorded in the version record system.

[0161] Step two: Input the user input information and the snapshot together into the teaching guidance model to obtain the expected snapshot (i.e., the expected state mentioned above).

[0162] User input information and snapshots are fed into the instructional guidance model to obtain the desired snapshot output by the instructional guidance model, and the desired snapshot is then fed into the difference analysis engine of the version record system.

[0163] The instructional guidance model can adopt the currently widely used large language models (e.g., GPT-4, DeepSeek, etc.). The instructional guidance model mainly includes the following modules: context fusion unit, retrieval controller, and inference engine. The context fusion unit is used to fuse snapshots and user input information to generate semantic vectors (i.e., the first fusion feature mentioned above). The retrieval controller retrieves the target software state, i.e. the result expected by the user, from the vector database based on the semantic vector. The inference engine is responsible for outputting the expected snapshot.

[0164] The teaching guidance model can be optimized by crawling the official documentation of the target software or Q&A data from the software community. In addition, the following methods can be used to further improve the performance of the teaching guidance model:

[0165] 1) In terms of prompt design optimization: preprocess the natural language instructions input by the user, add prompts containing user status information (e.g., software usage time and proficiency) to standardize the output of the teaching guidance model; suppress the model's fictional content by adding constraints, set output roles to limit the generation style, and introduce a multi-turn dynamic feedback mechanism to improve the accuracy of multi-turn dialogues in combination with context.

[0166] 2) In terms of Retrieval Enhancement Generation (RAG), a retrieval enhancement generation document library (RAG document library) based on sources such as official documents, Q&A communities and user logs is constructed, and the RAG document library is used as a vector database (i.e. the vector knowledge base mentioned above) to reduce the illusion phenomenon of teaching guidance models.

[0167] 3) In terms of evaluation mechanism design, the output quality of the teaching guidance model is quantified by evaluation signals such as the number of operation steps, the number of re-executions, and the feedback after the teaching guidance is completed. The behavior of the teaching guidance model is optimized accordingly. The evaluation signals may include explicit rating rewards for direct user ratings (i.e., the second reward value mentioned above) and implicit feedback rewards inferred from user behavior (such as dwell time and click sequence) (i.e., the first reward value mentioned above).

[0168] 4) In terms of data augmentation, semantic augmentation techniques are adopted, including but not limited to synonym replacement and question rewriting, to improve the teaching guidance model's ability to understand user input; and different UI states (e.g., layer selected and unselected states) are simulated through template insertion methods to enhance the teaching guidance model's adaptability and generalization ability to interface contexts.

[0169] Step 3: Based on the version history system, determine the sequence of differences between the target software's snapshot and the desired snapshot.

[0170] The version history system mainly consists of the following three parts:

[0171] 1) Learning Trajectory Tracker: Records snapshots of each user action.

[0172] 2) Delta Engine: Compares semantic changes between versions.

[0173] Unlike the instructional guidance model, the differential analysis engine operates based on an independently constructed knowledge graph. The inputs to the knowledge graph are a snapshot (a snapshot at the moment of input operation) and a desired snapshot, and the output is the sequence of operations that realizes the transition from the snapshot to the desired snapshot.

[0174] First, knowledge graph nodes are constructed based on the input snapshots. Each snapshot is an independent node, and each snapshot node contains several child nodes. For example, UI nodes are used to save the interface state, and function nodes are used to represent software functions (e.g., saving files). Then, different child nodes are connected by relational edges. For example, a relationship of "UI node - implementation → function node" and a relationship of "save button - implementation → save function" can be established.

[0175] By comparing the child nodes and their relationships under two snapshot nodes, a difference sequence is obtained. This sequence records the state changes between versions, such as UI state 1 → UI state 2. The difference sequence will be fed back to the teaching guidance model, and finally an operation sequence that can realize the state transition will be generated.

[0176] When the user finally confirms that the teaching plan corresponding to the operation sequence is feasible, the difference analysis engine will establish a new triple relationship based on the teaching plan, the functional nodes and path nodes in the current version, thereby realizing the update and expansion of the knowledge graph. This mechanism can continuously improve the accuracy of the system in difference analysis and gradually reduce the computational overhead caused by repeated reasoning.

[0177] 3) Branch Merge Coordinator (BMC).

[0178] The branch merge coordinator can perform merge operations on multi-user production environments. Leveraging the data stored in the learning process tracker, it can merge two or more production environments. For example, if users A and B use the same software but have different configuration parameters, such as resolution, font, and scaling, the branch merge coordinator can synchronize user A's modifications to the production environment with user B's, ensuring consistency across all users' production environments. The branch merge coordinator can also be used for production environment migration. Since migration is essentially a process of merging an empty production environment with the current one, the environment merge controller analyzes the background of the modifications made by each conflicting party, combines historical performance data, and automatically generates a merge plan. If conflicts cannot be resolved automatically, a comparison view is generated and submitted for manual adjudication.

[0179] The operation guidance method provided in this application embodiment includes an export function for the version history system. Users can export their operation sequence set as a separate file. During this process, the system will anonymize sensitive user data and proactively alert users to privacy risks to ensure user privacy and security. Furthermore, the operation guidance method provided in this application embodiment can, based on the user-submitted operation sequence set, combine it with a large language model for data analysis and learning to build an official "bot system," thereby providing users with a one-stop solution to their problems.

[0180] Furthermore, the operation guidance method provided in this application also supports sandbox mode. Sandbox mode relies on virtualization technology (e.g., Docker, virtual machine, etc.) to simulate the production environment and uses a version record mechanism to copy some configurations and states of the production environment to the sandbox, providing users with a safe and isolated operation exercise and training environment.

[0181] Step four: Input the difference sequence into the teaching guidance model to obtain the teaching guidance plan, and guide the user to operate based on the teaching guidance plan.

[0182] See Figure 6 , Figure 6 This is a schematic diagram of the sixth process of the operation guidance method provided in the embodiments of this application. The following will be combined with... Figure 6 The steps shown illustrate the data flow of the operation guidance method provided in the embodiments of this application.

[0183] Step 601: Obtain user input information.

[0184] Step 602: Record the state of the target software at the moment of information input, serialize the state of the target software into a snapshot, and record it in the version record system.

[0185] Step 603: Input the user input information and snapshot into the teaching guidance model to obtain the semantic vector.

[0186] Step 604: Retrieve the RAG document library based on semantic vectors to obtain the inferred expected snapshot.

[0187] Step 605: Input the desired snapshot and the snapshot into the difference analysis engine to obtain the difference sequence.

[0188] Step 606: Input the difference sequence into the teaching guidance model to obtain the teaching guidance scheme.

[0189] Step 607: Analyze the teaching guidance plan to obtain the operation sequence, and perform UI display according to the operation sequence to guide user operation.

[0190] The beneficial effects of the operation guidance method provided in this application include:

[0191] 1) Lowering the barrier to entry for users: Users can describe operation instructions in natural language without needing to master professional terminology, which is especially friendly to non-technical background and novice users.

[0192] 2) Improve the efficiency of using the target software, so that users can solve most operational problems without consulting manuals or technical forums.

[0193] 3) Supports continuous updates and personalized customization. It can optimize the teaching guidance model by combining the latest information and achieve user-level customization by fine-tuning the parameters of the teaching guidance model. At the same time, the teaching guidance model can also improve itself based on usage records.

[0194] 4) Visualize the teaching guidance process and deeply integrate it with the software user interface to make teaching guidance more intuitive and clear, thereby improving the quality and efficiency of teaching guidance.

[0195] 5) It supports convenient migration and sharing of production environments, and can simplify workflows, further improving the overall efficiency of software use.

[0196] In some embodiments, for complex software, detailed records of its state and configuration are required for software snapshot generation; for simple software, a "screenshot-image recognition" method can be used to quickly generate snapshots. In this scenario, image recognition capabilities can be introduced into the intent understanding module of the teaching guidance model to extract image information and use it as input for subsequent modules.

[0197] In some embodiments, the deployment of the teaching guidance model can adopt the following two schemes: one is cloud deployment, which is to integrate the model into the system by calling the application programming interface (API) of an external service provider through the Hypertext Transfer Protocol (HTTP) interface. This deployment method can shorten the project deployment time and has good scalability, but may have data privacy risks, rely on a stable external network connection, and make it difficult to achieve deep customization of the model; the other is local deployment, which is to deploy the teaching guidance model on a local server and store all data in an intranet environment. This deployment method has high data security and supports full customization, but requires certain deployment and maintenance costs.

[0198] In some embodiments, there are several optional implementation schemes for implementing teaching guidance based on the teaching guidance model output: First, the teaching guidance system is integrated into the software, encapsulated as part of the software. This integration method can maximize the use of software functions and achieve optimal teaching guidance quality and accuracy. However, due to the high coupling requirements between the system and the software, separate development and extension are needed for different software, resulting in high development and maintenance costs. Second, it is embedded into the target software as a plugin, for example, following VSCode's plugin management mechanism, and then utilizing the software's existing open interfaces for integration. This integration method does not require additional development of the target software, only adaptation of the system itself. However, it depends on whether the software provides complete open interfaces, thus limiting its implementation scope. Third, it is used in conjunction with the target software as a third-party independent software. This integration method does not require any modification to the target software, only adaptation of the system itself. However, its functionality is relatively limited, as it cannot directly call the software's data interface and can only obtain information through the user interface. Therefore, it is only suitable for software with relatively simple structures.

[0199] The following description continues to illustrate the exemplary structure of the operation guidance device 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the operation boot device 555 of the memory 550 may include:

[0200] The data acquisition module 5551 is used to determine the initial state of the software in response to an operation guidance request for a target function of the software.

[0201] The encoding module 5552 is used to encode the requirement text included in the operation guidance request to obtain the text features of the requirement text, and to encode the initial state to obtain the state features of the initial state.

[0202] The retrieval module 5553 is used to fuse the text features and the state features to obtain a first fused feature, and based on the first fused feature, to retrieve the vector knowledge base of the software, and to determine the candidate state corresponding to the retrieved candidate state feature as the expected state of the software. The vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature that represents each candidate state of the software.

[0203] The difference analysis module 5554 is used to perform difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state.

[0204] The interaction module 5555 is used to perform operation sequence prediction processing based on the difference sequence to obtain the target operation sequence, and to provide operation guidance for the target function based on the target operation sequence.

[0205] In some embodiments, the encoding module 5552 is further configured to acquire historical operation guidance requests for any function of the software and object behavior data of the software; construct prompt words for the requirement text based on the historical requirement text of the historical operation guidance requests and the object behavior data to obtain a first prompt word; adjust the first prompt word based on a preset prompt word template to obtain a second prompt word; and encode the second prompt word to obtain the text feature.

[0206] In some embodiments, the encoding module 5552 is further configured to encode the requirement text, the historical requirement text, and the object behavior data respectively to obtain a first encoding feature of the requirement text, a second encoding feature of the historical requirement text, and a third encoding feature of the object behavior data; to integrate the second encoding feature and the third encoding feature into the first encoding feature to obtain a second fusion feature; and to decode the second fusion feature to obtain the first prompt word.

[0207] In some embodiments, the difference analysis module 5554 is further configured to construct a first knowledge graph using the first attribute information contained in the initial state as a first node and the relationship between the first attribute information as a first edge; construct a second knowledge graph using the second attribute information contained in the desired state as a second node and the relationship between the second attribute information as a second edge; and determine the difference sequence based on the difference between the first knowledge graph and the second knowledge graph.

[0208] In some embodiments, the difference analysis module 5554 is further configured to: for each first node in the first knowledge graph, determine a second node of the same type as the first node from the second knowledge graph; if the first attribute information corresponding to the first node is different from the second attribute information corresponding to the second node, determine the difference between the first attribute information corresponding to the first node and the second attribute information corresponding to the second node as a first difference; for each first edge in the first knowledge graph, determine a second edge from the second knowledge graph, wherein the two second attribute information corresponding to the two second nodes connected by the second edge is the same as the two first attribute information corresponding to the two first nodes connected by the first edge; if the relationship represented by the first edge is different from the relationship represented by the second edge, determine the difference between the relationship represented by the first edge and the relationship represented by the second edge as a second difference; and in response to the fact that the number of both the first difference and the second difference is not zero, construct the difference sequence based on at least one first difference and at least one second difference.

[0209] In some embodiments, the interaction module 5555 is further configured to invoke a neural network model based on the difference sequence, and perform operation sequence prediction processing through the neural network model to obtain the target operation sequence.

[0210] In some embodiments, the interaction module 5555 is further configured to: collect user operation data for the operation guidance during the execution of the operation guidance; determine a first reward value based on the user operation data; determine a second reward value based on the evaluation information received for the operation guidance; merge the first reward value and the second reward value to obtain a third reward value; and adjust the parameters of the neural network model based on the third reward value.

[0211] In some embodiments, the retrieval module 5553 is further configured to acquire a product knowledge base of the software, wherein the product knowledge base contains descriptive text for each function of the software; for each descriptive text contained in the product knowledge base, perform state prediction processing based on the descriptive text to obtain a candidate state of the software, wherein the candidate state is the state of the software after executing the function corresponding to the descriptive text; encode the candidate state to obtain a state feature of the candidate state; create an initial vector knowledge base, and store the state feature of the candidate state corresponding to each descriptive text into the initial vector knowledge base to obtain the vector knowledge base.

[0212] In some embodiments, the retrieval module 5553 is further configured to determine, from the plurality of candidate state features contained in the vector knowledge base, at least one candidate state feature whose first similarity to the first fused feature is greater than a similarity threshold; and to determine the desired state of the software based on the at least one candidate state feature and the confidence level of each of the at least one candidate state feature.

[0213] In some embodiments, the retrieval module 5553 is further configured to: determine a first similarity between the candidate state feature and the first fusion feature for each of the at least one candidate state feature; adjust the first similarity based on the confidence level to obtain a second similarity; and decode the candidate state feature corresponding to the maximum second similarity to obtain the desired state of the software.

[0214] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the operation guidance method described in this application.

[0215] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the operation guidance method provided in this application.

[0216] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0217] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0218] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0219] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0220] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An operation guidance method, characterized in that, The method includes: In response to an operation guidance request for a target function of the software, determine the initial state of the software; The requirement text included in the operation guidance request is encoded to obtain the text features of the requirement text, and the initial state is encoded to obtain the state features of the initial state. The text features and the state features are fused to obtain the first fused feature; Obtain the product knowledge base of the software, wherein the product knowledge base contains descriptive text for each function of the software; For each of the description texts contained in the product knowledge base, state prediction processing is performed based on the description texts to obtain candidate states of the software, wherein the candidate states are the states of the software after executing the function corresponding to the description text; The candidate states are encoded to obtain the state features of the candidate states; An initial vector knowledge base is created, and the state features of the candidate states corresponding to each of the description texts are stored in the initial vector knowledge base to obtain the vector knowledge base; Based on the first fusion feature, the vector knowledge base is retrieved, and the candidate state corresponding to the retrieved candidate state feature is determined as the expected state of the software. The vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature that characterizes each candidate state of the software. A difference analysis is performed on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state; Based on the difference sequence, operation sequence prediction processing is performed to obtain the target operation sequence, and operation guidance for the target function is performed based on the target operation sequence.

2. The method according to claim 1, characterized in that, The process of encoding the request text included in the operation guidance request to obtain the textual features of the request text includes: Obtain historical operation guidance requests for any function of the software and object behavior data of the software; Based on the historical request text of the historical operation guidance request and the object behavior data, prompt words are constructed from the request text to obtain the first prompt word; Based on a preset prompt template, the first prompt is adjusted to obtain the second prompt; The second prompt word is encoded to obtain the text features.

3. The method according to claim 2, characterized in that, Based on the historical request text of the historical operation guidance request and the object behavior data, the request text is used to construct prompt words to obtain the first prompt word, including: The requirement text, the historical requirement text, and the object behavior data are encoded respectively to obtain the first encoding feature of the requirement text, the second encoding feature of the historical requirement text, and the third encoding feature of the object behavior data. The second coding feature and the third coding feature are incorporated into the first coding feature to obtain the second fused feature; The second fusion feature is decoded to obtain the first prompt word.

4. The method according to claim 1, characterized in that, The step of performing a difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state includes: A first knowledge graph is constructed using the first attribute information contained in the initial state as the first node and the relationship between the first attribute information as the first edge. A second knowledge graph is constructed using the second attribute information contained in the desired state as the second node and the relationship between the second attribute information as the second edge. The difference sequence is determined based on the differences between the first knowledge graph and the second knowledge graph.

5. The method according to claim 4, characterized in that, Determining the difference sequence based on the differences between the first knowledge graph and the second knowledge graph includes: For each first node in the first knowledge graph, a second node of the same type as the first node is determined from the second knowledge graph. If the first attribute information corresponding to the first node is different from the second attribute information corresponding to the second node, the difference between the first attribute information corresponding to the first node and the second attribute information corresponding to the second node is determined as the first difference. For each first edge in the first knowledge graph, a second edge is determined from the second knowledge graph. The two second attribute information corresponding to the two second nodes connected by the second edge is the same as the two first attribute information corresponding to the two first nodes connected by the first edge. If the relationship represented by the first edge is different from the relationship represented by the second edge, the difference between the relationship represented by the first edge and the relationship represented by the second edge is determined as the second difference. In response to the fact that the number of both the first difference and the second difference is not zero, the difference sequence is constructed based on at least one first difference and at least one second difference.

6. The method according to claim 1, characterized in that, The step of performing operation sequence prediction processing based on the differential sequence to obtain the target operation sequence includes: Based on the differential sequence, a neural network model is invoked, and the operation sequence prediction processing is performed through the neural network model to obtain the target operation sequence; After guiding the operation for the target function based on the target operation sequence, the method further includes: During the execution of the operation guidance, user operation data related to the operation guidance is collected, and a first reward value is determined based on the user operation data; In response to receiving evaluation information for the operation guidance, a second reward value is determined based on the evaluation information; The first reward value and the second reward value are combined to obtain a third reward value, and the parameters of the neural network model are adjusted based on the third reward value.

7. The method according to claim 1, characterized in that, The step of retrieving the vector knowledge base based on the first fusion feature and determining the candidate state corresponding to the retrieved candidate state feature as the desired state of the software includes: From the plurality of candidate state features contained in the vector knowledge base, at least one candidate state feature whose first similarity to the first fused feature is greater than a similarity threshold is determined. The desired state of the software is determined based on the at least one candidate state feature and the confidence level of each of the at least one candidate state feature.

8. The method according to claim 7, characterized in that, Determining the desired state of the software based on the at least one candidate state feature and the confidence level of each candidate state feature includes: For each of the at least one candidate state features, a first similarity is determined between the candidate state feature and the first fused feature; based on the confidence level, the first similarity is adjusted to obtain a second similarity. The candidate state features corresponding to the maximum second similarity are decoded to obtain the desired state of the software.

9. An operation guidance device, characterized in that, The device includes: A data acquisition module is used to determine the initial state of the software in response to an operation guidance request for a target function of the software. The encoding module is used to encode the requirement text included in the operation guidance request to obtain the text features of the requirement text, and to encode the initial state to obtain the state features of the initial state. A retrieval module is configured to fuse the text features and the state features to obtain a first fused feature; acquire the product knowledge base of the software, wherein the product knowledge base contains descriptive text for each function of the software; for each descriptive text contained in the product knowledge base, perform state prediction processing based on the descriptive text to obtain a candidate state of the software, wherein the candidate state is the state of the software after executing the function corresponding to the descriptive text; encode the candidate state to obtain a state feature of the candidate state; create an initial vector knowledge base and store the state features of the candidate state corresponding to each descriptive text into the initial vector knowledge base to obtain the vector knowledge base; and retrieve the vector knowledge base based on the first fused feature, and determine the candidate state corresponding to the retrieved candidate state feature as the expected state of the software, wherein the vector knowledge base includes multiple candidate state features of the software, and each candidate state feature is a state feature characterizing each candidate state of the software; The difference analysis module is used to perform difference analysis on the initial state and the desired state to obtain a difference sequence between the initial state and the desired state; The interaction module is used to perform operation sequence prediction processing based on the difference sequence to obtain the target operation sequence, and to provide operation guidance for the target function based on the target operation sequence.

Images