Operation guidance generation method and device, equipment, medium and program product

Abstract

The invention provides an operation guidance generation method and device, equipment, a medium and a program product, can be applied to the field of artificial intelligence and the field of financial science and technology, and particularly relates to application of a large model in a user interface guidance scene. The method comprises the steps of monitoring a page state change event in real time, responding to an abnormal state page fed back to a user, and based on the abnormal state page, calling a multi-mode large model to extract an abnormal information text and position coordinates; under the condition that the abnormal state page is triggered for the first time, according to the abnormal information text and the user operation data, performing anomaly analysis based on a text large model to obtain a first solution; based on the exception solution and the position coordinates, multi-modal operation guidance is generated through a multi-modal large model; wherein the exception solution comprises a first solution; and dynamically guiding the user based on the multi-modal operation guidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and the field of financial technology, in particular to the application of a large model in a user interface guidance scenario, more particularly to a method and device for generating operation guidance, equipment, medium and program product. BACKGROUND

[0002] Electronic banking (such as mobile banking, online banking, etc.) has various functions, covering complex operations such as transfer, payment, financial management, and account management. Some users (especially the elderly or users unfamiliar with digital technology) may be deterred by the interface and process. Clear operation guidance can help users quickly understand the function logic through graphic and text explanations, step-by-step disassembly, and other methods, allowing them to complete the operation independently and avoid missing convenient financial services due to "not knowing how to use" them.

[0003] Traditional operation guidance mainly uses static guidance pictures, text, and fixed pop-up prompts. For example, a transfer page only displays a general prompt "Please enter the amount". However, static guidance has limitations. When the user's operation encounters an interface error, only static error information is returned, and it is not possible to provide targeted guidance based on the user input error type (such as amount exceeding the limit or format error) or operation progress (such as not selecting a payee). This makes the interaction mode limited, the exception handling inefficient, and the operation guidance lacking in scene adaptability. SUMMARY

[0004] In view of the above problems, the present application provides a method and device for generating operation guidance to improve the efficiency of exception handling.

[0005] According to a first aspect of the present application, a method for generating operation guidance is provided, comprising: monitoring page state change events in real time, responding to an abnormal state page feedback to a user, and based on the abnormal state page, calling a multi-modal large model to extract abnormal information text and position coordinates; in the case of first triggering the abnormal state page, based on the abnormal information text and user operation data, performing abnormal analysis based on a text large model to obtain a first solution; based on the abnormal solution and the position coordinates, generating a multi-modal operation guidance through the multi-modal large model; wherein the abnormal solution includes the first solution; and based on the multi-modal operation guidance, dynamically guiding the user.

[0006] According to an embodiment of the present application, based on the abnormal solution and the position coordinates, the multi-modal large model generates a multi-modal operation guidance, comprising: based on the abnormal state page, the multi-modal large model analyzes the abnormal solution and the position coordinates to generate the multi-modal operation guidance; wherein the multi-modal operation guidance includes element positioning, operation action, and explanation text.

[0007] According to an embodiment of the present application, the dynamic guidance of the user based on the multi-modal operation guidance includes: determining a target operation control based on the element positioning, and visually highlighting the target operation control; converting and processing the instruction text through a speech synthesis engine to play the instruction text; and displaying the abnormal solution to the user through a folding panel component.

[0008] According to an embodiment of the present application, the dynamic guidance of the user based on the multi-modal operation guidance further includes: in response to the user input voice instruction, automatically performing the operation action according to the voice instruction; and real-time monitoring of the page state change event.

[0009] According to an embodiment of the present application, the calling of a multi-modal large model to extract abnormal information text and position coordinates based on the abnormal state page includes: obtaining a page screenshot and a node tree corresponding to the abnormal state page; and calling the multi-modal large model and analyzing the page screenshot and the node tree through the multi-modal large model to obtain the abnormal information text and the position coordinates.

[0010] According to an embodiment of the present application, the abnormal analysis based on a text large model according to the abnormal information text and user operation data to obtain a first solution includes: performing root cause analysis on the abnormal information text and the user operation data through the text large model to generate abnormal state information; and based on an abnormal knowledge base, retrieving and matching the abnormal state information to generate the first solution; wherein the abnormal knowledge base is a key-value pair database constructed based on historical abnormal state information and historical solution processes.

[0011] According to an embodiment of the present application, the abnormal solution further includes a second solution, and the method further includes: in a case where the abnormal state page is not triggered for the first time, obtaining a solution corresponding to the abnormal state page based on a solution database to obtain the second solution; wherein the solution database is constructed based on historical abnormal state pages and corresponding first solutions.

[0012] According to an embodiment of the present application, the method further includes: in response to an input operation of the user, receiving a business processing request corresponding to the input operation and performing data processing based on the business processing request; and in a case where the data processing result is abnormal, feeding back the abnormal state page to the user based on the data processing result.

[0013] The second aspect of the present application provides a generation device of operation guide, comprising: an abnormal data extraction module, configured to monitor a page state change event in real time, feed back an abnormal state page to a user, call a multi-modal large model to extract abnormal information text and position coordinates based on the abnormal state page; an abnormal analysis module, configured to, in a case where the abnormal state page is triggered for the first time, perform abnormal analysis based on a text large model according to the abnormal information text and user operation data, and obtain a first solution; an operation guide generation module, configured to generate a multi-modal operation guide by the multi-modal large model based on an abnormal solution and the position coordinates; wherein the abnormal solution comprises the first solution; and a dynamic guide module, configured to guide a user dynamically based on the multi-modal operation guide.

[0014] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0015] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0016] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0017] In the embodiments of the present application, the page state change is monitored in real time, when an abnormal state occurs, the multi-modal large model is called to extract abnormal information, the text large model is combined to analyze abnormal solution, and finally the multi-modal operation guide is generated by the multi-modal large model, thereby constructing a dynamic guide graphical user interface intelligent framework, intelligently generating operation guide, realizing dynamic guide of user to handle abnormal situation, providing multi-modal operation guide, dynamically assisting user operation, closely combining guide information with actual user interface, improving scene perception ability, greatly improving abnormal handling efficiency and user experience, thereby significantly improving fault tolerance and operation fluency of electronic banking transaction process. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following description of the embodiments of the present application taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 An application scenario diagram of the generation method, device, equipment, medium and program product of operation guide according to the embodiments of the present application is schematically shown;

[0020] Figure 2 A flowchart illustrating a method of generating operational guidance according to embodiments of the present application is shown schematically;

[0021] Figure 3 A graphical user interface agent framework diagram illustrating a method of generating operational guidance according to embodiments of the present application is shown schematically;

[0022] Figure 4 Another flowchart illustrating a method of generating operational guidance according to embodiments of the present application is shown schematically;

[0023] Figure 5 A dynamic guidance flowchart illustrating a method of generating operational guidance according to embodiments of the present application is shown schematically;

[0024] Figure 6 A block diagram illustrating a structure of a device for generating operational guidance according to embodiments of the present application is shown schematically; and

[0025] Figure 7 A block diagram illustrating an electronic device suitable for implementing a method of generating operational guidance according to embodiments of the present application is shown schematically. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description which follows is merely illustrative and is not intended to limit the scope of the present application. In the following detailed description of embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0027] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present application. The terms "include" and "have" and the like used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0028] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0029] In the case of using expressions such as "at least one of A, B, and C", it is generally intended that the inclusion of at least one of A, B, or C should be treated as if each of the following expressions were incorporated herein: A alone, B alone, C alone, A and B, A and C, B and C, and A and B and C.

[0030] The operation guide of electronic banking is related to user experience and benefit protection, and also affects the efficiency and security of bank services. Financial operations have very high accuracy requirements. Any mistake in the input of account information during a transfer, the confirmation of risks during financial management, and the process specification during password modification may lead to fund loss or information leakage. The operation guide can highlight key points (such as "how to check the payee information" and "do not disclose the verification code") and clearly prohibit operations and matters needing attention, thereby reducing the risk of user misoperation from the source and indirectly building a strong defense for the user's property safety.

[0031] Currently, mobile banking mainly provides users with guided operations through static guides during the user's transaction process. However, static guides have the following disadvantages:

[0032] (1) Low efficiency of error handling: When the user's operation encounters interface errors (such as verification failure, input overrun, and transaction interruption), the system only returns static error codes or short prompts, and the user needs to consult the help document or contact customer service, resulting in a long interruption time in the operation process.

[0033] (2) Lack of scene adaptability of the guide: Existing solutions (such as preset questions and answers) cannot provide dynamic guidance in combination with user historical behavior, current operation context, and interface element state, resulting in a disconnection between the guide information and the user's actual interface.

[0034] (3) Limitation of interaction mode: Existing static guides mainly use text, simple icons, guide pictures, pop-up windows, etc., and do not integrate multi-modal interaction methods such as voice prompts and interface annotations. This is not friendly to middle-aged and elderly users or visually impaired users, and guide pictures and pop-up windows need to be designed or developed specifically, which has a high development cost.

[0035] (4) Fragmentation of interaction experience: Users need to switch between multiple windows to view the guide steps, and cannot directly obtain visual operation instructions on the operation interface, which reduces the operation success rate and efficiency.

[0036] The embodiment of the present application provides a method for generating operation guidance, real-time page state change events are monitored, an abnormal state page is fed back to a user in response, abnormal information text and position coordinates are extracted based on the abnormal state page by calling a multi-modal large model; in the case of first triggering of the abnormal state page, abnormal analysis is performed based on a text large model according to the abnormal information text and user operation data, and a first solution is obtained; a multi-modal operation guidance is generated by the multi-modal large model based on the abnormal solution and the position coordinates; wherein the abnormal solution includes the first solution; and the user is dynamically guided based on the multi-modal operation guidance.

[0037] It should be noted that the method and device for generating operation guidance provided by the present application can be used in the field of artificial intelligence and the field of financial technology, and are related to the application of a large model in a user interface guidance scenario, and can also be used in any field other than the field of artificial intelligence and the field of financial technology, and the application field of the method and device for generating operation guidance provided by the present application is not limited.

[0038] Figure 1 An application scenario diagram of the method and device for generating operation guidance, equipment, medium and program product according to the embodiment of the present application is schematically shown.

[0039] As shown in Figure 1 The application scenario 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0040] The user can use the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0041] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0042] The server 105 can be a server providing various services, for example, a background management server providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing result (for example, a webpage, information, or data, or the like obtained or generated according to the user request) to the terminal device.

[0043] It should be noted that the operation guide generation method provided in the embodiments of the present application can generally be executed by the server 105. Correspondingly, the operation guide generation apparatus provided in the embodiments of the present application can generally be arranged in the server 105. The operation guide generation method provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the operation guide generation apparatus provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0044] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0045] The operation guide generation method according to the embodiments of the present application will be described in detail below based on the scenario described above. Figure 1 Figures 2-5

[0046] Figure 2 An illustrative flowchart of the operation guide generation method according to the embodiments of the present application is shown.

[0047] As shown in Figure 2 , the operation guide generation method of this embodiment includes operations S210-S240, and the operation guide generation method is not limited to a specific execution subject. The execution subject can be any electronic device, such as a terminal device or a server device, and the like. The execution subject can also be any software application or client.

[0048] In operation S210, a page state change event is monitored in real time, and in response to feeding back an abnormal state page to a user, based on the abnormal state page, a multi-modal large model is called to extract abnormal information text and position coordinates.

[0049] ​​Real-time page state change events can be captured in real time through front-end event listening mechanisms (such as document object model listening and interface callback). Interface abnormalities can be sensed in real time. When detecting abnormal page loading or rendering such as form error, feedback the abnormal state page to the user, and trigger the exception handling process.

[0050] When an abnormal state page appears, a multi-modal large model can be called to process the abnormal state page and extract abnormal information text and position coordinates corresponding to the abnormal state page. Position coordinates represent the location of controls in the page, which can guide the multi-modal large model to select the correct control to click and label the location of these controls. Abnormal information text represents the text content of the abnormal prompt in the page, such as "transfer operation failed" and "no pending bills to pay".

[0051] It is worth noting that a multi-modal large model is a deep learning model that can process multiple media data such as text, images, audio, and video. It has the ability to process multi-modal data, cross-modal learning, efficient information representation and understanding, and flexible output generation. The basic structure of the multi-modal large model includes a pre-trained multi-modal encoder and a pre-trained large language model, and a modal interface is used to narrow the gap between different modalities, so that the model can better understand and process multi-modal input.

[0052] In operation S220, in the case of first triggering an abnormal state page, an abnormal analysis is performed based on the text large model according to the abnormal information text and the user operation data, and a first solution is obtained.

[0053] If the abnormal state page is first encountered, it indicates that the abnormal state has not been encountered before, so the text large model is used to analyze the abnormal information text and user operation data to obtain a corresponding solution. The user operation data includes the user's current transaction type, the user's historical operation characteristics, and device environment information.

[0054] The first solution represents the solution corresponding to the abnormal state page obtained by the text large model when the abnormal state page first appears. The abnormal state page and the corresponding first solution are stored in the solution database, so that when the abnormal state page is triggered again, the solution corresponding to the abnormal state page can be directly retrieved from the solution database.

[0055] It is worth noting that the text large model is an artificial intelligence model based on deep learning technology, focusing on text data processing, and its core capability is to understand and generate human language. Through learning language rules, semantic logic and knowledge information from massive text data, it realizes deep analysis and creation of text. In the pre-training stage: learn general language rules through massive text, in the fine-tuning stage, adjust the model parameters with a small amount of specific financial field data (such as abnormal information, error prompt) to adapt to specific tasks.

[0056] In operation S230, based on the abnormal solution and the position coordinates, a multi-modal operation guide is generated by a multi-modal large model; wherein the abnormal solution includes a first solution.

[0057] After determining the abnormal solution and the position coordinates of the control, the multi-modal operation guide can be derived by using the multi-modal large model, and the multi-modal operation guide includes visual guide, voice guide and text guide, etc.

[0058] When the abnormal state page is first appeared, the abnormal solution is the first solution generated by the text large model, and when the abnormal state page is not first appeared, the abnormal solution is the second solution stored in the solution database.

[0059] In operation S240, based on the multi-modal operation guide, the user is dynamically guided.

[0060] Load the structured data of the multi-modal operation guide, integrate element positioning (such as control identification), operation action (such as click) and explanation text, and associate the current page scene. Compare the current interface elements with the target scene in the operation guide to further confirm the guide steps to be executed, and ensure that the guide is synchronized with the user operation progress. Combine the voice synthesis to play the explanation text, and at the same time mark the target element position on the page through visual prompts such as highlighting and pop-up windows. If the user operation deviates, adjust the guide content in real time. Listen to the user operation result, if executed correctly, proceed to the next guide, if failed, replay the explanation or supplement the prompt, until the whole process operation is completed, forming a dynamically adaptive guide closed loop.

[0061] In the embodiments of the present application, the page state changes are monitored in real time, when the abnormal state appears, the abnormal information is extracted by calling the multi-modal large model, and the abnormal solution is analyzed combined with the text large model, and finally the multi-modal operation guide is generated by the multi-modal large model, a dynamic guidance graphical user interface intelligent framework is constructed, the operation guide is intelligently generated, the user is guided to handle the abnormal situation, the multi-modal operation guide is provided, the user operation is dynamically assisted, the guide information is closely combined with the actual interface of the user, the scene perception ability is improved, the abnormal handling efficiency and user experience are greatly improved, thereby the fault tolerance and operation fluency of the electronic banking transaction process are significantly improved.

[0062] Exemplarily, Figure 3 A graphical user interface intelligent agent framework of a method for generating operation guidance according to an embodiment of the present application is schematically shown.

[0063] As Figure 3 shown, a dynamic guidance GUI Agent (Graphical User Interface Agent) framework is constructed based on a multi-modal large model and a text large model, and an intelligent GUI Agent is constructed, which dominates to realize a closed-loop process of "perception-analysis-decision- interaction". Real-time interface information is perceived, and user guidance is triggered when an error occurs. The interface is screenshot, information extraction is performed through a multi-modal large model, error perception, picture annotation position information and content understanding are realized through a multi-modal large model, in the initial processing of the error information, the error root cause is intelligently analyzed by a text large model combined with an error knowledge base, a solution is determined, in the non-initial processing of the error information, the previously stored solution can be called, and the solution and position information are analyzed by a multi-modal large model, a visual operation guidance is dynamically generated, accurate highlight indication operation area in the current user interface is realized, and voice / text synchronous guidance is provided, and the user can be assisted to operate, and the loading of the page is monitored in real time, so that the fault tolerance and operation fluency of the mobile banking transaction process are significantly improved.

[0064] The GUI Agent is an artificial intelligence agent system based on a multi-modal visual model, which can understand user requirements through natural language instructions, automatically reason and execute interactive operations of the user interface, perceive the operation environment through the graphical user interface like a human user, simulate a human to complete cross-platform tasks, understand and execute various operation instructions on the graphical interface, such as clicking, inputting, dragging, reading interface information, etc., and realize automatic processing of complex tasks required by the user. The GUI Agent has core capabilities such as multi-modal perception, dynamic task planning, and cross-platform versatility.

[0065] Figure 4 Another flowchart of a method for generating operation guidance according to an embodiment of the present application is schematically shown.

[0066] As Figure 4 shown, the method for generating operation guidance of this embodiment includes operations S410-S450.

[0067] In operation S410, a page state change event is monitored in real time, and in response to feeding back an abnormal state page to the user, based on the abnormal state page, a multi-modal large model is called to extract abnormal information text and position coordinates. It should be noted that the method of operation S410 is the same as that of operation S210 described above, and will not be repeated here.

[0068] In operation S420, in the case of first triggering the abnormal state page, an abnormal analysis is performed based on the text large model according to the abnormal information text and the user operation data to obtain a first solution. It should be noted that operation S420 is the same as the method of operation S220 described above, and will not be repeated here.

[0069] It should be noted that whether it is the first time to trigger the abnormal state page can be determined according to the core features of the abnormal state page (such as abnormal code, page element structure, error text).

[0070] In operation S430, in the case of non-first triggering of the abnormal state page, a solution database is used to obtain a solution corresponding to the abnormal state page to obtain a second solution; wherein the solution database is constructed based on historical abnormal state pages and corresponding first solutions.

[0071] The core features (such as abnormal code, page element structure, error text) of the non-first abnormal state page are extracted by a hash algorithm and a unique feature identifier is generated, and the solution database is queried with the identifier to match historical records and obtain the corresponding second solution. After the new abnormal state page is processed by the multi-modal large model and the text large model, the first solution is generated, the core features of the page are extracted by a hash algorithm to generate a feature identifier, and the feature identifier of the page is recorded in the solution database. The corresponding first solution is entered into the solution database, and the solution database is used to quickly match reliable solutions for non-first abnormalities. The second solution represents a solution corresponding to an abnormal state page stored in the solution database.

[0072] In an embodiment of the present application, when the abnormal state page is not the first time, the historical solution database is used to quickly call the generated solution, avoiding repeated analysis, and significantly improving the efficiency and consistency of abnormal processing, while continuously optimizing self-learning and fault tolerance.

[0073] In operation S440, a multi-modal operation guide is generated by the multi-modal large model based on the abnormal solution and the location coordinates; wherein the abnormal solution includes the first solution or the second solution. It should be noted that operation S440 is the same as the method of operation S230 described above, and will not be repeated here.

[0074] In operation S450, the user is dynamically guided based on the multi-modal operation guide. It should be noted that operation S450 is the same as the method of operation S240 described above, and will not be repeated here.

[0075] According to the embodiments of the present application, before feeding back the abnormal state page to the user, further comprising: in response to the input operation of the user, receiving a service processing request corresponding to the input operation, and performing data processing based on the service processing request; and in the case that the data processing result is abnormal, feeding back the abnormal state page to the user based on the data processing result.

[0076] The user performs an input operation on the page, such as filling in a form and then clicking a "submit" button, clicking a "save", "buy", "send" or any other button that triggers a network request, the operation event handling function of the user is triggered, at this time the front end needs to send an asynchronous service processing request to the server backend, the service processing request is asynchronous, the page will not be refreshed, and the user can still stay on the current page, waiting for the response of the server.

[0077] The server receives the service processing request sent by the front end, starts to process the business logic, such as verifying data, querying a database, calling other services, processing files and other data processing, during the backend processing, the data processing is abnormal, an abnormal situation occurs, the data processing result (whether success or failure) is wrapped into a response and returned to the front end, the response includes a state code corresponding to the abnormal situation. If the data processing is not abnormal, a response feeding back the data processing result is sent to the front end, the front end receives the response and renders the processing result in the page. Abnormal situations include internal server errors (such as code errors, database connection failures, syntax errors, etc.), validation errors (such as incorrect data format submitted by the front end, although passing the front-end check but not passing the back-end check), insufficient permissions (such as the user trying to perform an operation without permission), resource not existing (incorrect application program interface address or data to be accessed has been deleted), network timeout (such as server processing time is too long, or server communication fails) and other abnormalities.

[0078] After the front-end code receives the response sent back by the server, it first checks the state code, if the state code is not in the success range, it is determined as a failed request or an abnormal request, according to the captured error information, it is displayed to the user, if a single input item is wrong (such as a username already exists), an error prompt is usually displayed near the input box, if a fatal error seriously affects the operation process, the process will be prevented from continuing, and the document object model (DOM) structure of the current page is updated to render an abnormal state page designed, for example, to display an explicit state code or error name (such as "server internal error"), a descriptive text (such as "server is out of order, please try again later"), a page illustration (such as displaying a related illustration or icon to relieve user anxiety).

[0079] In the embodiments of the present application, data processing is performed on the input operation of the user, realizing closed-loop linkage of business request and exception handling, real-time response to user operation and automatic detection of data exception, timely feedback of problems through a visual page, ensuring rapid exposure and transparency of the abnormal state, greatly improving fault tolerance and user experience, and ensuring smooth and controllable business process.

[0080] According to the embodiments of the present application, according to the abnormal information text and the user operation data, the abnormal analysis is performed based on the large text model to obtain the first solution, including: performing root cause analysis on the abnormal information text and the user operation data through the large text model to generate abnormal state information; and based on the abnormal knowledge base, the abnormal state information is retrieved and matched to generate the first solution; wherein the abnormal knowledge base is a key-value pair database constructed based on historical abnormal state information and historical solution process.

[0081] The input multi-dimensional data is input to the large text model for root cause analysis, and the abnormal state information (for example: "failed due to single transfer exceeding the limit") is output. The multi-dimensional data includes abnormal information text and user operation data. The user operation data includes but is not limited to the user's current transaction type (such as transfer, payment), the user's historical operation characteristics (such as high-frequency error type, operation speed), device environment information (such as network status, system version).

[0082] In the technical solutions of the present application, the user information (including but not limited to user personal information, user operation information, user image information, user device information, such as location information) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0083] In the scene of using personal information for automatic decision-making, the method, device and system provided by the embodiments of the present application all provide corresponding operation portal for the user to choose to agree or refuse the automatic decision-making result; if the user chooses to refuse, the expert decision-making process is entered. The expression "automatic decision-making" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies or economic, health, credit status of individuals through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have special experience, knowledge and skills, and reach a certain professional level.

[0084] The multi-dimensional data is pre-processed to standardize the abnormal information text (remove redundant characters, complete missing fields); the user operation data is structured (transaction type is mapped to enumeration value, historical characteristics are quantified as frequency / time length, etc. Device information is converted to key-value pair). A fusion template is designed, which is spliced into a model input in the format of "abnormal text + transaction scenario + behavior characteristics + environmental parameters", for example, the abnormal prompt "transaction failure", the current operation is transfer, the history is 3 times due to limit failure, and the network delay is 200ms. Call the text large model, guide the model to associate multi-dimensional characteristics through prompt words, and use causal reasoning ability to locate the core factors. Finally, the model output is post-processed, the key conclusions are extracted and standardized into structured abnormal state information similar to "single transfer exceeds limit and fails", and the root cause analysis is completed.

[0085] The abnormal knowledge base is stored in the form of key-value pair, such as key-value pair <error code, solution step list>, error code identifies the corresponding abnormal state information, and solution step list is the solution process of the solution. After obtaining the abnormal state information, the corresponding operation process is retrieved from the pre-set structured abnormal knowledge base, and the process is as follows:

[0086] Abnormal information analysis: First, extract the core features of the abnormal state to be processed (such as "transfer triggers risk interception" and "account login verification fails"), and convert them into standard error codes (such as "transfer risk interception" corresponds to error code "E0001") through pre-set rules (such as keyword matching, state code mapping), to ensure consistency with the knowledge base key-value format.

[0087] Knowledge base retrieval: Take the parsed error code as the query key, call the knowledge base retrieval interface, and accurately match the corresponding <error code, solution step list> key-value pair. If there are multiple similar matching results, select the optimal item according to the "matching degree priority".

[0088] Solution generation and optimization: Extract the matched solution step list, check the logical coherence of the steps (such as the rationality of the process "verify identity → submit credentials → remove restrictions"), and then convert the technical expressions into natural language that users can understand (such as "go to the security center to upload an ID card" instead of "call the identity verification interface"), to generate structured solutions.

[0089] Retrieval record retention: Record the retrieval time and matching results of the error code in real time, and provide data support for subsequent knowledge base updates (such as supplementing the details of high-frequency error codes), to ensure process closure.

[0090] In the embodiments of the present application, by combining the text large model root cause analysis and the historical knowledge base precise matching, the intelligent diagnosis and rapid solution generation of abnormal problems are realized, the processing accuracy is significantly improved, the repeated troubleshooting is reduced, and the response efficiency is optimized relying on historical experience, to improve the level of system automation operation.

[0091] According to an embodiment of the present application, based on the abnormal state page, the multi-modal large model is called to extract abnormal information text and position coordinates, comprising: obtaining a page screenshot and a node tree corresponding to the abnormal state page; and calling the multi-modal large model and analyzing the page screenshot and the node tree through the multi-modal large model to obtain the abnormal information text and the position coordinates.

[0092] Through the screen flow monitoring module of the electronic banking application, the current interface screenshot (page screenshot) and the accessibility DOM node tree during user operation are captured, the multi-modal large model is called to analyze the page screenshot and the node tree, error feature elements (such as red warning icons, error prompt pop-up windows, and grayed buttons) in the page are identified, and error information text and position coordinates are extracted.

[0093] The process of the multi-modal large model analyzing the page screenshot and the node tree is as follows: first, the page screenshot is input into the multi-modal large model, and interface elements (buttons, input boxes, etc.) and spatial positions are identified by using a visual module; at the same time, element labels, attributes, and other structured information are obtained by analyzing the DOM node tree. The multi-modal large model fuses visual features and node tree data through a cross-modal attention mechanism, establishes a mapping relationship between pixel regions and DOM nodes, generates a structured description with semantic annotations, realizes the association of “screenshot-element-node”, and provides a unified understanding basis for subsequent operation guidance.

[0094] In an embodiment of the present application, by intelligently analyzing the screenshot and the node tree through the multi-modal large model, interface abnormalities are perceived in real time, abnormal text and coordinates are accurately extracted, automatic fault positioning is realized, precise operation guidance is generated, and detection efficiency and accuracy are greatly improved, reducing the cost of manual troubleshooting.

[0095] According to an embodiment of the present application, based on the abnormal solution and the position coordinates, a multi-modal operation guide is generated through the multi-modal large model, comprising: based on the abnormal state page, the multi-modal large model analyzes the abnormal solution and the position coordinates to generate a multi-modal operation guide; wherein the multi-modal operation guide includes element positioning, operation actions, and explanation text.

[0096] The abnormal solution and the current interface screenshot are input into the multi-modal large model, and the multi-modal operation guide is output, the current interface screenshot represents a screenshot corresponding to the abnormal state page, and the clickable controls are labeled in the page screenshot through the position coordinates.

[0097] The element positioning represents interface element positioning, such as control corresponding identification. The operation action represents the operation that should be performed, such as clicking, sliding, inputting, etc. The explanation text represents dynamic prompt text, and generates colloquial explanation text for each step, such as “Please modify the transfer amount to less than 50,000 yuan”.

[0098] The multimodal large model processing process is: first, information input and analysis, the text logic of the abnormal solution scheme (such as "transfer amount exceeds the limit") and the interface screenshot Figure 1 The same as the incoming, the large model first extracts the core target (such as adjusting the amount, clicking the confirmation) in the solution through the natural language processing module, and then identifies the controls (input box, button, etc.) in the screenshot using the computer vision module, matches the unique identifier and location coordinates of each control, and completes element positioning. Then, operation action mapping, the large model combines the parsed target and the positioned elements to determine the action to be performed (such as input box corresponds to "input", confirmation button corresponds to "click"), ensuring that the action matches the abnormal solution requirements (such as amount exceeds the limit corresponds to "input compliant amount" action). Finally, generate an explanatory text, the large model generates a colloquial guide text (such as "please enter a number within 50,000 yuan") according to the element positioning and operation action, and finally integrates the element positioning, operation action, and explanatory text to form an operation guide.

[0099] In the embodiments of the present application, the multimodal large model generates precise multimodal operation guides that integrate visual positioning, action steps, and text explanations, greatly improving the intuitiveness and operability of the guides, reducing user understanding costs, and effectively improving the efficiency and experience of solving abnormal problems.

[0100] Figure 5 A dynamic guide flowchart of the operation guide generation method according to an embodiment of the present application is schematically shown.

[0101] As Figure 5 shown, according to an embodiment of the present application, based on the multimodal operation guide, the user is dynamically guided, including operation S510~operation S530.

[0102] In operation S510, the target operation control is determined based on element positioning, and the target operation control is visually highlighted.

[0103] Visual guidance is performed on the visual layer, a semi-transparent highlight layer is superimposed on the current interface, and the target operation control is visually highlighted according to the element positioning, which includes but is not limited to highlight display, flicker, border animation, color change, such as making the button to be clicked red, making the text box to be input blue.

[0104] In operation S520, the explanatory text is converted and processed by a speech synthesis engine to play the explanatory text.

[0105] Voice guidance is performed at the voice layer, and the description text is played through a voice synthesis engine, supporting switch control. The voice synthesis engine first cleans the description text (such as correcting punctuation and formatting), then converts it into standardized text recognizable by the voice synthesis engine, then disassembles the standardized text into phonemes and syllables, generates a voice spectrum in combination with the rhythm rules, synthesizes an audio signal simulating human voice according to the voice spectrum, and plays it through the device speaker. Real-time response to switch instructions, turn on to execute the playback process, and turn off to immediately interrupt audio generation and playback.

[0106] At operation S530, the abnormal solution is displayed to the user by folding the panel component.

[0107] Text guidance is performed at the text layer, and a folding panel component is deployed at the bottom of the application interface, which is in a collapsed state initially, and the folding panel component integrates structured abnormal solution. User can manually expand to view, and user can manually switch to an expanded state by triggering the panel header control, and display the abnormal solution (complete operation step list) to facilitate user browsing of complete content.

[0108] In the embodiments of the present application, through multi-modal guidance, the interaction inclusiveness is improved, and visual highlighting + voice playing + text guidance are provided synchronously to adapt to different scene needs, such as relying on voice for visually impaired users and relying on text for noisy environment. The operation area is displayed by visual highlighting, the operation efficiency is improved, the average error processing time of user is shortened, the visual guidance avoids user misunderstanding of abstract prompts, the operation success rate is improved, and the operation accuracy is optimized.

[0109] According to the embodiments of the present application, based on multi-modal operation guidance, the user is dynamically guided, further comprising automatically performing operation actions according to voice instructions in response to user input voice instructions; and real-time page state change events are monitored.

[0110] Based on operation guidance, user can automatically click / input voice instructions. First, user's voice instructions can be received through a microphone, and voice signals are converted into structured voice text by means of automatic speech recognition technology, while background noise is filtered to improve recognition accuracy. Second, a natural language processing module is used to analyze voice text, extract core operation intent (such as "submit form" and "modify amount") and target object, and match interface element information (such as control identifier) output by multi-modal model. According to the analysis result, the preset operation action (click, input, etc.) is associated, and the element state (such as whether the button can be clicked) is verified. Finally, the system interface control application program interface is called to trigger the corresponding operation, and the result (such as "operation successful") is returned after execution. Voice synthesis technology can also be used to broadcast feedback operation results to the user.

[0111] After the abnormality processing step is completed according to the operation guide, the page loading / refresh completion event and the page state change event are monitored in real time, and the perception process is repeatedly entered to perceive whether an abnormal state page appears in real time, so as to facilitate timely processing of the abnormal state page.

[0112] In the embodiment of the present application, automatic operation is performed according to a voice instruction, most interface operation errors can be automatically solved, user operation is reduced, user experience is improved, customer service cost is reduced, and the number of manual customer service work orders is reduced.

[0113] Based on the operation guide generation method described above, the present application further provides an operation guide generation device. The following will be described in detail Figure 6 The device is described in detail.

[0114] Figure 6 The structure block diagram of the operation guide generation device according to the embodiment of the present application is schematically shown.

[0115] As Figure 6 shown, the operation guide generation device 600 of the embodiment includes an abnormal data extraction module 610, an abnormality analysis module 620, an operation guide generation module 630, and a dynamic guide module 640.

[0116] The abnormal data extraction module 610 is configured to monitor a page state change event in real time, respond to feedback of an abnormal state page to a user, and call a multi-modal large model to extract abnormal information text and position coordinates based on the abnormal state page. In an embodiment, the abnormal data extraction module 610 can be configured to perform the operation S210 described above, and details are not repeated here.

[0117] The abnormality analysis module 620 is configured to, in a case where the abnormal state page is triggered for the first time, perform abnormality analysis based on a text large model according to the abnormal information text and user operation data, and obtain a first solution. In an embodiment, the abnormality analysis module 620 can be configured to perform the operation S220 described above, and details are not repeated here.

[0118] The operation guide generation module 630 is configured to generate a multi-modal operation guide through the multi-modal large model based on an abnormal solution and the position coordinates, wherein the abnormal solution includes the first solution. In an embodiment, the operation guide generation module 630 can be configured to perform the operation S230 described above, and details are not repeated here.

[0119] The dynamic guide module 640 is configured to guide a user based on the multi-modal operation guide. In an embodiment, the dynamic guide module 640 can be configured to perform the operation S240 described above, and details are not repeated here.

[0120] According to an embodiment of the present application, the operation guide generation module 630 comprises a multi-modal large model analysis unit configured to analyze the abnormal solution and the location coordinates based on the abnormal state page by using the multi-modal large model to generate the multi-modal operation guide; wherein the multi-modal operation guide comprises element positioning, operation action and description text.

[0121] According to an embodiment of the present application, the dynamic guide module 640 comprises a visual display unit configured to determine a target operation control based on the element positioning and visually highlight the target operation control; a voice playing unit configured to convert and process the description text by using a voice synthesis engine to play the description text; and a solution display unit configured to display the abnormal solution to the user by using a folding panel component.

[0122] According to an embodiment of the present application, the dynamic guide module 640 further comprises an automatic operation unit configured to automatically execute the operation action according to the voice instruction in response to the user input voice instruction; and to listen to the page state change event in real time.

[0123] According to an embodiment of the present application, the abnormal data extraction module 610 comprises a page screenshot unit configured to obtain a page screenshot and a node tree corresponding to the abnormal state page; and a model analysis unit configured to call the multi-modal large model and analyze the page screenshot and the node tree by using the multi-modal large model to obtain the abnormal information text and the location coordinates.

[0124] According to an embodiment of the present application, the abnormal analysis module 620 comprises a root cause analysis unit configured to perform root cause analysis on the abnormal information text and the user operation data by using the text large model to generate abnormal state information; and a retrieval matching unit configured to perform retrieval matching on the abnormal state information based on an abnormal knowledge base to generate the first solution; wherein the abnormal knowledge base is a key-value pair database constructed based on historical abnormal state information and historical solution processes.

[0125] According to an embodiment of the present application, the abnormal solution further comprises a second solution, and the device 600 further comprises a historical solution acquisition module configured to, in a case where the abnormal state page is not triggered for the first time, acquire a solution corresponding to the abnormal state page based on a solution database to obtain the second solution; wherein the solution database is constructed based on historical abnormal state pages and corresponding first solutions.

[0126] According to an embodiment of the present application, the apparatus 600 further comprises a service processing module configured to, in response to the input operation of the user, receive a service processing request corresponding to the input operation, and perform data processing based on the service processing request; and in a case where a data processing result is abnormal, feed back the abnormal state page to the user based on the data processing result.

[0127] According to an embodiment of the present application, any one or more of the abnormal data extraction module 610, the abnormal analysis module 620, the operation guidance generation module 630, the dynamic guidance module 640, the historical solution acquisition module, and the service processing module can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the abnormal data extraction module 610, the abnormal analysis module 620, the operation guidance generation module 630, the dynamic guidance module 640, the historical solution acquisition module, and the service processing module can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of the three implementation ways of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the abnormal data extraction module 610, the abnormal analysis module 620, the operation guidance generation module 630, the dynamic guidance module 640, the historical solution acquisition module, and the service processing module can be at least partially implemented as a computer program module which can perform corresponding functions when it is run.

[0128] Figure 7 The block diagram of the electronic device suitable for implementing the operation guidance generation method according to an embodiment of the present application is schematically shown.

[0129] As Figure 7As shown, the electronic device 900 according to an embodiment of the present application includes a processor 901 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application.

[0130] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0131] According to embodiments of the present application, the electronic device 900 can also include an input / output (I / O) interface 905 which is also connected to the bus 904. The electronic device 900 can also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as necessary. A removable medium 911 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out from the removable medium 911 is installed in the storage section 908 as necessary.

[0132] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0133] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include one or more of the above-described ROM 902 and / or RAM 903 and / or one or more memories other than the ROM 902 and the RAM 903.

[0134] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the operation guide generation method provided by the embodiments of the application.

[0135] The above-described functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 901. According to the embodiments of the application, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0136] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 909, and / or installed from the detachable medium 911. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0137] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 909, and / or installed from the removable media 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be implemented by the computer program modules.

[0138] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0139] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0140] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations, even if such combinations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present application. All such combinations and / or integrations are within the scope of the present application.

Claims

1. A method for generating operation instructions, characterized in that, The method comprises: Real-time monitoring of page state change events, in response to feeding back an abnormal state page to a user, based on the abnormal state page, calling a multi-modal large model to extract abnormal information text and location coordinates; In the case of first triggering the abnormal state page, based on the abnormal information text and user operation data, abnormal analysis is performed based on a text large model to obtain a first solution; Based on the abnormal solution and the location coordinates, a multi-modal operation guide is generated through the multi-modal large model; wherein the abnormal solution includes the first solution; and Based on the multi-modal operation guide, the user is dynamically guided.

2. The method of claim 1, wherein, The multi-modal operation guide generated by the multi-modal large model based on the abnormal solution and the location coordinates comprises: Based on the abnormal state page, the multi-modal large model analyzes the abnormal solution and the location coordinates to generate the multi-modal operation guide; wherein the multi-modal operation guide includes element positioning, operation action and instruction text.

3. The method of claim 2, wherein, Based on the multi-modal operation guide, the user is dynamically guided, which comprises: Based on the element positioning, determine the target operation control, and visually highlight the target operation control; Convert the instruction text through a speech synthesis engine to play the instruction text; and Display the abnormal solution to the user through a folding panel component.

4. The method of claim 2, wherein, Based on the multi-modal operation guide, the user is dynamically guided, which further comprises: In response to the user input voice instruction, automatically execute the operation action according to the voice instruction; and Real-time monitoring of the page state change event.

5. The method of claim 1, wherein, Based on the abnormal state page, calling a multi-modal large model to extract abnormal information text and location coordinates comprises: Obtain the page screenshot and node tree corresponding to the abnormal state page; and Call the multi-modal large model and analyze the page screenshot and the node tree through the multi-modal large model to obtain the abnormal information text and the location coordinates.

6. The method of claim 1, wherein, Based on the abnormal information text and user operation data, the text large model is used to perform abnormal analysis to obtain a first solution, which comprises: Root cause analysis of the abnormal information text and the user operation data is performed through the text large model to generate abnormal state information; and Based on the abnormal knowledge base, the abnormal state information is retrieved and matched to generate the first solution; wherein the abnormal knowledge base is a key-value pair database constructed based on historical abnormal state information and historical solution process.

7. The method according to any one of claims 1 to 6, characterized in that, The abnormal solution also includes a second solution, and the method further comprises: In the case of non-first triggering of the abnormal state page, based on the solution database, the solution corresponding to the abnormal state page is obtained to obtain the second solution; wherein the solution database is constructed based on historical abnormal state pages and corresponding first solutions.

8. The method of any one of claims 1-6, wherein, The method further comprises: In response to the user's input operation, receive the business processing request corresponding to the input operation, and perform data processing based on the business processing request; and In the case of an abnormal data processing result, an abnormal state page is fed back to the user based on the data processing result.

9. An operation guide generation device, characterized by comprising: The device comprises: An abnormal data extraction module is configured to monitor page state change events in real time, and in response to feeding back an abnormal state page to a user, call a multi-modal large model to extract abnormal information text and location coordinates based on the abnormal state page; An abnormal analysis module is configured to, in the case of first triggering the abnormal state page, perform abnormal analysis based on a text large model according to the abnormal information text and user operation data, and obtain a first solution; An operation guide generation module is configured to generate a multi-modal operation guide based on an abnormal solution and the location coordinates through the multi-modal large model; wherein the abnormal solution includes the first solution; and A dynamic guide module is configured to guide the user based on the multi-modal operation guide.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8.

12. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8. The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-8.

Images