Real scene-based cross-software software operation mixed interaction help method and system

Through deep learning technology, a cross-software operation hybrid interactive help system is built with real-life semantic understanding, which solves the problems of users' difficulty in finding functions in software and complex cross-software operation, realizes accurate perception and efficient guidance, and improves the software user experience and efficiency .

CN118964566AActive Publication Date: 2024-11-15XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411033977.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

Users find it difficult to find the required functions when using electronic device software, and the instruction manual is not intuitive, resulting in low software usage efficiency and complex cross-software operation. The existing large models have insufficient capabilities in the language-video multimodal field, making it difficult to accurately perceive users. Continuous operation.

Method used

Using natural language processing technology based on deep learning combined with computer technology to build a cross-software operation hybrid interactive help system with real-life semantic understanding. By extracting software and hardware information, generating training data, building a hybrid interactive help model, accurately perceive the user's operating environment and Natural language issues, providing real-life guidance.

Benefits of technology

It improves the efficiency and user experience of software use, accurately perceives user operations, reduces conversation rounds, reduces training and usage costs, and achieves efficient interactive help across software and cross-platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964566A_ABST
    Figure CN118964566A_ABST
Patent Text Reader

Abstract

The invention discloses a live-action-based cross-software software operation mixed interaction help method and system, and the method comprises the steps: extracting software information and button information of electronic equipment, and generating corresponding training data; and pre-constructing a real-scene semantic comprehension-based cross-software software operation mixed interactive help model adopting deep learning, inputting the software operation mixed interactive help model according to question information, outputting a sorting score of each semantic analysis result, and selecting one or more semantic analysis results as user question comprehension results, and providing interactive operation help for the user, wherein the interactive operation help comprises step-by-step prompting or automatic button clicking operation. According to the method, the problem solving range of a software operation interaction help system based on real scene semantic understanding is greatly widened, the use convenience and the problem solving efficiency are improved, the problem that a user uses a general help instruction book is not intuitive is solved, and the good experience feeling of using a product by the user is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a cross-software software operation hybrid interactive help method and system based on real-scene semantic understanding for various computers, mobile terminals such as mobile phones, and various electronic devices such as household appliances and industrial appliances. Background Art

[0002] The convenience of using electronic devices is determined by whether the software in the current electronic equipment hardware is easy to use. Users have three main problems when using the software of electronic devices: first, they "don't know" whether a certain software has the function they need, or they don't know whether any software has developed the function they need; second, they "can't find" where the required function button is in the software; third, they know that a certain software has the function they need and where it is, but they just "don't know how to use it" or use it poorly. Although the software has help manuals, understanding the manual and finding functions are still far from the experience of having a professional guiding the operation. These three problems greatly affect the efficiency of software use and restrict the widespread popularization of software.

[0003] At the same time, some electrical appliances, such as household appliances, have many operating buttons, the operating interface is not very user-friendly, and the instruction manual is inconvenient to use. In addition, each appliance is equipped with an APP, which increases the difficulty of user management and search. It is imperative to change these problems through artificial intelligence technology to realize a unified electronic instruction manual that can answer questions in natural language and can intuitively guide users in real-life operations.

[0004] Since all software has its limitations, users often encounter the situation when the software they are using cannot meet their requirements well, or even cannot meet their requirements at all. For example, drawing a flowchart in Word is not as convenient as drawing it in Visio and inserting it into Word. Sometimes when operating a mobile phone, it is necessary to complete the work in the computer software first, and then finally complete the work through the mobile phone. At this time, cross-software solutions are needed. Since the operating system is also software, cross-software operations can not only complete cross-software interactive help of the local machine, but also cross-platform and electronic device interactive help. Of course, the premise is that different platforms and devices can communicate with each other.

[0005] With the development of natural language processing technology, image recognition technology and multimodal technology based on deep learning, it has become possible to understand users' natural language questions and then guide users to perform software operations. During the guidance process, there are two modes. One is to use automation technology to reduce users' operations as much as possible, and the other mode is to guide users to perform specific operations, such as clicking buttons to complete operations. The two modes each correspond to different needs and have different advantages and disadvantages. The automation mode is more efficient at certain times. Guiding users to perform specific operations step by step can help users learn specific operations. When encountering similar problems in the future, they can operate directly, saving the time of asking questions, which also improves efficiency.

[0006] In fact, people often have a lot of omissions and ambiguities in natural language conversations in real scenarios. For example, when asked "where is the translator", in reality it may be looking for the translator, while in Word it is looking for the translation button. Without scenario information, accurate understanding can only be achieved by increasing the number of conversation turns. However, when combined with actual scenarios, such problems sometimes have no ambiguity. Even if there is ambiguity, the number of conversation turns or the complexity of the conversation can be reduced. In today's rapidly developing artificial intelligence, it is critical to perceive the real scene of the software operation process and correctly handle the multimodal problem of language-continuous button clicking actions.

[0007] At present, the big model technology has developed from a simple large language (text) model (LLM) to an image-language multimodal model (VLM). In the field of language-video multimodality, continuous actions need to be perceived and understood in combination with language. The current big models are not strong in this regard. Even OpenAI's sora, which generates a one-minute video, has many loopholes. The software needs to perceive the user's continuous operation for real-life interactive help, and sometimes the operation time is far more than one minute. In addition, the guidance information returned by the big model is text and pictures, which is still far from accurate and intuitive guidance.

[0008] In fact, the perception and understanding of continuous actions is the most important content for understanding the real world, and is currently an important factor restricting the further improvement of the intelligence of large models. Selecting those real-life scenarios that can be accurately perceived and understood and organically combining them with existing large models is the only way to improve the capabilities of large models under existing conditions. Real-scene-based software operations are such scenarios, and cross-software software operation interactive assistance can further significantly enhance the ability of large models to handle real-world problems.

[0009] At the same time, the training and use costs in the language-video multimodal field are high and the success rate of accurately understanding videos is not ideal. By adopting natural language processing technology based on deep learning combined with computer technology to capture the continuous actions of user software operations, the success rate can be greatly improved and the training and use costs can be reduced. In this way, it is imperative to accurately perceive the user's software operation process, accurately understand the user's natural language problems in the actual software operation scene, and finally realize accurate guidance of user software operation technology. Summary of the invention

[0010] In order to solve the above-mentioned defects existing in the prior art, the purpose of the present invention is to provide a hybrid interactive help method and system for software operations based on real scenes and across software. The present invention greatly broadens the scope of problems solved by the software operation interactive help system based on real-scene semantic understanding, improves the ease of use and efficiency of problem solving, solves the problem that users are not intuitive when using general help manuals, improves the efficiency of solving problems encountered in the use of electronic equipment software, and enhances the quality of user experience when using the product.

[0011] The present invention is achieved through the following technical solutions.

[0012] One aspect of the present invention provides a real-scene-based cross-software software operation hybrid interactive assistance method, comprising the following steps:

[0013] Step 111, extracting software information in the electronic device, button information of operation buttons of interactively operated software, collecting usage problems and help information and help information of other problems, and generating training data of a cross-software software operation hybrid interactive help system based on real scene semantic understanding;

[0014] Step 112, extracting the software and hardware information in the user's electronic device and constructing a data model of the user's electronic device;

[0015] Step 113, pre-building a software operation hybrid interactive help model based on real-scene semantic understanding and cross-software using deep learning according to the training data;

[0016] Step 114, extracting natural language question information asked by the user;

[0017] Step 115, extracting the current real scene information of the software according to the constructed user electronic device data model, and generating real scene natural language question information according to the user's question and operation selection;

[0018] Step 116, inputting the real-scene natural language question information or the user question information into the cross-software software operation hybrid interactive help model based on real-scene semantic understanding, and outputting the ranking score of each semantic analysis result;

[0019] Step 117 , selecting one or more semantic parsing results as understanding results of the user's question according to the ranking scores, and providing interactive operation help to the user based on the cross-software hybrid interactive help model.

[0020] Preferably, the software information in the electronic device includes:

[0021] The operating system and its version name, software name, software version and specific function information in the electronic device;

[0022] Button information includes:

[0023] Button name, button range information, button relationship information and button function information;

[0024] Button range information refers to the range contained in the button border or vertex information and the transformation relationship information of the button range between different screen sizes and different resolutions;

[0025] Button relationship refers to the relationship between the current button and the sub-button that appears when the current button is clicked;

[0026] Button function information refers to the description information of the button function;

[0027] Cross-software refers to software on different electronic devices or different software on the same device, and the software includes an operating system.

[0028] Preferably, step 111 is specifically as follows:

[0029] Step 1-1, extracting button information of interactively operable software operation buttons according to a real-scene cross-software software operation hybrid interactive help system;

[0030] Step 1-2, based on the knowledge of button usage problems, a rule-based method or an if-then-based form is used to generate training data of real-scene natural language question information of usage problems combined with real-scene information, user intentions and help information, and generate a software button relationship tree;

[0031] The button relationship tree refers to the tree-like hierarchical relationship formed by the relationship between the parent and child buttons of the buttons of the interactively operable software;

[0032] The real-scene natural language question information, user intention and help information combined with the use problem of the real-scene information are combined by using the real-scene information in a form of natural language omission recovery as part of the question information, or using the real-scene information as part of the question information in the form of an attention mechanism;

[0033] Step 1-3, further generating training data based on the synonym database and the real-scene natural language question information combined with the usage problem of the real-scene information, user intention and help information;

[0034] Steps 1-4, collecting the usage problems and help information of the interactive software and help information of other problems, including problems that can be solved across software and their help information, collecting text, voice and picture data of other types of problems and solutions on the Internet and books, and annotating them as training data;

[0035] Help information for other problems refers to the types of problems that can be solved based on the existing generative artificial intelligence large model, but does not include problems with software operation and use;

[0036] Step 1-5: using the training data as training samples.

[0037] Preferably, step 112 is specifically as follows:

[0038] Extract the software and hardware information from the user's electronic devices, including the software and hardware information of the user's computer, mobile phone, pad, iPad and household appliances, to build a data model of the user's electronic devices;

[0039] The hardware and software information includes: the CPU, memory size, hard disk size and user screen resolution and zoom ratio hardware information in the user's electronic device, the operating system and its version name, the software name, software version and its storage location information.

[0040] Preferably, step 113 is specifically as follows:

[0041] The training samples and the help information are used to train a cross-software software operation hybrid interactive help model based on real-scene semantic understanding, wherein the software operation hybrid interactive help model adopts a dialogue form based on deep learning, and during the training, the software real-scene information corresponding to the software operation questions asked by the user is added to the training model in a weighted manner;

[0042] The software real-scene information is added to the training model in the form of dialogue using natural language processing technology in deep learning. This is a Transformer model that uses the attention mechanism, or a generative artificial intelligence model that uses the Transformer.

[0043] Preferably, when the Transformer model is adopted, the real scene encoding layer is fused by adding the embedding layer to obtain the real scene embedding vector, which is input into the encoding component of the Transformer to perform feature learning and weight change;

[0044] According to the embedding vector of the real scene fused into the encoding layer, multi-head self-attention calculation is performed and the obtained attention vector is used as the intermediate state of the encoding layer;

[0045] After passing through the layer-by-layer feedforward neural network, the hidden layer feature vectors corresponding to all the tags in the input sequence are obtained;

[0046] Select the cross entropy loss function and obtain the cross entropy loss of the module after scene omission restoration.

[0047] Preferably, step 115 is specifically as follows:

[0048] Step 5-1, extracting natural language question information asked by the user when using the interactive software;

[0049] Extract the current real-life information of the software, including the software name, the actual state of the menu bar, button information, button click information, and information on the screen size and resolution of the pop-up menu and button, as well as whether the software usage problem button has been clicked;

[0050] Acquisition of real-scene information, including using the "button information" provided by the operating system and software interface, or using the hook method, system monitoring or polling scheduling method, or deep learning-based image processing technology to identify button and menu operation information;

[0051] Software usage issues, including button location search, button function query, button usage help, and issues that need to be solved across software;

[0052] Step 5-2, when the user clicks the software usage question button, the real scene information and the natural language question information are combined to generate the real scene natural language question information, wherein the combination method includes using the real scene information as part of the question information in the form of language omission restoration, or using the real scene information as part of the question information in the form of an attention mechanism;

[0053] If the user does not click the software usage question button, the natural language question information based on the real scene only includes the user's question information;

[0054] Users can improve system response speed by selecting general questions or software operation questions.

[0055] Preferably, step 117 is specifically as follows:

[0056] Step 7-1, selecting one or more semantic parsing results according to the ranking scores as the understanding results of the user's question;

[0057] Step 7-2, providing interactive operation help to the user based on the cross-software hybrid interactive help model, the cross-software hybrid interactive help model includes:

[0058] Button location search help: Based on the location of the current menu state of the interactive software and the button relationship tree, prompts are given to click one or more buttons in sequence. The prompt is an arrow pointing to the button, or a highlight or flashing at the button location;

[0059] Button function query help: provide an introduction to button functions, including text, pictures and videos;

[0060] Button usage help: Combine button function query help and button location search help to introduce the real-life results;

[0061] Button click help: including step-by-step prompts and automatic button clicks, which can be set by the user;

[0062] The step-by-step prompts: provide operation instructions and guide the user to click buttons in sequence to complete;

[0063] The automatic button click: automatically completes the button click or calls the corresponding method to complete the button click, and the user completes the operation he needs to complete;

[0064] Cross-software recommendation: For user problems that can only be solved across software or platforms, recommended usage information is given based on the user's electronic device information model, including giving priority to software suitable for the local machine. If there is no suitable software on the local machine, other suitable software is recommended;

[0065] Cross-device step-by-step guidance or automatic clicks: For user problems that can only be solved across devices, with the user's permission, interactive help information will be transmitted to the corresponding device through network communication, and the corresponding software and functions will be activated to achieve cross-platform step-by-step prompts and automatic button clicks;

[0066] Software usage issues: button location search, button function query, button usage help, and issues that need to be solved across software;

[0067] Step 7-3, if the user does not click the software usage problem button, then extract the results matching the real scene information and the solution to the current software usage problem based on the real scene information, and have a dialogue with the user, initiate the next round of dialogue based on the real scene information, and let the user confirm the returned result;

[0068] If the user confirms that the software is being used or that the solution can be used across software, the user will be guided to solve the corresponding problem; if the user confirms that the solution is not being used, all the content reported by the system will be returned; if there is no content related to the real scene information in the returned results, all the content reported by the system will be returned;

[0069] Step 7-4: When the user’s purpose is not clearly understood, initiate a conversation to further clarify the user’s needs.

[0070] Preferably, the natural language question information is voice information or text information;

[0071] For voice information, before the natural language information is semantically parsed, the natural language information is converted into text information through voice recognition.

[0072] Another aspect of the present invention provides a real-scene-based cross-software software operation hybrid interactive help system, comprising:

[0073] The interactive software button information acquisition module is used to help system developers obtain button information by using software operation mixed interaction, where the button function information, button border or vertex information, current button and its parent button are filled in manually in the menu provided by the development platform, or automatically filled in by OCR, target detection in deep learning or other image recognition;

[0074] A real-scene natural language question information-answer pair generation module is used to generate a software button relationship tree based on the button information and the button usage problems; further generate training data; use the training data as training samples; generate text, voice, and picture data based on the synonym library and the real-scene natural language question information combined with the usage problems of the real-scene information, user intentions and help information, and other various problems and solutions on the Internet and books, and further generate training data;

[0075] A software operation interactive help model construction module is used to train the training samples to obtain a cross-software software operation hybrid interactive help model based on real-scene semantic understanding;

[0076] The real scene acquisition module is used to obtain real scene information including software name and version, current menu status information, operating system and its version, obtain "button information" provided by the software interface, or use the hook method, system monitoring method or polling scheduling method, or deep learning-based image processing technology to identify button and menu operation information;

[0077] A real-scene natural language question information generation module extracts the natural language question information of the user when using the interactive software according to the user's selection, combines the real-scene information with the natural language question information to generate the real-scene natural language question information or only uses the user's question information;

[0078] The cross-software hybrid interactive help module based on real-scene semantic understanding adopts a dialogue system based on deep learning. It is used to understand the user's natural language questions containing real-scene information. The user can select general questions or software operation questions to improve the system response speed, and interactive help is given according to the questions and the above choices, including button function query help, button location search help or button how to use help;

[0079] Hints are arrows pointing to buttons, or highlights or flashes at the button locations; button click help modes include step-by-step prompts and automatic button clicks, which are set by the user;

[0080] The system also includes: a speech recognition module, which is used to convert the natural language information into text information through speech recognition before the semantic analysis module performs semantic analysis on the natural language information.

[0081] The present invention adopts the above technical solution, which has the following beneficial effects:

[0082] The solution of the present invention accurately perceives the user's software operating environment and continuous operating actions, combines natural language questions asked by the software operation, and provides users with precise guidance on solving problems. It solves the understanding difficulties and even errors caused by the lack of scene semantic information in general deep learning methods, improves the human-computer interaction effect and interaction efficiency of software operations that can cross software, and gradually prompts to solve problems and automatically clicks to solve problems, as well as the integration of problem solving in specific software operation scenarios and problem solving of other types of problems to further improve the efficiency of software interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute an improper limitation of the present invention. In the drawings:

[0084] Figure 1 It is a flow chart of a software operation hybrid interactive assistance method based on real scenes and capable of crossing software in an embodiment of the present invention;

[0085] Figure 2 It is a structural schematic diagram of a development module of a software operation hybrid interactive help system based on real scenes and capable of crossing software according to an embodiment of the present invention;

[0086] Figure 3 It is a structural schematic diagram of a service module of a software operation hybrid interactive help system based on real scenes and capable of crossing software in an embodiment of the present invention;

[0087] Figure 4 It is a step-by-step guide or automatic click structure diagram across devices. DETAILED DESCRIPTION

[0088] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0089] like Figure 1 FIG. 1 is a flow chart of a natural language understanding method provided by an embodiment of the present invention. The present invention provides a software operation hybrid interactive assistance method based on real scenes and capable of crossing software, comprising the following steps:

[0090] Step 111, extracting software information in the electronic device, button information of operation buttons of interactively operated software, collecting usage problems and help information and help information of other problems, and generating training data of a cross-software software operation hybrid interactive help system based on real-scene semantic understanding.

[0091] The specific steps include:

[0092] Step 1-1, extracting button information of interactively operable software operation buttons according to a real-scene-based cross-software software operation hybrid interactive help system.

[0093] Cross-software refers to software on different electronic devices or different software on the same device. The software here includes the operating system.

[0094] The software information in the electronic device includes: the operating system in the electronic device and its version name, the software name, software version and its specific function information.

[0095] Button information includes button name, button range information, button relationship information between buttons and button function information. Button range information refers to the range contained in the button border or vertex information and the transformation relationship information between button ranges in different screen sizes and resolutions; button relationship refers to the relationship between the current button and the sub-button that appears when the current button is clicked; button function information refers to the description information of the button function; cross-software refers to software on different electronic devices or different software on the same device, and software includes operating system.

[0096] In one embodiment: To obtain the button information of "Flowchart Arrow" in "Word" software, first start the "Word" software and the real-scene cross-software software operation hybrid interactive help system, put the mouse on the upper left corner and lower right corner of the "Arrow Collection" button, and follow the system prompts to obtain the button range information, and then manually fill in the button name "Arrow Collection", the button parent node "Shape" button, and then fill in the function information and usage steps. For functions that need to cross software, the corresponding advantage information of the software Visio that needs to cross software should be filled in the Word function information, such as more convenient drawing and adjustment tools.

[0097] Step 1-2, based on the knowledge of button usage problems, a rule-based method or an if-then-based form is used to generate training data of real-life natural language question information of usage problems combined with real-life information, user intentions and help information, and generate a software button relationship tree.

[0098] The button relationship tree refers to the tree-like hierarchical relationship formed by the relationship between parent and child buttons of the interactive software.

[0099] It combines the usage problems of real-scene information with real-scene natural language question information, user intention and help information. The combined method uses the real-scene information as part of the question information in the form of natural language omission recovery, or uses the attention mechanism to use the real-scene information as part of the question information.

[0100] Steps 1-3, further generate training data based on the synonym database and usage questions combined with real-life information, user intentions and help information.

[0101] Steps 1-4, collect the usage problems and help information of the interactive software and help information for other problems, including problems that can be solved across software and their help information, collect text, voice and image data of other types of problems and solutions on the Internet and books, and annotate them as training data, and use the above training data as training samples.

[0102] Other types of problems refer to the types of problems that can be solved by existing generative artificial intelligence large models, but do not include problems related to software operation and usage.

[0103] In one embodiment, a synonym for "save" may be: save a file.

[0104] Collect other usage questions and help information of users of interactive software, mark them, and use them as training data; in one embodiment, "save" can also be described as: save the file, etc.

[0105] The input sequence INPUT1 for the user query question is:

[0106] INPUT1= <x1,x2,…,x m >

[0107] Then the input sequence INPUT2 combined with the real scene should be:

[0108] INPUT2= <y1,y2,…,y n ,x1,x2,…,x m >

[0109] Among them, y1,y2,…,y n Real scene environment information and user operation information

[0110] The processed combined real scene input sequence INPUT3 is:

[0111] INPUT3= <z,x1,x2,…,x m >

[0112] Where z is the content corresponding to the real scene, which can be the omitted restored dialogue content of the real scene, or the content corresponding to the real scene after being processed by the attention mechanism.

[0113] Step 1-5: Use the above training data as training samples.

[0114] Step 112, extracting the software and hardware information in the user's electronic device and constructing a user's electronic device data model, specifically includes:

[0115] Extract the software and hardware information from the user's electronic devices, including the user's computer, mobile phone, pad (including iPad) and household appliances, to build a data model of the user's electronic devices.

[0116] The hardware and software information includes: the CPU, memory size, hard disk size, user screen resolution and zoom ratio and other hardware information in the user's electronic device, the operating system and its version name, software name, software version and its storage location information.

[0117] Step 113, based on the training data, pre-build a software operation hybrid interactive help model based on real scene semantic understanding and cross-software using deep learning, specifically including:

[0118] The software operation hybrid interactive help model that can cross software and is based on real-scene semantic understanding is obtained by training with training samples and help information. The software operation hybrid interactive help model adopts a dialogue form based on deep learning, and during training, the software real-scene information corresponding to the software operation questions asked by users is added to the training model in a weighted manner.

[0119] The software real-scene information is added to the training model in the form of dialogue using natural language processing technology in deep learning. It can be a Transformer model using the attention mechanism or a generative artificial intelligence model using the Transformer.

[0120] In the case of the Transformer model, the scene encoding layer is added and fused through the embedding layer to obtain the scene embedding vector E sense , and input it into the encoding component of Transformer for feature learning and weight change. The encoding component is composed of multiple identical encoding layers, each of which contains a multi-head self-attention sublayer and a layer-by-layer feedforward network sublayer. Each sublayer uses residual connections and layer normalization operations.

[0121] After the embedding vector of the fused real scene is input into the encoding layer, multi-head self-attention calculation is first performed and the obtained attention vector is represented as the intermediate state of the encoding layer, which is expressed as follows:

[0122]

[0123] in, represents the input vector of the i-th encoding layer, which is used as the query vector, key vector, and value vector of the attention machine function. represents the attention vector obtained by the i-th encoding layer, d k is the dimension of the input key vector.

[0124] After passing through the layer-by-layer feedforward neural network, the feature vector of the current coding layer fused with the real scene information is obtained. Finally, after passing through the last coding layer, the hidden layer feature vector corresponding to all the tags in the input sequence is obtained, which can be expressed as:

[0125] H sense =FFN(Z sense )

[0126] H sense =(h1,h2,…,h l ),h i ∈R M

[0127] Among them, FFN is a feedforward neural network, H sense is the feature vector representation of the scene output by the last encoder, Z sense is the attention vector output by the last encoder, h i represents the i-th input vector, l is the length of the input sequence including the special label, and M represents the vector dimension of the hidden layer.

[0128] Loss function, select the cross entropy loss function, the specific calculation formula is as follows:

[0129]

[0130] y max =max(P(y I ,y S |I * ))

[0131] L Sense =∑ i (y i ln(y max )+(1-y i ln(1-y max )))

[0132] Among them, y i Represents the probability output of all intent categories and slot names corresponding to the input sequence of fused real scene information, y max represents the maximum value of the probability of the intent category and slot name in the probability output, L Sense Represents the cross entropy loss of the module after scene omission restoration.

[0133] Step 114, extracting the user's natural language question information, including: if the user does not click the software usage question button, the user's question is solved as a non-software usage question.

[0134] Step 115, based on the construction of the user electronic device data model, extract the current real scene information of the software, and generate real scene natural language question information according to the user's question and operation selection, including:

[0135] Step 5-1, extracting the natural language question information of the user when using the interactive software and the current real scene information of the software.

[0136] Extracting the current real-scene information of the software means: when the user is using the software, it includes the software name, the actual status of the menu bar, button information, button click information, and the pop-up menu and button screen size and resolution information, and whether the software usage problem button is clicked; the real-scene information is obtained using the "button information" provided by the operating system and software interface, or the button and menu operation information obtained by using the hook method, system monitoring method or polling scheduling method, and deep learning-based image processing technology to identify; software usage problems, including button location search, button function query and button usage help, as well as problems that need to be solved across software.

[0137] Step 5-2: If the user clicks the software usage question button, the real scene information and the natural language question information are combined to generate the real scene natural language question information. The combination method includes restoring the real scene information in the form of language omission as part of the question information, or using the attention mechanism to make the real scene information part of the question information. If the user does not click the software usage question button, the natural language question information based on the real scene only includes the user's question information.

[0138] Users can improve system response speed by selecting general questions or software operation questions.

[0139] Step 116, inputting the natural language question information based on the real scene or the user question information into the cross-software software operation hybrid interactive help model based on the real scene semantic understanding, and outputting the ranking score of each semantic analysis result, including:

[0140] The user's natural language question information or natural language question information based on real scenes is input into a cross-software hybrid interactive help model based on real-scene semantic understanding using deep learning; the ranking score of each semantic analysis result is output, and the semantic analysis result combined with the real scene and its ranking score are given.

[0141] Step 117, one or more semantic parsing results are selected according to the ranking scores as the understanding results of the user's question, and then interactive operation help is provided to the user based on the cross-software hybrid interactive help model, including:

[0142] Step 7-1, selecting one or more semantic parsing results according to the ranking scores as the understanding results of the user's question;

[0143] Step 7-2, providing interactive operation help to the user based on the cross-software hybrid interactive help model, the cross-software hybrid interactive help model includes:

[0144] Button location search help: Based on the location of the current menu state of the interactive software and the button relationship tree, prompts are given for clicking one or more buttons in sequence. The prompt can be an arrow pointing to the button, or it can be highlighted or flashing at the button location.

[0145] Button function query help: provide an introduction to button functions, including text, pictures and videos;

[0146] Button usage help: The real-life results of combined button function query help and button location search help are introduced.

[0147] Button click help mode: including step-by-step prompts and automatic button clicks, which can be set by the user.

[0148] Step-by-step instructions: Provide operation instructions and guide users to click buttons one by one to complete the task.

[0149] Automatic button click: Automatically complete button clicks or call corresponding methods to complete button clicks, allowing users to complete the operations they need to complete.

[0150] Cross-software recommendations: For user problems that can only be solved across software or platforms, recommended usage information is given based on the user's electronic device information model, including priority recommendations for software suitable for the local machine. If the local machine does not have suitable software, other suitable software can be recommended.

[0151] Cross-device step-by-step guidance or automatic clicks: For user problems that can only be solved across devices, with the user's permission, interactive help information will be transmitted to the corresponding device via network communication, and the corresponding software and functions will be launched to achieve cross-platform step-by-step prompts and automatic button clicks.

[0152] Software usage issues: finding button locations, querying button functions, and helping with how to use buttons, as well as issues that need to be solved across software.

[0153] Step 7-3, if the user does not click the software usage problem button, then extract the results matching the real scene information and the solution to the current software usage problem in the returned results according to the real scene information, and have a dialogue with the user, initiate the next round of dialogue based on the real scene information, and let the user confirm the returned results;

[0154] If the user confirms that he is currently using the software or a cross-software solution to the problem, he will be guided to solve the corresponding problem. If the user confirms that he is not currently using the software solution to the problem, all the content fed back by the system will be returned. If the returned results do not contain content related to the real-life information, all the content fed back by the system will be returned.

[0155] After task-based dialogue processing using deep learning, the output is:

[0156] OUTPUT1= <y1,y2,…,y n >

[0157] The output sequence includes button operation commands, cross-software recommendations, and cross-device operation commands.

[0158] This solution solves the problem that general help manuals are not intuitive for users. Intuitive help combined with real scenes greatly enhances the user's good experience in using the product and improves usage efficiency. At the same time, cross-software recommendations and cross-device operations further improve the quality of problem solving.

[0159] The predicted probability of the solution to the problem combined with the real scene information and the user's question is:

[0160] p1=(y|x1,x2,…,x i-1 , x i , x i+1 , …, x n )

[0161]

[0162] x1,x2,…,x i-1 Indicates the real-life information of the software operation including the action sequence of the user, x i ,x i+1 ,…,x n Represents the user's natural language problem description;

[0163] The predicted probability p2 of the problem solution generated by combining the real scene information with the natural language question information is:

[0164] p2=q(y|x0,x1,x2,…,x m )

[0165]

[0166] x1,x2,…,x m Represents the user's natural language problem description;

[0167] p1>=p2

[0168] Obviously, based on the fact that computer technology can accurately capture users' continuous operations, the probability of understanding users' natural language questions is greater than or equal to the probability of understanding users' natural language questions through deep learning technology processing language-video multimodal technology, especially when the existing video multimodal success rate is not ideal.

[0169] During use, in one embodiment 1, when a user is using Word and asks how to draw a flowchart, if it is detected that the user has Visio software on his / her computer, the system can recommend him / her to use Visio to draw a flowchart and attach reasons for the recommendation. After the user agrees, the system can automatically start or guide him / her to start his / her Visio software, and the system will automatically switch to Visio interactive help. After the flowchart is drawn, the user can be guided to insert it into the Word text.

[0170] In one embodiment 2, when a user is using Word, if the user asks how to extract a large paragraph of text from a picture in a Word text, when it is detected that the corresponding software for extracting text from a picture is not installed, and the user often uses WeChat on a mobile phone, the user can be prompted to select a picture and click the "Save Picture As" button in the right button. After the picture is saved, the user opens WeChat on the computer and can select a friend or a tool such as a file transfer assistant to send the picture to WeChat on the mobile phone. The user then opens the picture, long presses the picture, and clicks the "Extract Text" button in WeChat to extract the information. The user selects all the information, and then sends the information in WeChat. After the text is extracted on the computer, the text can be inserted into the corresponding position in Word. The button prompt can be a step-by-step prompt to click the corresponding button, or the button can be used to automatically click to complete the operation by completing the corresponding operation.

[0171] Step 7-4: When the user’s purpose is not clearly understood, initiate a conversation to further clarify the user’s needs.

[0172] Furthermore, the natural language question information is voice information or text information;

[0173] For voice information, natural language information is converted into text information through voice recognition before semantic analysis of natural language information.

[0174] Accordingly, the embodiment of the present invention also provides a hybrid interactive help system for software operation based on real scenes and capable of crossing software. The structural diagram of the development module is as follows: Figure 2 As shown; the structural diagram of the service module is as follows Figure 3 as shown; and a schematic diagram of a step-by-step guide or automatic click structure across devices, such as Figure 4 shown.

[0175] The real-scene-based, cross-software hybrid interactive help system includes:

[0176] Interactive software button information acquisition module 201, such as Figure 2As shown, it is used to use software operation mixed interaction to help system developers obtain button information, where button function information, button border or vertex information, current button and its parent button can be filled in manually in the menu provided by the development platform, or automatically filled in after OCR, target detection in deep learning or other image recognition;

[0177] The real-scene natural language question information-answer pair generation module 202 is used to generate a software button relationship tree based on the button information and the button usage problems; further generate training data; use the training data as training samples; generate text, voice, and picture data based on the synonym database and the real-scene natural language question information combined with the usage problems of the real-scene information, user intentions and help information, and other various problems and solutions on the Internet and books, and further generate training data;

[0178] A software operation interactive help model construction module 203 is used to obtain a software operation hybrid interactive help model that can cross software based on real scene semantic understanding according to training samples;

[0179] Real scene acquisition module 301, such as Figure 3 As shown, it is used to obtain real-scene information including software name and version, current menu status information, operating system and its version, obtain "button information" provided by the software interface, or use the hook method, system monitoring method or polling scheduling method, or deep learning-based image processing technology to identify button and menu operation information;

[0180] A real-scene natural language question information generating module 302 extracts the natural language question information of the user when using the interactive operation software according to the user's selection, combines the real-scene information with the natural language question information to generate the real-scene natural language question information or only uses the user's question information;

[0181] The cross-software hybrid interactive help module 303 based on real-scene semantic understanding adopts a dialogue system based on deep learning; it is used to understand the user's natural language questions containing real-scene information. The user can choose between general questions or software operation questions to improve the system response speed, and interactive help is given based on the questions and the above choices, and the real-scene results of the combined button function query help and button location search help are introduced.

[0182] The prompt can be an arrow pointing to the button, or it can be highlighted or flashing at the button position; the button click help mode includes step-by-step prompts and automatic button clicking, which can be set by the user.

[0183] Step-by-step instructions: Provide operation instructions and guide users to click buttons one by one.

[0184] Automatic button click: Automatically complete button clicks or call corresponding methods to complete button clicks, allowing users to complete the operations they need to complete.

[0185] Cross-software recommendations: For user problems that can only be solved across software or platforms, recommended usage information is given based on the user's electronic device information model, including giving priority to software suitable for the local machine. If the software does not exist, other suitable software is recommended.

[0186] Cross-device step-by-step guidance or automatic clicks: Figure 4 For user problems that need to be solved across devices, with the user's permission, electronic device I 401 sends information to electronic device II 402 through step-by-step prompts or automatic clicks on the information. Electronic device II 402 transmits the interactive help information to the corresponding electronic device I 401 through network communication, and starts the corresponding software and functions to achieve cross-platform step-by-step prompts and automatic button clicks.

[0187] Software usage issues: finding button locations, querying button functions, and helping with how to use buttons, as well as issues that need to be solved across software.

[0188] If the user does not click the software usage problem button, the results matching the real scene information and the solution to the problem of the current software usage are extracted from the returned results based on the real scene information, and a dialogue is conducted with the user. The next round of dialogue is initiated based on the real scene information to ask the user to confirm the returned results; if the user confirms that the software is being used or the solution to the problem can be cross-software, the user is guided to solve the corresponding problem; if the user confirms that the software problem solution is not being used, all the content fed back by the system is returned. If there is no content related to the real scene information in the returned results, all the content fed back by the system is returned.

[0189] The system also includes: a speech recognition module, which is used to convert the natural language information into text information through speech recognition before the semantic analysis module performs semantic analysis on the natural language information.

[0190] The present invention adopts a hybrid interactive help method for software operation based on real scenes and across software, which solves the problems of software usage efficiency and difficulty in finding software; the present invention can accurately perceive continuous actions-accurate understanding-accurate guidance, and can achieve high-quality interactive help for software operations. By integrating its training data with the existing general large model training data for training, it can achieve the expansion of the application scenarios of large models from simple large language (text), image-language multimodal model (VLM) to software operation continuous action-language multimodal model. Since the real-scene interactive help system has a wide user group, it can not only realize the application of large models in the field of software operation problem solving, but also promote the popularization of large models.

[0191] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A hybrid interactive help method for software operation based on real scenes and capable of crossing software, characterized in that: The following steps are involved: Step 111, extracting software information in the electronic device, button information of operation buttons of interactively operated software, collecting usage problems and help information and help information of other problems, and generating training data of a cross-software software operation hybrid interactive help system based on real scene semantic understanding; Step 112, extracting the software and hardware information in the user's electronic device and constructing a data model of the user's electronic device; Step 113, pre-building a software operation hybrid interactive help model based on real-scene semantic understanding and cross-software using deep learning according to the training data; Step 114, extracting natural language question information asked by the user; Step 115, extracting the current real scene information of the software according to the constructed user electronic device data model, and generating real scene natural language question information according to the user's question and operation selection; Step 116, inputting the real-scene natural language question information or the user question information into the cross-software software operation hybrid interactive help model based on real-scene semantic understanding, and outputting the ranking score of each semantic analysis result; Step 117 , selecting one or more semantic parsing results as understanding results of the user's question according to the ranking scores, and providing interactive operation help to the user based on the cross-software hybrid interactive help model.

2. The method according to claim 1, characterized in that: Software information in electronic devices includes: The operating system and its version name, software name, software version and specific function information in the electronic device; Button information includes: Button name, button range information, button relationship information and button function information; Button range information refers to the range contained in the button border or vertex information and the transformation relationship information of the button range between different screen sizes and different resolutions; Button relationship refers to the relationship between the current button and the sub-button that appears when the current button is clicked; Button function information refers to the description information of the button function; Cross-software refers to software on different electronic devices or different software on the same device, and the software includes an operating system.

3. The method according to claim 1, characterized in that Step 111 is specifically as follows: Step 1-1, extracting button information of interactively operable software operation buttons according to a real-scene cross-software software operation hybrid interactive help system; Step 1-2, based on the knowledge of button usage problems, a rule-based method or an if-then-based form is used to generate training data of real-scene natural language question information of usage problems combined with real-scene information, user intentions and help information, and generate a software button relationship tree; The button relationship tree refers to the tree-like hierarchical relationship formed by the relationship between the parent and child buttons of the buttons of the interactively operable software; The real-scene natural language question information, user intention and help information combined with the use problem of the real-scene information are combined by using the real-scene information in a form of natural language omission recovery as part of the question information, or using the real-scene information as part of the question information in the form of an attention mechanism; Step 1-3, further generating training data based on the synonym database and the real-scene natural language question information combined with the usage problem of the real-scene information, user intention and help information; Steps 1-4, collecting the usage problems and help information of the interactive software and help information of other problems, including problems that can be solved across software and their help information, collecting text, voice and picture data of other types of problems and solutions on the Internet and books, and annotating them as training data; Help information for other problems refers to the types of problems that can be solved based on the existing generative artificial intelligence large model, but does not include problems with software operation and use; Step 1-5: using the training data as training samples.

4. The method according to claim 1, characterized in that: Step 112 is specifically as follows: Extract the software and hardware information of the user's electronic devices, including the software and hardware information of the user's computer, mobile phone, pad, iPad and household appliances, to build a data model of the user's electronic devices; The hardware and software information includes: the CPU, memory size, hard disk size and user screen resolution and zoom ratio hardware information in the user's electronic device, the operating system and its version name, the software name, software version and its storage location information.

5. The method according to claim 1, characterized in that Step 113 is specifically as follows: The training samples and the help information are used to train a cross-software software operation hybrid interactive help model based on real-scene semantic understanding, wherein the software operation hybrid interactive help model adopts a dialogue form based on deep learning, and during the training, the software real-scene information corresponding to the software operation questions asked by the user is added to the training model in a weighted manner; The software real-scene information is added to the training model in the form of dialogue using natural language processing technology in deep learning. This is a Transformer model that uses the attention mechanism, or a generative artificial intelligence model that uses the Transformer.

6. The method according to claim 5, characterized in that In the case of the Transformer model, the scene encoding layer is added and fused through the embedding layer to obtain the scene embedding vector E sense , input it into the encoding component of Transformer for feature learning and weight change; According to the embedding vector of the real scene fused into the encoding layer, multi-head self-attention calculation is performed and the obtained attention vector is used as the intermediate state of the encoding layer. The attention vector of the encoding layer is as follows: in, represents the input vector of the i-th encoding layer, represents the attention vector obtained by the i-th encoding layer, d k is the dimension of the input key vector; After the layer-by-layer feedforward neural network, the hidden layer feature vectors corresponding to all the tags in the input sequence are obtained, which can be expressed as: H sense =FFN(Z sense ) H sense =(h1,h2,…,h l ),h i ∈R M Among them, FFN is a feedforward neural network, H sense is the feature vector representation of the scene output by the last encoder, Z sense is the attention vector output by the last encoder, h i represents the i-th input vector, l is the length of the input sequence containing special labels, and M represents the vector dimension of the hidden layer; Select the cross entropy loss function, the specific calculation formula is as follows: and max =max(P(y I ,and S |I * )) L Sense =∑ i (y i ln(y max )+(1-y i ln(1-y max ))) Among them, y i Represents the probability output of all intent categories and slot names corresponding to the input sequence of fused real scene information, y max represents the maximum value of the probability of the intent category and slot name in the probability output, L Sense Represents the cross entropy loss of the module after scene omission restoration.

7. The method according to claim 1, characterized in that Step 115 is specifically as follows: Step 5-1, extracting natural language question information asked by the user when using the interactive software; Extract the current real-life information of the software, including the software name, the actual state of the menu bar, button information, button click information, and information on the screen size and resolution of the pop-up menu and button, as well as whether the software usage problem button has been clicked; Acquisition of real-scene information, including using "button information" provided by the operating system and software interface, or using hook methods, system monitoring or polling scheduling methods, or deep learning-based image processing technology to identify button and menu operation information; Software usage issues, including button location search, button function query, button usage help, and issues that need to be solved across software; Step 5-2, when the user clicks the software usage question button, the real scene information and the natural language question information are combined to generate the real scene natural language question information, wherein the combination method includes using the real scene information as part of the question information in the form of language omission restoration, or using the real scene information as part of the question information in the form of an attention mechanism; If the user does not click the software usage question button, the natural language question information based on the real scene only includes the user's question information; Users can improve system response speed by selecting general questions or software operation questions.

8. The method according to claim 1, characterized in that Step 117 is specifically as follows: Step 7-1, selecting one or more semantic parsing results according to the ranking scores as the understanding results of the user's question; Step 7-2, providing interactive operation help to the user based on the cross-software hybrid interactive help model, the cross-software hybrid interactive help model includes: Button location search help: Based on the location of the current menu state of the interactive software and the button relationship tree, prompts are given to click one or more buttons in sequence. The prompt is an arrow pointing to the button, or a highlight or flashing at the button location; Button function query help: provide an introduction to button functions, including text, pictures and videos; Button usage help: Combine button function query help and button location search help to introduce the real-life results; Button click help: including step-by-step prompts and automatic button clicks, which can be set by the user; The step-by-step prompts: provide operation instructions and guide the user to click buttons in sequence to complete; The automatic button click: automatically completes the button click or calls the corresponding method to complete the button click, and the user completes the operation he needs to complete; Cross-software recommendation: For user problems that can only be solved across software or platforms, recommended usage information is given based on the user's electronic device information model, including giving priority to software suitable for the local device. If there is no suitable software on the local device, other suitable software is recommended; Cross-device step-by-step guidance or automatic clicks: For user problems that can only be solved across devices, with the user's permission, interactive help information is transmitted to the corresponding device through network communication, and the corresponding software and functions are activated to achieve cross-platform step-by-step prompts and automatic button clicks; Software usage issues: button location search, button function query, button usage help, and issues that need to be solved across software; Step 7-3, if the user does not click the software usage problem button, extract the results matching the real scene information and the solution to the current software usage problem according to the real scene information, and have a dialogue with the user, initiate the next round of dialogue based on the real scene information, and let the user confirm the returned result; If the user confirms that the software is being used or that the solution can be used across software, the user will be guided to solve the corresponding problem; if the user confirms that the solution is not being used, all the content reported by the system will be returned; if there is no content related to the real scene information in the returned results, all the content reported by the system will be returned; Step 7-4: When the user’s purpose is not clearly understood, initiate a conversation to further clarify the user’s needs.

9. The method according to any one of claims 1 to 8, characterized in that: The natural language question information is voice information or text information; For voice information, before the natural language information is semantically parsed, the natural language information is converted into text information through voice recognition.

10. A real-scene-based cross-software software operation hybrid interactive help system adopted by the method as claimed in claim 9, characterized in that: include: The interactive software button information acquisition module is used to help system developers obtain button information by using software operation mixed interaction, where the button function information, button border or vertex information, current button and its parent button are filled in manually in the menu provided by the development platform, or automatically filled in by OCR, target detection in deep learning or other image recognition; A real-life natural language question information-answer pair generation module is used to generate a software button relationship tree based on button information and button usage questions; Further generate training data; Using the training data as training samples; Generate text, voice, and picture data based on the synonym database and the real-scene natural language question information combined with the real-scene information, user intention and help information, and other various problems and solutions on the Internet and books, and further generate training data; A software operation interactive help model construction module is used to train the training samples to obtain a cross-software software operation hybrid interactive help model based on real-scene semantic understanding; The real scene acquisition module is used to obtain the real scene information including the software name and version, the current status information of the menu, the operating system and its version, and obtain the "button information" provided by the software interface, or use the hook method, the system monitoring method or the polling scheduling method, or the image processing technology based on deep learning to identify the button and menu operation information; A real-scene natural language question information generation module extracts the natural language question information of the user when using the interactive software according to the user's selection, combines the real-scene information with the natural language question information to generate the real-scene natural language question information or only uses the user's question information; The cross-software hybrid interactive help module based on real-scene semantic understanding adopts a dialogue system based on deep learning. It is used to understand the user's natural language questions containing real-scene information. The user can select general questions or software operation questions to improve the system response speed, and interactive help is given according to the questions and the above choices, including button function query help, button location search help or button how to use help; The hint is an arrow pointing to the button, or a highlight or flash at the button location; The button click help modes include step-by-step prompts and automatic button clicks, which can be set by the user; The system also includes: a speech recognition module, which is used to convert the natural language information into text information through speech recognition before the semantic analysis module performs semantic analysis on the natural language information.

Citation Information

Patent Citations

  • Voice interaction method, system and platform for software operation live-action semantic understanding

    CN113223520A

  • Ship man-machine interaction method under system collaboration

    CN114546548A