Software learning method, system and device, storage medium and program product

Abstract

The invention discloses a software learning method, system and device, a storage medium and a program product. The method comprises the following steps: receiving learning materials for target software uploaded by a first user; wherein the learning materials comprise audio learning materials, video learning materials and / or guidance instructions; generating a training data set according to the learning materials; performing model training by adopting the training data set to generate a software learning model; wherein the software learning model is used for generating voice explanation and / or screen display teaching for the software learning problem of the second user. By adopting the technical means of the invention, the user can be assisted in learning software functions, the pertinence of software learning is improved, and the complexity of using software tools by the user is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a software learning method, system, device, storage medium and program product. Background Art

[0002] When using PC, tablet and other screen software in daily work and life, users often have problems with using some complex software or not being proficient when using in-depth functions, and need to take training courses, ask for advice or slowly explore. At present, in order to familiarize themselves with software functions, users usually use search technology or AI big model question and answer methods to explore. AI big models currently tend to be more question-based interactions, and are not friendly enough in teaching interactions. It can be seen that the current technology lacks a solution for virtual training that can achieve similar results to practical training teaching, and it is difficult to provide users with a good learning experience. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a software learning method, system, device, storage medium and program product, which can assist users in learning software functions, improve the pertinence of software learning, and reduce the complexity of users using software tools.

[0004] To achieve the above object, an embodiment of the present invention provides a software learning method, comprising:

[0005] Receiving learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instruction instructions;

[0006] Generate a training data set based on the learning material;

[0007] The training data set is used to perform model training to generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the software learning problems of the second user.

[0008] As an improvement of the above solution, before generating a training data set according to the learning material, the method further includes:

[0009] The learning materials are cleaned to obtain target learning materials associated with the target software, and the target learning materials are stored.

[0010] As an improvement of the above solution, generating a training data set according to the learning material includes:

[0011] Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data;

[0012] The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

[0013] As an improvement of the above solution, the cleaning process of the learning materials to obtain the target learning materials associated with the target software includes:

[0014] Identify sensitive information in the learning materials and delete the sensitive information;

[0015] The relevance between the learning materials and the target software is identified, and irrelevant learning materials are deleted to obtain target learning materials associated with the target software.

[0016] As an improvement of the above solution, the software operation interface included in the target learning material is subjected to atomic slicing processing to obtain atomic slice data, including:

[0017] Identify each interface element of the software operation interface contained in the target learning material, and form an atomic cut graph of the interface element;

[0018] A corresponding description file is matched to each of the atomic slices to form atomic slice data; wherein the description file includes the name, function, operation, effect corresponding to the operation and associated subsequent operations corresponding to the atomic slice.

[0019] As an improvement of the above solution, the step of performing data enhancement processing on the atomic slice data includes:

[0020] Comparing the similarities between the atomic slice data;

[0021] Acquire atomic graph cutting data whose similarity is greater than a preset similarity threshold as atomic graph cutting data to be processed;

[0022] At least one atomic graph cutting data is matched and associated with the atomic graph cutting data to be processed to form an atomic graph cutting data combination.

[0023] As an improvement of the above solution, the method further includes:

[0024] When a search instruction from a second user is received, target learning materials matching the search instruction are obtained for display.

[0025] As an improvement of the above solution, the method further includes:

[0026] receiving a rating of the target learning material by a second user;

[0027] Obtaining the query presentation volume and playback download volume of the target learning material within a unit time;

[0028] Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount;

[0029] During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

[0030] As an improvement of the above solution, the method further includes:

[0031] When receiving the first teaching instruction from the second user, starting the AI ​​teaching function;

[0032] Obtaining software learning questions input by a second user;

[0033] The software learning problem is input into the software learning model for processing, and the answer content for the software learning problem is output.

[0034] As an improvement of the above solution, after inputting the software learning problem into the software learning model for processing and outputting the solution content for the software learning problem, the method further includes:

[0035] When a second teaching instruction from a second user is received, the software learning model outputs keyboard and mouse control instructions and voice synchronization explanation instructions to control keyboard and mouse operations for screen display teaching and voice synchronization explanation.

[0036] The embodiment of the present invention further provides a software learning system, including:

[0037] A material acquisition module, configured to receive learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instruction instructions;

[0038] A data preprocessing module, used to generate a training data set based on the learning material;

[0039] The AI ​​model capability module is used to use the training data set to perform model training and generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the software learning problems of the second user.

[0040] As an improvement of the above solution, the data preprocessing module is specifically used for:

[0041] Cleaning the learning materials to obtain target learning materials associated with the target software and storing the target learning materials;

[0042] Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data;

[0043] The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

[0044] As an improvement of the above solution, the data acquisition module includes a recording module and a text recognition module;

[0045] The recording module is used to record the audio learning materials and video learning materials of the target software;

[0046] The text recognition module is used to receive instructions and perform text recognition on the instructions; wherein the instructions include documents and pictures.

[0047] As an improvement of the above solution, the system further includes an interaction and scoring module; the interaction and scoring module is used to:

[0048] Receiving a search instruction from a second user, and obtaining target learning materials matching the search instruction for display;

[0049] receiving a rating of the target learning material by a second user;

[0050] Obtaining the query presentation volume and playback download volume of the target learning material within a unit time;

[0051] The AI ​​model capability module is also used to:

[0052] Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount;

[0053] During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

[0054] As an improvement of the above solution, the method further includes a voice question and answer module and a keyboard and mouse control module;

[0055] The AI ​​model capability module is also used to:

[0056] When receiving the first teaching instruction from the second user, starting the AI ​​teaching function;

[0057] Obtaining software learning questions input by a second user;

[0058] Inputting the software learning problem into the software learning model for processing, and outputting the answer to the software learning problem;

[0059] When receiving a second teaching instruction from a second user, outputting keyboard and mouse control instructions and voice synchronization explanation instructions through the software learning model;

[0060] The keyboard and mouse control module is used to control the keyboard and mouse operation to perform screen display teaching according to the keyboard and mouse control instructions;

[0061] The voice question and answer module is used to output voice synchronization explanation according to the voice synchronization explanation instruction.

[0062] An embodiment of the present invention also provides a software learning device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any one of the software learning methods described above when executing the computer program.

[0063] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the software learning methods described above.

[0064] An embodiment of the present invention further provides a computer program product, which includes a computer program or a computer instruction. When the computer program or the computer instruction is executed by a processor, the software learning method as described in any one of the above is implemented.

[0065] Compared with the prior art, the software learning method, system, device, storage medium and program product disclosed in the present invention assist users in software learning based on the AI ​​big model. The training data of the AI ​​model comes from the learning materials uploaded by contributors who are familiar with software applications, which effectively improves the effectiveness of the training data source and is conducive to improving the model training effect. By providing targeted voice answers and screen display teaching to users' software learning problems through the trained software learning model, it can assist users in learning software functions, improve the pertinence of software learning, and reduce the complexity of users using software tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flowchart of a software learning method provided by an embodiment of the present invention;

[0067] Figure 2 Schematic diagram of the principle of data enhancement in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the SaaS software architecture in an embodiment of the present invention;

[0069] Figure 4 It is a structural diagram of a software learning system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0072] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0073] See also Figure 1 , is a flowchart of a software learning method provided by an embodiment of the present invention. The embodiment of the present invention provides a software learning method, including steps S11 to S13:

[0074] S11, receiving learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instructions;

[0075] S12, generating a training data set according to the learning material;

[0076] S13. Perform model training using the training data set to generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the second user's software learning problems.

[0077] The embodiments of the present invention are applicable to screen software learning scenarios, and the screen software includes but is not limited to programming, PS, PPT writing, game teaching, etc.

[0078] In an embodiment of the present invention, in order to solve the problem of limited training effect due to insufficient training data sources and effectiveness evaluation in the prior art of AI large model teaching, an embodiment of the present invention provides a software learning system to execute the software learning method, through which the learning materials of the target software uploaded by the first user are received, including but not limited to audio learning materials, video learning materials, instructions, etc., and a training data set is generated according to the learning materials to implement the training of the AI ​​large model and generate a software learning model. In the subsequent application process, when the second user has a learning need for the target software, the software learning problem of the second user is processed through the software learning model, and voice explanations and / or screen display teaching are generated to answer the second user's questions in a targeted manner.

[0079] It should be noted that the first user is a contributor of software learning materials, that is, a user who is familiar with the usage functions and operation procedures of the target software, and the second user is a learner who is not familiar with the usage functions and operation procedures of the target software.

[0080] By adopting the technical means of the embodiments of the present invention, users are assisted in software learning based on the AI ​​big model. The training data of the AI ​​model comes from the learning materials uploaded by contributors who are familiar with software applications, which effectively improves the effectiveness of the training data source and is conducive to improving the model training effect. By providing targeted voice answers and screen display teaching to users' software learning problems through the trained software learning model, users can learn software functions, improve the pertinence of software learning, and reduce the complexity of users using software tools.

[0081] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of the above embodiment. Before generating the training data set according to the learning material, the method further includes:

[0082] The learning materials are cleaned to obtain target learning materials associated with the target software, and the target learning materials are stored.

[0083] In an embodiment of the present invention, after receiving the learning materials uploaded by the first user, it is necessary to perform a cleaning process on the data, wherein the cleaning process includes sensitive information deletion and relevance screening.

[0084] Preferably, the cleaning process of the learning materials to obtain target learning materials associated with the target software includes:

[0085] Identify sensitive information in the learning materials and delete the sensitive information;

[0086] The relevance between the learning materials and the target software is identified, and irrelevant learning materials are deleted to obtain target learning materials associated with the target software.

[0088] Filter data based on relevance. For example, only learning materials related to wps are allowed to be uploaded to the wps software portal. Filtering and identification methods include:

[0089] Text type screening: Set relevant keywords for the target software, and identify the uploaded text and match the keywords. Only when the matching rate is greater than n% can it be uploaded, otherwise the data will be removed.

[0090] Image and video type screening: By setting m typical pictures of the target software operation interface, the similarity of the uploaded images is compared. If the similarity exceeds x%, it can be uploaded, otherwise the data will be removed. If it is a video, a set length of frame is taken for comparison, and the average similarity exceeds s% before it can be uploaded, otherwise the data will be removed.

[0091] Audio can be converted into text for screening and matching. Under normal circumstances, audio is matched with video.

[0092] As a preferred implementation manner, the embodiment of the present invention is further implemented on the basis of any of the above embodiments, wherein generating a training data set according to the learning material includes:

[0093] Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data;

[0094] The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

[0095] In the embodiment of the present invention, after receiving the learning materials uploaded by the first user, it is also necessary to perform pre-processing operations such as standardization and feature classification on the data.

[0096] Preferably, the step of atomically slicing the software operation interface contained in the target learning material to obtain atomic slice data includes:

[0097] Identify each interface element of the software operation interface contained in the target learning material, and form an atomic cut graph of the interface element;

[0098] A corresponding description file is matched to each of the atomic slices to form atomic slice data; wherein the description file includes the name, function, operation, effect corresponding to the operation and associated subsequent operations corresponding to the atomic slice.

[0099] Specifically, the software operation interface is generally divided into regions. Common divisions include the following: make a distinction based on operability, independence, etc., cut each atomic region, and provide a description file for the atomic cut, explaining the name, action, effect, and associated subsequent operations corresponding to the cut. Common area divisions of software, different feature tables can be made according to common area divisions to implement data feature engineering: a. Top navigation bar; b. Sidebar; c. Main workspace; d. Bottom status bar; e. Operation button bar; f. Input bar; g. Pop-up window and modal dialog box; h. Paging control; i. Tree view and list view; j. Parameter setting area; k. Function setting area...

[0100] The traversal operation of the software interface and the atomic cutting of each button / element are implemented as follows: Use UI automation tools to identify and click interface elements, create a queue to store the elements to be processed, add all operable buttons on the initial interface to the queue, and use breadth-first shading (BFS) to take out an element A (button) from the queue each time, perform the click operation, and then add all the new buttons generated to the end of the queue. The newly added buttons include paths such as AB. If a button will cause the page to jump or return after clicking, it is necessary to maintain a history record (such as a stack) to manage the page status so that you can return to the previous page when needed. When the queue or stack is empty and no new buttons are added, it can be considered that the shading has been completed.

[0101] Before executing the click action, the corresponding button image will be generated into a slice image. Data annotation: Before executing the click action, the corresponding button screenshot will be generated into an atomic slice image. For the specific slice image, the relevant operation attributes will be synchronized, including the optional operation corresponding to the element, the corresponding effect of the operation and the slice image, etc.

[0102] As an example, an example of atomic slice data is shown in Table 1:

[0103] Table 1

[0104]

[0105]

[0106] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of any of the above embodiments, wherein the atomic slice data is subjected to data enhancement processing, including:

[0107] Comparing the similarities between the atomic slice data;

[0108] Acquire atomic graph cutting data whose similarity is greater than a preset similarity threshold as atomic graph cutting data to be processed;

[0109] At least one atomic graph cutting data is matched and associated with the atomic graph cutting data to be processed to form an atomic graph cutting data combination.

[0110] In an embodiment of the present invention, for the atomic slice data generated by the above traversal, for the picture part, in order to make AI recognition more accurate, in addition to using common image processing libraries such as OpenCV or PIL for image enhancement, such as clarity enhancement, edge detection, image rotation and scaling, color adjustment and other processing, the following scheme can also be used to enhance the AI ​​training effect: compare the similarity of each atomic slice, if there is a slice with a similarity > S (S can be set as needed, such as 80%), use a combined recognition method to simultaneously associate the upper, lower, left and right atomic slices when identifying the main body features, and form an atomic slice data combination for feature recognition.

[0111] For example, see Figure 2 , is a schematic diagram of the principle of data enhancement in an embodiment of the present invention. Taking a magnifying glass cut image as an example, there may be many similar points in the magnifying glass cut image. The embodiment of the present invention uses the magnifying glass cut image as the center to make a nine-square grid diffusion, and introduces other features to better locate the corresponding button / element. When locating, it can be located in the middle part of the cut image. The combined recognition method data enhancement method provided in an embodiment of the present invention is only used when the similarity exceeds the specified value. This method reduces the similarity of each atomic cut image to less than S.

[0112] Preferably, the atomic slice data is processed for data set construction. A single data set is easily disturbed in a complex environment. A large number of diverse and representative slice image data sets of actual operation data of different users can be constructed through videos and pictures recorded by users, etc., and different operating environments and action changes are integrated to ensure that the AI ​​model can fully learn the features and patterns in different operating situations.

[0113] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of the above embodiment, and the method further includes:

[0114] When a search instruction from a second user is received, target learning materials matching the search instruction are obtained for display.

[0115] In the embodiment of the present invention, the learning materials provided by the contributor, i.e. the first user, are not only used for model training, but also stored in the storage module for users to study in the usual way of searching for learning materials. After the user searches for teaching videos or instructions, he can give scores and comments on each teaching video or instruction, and the scores are presented as the average of each user; he can also evaluate, and the scores and evaluations will be displayed to the contributors, and the contributors can interact in the comment area for in-depth communication.

[0116] Preferably, the method further comprises:

[0117] receiving a rating of the target learning material by a second user;

[0118] Obtaining the query presentation volume and playback download volume of the target learning material within a unit time;

[0119] Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount;

[0120] During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

[0121] In an embodiment of the present invention, a weight (learning value) is assigned to the learning material based on the score given by the first user, which is used for feedback optimization of subsequent training of the AI ​​large model.

[0122] Preferably, the calculating the learning value of the target learning material according to the average score Px of the target learning material X, the query presentation amount Cx and the playback download amount Bx includes:

[0123] Determine whether the query presentation amount is greater than a first preset threshold, and whether the playback download amount is greater than a second preset threshold;

[0124] When the query presentation volume Cx is greater than the first threshold C, and the playback download volume Bx is greater than the second threshold B, if the average score Px is greater than or equal to the third threshold P1, the learning value is the positive learning value Sx=1+(Px-P1) / Pm; wherein Pm is the total score value; if the average score Px is less than the fourth threshold P2, the learning value is the negative learning value Sx=1-(Px-P2) / Pm.

[0125] As an example, for the teaching video X uploaded by the second user, the query presentation volume Cx, the playback volume or download volume Bx, and the average user rating Px (10-point system, 0<=Px<=10) within a unit time period.

[0126] When Cx>C and Bx>B:

[0127] If Px>=8 points, video X is assigned a positive learning value as a data source for retraining the AI ​​model, and the learning value is Sx=1+(Jx-8) / 10;

[0128] If 6<=Px<8, video X is not processed;

[0129] If Px<6, the video X is assigned a negative learning value for the AI ​​big model to feedback and correct, and the video is deleted from the repository, and the learning value Sx=1-(Jx-8) / 10;

[0130] When Cx>C and Bx>B are not satisfied, video X is not processed temporarily.

[0131] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of the above embodiment, and the method further includes:

[0132] When receiving the first teaching instruction from the second user, starting the AI ​​teaching function;

[0133] Obtaining software learning questions input by a second user;

[0134] The software learning problem is input into the software learning model for processing, and the answer content for the software learning problem is output.

[0135] When a second teaching instruction from a second user is received, the software learning model outputs keyboard and mouse control instructions and voice synchronization explanation instructions to control keyboard and mouse operations for screen display teaching and voice synchronization explanation.

[0136] In an embodiment of the present invention, the user learns software through the software learning system in the following manner: after logging into the software learning system, the user enters the software he wants to learn, searches and views learning videos, audios or instructions for learning. If the user encounters a problem while using the software, he enters the corresponding learning software and searches for the corresponding instruction materials by keywords, rates the instruction materials, and if he has any questions, comments on the corresponding materials and waits for the author to provide in-depth answers.

[0137] For software with complex operations and advanced functions such as PS and Excel, users operate the software with specific tasks, open the software learning system, start AI teaching, and consult AI-related software learning questions through voice, such as how to use a certain function. The software uses the AI ​​capabilities provided by the AI ​​big model background to provide accurate answers. If the user still does not know how to operate, the user can send corresponding instructions, and the voice will tell the user to control the mouse and keyboard for teaching, and the teaching will be carried out synchronously through voice, thereby simulating the hands-on teaching of a real teacher.

[0138] The technical means adopted in the embodiments of the present invention can assist users in learning software functions, realize the integration of learning and operation, provide more focused field learning materials and video queries, and provide AI teaching just like having an expert in the relevant field to tutor at all times, guiding users to quickly get started and use it proficiently, effectively improving the efficiency of knowledge learning, especially for some learning that requires repeated practice, such as programming, PS, PPT writing, game teaching, etc., which is fast and efficient, avoiding a lot of disorderly exploration, and can replace many offline trainings on the market, and can provide targeted teaching guidance according to the learning characteristics of each person; at the same time, for the data source of the AI ​​large model, a mechanism is established to encourage creators to actively upload learning explanations, learning videos, etc., and then improve the quality of the data source through the student learning feedback mechanism.

[0139] See also Figure 3 and Figure 4 , Figure 3 is a schematic diagram of the SaaS software architecture in an embodiment of the present invention, Figure 4 It is a structural diagram of a software learning system provided by an embodiment of the present invention. The embodiment of the present invention further explains the specific structure and functions of the software learning system. The embodiment of the present invention provides a software learning system, which can be implemented through a SaaS (Software-as-a-service) software architecture. The software learning system includes a software management module, a keyboard and mouse control module, an operation recognition module, an intelligent voice question and answer module, an interaction and scoring module, a billing module, a data preprocessing module, a recording module, a text recognition module, a contribution calculation module and a storage module.

[0140] 1. Software management module:

[0141] A first user-side software management module and a second user-side software management module are distinguished, wherein the first user (contributor) refers to a party who is good at using the target software and provides software usage guidance data for training the AI ​​big model, and the second user refers to a party who needs to use the AI ​​learning software system.

[0142] The important functions of the software management module include:

[0143] 1. Summarize all the software included in the teaching software, provide a presentation interface and the entrance to each software; each software needs to include the following options as associated items for large model data input: software name, software alias, software version, version corresponding to the data source, and operating system corresponding to the data source.

[0144] 2. Arrange the software by category to facilitate quick selection by the first and second users, and provide search function;

[0145] 3. Categorize vertical data on the input side for AI large model training to provide more accurate training data. That is, if the teaching is about the use of wps software, then this vertical field only focuses on wps software usage guidance and video teaching data, and completes the preliminary cleaning function of data preprocessing well.

[0146] 4. Creation, evaluation and data synchronization of software; if the second user finds that the software he needs to learn is not available during use, he can create the software in the software management module, and the operator will evaluate whether to add it. If the addition is allowed, the data synchronization will also be presented to the software management presentation interface on the first user side; conversely, the first user can also create new software, provide training data, and enter the data into the storage module.

[0147] 2. Keyboard and mouse control module:

[0148] SaaS software captures and simulates keyboard and mouse operations and presents users with real keyboard and mouse operations in a visualized way. This is achieved using a common Windows example:

[0149] 1. Capture keyboard and mouse events:

[0150] On Windows systems, you can use Windows API to capture keyboard and mouse events. For example, you can use the SetWindowsHookEx function to install a global hook to capture system-level keyboard and mouse events.

[0151] For keyboard events, you can capture key press (WM_KEYDOWN) and key release (WM_KEYUP) messages.

[0152] For mouse events, you can capture mouse movement (WM_MOUSEMOVE), mouse button press (WM_LBUTTONDOWN, WM_RBUTTONDOWN, etc.), and mouse button release (WM_LBUTTONUP, WM_RBUTTONUP, etc.) messages.

[0153] 2. Transmit keyboard and mouse events:

[0154] Convert the captured keyboard and mouse event data into a format suitable for network transmission. This usually involves serializing the event data into JSON, XML, or other formats. Establish a secure network connection (such as HTTPS) and send the serialized keyboard and mouse event data to the SaaS server. On the SaaS server, a service is required to receive and parse the keyboard and mouse event data.

[0155] 3. Handle keyboard and mouse events in SaaS applications:

[0156] Parsing keyboard and mouse event data:

[0157] On the SaaS server, after receiving the keyboard and mouse event data from the Windows system, it needs to be parsed back to the original event format. This may involve converting data in formats such as JSON and XML back to keyboard and mouse event objects that the program can understand.

[0158] Simulate keyboard and mouse operation:

[0159] Based on the parsed keyboard and mouse event data, the corresponding keyboard and mouse operations are simulated within the SaaS application or on the connected remote desktop.

[0160] For keyboard events, you can simulate key presses and key releases. On Windows, you can use the keybd_event function to simulate keyboard presses.

[0161] For mouse events, you can simulate mouse movements, clicks, and drags. On Windows, you can use the mouse_event function to simulate mouse events.

[0162] 4. Feedback and monitoring:

[0163] Real-time feedback: In SaaS applications, the real-time screen content of the Windows system is displayed to users so that users can see the effects of keyboard and mouse operations. This can be achieved by using remote desktop protocols (such as RDP) or screen sharing technology.

[0164] Monitoring and logging: Optional video recording of actions to facilitate subsequent practice.

[0165] User experience optimization: You can use caching, compression and other technologies to reduce the amount of data transmitted over the network. Optimize the transmission and simulation process of keyboard and mouse operations to reduce latency and increase response speed, thereby providing a better user experience.

[0166] It should be noted that keyboard and mouse operations need to be bound to software operations and voice introductions after learning the AI ​​large model.

[0167] 3. User operation identification module:

[0168] The core of this module is to identify the user's operation on the software interface to confirm the correctness and intention of the operation. The core includes:

[0169] 1. Capture of keyboard and mouse operations in the keyboard and mouse control module;

[0170] 2. The correspondence between operations and software interfaces, and the correlation between operations and responses (this item is the core capability output of the AI ​​big model, and the capability is obtained by training the big model based on data preprocessing using the contributor’s materials).

[0171] 4. Intelligent Voice Question and Answer Module:

[0172] The core of this module is to complete real-time voice interaction with users and automatically control the keyboard and mouse module or provide analysis and explanation when recognizing user operations. It includes: cooperating with the keyboard and mouse operation module to explain, outputting the keyboard and mouse control for the software to implement a certain function A according to the large model capability and cooperating to complete the voice explanation; recognizing user operations, voice outputting possible errors and corresponding effects of operations; general semantic understanding and interactive Q&A.

[0173] 5. Interaction and scoring module:

[0174] The core of this module is to score the materials of each contributor. In addition to being used for training, the materials of the contributors will also be stored in the storage module for users to learn by the usual way of searching for learning materials. After the user queries the teaching video or guidance, they can give scores and comments on each teaching video or guidance. The scores are presented as the average value of each user, and they can also be evaluated. The scores and evaluations will be shown to the contributors, and the contributors can interact in the comment area for in-depth communication. At the same time, the score will give the data source a weight (learning value) in the data preprocessing module, which will be used for feedback optimization of subsequent training of the AI ​​large model.

[0175] 6. Billing module:

[0176] As commercial SaaS software, a business model can be formulated to make profits, including conventional billing methods such as VIP level fees generated by user-side VIP registration, or charging by class hours, or charging by advertising, etc. The operator can customize it according to its own needs.

[0177] 7. Data preprocessing module:

[0178] This module focuses on preprocessing operations such as cleaning, standardization, and feature classification of the training data sources uploaded by contributors, including: sensitive information deletion, data correlation screening, atomic slicing processing, data enhancement processing, and data set construction processing.

[0179] Sensitive information deletion: Delete sensitive information.

[0180] Data relevance screening: Screening is performed based on data relevance. For example, the learning materials uploaded to the wps software portal are only allowed to be related to wps. The screening and identification methods include:

[0181] Text type screening: Set relevant keywords for the target software, and identify the uploaded text and match the keywords. Only when the matching rate is greater than n% can it be uploaded, otherwise the data will be removed.

[0182] Image and video type screening: By setting m typical pictures of the target software operation interface, the similarity of the uploaded images is compared. If the similarity exceeds x%, it can be uploaded, otherwise the data will be removed. If it is a video, a set length of frame is taken for comparison, and the average similarity exceeds s% before it can be uploaded, otherwise the data will be removed.

[0183] Audio can be converted into text for screening and matching. Under normal circumstances, audio is matched with video.

[0184] Atomic slicing: The software operation interface is generally divided into areas. The common division methods include the following: make a distinction based on operability, independence, etc., cut each atomic area, and provide a description file for the atomic cut, explaining the name, action, effect of the operation, and associated subsequent operations corresponding to the cut. Common area divisions of software, different feature tables can be made according to common area divisions to implement data feature engineering: a. Top navigation bar; b. Sidebar; c. Main workspace; d. Bottom status bar; e. Operation button bar; f. Input bar; g. Pop-up window and modal dialog box; h. Paging control; i. Tree view and list view; j. Parameter setting area; k. Function setting area...

[0185] The traversal operation of the software interface and the atomic cutting of each button / element are implemented as follows: Use UI automation tools to identify and click interface elements, create a queue to store the elements to be processed, add all operable buttons on the initial interface to the queue, and use breadth-first shading (BFS) to take out an element A (button) from the queue each time, perform the click operation, and then add all the new buttons generated to the end of the queue. The newly added buttons include paths such as AB. If a button will cause the page to jump or return after clicking, it is necessary to maintain a history record (such as a stack) to manage the page status so that you can return to the previous page when needed. When the queue or stack is empty and no new buttons are added, it can be considered that the shading has been completed.

[0186] Before executing the click action, the corresponding button image will be generated into a slice image. Data annotation: Before executing the click action, the corresponding button screenshot will be generated into an atomic slice image. For the specific slice image, the relevant operation attributes will be synchronized, including the optional operation corresponding to the element, the corresponding effect of the operation and the slice image, etc.

[0187] Data enhancement processing: For the list data generated by the above traversal, for the picture part, in order to make AI recognition more accurate, in addition to using common image processing libraries such as OpenCV or PIL to perform image enhancement such as clarity enhancement, edge detection, image rotation and scaling, color adjustment, etc., the following scheme can be used to enhance the AI ​​training effect: Compare the similarity of each atomic cut image. If there is a cut image with a similarity > S (S can be set as needed, such as 80%), use a combined recognition method. When identifying the main body features, simultaneously associate the icons up, down, left, and right for feature recognition. For example, the cut image of a magnifying glass may have many similar points, but a nine-square grid diffusion is made with this as the center. Introducing other features can better locate the corresponding button / element. When locating, you can locate the middle part of the cut image. It is only used when the similarity exceeds the specified value. This method reduces the similarity between each atomic cut image to below S.

[0188] Dataset construction: A single dataset can be easily disturbed in a complex environment. We can build a large number of diverse and representative sliced ​​image datasets of actual operation data of different users through videos and pictures recorded by users, and integrate factors such as different operating environments and action changes to ensure that the AI ​​model can fully learn the features and patterns in different operating situations.

[0189] In addition, it also includes evaluation and feedback: for the data set constructed above, the learning value of the initial training data is 1. Combined with the scoring in the interaction and scoring module, the training data is assigned positive and negative features and feedback is retrained. The learning value can be assigned an algorithm based on the scoring situation.

[0190] Migration learning during version upgrades: The version of the user learning software will have some minor changes during the iteration of small versions. For such changes, most of the time, it is not necessary to completely retrain. You can use fine-tuning to regularly collect new data and use incremental learning or online learning methods to update the changed data to update and optimize the model. During the fine-tuning process, freeze the unchanged parameters of the training model, only train the parameters of the changed layers, and gradually unfreeze and fine-tune more layers; for major version changes, if the parameter changes reach more than 50%, you can consider creating a new software version for new training.

[0191] The online learning implementation method can adopt online stochastic gradient descent, and update the model parameters every time new slice and attribute data are received. The data can be received in real time based on the server-side model service, and online learning updates can be performed.

[0192] Incremental learning can use incremental principal component analysis (IPA) to combine new data samples with existing knowledge and update the parameters and feature representation of the model.

[0193] By adopting the above implementation solutions and details, we can better realize the atomic slice screenshots and feed the data to AI for precise training, thereby improving the accuracy and generalization ability of the model.

[0194] 8. Recording module:

[0195] The software provides recording-related modules, which are mainly used to record software operation videos with audio commentary and complete the entry and upload of audio and video related files. It can control the clarity and video size and perform encryption and other functions. This platform provides recording to ensure that the recorded operation videos are more in line with the input data requirements, and better control the clarity, duration, segmentation, etc., so as to ensure the validity and accuracy of audio and video data and ensure more accurate training of AI large models. At the same time, the recorded videos are well classified and stored with keywords and entered into the storage module for subsequent AI indexing and user search for teaching demonstration video playback.

[0196] 9. Text recognition module:

[0197] Used for the recognition of various types of texts. Most of the software includes basic instruction documents, help documents, etc. The document formats include PDF, word, html, chm, Markdown, online help documents, pictures, etc. The text recognition module can support the reading of documents in all formats and convert them into general training data. Such data is generally used as the basic data for the software AI large model capabilities. When the new software is created, the data is collected and stored in the storage module. Parse the instructions for specific operations given by search engine websites and store the data in the storage module.

[0198] 10. Contribution calculation module:

[0199] Combined with the scoring and interaction modules, the materials uploaded by the contributors can be queried for presentation volume, playback volume or download volume, ratings and other data. The contributor's contribution can be calculated based on the above data, and the contribution can be used to calculate remuneration. Through the remuneration mechanism, more high-quality teaching contributors are attracted to upload materials for AI learning and user learning, creating a virtuous cycle platform.

[0200] In the actual application process, the steps of using the software learning system to implement model training and software learning are as follows:

[0201] A material acquisition module, configured to receive learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instruction instructions;

[0202] A data preprocessing module, used to generate a training data set based on the learning material;

[0203] The AI ​​model capability module is used to use the training data set to perform model training and generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the software learning problems of the second user.

[0204] Preferably, the data preprocessing module is specifically used for:

[0205] Cleaning the learning materials to obtain target learning materials associated with the target software and storing the target learning materials;

[0206] Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data;

[0207] The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

[0208] Preferably, the data acquisition module includes a recording module and a text recognition module;

[0209] The recording module is used to record the audio learning materials and video learning materials of the target software;

[0210] The text recognition module is used to receive instructions and perform text recognition on the instructions; wherein the instructions include documents and pictures.

[0211] Preferably, the interaction and scoring module is used to:

[0212] Receiving a search instruction from a second user, and obtaining target learning materials matching the search instruction for display;

[0213] receiving a rating of the target learning material by a second user;

[0214] Obtaining the query presentation volume and playback download volume of the target learning material within a unit time;

[0215] The AI ​​model capability module is also used to:

[0216] Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount;

[0217] During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

[0218] Preferably, the AI ​​model capability module is also used to:

[0219] When receiving the first teaching instruction from the second user, starting the AI ​​teaching function;

[0220] Obtaining software learning questions input by a second user;

[0221] Inputting the software learning problem into the software learning model for processing, and outputting the answer to the software learning problem;

[0222] When receiving a second teaching instruction from a second user, outputting keyboard and mouse control instructions and voice synchronization explanation instructions through the software learning model;

[0223] The keyboard and mouse control module is used to control the keyboard and mouse operation to perform screen display teaching according to the keyboard and mouse control instructions;

[0224] The voice question and answer module is used to output voice synchronization explanation according to the voice synchronization explanation instruction.

[0225] It should be noted that a software learning system provided in an embodiment of the present invention is used to execute all process steps of a software learning method in the above embodiment, and the working principles and beneficial effects of the two correspond one to one, so they will not be repeated here.

[0226] The technical means of the embodiments of the present invention have broken through the limitation of one-to-one teaching venues. You can experience the practical teaching guidance of famous teachers by studying alone at home, quickly become familiar with and master the use of software, and pay more attention to the ability and learning authenticity and effectiveness experience compared to existing AR teaching and other forms. AI ability training data is more accurate. Based on the classification of learning software, the screening of data sources is controlled by the system within the scope of the software. The data preprocessing solution specifically solves the learning cost of AI for software. It can optimize learning according to user feedback and improve AI capabilities to better and faster meet users' real learning demands. Create a virtuous growth cycle for learning users and teaching material providers. Learning users can obtain learning materials and better AI teaching more accurately; contributor users can obtain income by making teaching materials.

[0227] An embodiment of the present invention also provides a software learning device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the software learning method as described in any one of the above embodiments is implemented.

[0228] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the software learning method as described in any one of the above embodiments.

[0229] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the software learning method as described in any one of the above embodiments is implemented.

[0230] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0231] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A software learning method, characterized in that: include: Receiving learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instruction instructions; Generate a training data set based on the learning material; The training data set is used to perform model training to generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the software learning problems of the second user.

2. The software learning method according to claim 1, characterized in that: Before generating a training data set according to the learning material, the method further includes: The learning materials are cleaned to obtain target learning materials associated with the target software, and the target learning materials are stored.

3. The software learning method according to claim 2, characterized in that: The step of generating a training data set according to the learning material includes: Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data; The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

4. The software learning method according to claim 2, characterized in that: The cleaning process of the learning materials to obtain target learning materials associated with the target software includes: Identify sensitive information in the learning materials and delete the sensitive information; The relevance between the learning materials and the target software is identified, and irrelevant learning materials are deleted to obtain target learning materials associated with the target software.

5. The software learning method according to claim 3, characterized in that: The step of atomically slicing the software operation interface contained in the target learning material to obtain atomic slice data includes: Identify each interface element of the software operation interface contained in the target learning material, and form an atomic cut graph of the interface element; A corresponding description file is matched to each of the atomic slices to form atomic slice data; wherein the description file includes the name, function, operation, effect corresponding to the operation and associated subsequent operations corresponding to the atomic slice.

6. The software learning method according to claim 5, characterized in that: The step of performing data enhancement processing on the atomic slice data includes: Comparing the similarities between the atomic slice data; Acquire atomic graph cutting data whose similarity is greater than a preset similarity threshold as atomic graph cutting data to be processed; At least one atomic graph cutting data is matched and associated with the atomic graph cutting data to be processed to form an atomic graph cutting data combination.

7. The software learning method according to claim 2, characterized in that: The method further comprises: When a search instruction from a second user is received, target learning materials matching the search instruction are obtained for display.

8. The software learning method according to claim 7, characterized in that: The method further comprises: receiving a rating of the target learning material by a second user; Obtaining the query presentation volume and playback download volume of the target learning material within a unit time; Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount; During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

9. The software learning method according to claim 1, characterized in that: The method further comprises: When receiving the first teaching instruction from the second user, starting the AI ​​teaching function; Obtaining software learning questions input by a second user; The software learning problem is input into the software learning model for processing, and the answer content for the software learning problem is output.

10. The software learning method according to claim 9, characterized in that: After inputting the software learning problem into the software learning model for processing and outputting the solution content for the software learning problem, the method further includes: When a second teaching instruction from a second user is received, the software learning model outputs keyboard and mouse control instructions and voice synchronization explanation instructions to control keyboard and mouse operations for screen display teaching and voice synchronization explanation.

11. A software learning system, characterized in that: include: A material acquisition module, configured to receive learning materials for the target software uploaded by the first user; wherein the learning materials include audio learning materials, video learning materials and / or instruction instructions; A data preprocessing module, used to generate a training data set based on the learning material; The AI ​​model capability module is used to use the training data set to perform model training and generate a software learning model; wherein the software learning model is used to generate voice explanations and / or screen display teaching for the software learning problems of the second user.

12. The software learning system according to claim 11, characterized in that: The data preprocessing module is specifically used for: Cleaning the learning materials to obtain target learning materials associated with the target software and storing the target learning materials; Atomizing and slicing the software operation interface included in the target learning material to obtain atomic slice data; The atomic slice data is subjected to data enhancement processing and data set construction processing to generate a training data set.

13. The software learning system according to claim 11, characterized in that: The data acquisition module includes a recording module and a text recognition module; The recording module is used to record the audio learning materials and video learning materials of the target software; The text recognition module is used to receive instructions and perform text recognition on the instructions; wherein the instructions include documents and pictures.

14. The software learning system according to claim 12, characterized in that: The system also includes an interaction and scoring module; the interaction and scoring module is used to: Receiving a search instruction from a second user, and obtaining target learning materials matching the search instruction for display; receiving a rating of the target learning material by a second user; Obtaining the query presentation volume and playback download volume of the target learning material within a unit time; The AI ​​model capability module is also used to: Calculating the learning value of the target learning material according to the average score of the target learning material, the query presentation amount and the playback download amount; During the training process of the software learning model, the learning value is assigned to the target learning material, and feedback modification is performed on the software learning model.

15. The software learning system according to claim 11, characterized in that: The method also includes a voice question and answer module and a keyboard and mouse control module; The AI ​​model capability module is also used to: When receiving the first teaching instruction from the second user, starting the AI ​​teaching function; Obtaining software learning questions input by a second user; Inputting the software learning problem into the software learning model for processing, and outputting the answer to the software learning problem; When receiving a second teaching instruction from a second user, outputting keyboard and mouse control instructions and voice synchronization explanation instructions through the software learning model; The keyboard and mouse control module is used to control the keyboard and mouse operation to perform screen display teaching according to the keyboard and mouse control instructions; The voice question and answer module is used to output voice synchronization explanation according to the voice synchronization explanation instruction.

16. A software learning device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the software learning method according to any one of claims 1 to 10 when executing the computer program.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the software learning method according to any one of claims 1 to 10.

18. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the software learning method according to any one of claims 1 to 10 is implemented.

Images