Method and system for generating software electronic specification based on large language model

Abstract

The invention relates to the field of specifications, and discloses a method and system for generating a software electronic specification based on a large language model.The method comprises the steps that each video frame of a software operation video is extracted, a corresponding video frame image is obtained according to each video frame, all the video frame images and interface screenshots are processed, and the processed video frame images and interface screenshots are obtained; determining the interface level of the interface element image according to the number of the information features, determining to perform interface extraction on the interface element image by using a convolutional neural network model or image processing based on the interface level, verifying whether the interface element of the interface element image is correct based on historical data, and if the interface element of the interface element image is incorrect, performing interface extraction on the interface element image. Re-determining the interface elements of the interface element image by using a clustering algorithm; text information of an interface element image is obtained by adopting an OCR technology, and a software electronic specification is generated by adopting a large language model. The method has the characteristics of comprehensively processing the data and automatically generating the software electronic specification.

Description

Technical Field

[0001] The present invention relates to the technical field of instructions, and in particular to a method and system for generating software electronic instructions based on a large language model. Background Art

[0002] With the advancement of technology, in the field of software development and user support, manuals serve as an important help document that can help users understand and use various software. With the increase in the types of smart devices and software, the complexity of software interfaces is also increasing, and it is relatively difficult for users to understand and use software functions. However, traditional manuals are written manually, and the frequent updates of software interfaces and functional changes will lead to missing or delayed manual content, and will not be able to reflect the latest changes in software functions in a timely manner, which will bring many inconveniences and affect the normal use of the software. At present, although there are some tools for automatically generating manuals, they rely on static information input or manual annotation, and are difficult to achieve intelligence and automation. Especially when facing complex software interfaces and cross-platform support, the existing manual generation has limitations, cannot meet the adaptation of multiple platforms, and cannot conduct comprehensive and integrated analysis of data.

[0003] Therefore, how to provide a method and system for generating software electronic manuals based on a large language model is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention proposes a method and system for generating software electronic manuals based on a large language model, aiming to solve the problems of being unable to meet multi-platform adaptation and unable to conduct comprehensive analysis of data when faced with complex software interfaces and cross-platform support.

[0005] In one aspect, the present invention provides a method for generating software electronic manuals based on a large language model, comprising: Determine the original data of the electronic manual of the software to be generated, the original data including the software operation video, the interface screenshot and the software installation package, extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, process all the video frame images and the interface screenshot, and obtain the interface element image of the electronic manual of the software to be generated; Extracting information features from the interface element image, determining an interface level of the interface element image according to the number of the information features, determining to use a convolutional neural network model or image processing to perform interface extraction on the interface element image based on the interface level, deriving interface elements of the interface element image based on the result of the interface extraction, and when using the convolutional neural network model to perform the interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data, and when determining that the interface elements of the interface element image are incorrect, using a clustering algorithm to redetermine the interface elements of the interface element image; The OCR technology is used to obtain the text information of the interface element image, the software installation package is installed and run, the interaction data during the running process is captured, the operation steps are generated according to the interface elements of the interface element image, the text information and the interaction data, and the large language model is used to generate the software electronic manual.

[0006] Furthermore, when all the video frame images and the interface screenshots are processed to obtain the interface element images of the electronic manual of the software to be generated, it includes: All video frame images and the interface screenshots are preprocessed, the preprocessing includes denoising, contrast adjustment, color balancing and desharpening, feature points are extracted from the preprocessed interface screenshots and all video frame images according to a first preset algorithm, all extracted feature points are associated and matched according to a second preset algorithm, and associated data between the preprocessed interface screenshots and all video frame images is determined, the associated data includes area location and image attributes, and the preprocessed interface screenshots and all video frame images are corrected according to the associated data to obtain the interface element image of the electronic manual of the software to be generated.

[0007] Furthermore, when extracting information features from the interface element image and determining the interface level of the interface element image according to the number of the information features, it includes: The information features include interface text, interface tables and interface text boxes; Counting and summing the numbers of the interface text, the interface tables, and the interface text boxes, and recording the summed number as a complex quantity; The interface levels include primary complexity and secondary complexity; When the complexity is greater than or equal to the complexity threshold, the interface level of the interface element image is determined to be the first level complexity; When the complexity is less than the complexity threshold, the interface level of the interface element image is determined to be the second-level complexity; The complexity levels of the primary complexity and the secondary complexity decrease in sequence.

[0008] Furthermore, when determining to extract the interface element image using a convolutional neural network model or image processing based on the interface level, and deriving the interface element of the interface element image based on the result of the interface extraction, it includes: According to the first level of complexity, the image processing is used to perform interface extraction on the interface element image, and according to the second level of complexity, the convolutional neural network model is used to perform interface extraction on the interface element image; When the interface element image is subjected to interface extraction by image processing, the method includes: obtaining the gradient value of each pixel in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel, determining the texture similarity based on the texture feature, merging or separating adjacent pixels according to the texture similarity, and determining the interface element of the interface element image according to the result of merging or separating; When the convolutional neural network model is used to perform interface extraction on the interface element image, it includes: extracting local features and global features of the interface element image according to the convolution layer and pooling layer of the convolutional neural network model, and mapping the local features and the global features to output labels through a fully connected layer, and determining the interface elements of the interface element image according to the output labels.

[0009] Furthermore, when the convolutional neural network model is used to perform the interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data includes: Determine a historical data set of the same interface element as the interface element image according to the historical data, and obtain historical element features according to the historical data set, wherein the historical element features include historical icon size, historical button size, and historical menu options; The similarity between the interface element and the historical element features is calculated by the following formula: ; in, Indicates similarity, , and represents the weight coefficient, and , Indicates the icon size. Indicates the history icon size, Indicates the button size. Indicates the size of the history button. Indicates a menu option. Represents the history menu option; Pre-set similarity threshold ; when Greater than or equal to , then verifying that the interface element of the interface element image is correct; when Less than , it is verified that the interface element of the interface element image is incorrect.

[0010] Further, when it is determined that the interface element of the interface element image is incorrect, re-determining the interface element of the interface element image using a clustering algorithm includes: Representative data of each interface element is obtained according to the historical data, and an aggregated data set is established by combining the representative data and the historical element features. The feature vector of each data in the aggregated data set is extracted, and the expected number of clusters k is determined to be 3. The parameters of the Gaussian distribution are initialized, and the probability that each data in the aggregated data set belongs to each Gaussian distribution is calculated to obtain the responsibility value, and the cluster with the largest responsibility value is selected as the interface element of the interface element image.

[0011] Furthermore, when the text information of the interface element image is obtained by using OCR technology, the software installation package is installed and run, and the interactive data during the running process is captured, it includes: Extracting the interface interval where the interface element is located in the interface element image based on OCR technology, converting the interface element in the interface interval into characters, and recognizing text information of the characters; The software installation package is stored in a virtual environment of a virtual machine, and the software is decompressed, installed, and run. Input events in the virtual environment are captured according to an API protocol to obtain the interaction data.

[0012] Furthermore, when generating operation steps according to the interface elements of the interface element image, the text information and the interaction data, it includes: The interface elements are matched according to the interaction data and the text information to obtain a first result, the text information is matched according to the interaction data and the interface elements to obtain a second result, the interaction data is matched according to the text information and the interface elements to obtain a third result, and the operation steps are generated in chronological order based on the first result, the second result and the third result.

[0013] Furthermore, when a large language model is used to generate an electronic manual of a software, it includes: Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find model parameters of a large language model, establish a large language model, use the model training set to fit the large language model, and input the model test set into the large language model to verify the output accuracy of the large language model. When the output accuracy reaches a preset accuracy threshold, generate a software electronic manual according to the operation steps.

[0014] Compared with the prior art, the beneficial effects of the present invention are: through software operation videos, interface screenshots and software installation packages, combined with a large language model, the efficiency of generating software electronic manuals is improved, and the workload of manual writing and editing is reduced. The interface extraction is performed using a convolutional neural network model or image processing, which can accurately identify interface elements and ensure the accuracy of the software electronic manual. On the basis of the convolutional neural network model, the correctness of the interface elements is verified in combination with historical data, and the incorrect interface elements are re-determined using a clustering algorithm, which improves the adaptability of automatically generating software electronic manuals. The text information in the interface element image is extracted by OCR technology to ensure that the content of the generated software electronic manual is comprehensive and accurate. During the installation and operation of the software, the interactive data can be captured and the operation steps can be generated in combination with the interface elements and text information, so that the software electronic manual can reflect the actual process of the software. Based on the large language model, the manual content that meets the user's needs can be intelligently generated according to the functional characteristics and requirements of the software, and the corresponding software electronic manual is output for different platforms, which improves the adaptability of the software electronic manual.

[0015] On the other hand, the present application also provides a system for generating software electronic manuals based on a large language model, which is used to apply the above-mentioned method for generating software electronic manuals based on a large language model, including: The acquisition unit is configured to determine the original data of the electronic manual of the software to be generated, wherein the original data includes a software operation video, an interface screenshot, and a software installation package, extract each video frame of the software operation video, obtain a corresponding video frame image according to each video frame, and process all the video frame images and the interface screenshot to obtain an interface element image of the electronic manual of the software to be generated; a processing unit configured to extract information features from the interface element image, determine an interface level of the interface element image according to the number of the information features, determine to perform interface extraction on the interface element image using a convolutional neural network model or image processing based on the interface level, derive interface elements of the interface element image based on the result of the interface extraction, and when performing the interface extraction using the convolutional neural network model, verify whether the interface elements of the interface element image are correct based on historical data, and when it is determined that the interface elements of the interface element image are incorrect, redetermine the interface elements of the interface element image using a clustering algorithm; The generation unit is configured to use OCR technology to obtain text information of the interface element image, install and run the software installation package, capture interaction data during the operation process, generate operation steps according to the interface elements of the interface element image, the text information and the interaction data, and use a large language model to generate an electronic manual of the software.

[0016] It is understandable that the above-mentioned method and system for generating software electronic manuals based on a large language model have the same beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A flowchart of a system for generating software electronic manuals based on a large language model provided by an embodiment of the present invention; Figure 2 A functional block diagram of a system for generating software electronic manuals based on a large language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] In some embodiments of the present application, see Figure 1As shown, a method for generating software electronic manual based on a large language model includes: S100: determining the original data of the electronic manual of the software to be generated, the original data including the software operation video, the interface screenshot and the software installation package, extracting each video frame of the software operation video, obtaining the corresponding video frame image according to each video frame, processing all the video frame images and the interface screenshot, and obtaining the interface element image of the electronic manual of the software to be generated; S200: extracting information features from the interface element image, determining the interface level of the interface element image according to the number of information features, determining to use a convolutional neural network model or image processing to perform interface extraction on the interface element image based on the interface level, obtaining interface elements of the interface element image based on the result of the interface extraction, and when using the convolutional neural network model to perform interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data, and when determining that the interface elements of the interface element image are incorrect, using a clustering algorithm to redetermine the interface elements of the interface element image; S300: Use OCR technology to obtain text information of interface element images, install and run the software installation package, capture interaction data during the operation process, generate operation steps based on interface elements, text information and interaction data of interface element images, and use a large language model to generate software electronic manuals.

[0020] Specifically, in S100, the software operation video, interface screenshots and software installation package are used as raw data, and each video frame in the operation video is extracted and converted into a video frame image, and combined with the interface screenshot to form an interface element image, aiming to capture the software interface and its dynamic change process through different forms of input data, and provide data support for the subsequent interface element extraction and generation of operation steps. Multidimensional data lays the foundation for generating accurate software electronic manuals. In S200, information features in the interface element image are extracted, and the interface level of the interface element image is determined according to the number of information features. According to the interface level, the interface element image is extracted by using a convolutional neural network model or image processing, which improves the dynamic processing capability of different interface levels. When the convolutional neural network (CNN) model is used for interface extraction, the interface elements of the interface element image are verified according to historical data to avoid the risk of inaccurate interface elements caused by the simple use of the model, and the accuracy of extracting interface elements is improved. When the interface elements of the interface element image are judged to be incorrect, the interface elements of the interface element image are re-determined by using a clustering algorithm, which further improves the accuracy and automation of extraction, and avoids errors in the software electronic manual caused by human errors. S300 uses OCR technology to extract text information from interface element images to further improve the description of interface element images. Install and run the software installation package to capture the interaction data during the software operation process. The interaction data includes clicks, inputs, scrolling and other behaviors. The interaction data reflects the interaction with the interface, and combines the interface elements, text information and interaction data of the interface element image to automatically generate operation steps. The operation steps not only reflect how to interact with the software, but also clarify the interface elements that need to be operated in each step, so that these operation steps are converted into coherent and clear language descriptions through the large language model, and then generate the software electronic manual.

[0021] It is understandable that the software operation video, interface screenshots and software installation package are used as raw data, and analyzed and processed, which avoids the tedious process required for traditional manual writing of instructions. The interface element image is extracted by using a convolutional neural network model or image processing, and the text information in the interface element image is extracted by using OCR technology. The operation steps are accurately generated in combination with the interactive data, which not only saves time, but also ensures the accuracy of the operation steps. In the traditional manual writing process, it is easy to overlook some details or make errors in the operation steps, especially when the software interface is complex or frequently updated. Through convolutional neural network and OCR technology, combined with the optimization of clustering algorithms, it can automatically adapt to different interface changes, ensure the matching degree of operation steps and interface elements, and even if the software interface is updated or changed, it can still be automatically adjusted according to historical data, reducing the burden of maintaining and updating the manual and improving the efficiency of generating electronic manuals for software.

[0022] In some embodiments of the present application, when all video frame images and interface screenshots are processed to obtain interface element images of the electronic manual of the software to be generated, the following steps are included: All video frame images and interface screenshots are preprocessed, and the preprocessing includes denoising, contrast adjustment, color balancing and desharpening. Feature points are extracted from the preprocessed interface screenshots and all video frame images according to a first preset algorithm. All extracted feature points are associated and matched according to a second preset algorithm to determine the associated data between the preprocessed interface screenshots and all video frame images, and the associated data includes area location and image attributes. The preprocessed interface screenshots and all video frame images are corrected according to the associated data to obtain the interface element images of the electronic manual of the software to be generated.

[0023] Specifically, denoising is usually achieved by using methods such as Gaussian filtering or median filtering. Contrast adjustment uses gamma correction to enhance the brightness range, especially the brightness of dark areas. Color balance adjusts the exposure and color distribution to make them visually consistent, so as to eliminate unnecessary information that affects subsequent processing, thereby improving the quality of interface element images. The first preset algorithms include Harris corner detection algorithm and AKAZE (accelerated KAZE feature), which are used to extract feature points. One of them can be used according to actual conditions. The second preset algorithms include KNN (K-Nearest Neighbors) matching and RANSAC (random sampling consensus algorithm), which can determine the region location and image attributes. All extracted feature points are associated and matched, and one of them can be used according to actual conditions. In addition, based on nonlinear transformation and resampling, sparse and dense optical flow, the preprocessed interface screenshots and all video frame images are corrected to improve the quality and accuracy of interface element images.

[0024] In some embodiments of the present application, when extracting information features from an interface element image and determining the interface level of the interface element image according to the number of information features, the process includes: Information features include interface text, interface tables, and interface text boxes; Count the number of interface texts, interface tables and interface text boxes and sum them up, and record the sum result as the complex quantity; Interface levels include primary complexity and secondary complexity; When the complexity is greater than or equal to the complexity threshold, the interface level of the interface element image is determined to be first-level complexity; When the complexity is less than the complexity threshold, the interface level of the interface element image is determined to be secondary complexity; The complexity of primary complexity and secondary complexity decreases in sequence.

[0025] In some embodiments of the present application, when the interface element image is extracted by using a convolutional neural network model or image processing based on the interface level, and the interface element of the interface element image is obtained based on the result of the interface extraction, the method includes: According to the first-level complexity, image processing is used to extract the interface element image, and according to the second-level complexity, a convolutional neural network model is used to extract the interface element image; When the interface element image is subjected to interface extraction by image processing, the method includes: obtaining the gradient value of each pixel in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel, determining the texture similarity based on the texture feature, merging or separating adjacent pixels according to the texture similarity, and determining the interface element of the interface element image according to the result of merging or separating; When a convolutional neural network model is used to perform interface extraction on an interface element image, it includes: extracting local features and global features of the interface element image according to the convolution layer and pooling layer of the convolutional neural network model, mapping the local features and global features to output labels through a fully connected layer, and determining the interface elements of the interface element image according to the output labels.

[0026] Specifically, different processing methods are selected according to the interface level of the interface element image, which can improve the efficiency of the interface element image processing process. The complexity threshold is preferably 20, which can be adjusted according to actual needs. For the first level of complexity, image processing includes image feature analysis and region segmentation algorithm. Image processing is responsible for processing interface element images with higher complexity, deriving the spatial relationship between interface elements, and then constructing the hierarchy of interface elements, for example: determining whether a button is located in a pop-up window. For the second level of complexity, a convolutional neural network (CNN) model is used. The convolutional neural network model is suitable for processing interface element images with lower complexity, and the convolutional neural network model has good adaptability and accuracy, and can automatically learn interface element images and record related data, so as to accurately determine the interface elements of the interface element image. The interface elements include buttons, menus, icons and other related data.

[0027] It can be understood that, through the automatic selection strategy, it is possible to flexibly adjust and process according to the interface level of the interface element image, so that the extraction process is targeted, which helps to improve the accuracy of interface element extraction, obtain the gradient value of each pixel in the interface element image, thereby determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel, and then determining the texture similarity based on the texture feature. First, the texture feature corresponding to a pixel is determined as the initial feature, and then the texture feature corresponding to an adjacent pixel is obtained and recorded as the target feature. The texture similarity is determined according to the ratio of the initial feature to the target feature until all the texture similarities are obtained. When the texture similarity is greater than or equal to 0.8, the texture features of the two adjacent pixels are considered to be similar, and the adjacent pixels are merged. When the texture similarity is less than 0.8, the texture features of the two adjacent pixels are considered to be dissimilar, and the adjacent pixels are separated, thereby determining the interface elements of the interface element image. The convolutional neural network method extracts features through a multi-layer network, provides accurate output labels of interface elements, and thus determines the interface elements of the interface element image according to the output labels.

[0028] In some embodiments of the present application, when a convolutional neural network model is used for interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data includes: Determine a historical data set of the interface element identical to the interface element image according to the historical data, and obtain historical element features according to the historical data set, wherein the historical element features include historical icon size, historical button size, and historical menu options; The similarity between the interface element and the historical element features is calculated using the following formula: ; in, Indicates similarity, , and represents the weight coefficient, and , Indicates the icon size. Indicates the history icon size, Indicates the button size. Indicates the size of the history button. Indicates a menu option. Represents the history menu option; Pre-set similarity threshold ; when Greater than or equal to , it verifies that the interface elements of the interface element image are correct; when Less than , the interface element image of the verification interface element is incorrect.

[0029] In some embodiments of the present application, when it is determined that the interface element of the interface element image is incorrect, re-determining the interface element of the interface element image using a clustering algorithm includes: The representative data of each interface element is obtained according to the historical data, and the representative data and the historical element features are combined to establish an aggregated data set. The feature vector of each data in the aggregated data set is extracted, and the expected number of clusters k is determined to be 3. The parameters of the Gaussian distribution are initialized, and the probability that each data in the aggregated data set belongs to each Gaussian distribution is calculated to obtain the responsibility value. The cluster with the largest responsibility value is selected as the interface element of the interface element image.

[0030] Specifically, by comparing with historical data, it is possible to accurately verify whether the interface elements of the interface element image are correct, reducing reliance on manual experience, improving the degree of automation, and being able to continuously accumulate and update historical data, gradually improving judgment capabilities and adaptability. When faced with new or unseen interface elements, it can still provide reliable judgment results. Moreover, when the interface elements of the interface element image are judged to be incorrect, a clustering algorithm is used to re-determine the interface elements of the interface element image, avoiding errors caused by relying solely on judgment and improving classification accuracy. The Gaussian mixture model allows automatic adjustment of classification parameters according to the natural distribution of the data, thereby adapting to different data and interface elements.

[0031] In some embodiments of the present application, when the OCR technology is used to obtain text information of the interface element image, the software installation package is installed and run, and the interactive data during the running process is captured, it includes: Extracting the interface interval where the interface element is located in the interface element image based on OCR technology, converting the interface element in the interface interval into characters, and recognizing the text information of the characters; The software installation package is stored in the virtual environment of the virtual machine, and the software is decompressed, installed and run. Input events in the virtual environment are captured according to the API protocol to obtain interaction data.

[0032] Specifically, the use of OCR technology and the capture of input events according to the API protocol in the virtual environment effectively improve the accuracy and efficiency of the automatic generation process of software electronic manuals. The OCR technology is used to convert the interface elements in the interface interval into characters, thereby identifying the text information of the characters and ensuring the accuracy of information extraction. In this context, the OCR technology is used to convert the interface elements into characters and identify the text information. The text information represents the specific content of the interface elements, including prompt feedback information, such as error messages and success prompts. Secondly, the software is installed and run using a virtual machine environment. The virtual machine environment can be configured as Android, iOS, Windows, macOS, and Web, etc., so as to adapt to different software installation conditions, and the API protocol is used to capture input events in the virtual environment. The input events include clicks, double-clicks, etc., so as to simulate the actual software operation process, and then collect interaction data. Interaction data, such as data of continuing to click a button after clicking an icon, improves the reproducibility of subsequent software operations and improves the automation level of generating software electronic manuals.

[0033] In some embodiments of the present application, when generating operation steps according to the interface elements, text information and interaction data of the interface element image, it includes: A first result is obtained by matching interface elements based on interaction data and text information, a second result is obtained by matching text information based on interaction data and interface elements, a third result is obtained by matching interaction data based on text information and interface elements, and operation steps are generated in chronological order based on the first result, the second result, and the third result.

[0034] Specifically, the operation steps are generated based on the comprehensive matching of interaction data, text information and interface elements, which can comprehensively and accurately reconstruct the operation steps of the software. Assuming that the "Save" button is clicked on the interface during the software operation, the interaction data matches the text "Save" displayed on the interface, and the first result is obtained: "Click the Save button". Assuming that the user enters the user name and clicks the "Login" button. The interaction data is matched with the interface elements and text information to obtain the second result: "Enter the user name in the user name field and click the Login button". Assuming that there is a prompt text "Please enter a correct email address" on the interface, the text information and interface elements match the interaction data to obtain the third result: "Enter a valid email address and follow the prompt requirements" to ensure the consistency of the generated operation steps with the actual operation, and based on the time sequence generation method, it is ensured that the generated operation steps are in line with the logical order of the actual operation, thereby generating coherent and clear operation steps, and then automatically generating the software electronic manual, which helps to enhance the readability of the software electronic manual.

[0035] In some embodiments of the present application, when a large language model is used to generate an electronic manual of a software, the process includes: Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find the model parameters of the large language model, establish a large language model, use the model training set to fit the large language model, and input the model test set into the large language model to verify the output accuracy of the large language model. When the output accuracy reaches the preset accuracy threshold, generate an electronic manual of the software according to the operation steps.

[0036] Specifically, the historical operation step data records the running status of the software in different time periods. The historical operation step data is divided into a model training set and a model test set. Usually 60%-70% of the data is used as the model training set, and the rest is used as the model test set. Ensure that both the model training set and the model test set contain data of various running conditions to improve the generalization ability of the large language model. Use cross-validation combined with grid search to find the model parameters of the large language model. Cross-validation divides the data into several parts and trains the model multiple times to verify its stability and performance. Grid search finds parameter combinations in the parameter space and uses the model test set to fit the large language model, reducing the risk of overfitting and improving the accuracy and stability of the model. Input the model test set into the trained large language model and verify the output accuracy of the large language model. The output accuracy reflects the performance of the model on unknown data and is an important indicator for evaluating model performance. After the model reaches the preset accuracy threshold, the current operation steps can be generated into accurate software electronic manuals in real time, which improves the reliability and automation of generating software electronic manuals.

[0037] It is understandable that, since the large language model is trained based on historical operation step data, when the large language model is generated, it can establish a corresponding processing mechanism based on the historical operation step data, and store data for each generation to iteratively upgrade the large language model, so as to timely adjust the content of the generated software electronic manual according to the changes in the interface and functions of the acquired software, thereby ensuring the accuracy and real-time performance of the software electronic manual. In addition, the large language model can generate multiple output formats, such as: graphic operation step diagrams combined with text and pictures. If text descriptions are required, detailed language descriptions are provided based on historical operation step data and current operation steps to meet the needs of different users. Since the interaction data is obtained from input events in a virtual environment, and the virtual machine environment is configured as Android, iOS, Windows, macOS, and Web, etc., the interaction data includes multi-platform interactions, so that the large language model can output corresponding software electronic manuals for different platforms, improving the adaptability of software electronic manuals to different platforms.

[0038] In summary, the beneficial effects of the present invention are: through software operation videos, interface screenshots and software installation packages, combined with a large language model, the efficiency of generating software electronic instructions is improved, and the workload of manual writing and editing is reduced. The interface extraction is performed using a convolutional neural network model or image processing, which can accurately identify interface elements and ensure the accuracy of the software electronic instructions. On the basis of the convolutional neural network model, the correctness of the interface elements is verified in combination with historical data, and the incorrect interface elements are re-determined using a clustering algorithm, which improves the adaptability of automatically generating software electronic instructions. The text information in the interface element image is extracted by OCR technology to ensure that the content of the generated software electronic instructions is comprehensive and accurate. During the installation and operation of the software, the interactive data can be captured and the operation steps can be generated in combination with the interface elements and text information, so that the software electronic instructions can reflect the actual process of the software. Based on the large language model, the content of the instructions that meet the user's needs can be intelligently generated according to the functional characteristics and requirements of the software, and the corresponding software electronic instructions are output for different platforms, which improves the adaptability and automation of the software electronic instructions.

[0039] In another preferred embodiment based on the above embodiment, refer to Figure 2 As shown, this embodiment provides a system for generating software electronic manuals based on a large language model, which is used to apply the above-mentioned method for generating software electronic manuals based on a large language model, including: The acquisition unit is configured to determine the original data of the electronic manual of the software to be generated, the original data including the software operation video, the interface screenshot and the software installation package, extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, process all the video frame images and the interface screenshot, and obtain the interface element image of the electronic manual of the software to be generated; a processing unit configured to extract information features from an interface element image, determine an interface level of the interface element image according to the number of information features, determine to perform interface extraction on the interface element image using a convolutional neural network model or image processing based on the interface level, derive interface elements of the interface element image based on the result of the interface extraction, and when performing interface extraction using the convolutional neural network model, verify whether the interface elements of the interface element image are correct based on historical data, and when determining that the interface elements of the interface element image are incorrect, use a clustering algorithm to redetermine the interface elements of the interface element image; The generation unit is configured to use OCR technology to obtain text information of the interface element image, install and run the software installation package, capture the interaction data during the operation, generate operation steps according to the interface elements, text information and interaction data of the interface element image, and use a large language model to generate an electronic manual of the software.

[0040] Specifically, the acquisition unit obtains raw data, including software operation videos, interface screenshots, and software installation packages. It provides data support for subsequent processing. The processing of video frame images and interface screenshots enables the system to capture dynamic and static information of software operations, ensuring that the generated electronic manual has good accuracy and completeness. The processing unit determines the interface level of the interface element image by analyzing the information characteristics of the interface element image, and selects convolutional neural network (CNN) or image processing technology to extract the interface elements. Through intelligent judgment of the interface level, the system can flexibly select the appropriate extraction method, thereby improving the accuracy and efficiency of the extraction. When using the convolutional neural network model, the system will verify the extraction results based on historical data to further ensure the correctness of the interface elements. When it is inaccurate, the clustering algorithm is used for adjustment to ensure the reliability and comprehensiveness of the extraction. According to the OCR technology, the text information in the interface element image is obtained. The system simulates the operating environment of the software, captures the interactive data of the software during operation, and generates operation steps according to the interface elements, text information and interactive data of the interface element image. The generation unit uses a large language model to automatically write the software electronic manual, making the software electronic manual logical. Improved the automation and accuracy of generating software electronic manuals.

[0041] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0042] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0043] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0044] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating software electronic manual based on a large language model, characterized in that: include: Determine the original data of the electronic manual of the software to be generated, the original data including the software operation video, the interface screenshot and the software installation package, extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, process all the video frame images and the interface screenshot, and obtain the interface element image of the electronic manual of the software to be generated; Extracting information features from the interface element image, determining an interface level of the interface element image according to the number of the information features, determining to use a convolutional neural network model or image processing to perform interface extraction on the interface element image based on the interface level, deriving interface elements of the interface element image based on the result of the interface extraction, and when using the convolutional neural network model to perform the interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data, and when determining that the interface elements of the interface element image are incorrect, using a clustering algorithm to redetermine the interface elements of the interface element image; The OCR technology is used to obtain the text information of the interface element image, the software installation package is installed and run, the interaction data during the running process is captured, the operation steps are generated according to the interface elements of the interface element image, the text information and the interaction data, and the large language model is used to generate the software electronic manual.

2. The method for generating software electronic manual based on a large language model according to claim 1, characterized in that: When all the video frame images and the interface screenshots are processed to obtain the interface element images of the electronic manual of the software to be generated, the method includes: All video frame images and the interface screenshots are preprocessed, the preprocessing includes denoising, contrast adjustment, color balancing and desharpening, feature points are extracted from the preprocessed interface screenshots and all video frame images according to a first preset algorithm, all extracted feature points are associated and matched according to a second preset algorithm, and associated data between the preprocessed interface screenshots and all video frame images is determined, the associated data includes area location and image attributes, and the preprocessed interface screenshots and all video frame images are corrected according to the associated data to obtain the interface element image of the electronic manual of the software to be generated.

3. The method for generating software electronic manual based on a large language model according to claim 2, characterized in that: When extracting information features from the interface element image and determining the interface level of the interface element image according to the number of the information features, the method includes: The information features include interface text, interface tables and interface text boxes; Counting and summing the numbers of the interface text, the interface tables, and the interface text boxes, and recording the summed number as a complex quantity; The interface levels include primary complexity and secondary complexity; When the complexity is greater than or equal to the complexity threshold, the interface level of the interface element image is determined to be the first level complexity; When the complexity is less than the complexity threshold, the interface level of the interface element image is determined to be the second-level complexity; The complexity levels of the primary complexity and the secondary complexity decrease in sequence.

4. The method for generating software electronic manual based on a large language model according to claim 3, characterized in that: When determining to extract the interface element image using a convolutional neural network model or image processing based on the interface level, and obtaining the interface element of the interface element image based on the result of the interface extraction, the method includes: According to the first level of complexity, the image processing is used to perform interface extraction on the interface element image, and according to the second level of complexity, the convolutional neural network model is used to perform interface extraction on the interface element image; When the interface element image is subjected to interface extraction by image processing, the method includes: obtaining the gradient value of each pixel in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel, determining the texture similarity based on the texture feature, merging or separating adjacent pixels according to the texture similarity, and determining the interface element of the interface element image according to the result of merging or separating; When the convolutional neural network model is used to perform interface extraction on the interface element image, it includes: extracting local features and global features of the interface element image according to the convolution layer and pooling layer of the convolutional neural network model, and mapping the local features and the global features to output labels through a fully connected layer, and determining the interface elements of the interface element image according to the output labels.

5. The method for generating software electronic manual based on a large language model according to claim 4, characterized in that: When the convolutional neural network model is used to perform the interface extraction, verifying whether the interface elements of the interface element image are correct based on historical data includes: Determine a historical data set of the same interface element as the interface element image according to the historical data, and obtain historical element features according to the historical data set, wherein the historical element features include historical icon size, historical button size, and historical menu options; The similarity between the interface element and the historical element features is calculated by the following formula: ； in, Indicates similarity, , and represents the weight coefficient, and , Indicates the icon size. Indicates the history icon size, Indicates the button size. Indicates the history button size. Indicates a menu option. Represents the history menu option; Pre-set similarity threshold ; when Greater than or equal to , then verifying that the interface element of the interface element image is correct; when Less than , it is verified that the interface element of the interface element image is incorrect.

6. The method for generating software electronic manual based on a large language model according to claim 5, characterized in that: When it is determined that the interface element of the interface element image is incorrect, re-determining the interface element of the interface element image using a clustering algorithm includes: Representative data of each interface element is obtained according to the historical data, and an aggregated data set is established by combining the representative data and the historical element features. The feature vector of each data in the aggregated data set is extracted, and the expected number of clusters k is determined to be 3. The parameters of the Gaussian distribution are initialized, and the probability that each data in the aggregated data set belongs to each Gaussian distribution is calculated to obtain the responsibility value, and the cluster with the largest responsibility value is selected as the interface element of the interface element image.

7. The method for generating software electronic manual based on a large language model according to claim 6, characterized in that: When the text information of the interface element image is obtained by using the OCR technology, the software installation package is installed and run, and the interactive data during the running process is captured, it includes: Extracting the interface interval where the interface element is located in the interface element image based on OCR technology, converting the interface element in the interface interval into characters, and recognizing text information of the characters; The software installation package is stored in a virtual environment of a virtual machine, and the software is decompressed, installed, and run. Input events in the virtual environment are captured according to an API protocol to obtain the interaction data.

8. The method for generating software electronic manual based on a large language model according to claim 7, characterized in that: When generating operation steps according to the interface elements of the interface element image, the text information and the interaction data, it includes: The interface elements are matched according to the interaction data and the text information to obtain a first result, the text information is matched according to the interaction data and the interface elements to obtain a second result, the interaction data is matched according to the text information and the interface elements to obtain a third result, and the operation steps are generated in chronological order based on the first result, the second result and the third result.

9. The method for generating software electronic manual based on a large language model according to claim 8, characterized in that: When using a large language model to generate software electronic manuals, it includes: Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find model parameters of a large language model, establish a large language model, use the model training set to fit the large language model, and input the model test set into the large language model to verify the output accuracy of the large language model. When the output accuracy reaches a preset accuracy threshold, generate a software electronic manual according to the operation steps.

10. A system for generating software electronic manuals based on a large language model, used for applying the method for generating software electronic manuals based on a large language model as described in any one of claims 1 to 9, characterized in that: include: The acquisition unit is configured to determine the original data of the electronic manual of the software to be generated, wherein the original data includes a software operation video, an interface screenshot, and a software installation package, extract each video frame of the software operation video, obtain a corresponding video frame image according to each video frame, and process all the video frame images and the interface screenshot to obtain an interface element image of the electronic manual of the software to be generated; a processing unit configured to extract information features from the interface element image, determine an interface level of the interface element image according to the number of the information features, determine to perform interface extraction on the interface element image using a convolutional neural network model or image processing based on the interface level, derive interface elements of the interface element image based on the result of the interface extraction, and when performing the interface extraction using the convolutional neural network model, verify whether the interface elements of the interface element image are correct based on historical data, and when it is determined that the interface elements of the interface element image are incorrect, redetermine the interface elements of the interface element image using a clustering algorithm; The generation unit is configured to use OCR technology to obtain text information of the interface element image, install and run the software installation package, capture interaction data during the operation process, generate operation steps according to the interface elements of the interface element image, the text information and the interaction data, and use a large language model to generate an electronic manual of the software.

Images