A Method and System for Generating Software Electronic Instruction Manuals Based on Large Language Models

Abstract

The present invention relates to the field of specifications, and discloses a method and system for generating an electronic software specification based on a large language model. The method includes: extracting each video frame of a software operation video, obtaining a corresponding video frame image according to each video frame, processing all video frame images and interface screenshots, determining the interface level of the interface element image according to the number of information features, determining whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element image based on the interface level, verifying whether the interface elements of the interface element image are correct based on historical data, and when it is determined that the interface elements of the interface element image are incorrect, using a clustering algorithm to re-determine the interface elements of the interface element image; obtaining the text information of the interface element image by using OCR technology, and generating an electronic software specification by using a large language model. The present invention has the characteristics of comprehensively processing data and automatically generating an electronic software specification.

Description

Technical Field

[0001] The present invention relates to the technical field of specifications, and in particular, to a method and system for generating an electronic specification for software based on a large language model. Background Art

[0002] With technological progress, in the fields of software development and user support, specifications, as an important help document, can assist users in understanding and using various software. With the increase in the types of intelligent devices and software, the interface complexity of software is also continuously increasing, and it has become relatively difficult for users to understand and use software functions. However, traditional specifications are written manually, and for frequently updated software interfaces and functional changes, the content of the specifications may be missing or lagging, unable to reflect the latest changes in software functions in a timely manner, thereby bringing many inconveniences and affecting the normal use of the software. Currently, although there are some tools for automatically generating specifications, they rely on static information input or manual annotation, making it difficult to achieve intelligence and automation. Especially when facing the interfaces of complex software and cross-platform support, there are limitations in the existing specification generation, unable to meet the adaptation of multiple platforms and unable to comprehensively and synthetically analyze data.

[0003] Therefore, how to provide a method and system for generating an electronic specification for software based on a large language model is an urgent technical problem to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for generating an electronic specification for software based on a large language model, aiming to solve the problems that it is unable to meet the adaptation of multiple platforms and unable to comprehensively analyze data when facing the interfaces of complex software and cross-platform support.

[0005] On the one hand, the present invention proposes a method for generating an electronic specification for software based on a large language model, including:

[0006] Determine the original data of the software electronic specification to be generated, where the original data includes software operation videos, interface screenshots, and software installation packages. Extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, and process all the video frame images and the interface screenshots to obtain the interface element images of the software electronic specification to be generated;

[0007] Extract the information features in the interface element image, determine the interface level of the interface element image according to the number of the information features, determine whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element image based on the interface level, obtain the interface elements of the interface element image based on the result of the interface extraction. When using the convolutional neural network model to perform the interface extraction, verify whether the interface elements of the interface element image are correct based on historical data. When it is determined that the interface elements of the interface element image are incorrect, use a clustering algorithm to re-determine the interface elements of the interface element image;

[0008] Use OCR technology to obtain the text information of the interface element image, install and run the software installation package, capture the interaction data during the running process, generate operation steps according to the interface elements of the interface element image, the text information, and the interaction data, and generate a software electronic manual using a large language model.

[0009] Further, when processing all video frame images and the interface screenshots to obtain the interface element image of the software electronic manual to be generated, it includes:

[0010] Preprocess all video frame images and the interface screenshots. The preprocessing includes denoising, contrast adjustment, color balance, and de-sharpening. Extract feature points from the preprocessed interface screenshots and all video frame images according to a first preset algorithm, perform correlation matching on all the extracted feature points according to a second preset algorithm, determine the correlation data between the preprocessed interface screenshots and all video frame images. The correlation data includes regional location and image attributes, and correct the preprocessed interface screenshots and all video frame images according to the correlation data to obtain the interface element image of the software electronic manual to be generated.

[0011] Further, when extracting the information features in the interface element image and determining the interface level of the interface element image according to the number of the information features, it includes:

[0012] The information features include interface text, interface tables, and interface text boxes;

[0013] Count the number of the interface text, the interface tables, and the interface text boxes and sum them up, and record the sum result as the complexity quantity;

[0014] The interface levels include first-level complexity and second-level complexity;

[0015] When the complexity quantity is greater than or equal to the complexity quantity threshold, then determine the interface level of the interface element image as the first-level complexity;

[0016] When the complexity is less than the complexity threshold, the interface level of the interface element image is determined to be the secondary complexity;

[0017] The complexity of the primary complexity and the secondary complexity decreases in turn.

[0018] Further, when determining to use a convolutional neural network model or image processing to perform interface extraction on the interface element image and obtaining the interface elements of the interface element image based on the result of the interface extraction, it includes:

[0019] According to the primary complexity, use the image processing to perform interface extraction on the interface element image, and according to the secondary complexity, use the convolutional neural network model to perform interface extraction on the interface element image;

[0020] When using image processing to perform interface extraction on the interface element image, it includes: obtaining the gradient value of each pixel point in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel point, determining the texture similarity based on the texture feature, merging or separating adjacent pixel points according to the texture similarity, and determining the interface elements of the interface element image according to the result of the merging or separating;

[0021] When using the convolutional neural network model to perform interface extraction on the interface element image, it includes: extracting the local features and global features of the interface element image according to the convolutional layer and pooling layer of the convolutional neural network model, and mapping the local features and the global features to the output label through the fully connected layer, and determining the interface elements of the interface element image according to the output label.

[0022] Further, when using the convolutional neural network model to perform the interface extraction, when verifying whether the interface elements of the interface element image are correct based on historical data, it includes:

[0023] Determine the historical data set with the same interface elements as the interface element image according to the historical data, obtain the historical element features according to the historical data set, and the historical element features include historical icon size, historical button size and historical menu options;

[0024] Calculate the similarity between the interface elements and the historical element features through the following formula:

[0025] ;

[0026] where, represents the similarity, 、 and represent the weight coefficients, and , represents the icon size, represents the historical icon size, represents the button size, represents the historical button size, represents the menu option, represents the historical menu option;

[0027] Preset the similarity threshold ;

[0028] When is greater than or equal to , then verify that the interface element of the interface element image is correct;

[0029] When is less than , then verify that the interface element of the interface element image is incorrect.

[0030] Further, when it is determined that the interface element of the interface element image is incorrect and the clustering algorithm is used to re-determine the interface element of the interface element image, it includes:

[0031] Obtain the representative data of each interface element according to the historical data, combine the representative data and the historical element features to establish an aggregation data set, extract the feature vectors of each data in the aggregation data set, and determine that the expected number of clusters k is 3, initialize the parameters of the Gaussian distribution, calculate the probability that each data in the aggregation data set belongs to each Gaussian distribution, obtain the responsibility value, and select the cluster with the largest responsibility value as the interface element of the interface element image.

[0032] Further, when using the OCR technology to obtain the text information of the interface element image, install and run the software installation package, and capture the interaction data during the running process, it includes:

[0033] Extract the interface interval where the interface element is located in the interface element image based on the OCR technology, convert the interface element in the interface interval into characters, and recognize the text information of the characters;

[0034] Store the software installation package in the virtual environment of the virtual machine, decompress, install and run the software, and capture the input events in the virtual environment according to the API protocol to obtain the interaction data.

[0035] Further, when generating the operation steps according to the interface element of the interface element image, the text information, and the interaction data, it includes:

[0036] Match the interface elements based on the interaction data and the text information to obtain a first result, match the text information based on the interaction data and the interface elements to obtain a second result, match the interaction data based on the text information and the interface elements to obtain a third result, and generate the operation steps based on the first result, the second result, and the third result in chronological order.

[0037] Further, when generating a software electronic manual using a large language model, it includes:

[0038] Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find the model parameters of the large language model, establish the large language model, fit the large language model using the model training set, and input the model test set into the large language model to verify the output accuracy of the large language model. When the output accuracy reaches a preset accuracy threshold, generate a software electronic manual according to the operation steps.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows: By combining a software operation video, interface screenshots, and a software installation package with a large language model, the efficiency of generating a software electronic manual is improved, and the workload of manual writing and editing is reduced. Using a convolutional neural network model or image processing for interface extraction can accurately identify interface elements and ensure the accuracy of the software electronic manual. Based on the convolutional neural network model, historical data is combined to verify the correctness of the interface elements, and a clustering algorithm is used to re-determine incorrect interface elements, improving the adaptability of automatically generating a software electronic manual. The text information in the interface element image is extracted through OCR technology to ensure that the content of the generated software electronic manual is comprehensive and accurate. During the software installation and operation process, interaction data can be captured and combined with interface elements and text information to generate operation steps, enabling the software electronic manual to reflect the actual process of the software. Based on the large language model, the content of the manual that meets the user's needs can be intelligently generated according to the functional characteristics and requirements of the software, and corresponding software electronic manuals can be output for different platforms, improving the adaptability of the software electronic manual.

[0040] On the other hand, the present application also provides a system for generating a software electronic manual based on a large language model, which is used to apply the method for generating a software electronic manual based on a large language model as described above, and includes:

[0041] The acquisition unit is configured to determine the original data for generating the software electronic manual. The original data includes software operation videos, interface screenshots, and software installation packages. Each video frame of the software operation video is extracted, and the corresponding video frame image is obtained according to each video frame. All video frame images and the interface screenshots are processed to obtain the interface element images of the software electronic manual to be generated;

[0042] The processing unit is configured to extract the information features in the interface element images, determine the interface level of the interface element images according to the number of the information features, determine whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element images based on the interface level, obtain the interface elements of the interface element images based on the results of the interface extraction. When using the convolutional neural network model for the interface extraction, verify whether the interface elements of the interface element images are correct based on historical data. When it is determined that the interface elements of the interface element images are incorrect, use a clustering algorithm to re-determine the interface elements of the interface element images;

[0043] The generation unit is configured to use OCR technology to obtain the text information of the interface element images, install and run the software installation package, capture the interaction data during the running process, generate operation steps according to the interface elements of the interface element images, the text information, and the interaction data, and generate a software electronic manual using a large language model.

[0044] It can be understood that the above method and system for generating a software electronic manual based on a large language model have the same beneficial effects, which will not be elaborated here. Description of the Drawings

[0045] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0046] Figure 1 is the flowchart of the system for generating a software electronic manual based on a large language model provided by an embodiment of the present invention;

[0047] Figure 2 is the functional block diagram of the system for generating a software electronic manual based on a large language model provided by an embodiment of the present invention. Detailed Embodiments

[0048] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0049] In some embodiments of the present application, referring to Figure 1 as shown, a method for generating an electronic software manual based on a large language model includes:

[0050] S100: Determine the original data of the software electronic manual to be generated. The original data includes software operation videos, interface screenshots, and software installation packages. Extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, and process all the video frame images and interface screenshots to obtain the interface element images of the software electronic manual to be generated;

[0051] S200: Extract the information features in the interface element images, determine the interface level of the interface element images according to the number of information features, determine whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element images based on the interface level, obtain the interface elements of the interface element images based on the results of the interface extraction. When using a convolutional neural network model for interface extraction, verify whether the interface elements of the interface element images are correct based on historical data. When it is determined that the interface elements of the interface element images are incorrect, use a clustering algorithm to re-determine the interface elements of the interface element images;

[0052] S300: Use OCR technology to obtain the text information of the interface element images, install and run the software installation package, capture the interaction data during the running process, and generate operation steps based on the interface elements, text information, and interaction data of the interface element images, and generate an electronic software manual using a large language model.

[0053] Specifically, in S100, the software operation video, interface screenshots, and software installation package are used as the original data. Each video frame in the operation video is extracted and converted into a video frame image, and combined with the interface screenshots to form an interface element image, aiming to capture the software interface and its dynamic changes through different forms of input data, providing data support for subsequent interface element extraction and operation step generation. The multi-dimensional data lays the foundation for generating an accurate software electronic manual. In S200, the information features in the interface element image are extracted, and the interface level of the interface element image is determined according to the number of information features. According to the interface level, a convolutional neural network model or image processing is used to perform interface extraction on the interface element image, improving the dynamic processing ability for different interface levels. When using the convolutional neural network (CNN) model for interface extraction, the interface elements of the interface element image are verified according to historical data to avoid the risk of inaccurate interface elements caused by simply using the model, improving the accuracy of interface element extraction. When it is determined that the interface elements of the interface element image are incorrect, a clustering algorithm is used to re-determine the interface elements of the interface element image, further improving the accuracy and automation of the extraction and avoiding errors in the software electronic manual caused by human errors. In S300, OCR technology is used to extract the text information in the interface element image, further improving the description of the interface element image. The software installation package is installed and run, and the interaction data during the software operation is captured. The interaction data includes behaviors such as clicks, inputs, and scrolls. The interaction data reflects the situation of interacting with the interface, and combined with the interface elements, text information, and interaction data of the interface element image, the operation steps are automatically generated. The operation steps not only reflect how to interact with the software but also clarify the interface elements that need to be operated in each step. Thus, through the large language model, these operation steps are transformed into a coherent and clear language description, and then the software electronic manual is generated.

[0054] It can be understood that using the software operation video, interface screenshots, and software installation package as the original data and performing analysis and processing avoids the cumbersome process required for traditional manual writing of the manual. Using a convolutional neural network model or image processing to perform interface extraction on the interface element image and using OCR technology to extract the text information in the interface element image, and accurately generating the operation steps in combination with the interaction data not only saves time but also ensures the accuracy of the operation steps. In the traditional process of writing the manual, it is easy to overlook some details or make mistakes in the operation steps, especially when the software interface is complex or frequently updated. Through the convolutional neural network and OCR technology, combined with the optimization of the clustering algorithm, it can automatically adapt to different interface changes, ensuring the matching degree of the operation steps and interface elements. Even if the software interface is updated or changed, it can still be automatically adjusted according to historical data, reducing the burden of maintaining and updating the manual and improving the efficiency of generating the software electronic manual.

[0055] In some embodiments of the present application, when processing all video frame images and interface screenshots to obtain the interface element images to be used for generating a software electronic manual, it includes:

[0056] Preprocess all video frame images and interface screenshots. The preprocessing includes denoising, contrast adjustment, color equalization, and de-sharpening. Extract feature points from the preprocessed interface screenshots and all video frame images according to a first preset algorithm. Perform correlation matching on all the extracted feature points according to a second preset algorithm to determine the correlation data between the preprocessed interface screenshots and all video frame images. The correlation data includes regional positions and image attributes. Correct the preprocessed interface screenshots and all video frame images according to the correlation data to obtain the interface element images to be used for generating a software electronic manual.

[0057] Specifically, denoising is usually achieved by methods such as Gaussian filtering or median filtering. Gamma correction is used for contrast adjustment to enhance the brightness range, especially for adjusting the brightness of dark parts. Color equalization adjusts the exposure and color distribution to make them visually consistent, eliminating unnecessary information that affects subsequent processing, thereby improving the quality of the interface element images. The first preset algorithm includes the Harris corner detection algorithm and AKAZE (Accelerated-KAZE Features). For specific feature point extraction, one of them can be adopted according to the actual situation. The second preset algorithm includes KNN (K-Nearest Neighbors) matching and RANSAC (Random Sample Consensus algorithm), which can determine the regional positions and image attributes. For specific correlation matching of all the extracted feature points, one of them can be adopted according to the actual situation. And based on non-linear transformation and resampling, sparse and dense optical flow, etc., the preprocessed interface screenshots and all video frame images are corrected, improving the quality and accuracy of the interface element images.

[0058] In some embodiments of the present application, when extracting the information features in the interface element images and determining the interface level of the interface element images according to the number of information features, it includes:

[0059] The information features include interface text, interface tables, and interface text boxes;

[0060] Count the number of interface text, interface tables, and interface text boxes and sum them up, and record the sum result as the complexity quantity;

[0061] The interface levels include first-level complexity and second-level complexity;

[0062] When the complexity quantity is greater than or equal to the complexity quantity threshold, the interface level of the interface element image is determined to be first-level complexity;

[0063] When the complexity measure is less than the complexity threshold, the interface level of the interface element image is determined to be second-level complex;

[0064] The complexity levels of first-level complex and second-level complex decrease in sequence.

[0065] In some embodiments of the present application, when performing interface extraction on the interface element image based on the interface level using a convolutional neural network model or image processing, and obtaining the interface elements of the interface element image based on the result of the interface extraction, it includes:

[0066] Performing interface extraction on the interface element image using image processing according to first-level complex, and performing interface extraction on the interface element image using a convolutional neural network model according to second-level complex;

[0067] When performing interface extraction on the interface element image using image processing, it includes: obtaining the gradient value of each pixel point in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel point, determining the texture similarity based on the texture feature, merging or separating adjacent pixel points according to the texture similarity, and determining the interface elements of the interface element image according to the result of the merging or separation;

[0068] When performing interface extraction on the interface element image using a convolutional neural network model, it includes: extracting the local features and global features of the interface element image according to the convolutional layer and pooling layer of the convolutional neural network model, and mapping the local features and global features to the output label through the fully connected layer, and determining the interface elements of the interface element image according to the output label.

[0069] Specifically, selecting different processing methods according to the interface level of the interface element image can improve the efficiency of the interface element image processing process. The complexity threshold is preferably 20, and can be specifically adjusted according to actual needs. For first-level complex, image processing includes image feature analysis and region segmentation algorithms. Image processing is responsible for processing interface element images with a higher degree of complexity, obtaining the spatial relationship between interface elements, and further constructing the hierarchical structure of interface elements. For example, determining whether a certain button is within a pop-up window. For second-level complex, a convolutional neural network (CNN) model is used. The convolutional neural network model is suitable for processing interface element images with a lower degree of complexity, and has good adaptability and accuracy. It can automatically learn the interface element image and record relevant data, so as to accurately determine the interface elements of the interface element image. The interface elements include relevant data such as buttons, menus, and icons.

[0070] It can be understood that through the automatic selection strategy, flexible adjustment processing can be performed according to the interface level of the interface element image, making the extraction process targeted, which helps to improve the accuracy of interface element extraction, obtain the gradient value of each pixel point in the interface element image, thereby determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel point, and then determining the texture similarity based on the texture feature. First, determine the texture feature corresponding to a pixel point as the initial feature, then obtain the texture feature corresponding to an adjacent pixel point as the target feature, and determine the texture similarity according to the ratio of the initial feature and the target feature until all texture similarities are obtained. When the texture similarity is greater than or equal to 0.8, it is considered that the texture features of two adjacent pixel points are similar, and the adjacent pixel points are merged. When the texture similarity is less than 0.8, it is considered that the texture features of two adjacent pixel points are not similar, and the adjacent pixel points are separated, thereby determining the interface element of the interface element image. The convolutional neural network method performs feature extraction through multiple layers of networks, providing an accurate output label of the interface element, thereby determining the interface element of the interface element image according to the output label.

[0071] In some embodiments of the present application, when using a convolutional neural network model for interface extraction, when verifying whether the interface element of the interface element image is correct based on historical data, it includes:

[0072] Determine a historical data set of the same interface element as the interface element image according to historical data, and obtain historical element features from the historical data set. The historical element features include historical icon size, historical button size, and historical menu options;

[0073] Calculate the similarity between the interface element and the historical element features through the following formula:

[0074] ;

[0075] where represents the similarity, 、 and represent weight coefficients, and , represents the icon size, represents the historical icon size, represents the button size, represents the historical button size, represents the menu option, represents the historical menu option;

[0076] Preset a similarity threshold ;

[0077] When is greater than or equal to When it is, the interface elements of the verified interface element image are correct;

[0078] When is less than When it is, the interface elements of the verified interface element image are incorrect.

[0079] In some embodiments of the present application, when it is determined that the interface elements of the interface element image are incorrect, when using a clustering algorithm to re-determine the interface elements of the interface element image, it includes:

[0080] Obtain the representative data of each interface element according to historical data, combine the representative data and historical element features to establish an aggregation data set, extract the feature vectors of each data in the aggregation data set, and determine that the expected number of clusters k is 3, initialize the parameters of the Gaussian distribution, calculate the probability that each data in the aggregation data set belongs to each Gaussian distribution, obtain the responsibility value, and select the cluster with the largest responsibility value as the interface elements of the interface element image.

[0081] Specifically, by comparing with historical data, it can accurately verify whether the interface elements of the interface element image are correct, reduce the dependence on manual experience, improve the degree of automation, and can continuously accumulate and update historical data, gradually improving the judgment ability and adaptability. In the face of new or unseen interface elements, it can still provide reliable judgment results. And when it is determined that the interface elements of the interface element image are incorrect, using a clustering algorithm to re-determine the interface elements of the interface element image avoids the errors caused by relying solely on judgment, improves the accuracy of classification, and the Gaussian mixture model allows automatic adjustment of classification parameters according to the natural distribution of data, so as to adapt to different data and interface elements.

[0082] In some embodiments of the present application, when using OCR technology to obtain the text information of the interface element image, installing and running the software installation package, and capturing the interaction data during the running process, it includes:

[0083] Based on OCR technology, extract the interface interval where the interface elements in the interface element image are located, convert the interface elements in the interface interval into characters, and recognize the text information of the characters;

[0084] Store the software installation package in the virtual environment of the virtual machine, and perform decompression installation and run the software. According to the API protocol, capture the input events in the virtual environment to obtain interaction data.

[0085] Specifically, by adopting OCR technology and capturing input events according to the API protocol in the virtual environment, the accuracy and efficiency of the automatic generation process of software electronic manuals are effectively improved. The OCR technology is used to convert the interface elements within the interface area into characters, thereby identifying the text information of the characters, ensuring the accuracy of information extraction. In this context, using OCR technology to convert interface elements into characters and identify text information, the text information represents the specific content of the interface elements, including prompt feedback information, such as error messages and success prompts, etc. Secondly, the virtual machine environment is used to install and run the software. The virtual machine environment can be configured as Android, iOS, Windows, macOS, and Web, etc., so as to adapt to different software installation conditions, and the API protocol is used to capture input events in the virtual environment. The input events include clicks, double-clicks, etc., thereby simulating the running process of the actual software, and then collecting interaction data. The interaction data, such as the data of clicking the button after clicking the icon, thereby improving the reproducibility of subsequent software operations and enhancing the automation level of generating software electronic manuals.

[0086] In some embodiments of the present application, when generating operation steps based on the interface elements, text information, and interaction data of the interface element image, it includes:

[0087] Matching the interface elements according to the interaction data and text information to obtain a first result, matching the text information according to the interaction data and interface elements to obtain a second result, matching the interaction data according to the text information and interface elements to obtain a third result, and generating operation steps based on the first result, second result, and third result in chronological order.

[0088] Specifically, generating operation steps based on the comprehensive matching of interaction data, text information, and interface elements can comprehensively and accurately reconstruct the operation steps of the software. Suppose that during the operation of the software, the "Save" button is clicked on the interface, and the interaction data matches the text "Save" displayed on the interface, then the first result is obtained: "Click the Save button". Suppose the user enters a username and clicks the "Login" button. The interaction data matches the interface elements and text information, and the second result is obtained: "Enter the username in the username field and click the Login button". Suppose there is a prompt text "Please enter a correct email address" on the interface, and the text information and interface elements match the interaction data to obtain the third result: "Enter a valid email address and operate according to the prompt requirements" to ensure the consistency between the generated operation steps and the actual operations, and the generation method based on chronological order ensures that the generated operation steps conform to the logical order of the actual operations, thereby generating coherent and clear operation steps, and then automatically generating software electronic manuals, which helps to enhance the readability of software electronic manuals.

[0089] In some embodiments of the present application, when generating a software electronic manual using a large language model, it includes:

[0090] Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find the model parameters of the large language model, establish the large language model, use the model training set to fit the large language model, and input the model test set into the large language model to verify the output accuracy rate of the large language model. When the output accuracy rate reaches the preset accuracy rate threshold, generate a software electronic manual according to the operation steps.

[0091] Specifically, the historical operation step data records the running status of the software at different time periods. Divide the historical operation step data into a model training set and a model test set. Usually, 60% - 70% of the data is used as the model training set, and the rest is used as the model test set to ensure that both the model training set and the model test set contain data of various running statuses to improve the generalization ability of the large language model. Use cross-validation combined with grid search to find the model parameters of the large language model. Cross-validation divides the data into several parts and trains the model multiple times to verify its stability and performance. Grid search searches for parameter combinations in the parameter space and uses the model test set to fit the large language model, reducing the risk of overfitting and improving the accuracy and stability of the model. Input the model test set into the already trained large language model and verify the output accuracy rate of the large language model. The output accuracy rate reflects the performance of the model on unknown data and is an important indicator for evaluating the model's performance. After the model reaches the preset accuracy rate threshold, an accurate software electronic manual can be generated in real time based on the current operation steps, improving the reliability and automation of generating the software electronic manual.

[0092] It can be understood that since the large language model is trained based on the historical operation step data, when generating a software electronic manual, the established large language model can establish a corresponding processing mechanism according to the historical operation step data, and store the data for each generation to iteratively upgrade the large language model. Thus, according to the obtained changes in the interface and functions of the software, the content of the generated software electronic manual can be adjusted in a timely manner, ensuring the accuracy and timeliness of the software electronic manual. Moreover, the large language model can generate multiple output formats, such as: an illustrated operation step diagram combining pictures and texts. If a text description is required, a detailed language description is provided according to the historical operation step data and the current operation steps to meet the needs of different users. Since the interaction data is obtained from input events in a virtual environment, and the virtual machine environment is configured as Android, iOS, Windows, macOS, and Web, etc., the interaction data includes interactions on multiple platforms, enabling the large language model to output corresponding software electronic manuals for different platforms, improving the adaptability of the software electronic manual to different platforms.

[0093] In summary, the beneficial effects of the present invention are as follows: By means of software operation videos, interface screenshots, and software installation packages, combined with large language models, the efficiency of generating software electronic manuals is improved, and the workload of manual writing and editing is reduced. Using a convolutional neural network model or image processing for interface extraction can accurately identify interface elements and ensure the accuracy of software electronic manuals. Based on the convolutional neural network model, historical data is combined to verify the correctness of interface elements, and a clustering algorithm is used to re-determine incorrect interface elements, improving the adaptability of automatically generating software electronic manuals. By using OCR technology to extract text information from interface element images, it is ensured that the content of the generated software electronic manual is comprehensive and accurate. During the installation and operation of the software, interaction data can be captured and combined with interface elements and text information to generate operation steps, enabling the software electronic manual to reflect the actual process of the software. Based on large language models, it is possible to intelligently generate manual content that meets user needs according to the functional characteristics and requirements of the software, and output corresponding software electronic manuals for different platforms, improving the adaptability and automation of software electronic manuals.

[0094] In another preferred manner based on the above embodiments, refer to Figure 2 As shown, this embodiment provides a system for generating software electronic manuals based on large language models, which is used to apply the method for generating software electronic manuals based on large language models as described above, and includes:

[0095] An acquisition unit, configured to determine the original data for generating a software electronic manual. The original data includes software operation videos, interface screenshots, and software installation packages. Each video frame of the software operation video is extracted, and a corresponding video frame image is obtained according to each video frame. All video frame images and interface screenshots are processed to obtain interface element images for generating the software electronic manual;

[0096] A processing unit, configured to extract information features from the interface element images, determine the interface level of the interface element images according to the number of information features, and based on the interface level, determine whether to use a convolutional neural network model or image processing for interface extraction of the interface element images. Based on the result of the interface extraction, the interface elements of the interface element images are obtained. When using a convolutional neural network model for interface extraction, historical data is used to verify whether the interface elements of the interface element images are correct. When it is determined that the interface elements of the interface element images are incorrect, a clustering algorithm is used to re-determine the interface elements of the interface element images;

[0097] A generation unit, configured to use OCR technology to obtain the text information of the interface element images, install and run the software installation package, capture interaction data during the running process, and generate operation steps according to the interface elements, text information, and interaction data of the interface element images, and generate a software electronic manual using a large language model.

[0098] Specifically, the acquisition unit obtains the original data, including software operation videos, interface screenshots, and software installation packages, providing data support for subsequent processing. The processing of video frame images and interface screenshots enables the system to capture the dynamic and static information of software operations, ensuring that the generated electronic manual has good accuracy and integrity. The processing unit determines the interface level of the interface element image by analyzing the information features of the interface element image, and selects a convolutional neural network (CNN) or image processing technology to extract the interface element. Through the intelligent judgment of the interface level, the system can flexibly select an appropriate extraction method, thereby improving the extraction accuracy and efficiency. When using the convolutional neural network model, the system validates the extraction results based on historical data to further ensure the correctness of the interface elements. When it is inaccurate, the clustering algorithm is used for adjustment to ensure the reliability and comprehensiveness of the extraction. According to the OCR technology, the text information in the interface element image is obtained. The system simulates the software operation environment and captures the interaction data during the software operation. The operation steps are generated based on the interface elements, text information, and interaction data of the interface element image. The generation unit uses a large language model to automatically write the software electronic manual, making the software electronic manual logical, improving the automation degree and accuracy of generating the software electronic manual.

[0099] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowcharts and / or block diagrams can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0101] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or a plurality of processes and / or boxes Figure 1 in the boxes or a plurality of boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process Figure 1 one process or a plurality of processes and / or boxes Figure 1 in the boxes or a plurality of boxes.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific implementation manners of the present invention. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for generating an electronic instruction manual for software based on a large language model, characterized in that, Including: Determine the original data for generating the software electronic manual. The original data includes software operation videos, interface screenshots, and software installation packages. Extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, and process all the video frame images and the interface screenshots to obtain the interface element images of the software electronic manual to be generated; Extract the information features in the interface element images, determine the interface level of the interface element images according to the number of the information features, and based on the interface level, determine whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element images. Based on the result of the interface extraction, obtain the interface elements of the interface element images. When using the convolutional neural network model to perform the interface extraction, verify whether the interface elements of the interface element images are correct based on historical data. When it is determined that the interface elements of the interface element images are incorrect, use a clustering algorithm to re-determine the interface elements of the interface element images; Use OCR technology to obtain the text information of the interface element images, install and run the software installation package, capture the interaction data during the running process, and generate operation steps according to the interface elements of the interface element images, the text information, and the interaction data, and use a large language model to generate a software electronic manual; When processing all the video frame images and the interface screenshots to obtain the interface element images of the software electronic manual to be generated, it includes: Preprocess all the video frame images and the interface screenshots. The preprocessing includes denoising, contrast adjustment, color balance, and de-sharpening. Extract feature points from the preprocessed interface screenshots and all the video frame images according to a first preset algorithm, perform correlation matching on all the extracted feature points according to a second preset algorithm, determine the correlation data between the preprocessed interface screenshots and all the video frame images. The correlation data includes regional positions and image attributes, and correct the preprocessed interface screenshots and all the video frame images according to the correlation data to obtain the interface element images of the software electronic manual to be generated.

2. The method for generating an electronic specification of software based on a large language model according to claim 1, wherein, When extracting the information features in the interface element images and determining the interface level of the interface element images according to the number of the information features, it includes: The information features include interface text, interface tables, and interface text boxes; Count the number of the interface text, the interface tables, and the interface text boxes and sum them up, and record the sum result as the complexity quantity; The interface levels include first-level complexity and second-level complexity; When the complexity quantity is greater than or equal to the complexity quantity threshold, determine the interface level of the interface element images as the first-level complexity; When the complexity quantity is less than the complexity quantity threshold, determine the interface level of the interface element images as the second-level complexity; The complexity degrees of the first-level complexity and the second-level complexity decrease in sequence.

3. The method for generating an electronic specification of software based on a large language model according to claim 2, wherein When determining whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element images based on the interface level, and obtaining the interface elements of the interface element images based on the result of the interface extraction, it includes: According to the first-level complexity, the interface extraction of the interface element image is performed using the image processing, and according to the second-level complexity, the interface extraction of the interface element image is performed using the convolutional neural network model; When performing the interface extraction of the interface element image using the image processing, it includes: obtaining the gradient value of each pixel point in the interface element image, determining the edge position of the interface element image, and determining the corresponding texture feature according to each pixel point, determining the texture similarity based on the texture feature, merging or separating adjacent pixel points according to the texture similarity, and determining the interface element of the interface element image according to the result of the merging or separation; When performing the interface extraction of the interface element image using the convolutional neural network model, it includes: extracting the local feature and the global feature of the interface element image according to the convolutional layer and the pooling layer of the convolutional neural network model, and mapping the local feature and the global feature to the output label through the fully connected layer, and determining the interface element of the interface element image according to the output label.

4. The method for generating an electronic specification of software based on a large language model according to claim 3, wherein, When performing the interface extraction using the convolutional neural network model, when verifying whether the interface element of the interface element image is correct based on the historical data, it includes: Determining the historical data set with the same interface element as the interface element image according to the historical data, and obtaining the historical element features from the historical data set, where the historical element features include the historical icon size, the historical button size, and the historical menu options; Calculate the similarity between the interface element and the historical element features ; Preset similarity threshold ; When is greater than or equal to then the interface element of the interface element image is verified to be correct; When less than , the interface element of the interface element image is incorrect.

5. The method for generating an electronic specification of software based on a large language model according to claim 4, wherein, When it is determined that the interface element of the interface element image is incorrect and the clustering algorithm is used to re-determine the interface element of the interface element image, it includes: Obtaining the representative data of each interface element according to the historical data, combining the representative data and the historical element features to establish an aggregated data set, extracting the feature vectors of each data in the aggregated data set, determining the expected number of clusters k as 3, initializing the parameters of the Gaussian distribution, calculating the probability that each data in the aggregated data set belongs to each Gaussian distribution, obtaining the responsibility value, and selecting the cluster with the largest responsibility value as the interface element of the interface element image.

6. The method for generating an electronic specification of software based on a large language model according to claim 5, wherein When using the OCR technology to obtain the text information of the interface element image, installing and running the software installation package, and capturing the interaction data during the running process, it includes: Extracting the interface interval where the interface element is located in the interface element image based on the OCR technology, converting the interface elements in the interface interval into characters, and recognizing the text information of the characters; Storing the software installation package in the virtual environment of the virtual machine, decompressing, installing, and running the software, and capturing the interaction data according to the input events in the virtual environment according to the API protocol.

7. The method for generating an electronic specification of software based on a large language model according to claim 6, wherein, When generating the operation steps according to the interface element of the interface element image, the text information, and the interaction data, it includes: Match the interface elements based on the interaction data and the text information to obtain a first result, match the text information based on the interaction data and the interface elements to obtain a second result, match the interaction data based on the text information and the interface elements to obtain a third result, and generate the operation steps based on the first result, the second result, and the third result in chronological order.

8. The method for generating an electronic specification of software based on a large language model according to claim 7, wherein When generating a software electronic manual using a large language model, it includes: Obtain historical operation step data, divide the historical operation step data into a model training set and a model test set, use cross-validation and grid search to find the model parameters of the large language model, establish the large language model, fit the large language model using the model training set, and input the model test set into the large language model to verify the output accuracy of the large language model. When the output accuracy reaches the preset accuracy threshold, generate a software electronic manual according to the operation steps.

9. A system for generating an electronic software instruction manual based on a large language model, which is used to apply the method for generating an electronic software instruction manual based on a large language model according to any one of claims 1-8, characterized in that, It includes: A collection unit configured to determine the original data of the software electronic manual to be generated. The original data includes software operation videos, interface screenshots, and software installation packages. Extract each video frame of the software operation video, obtain the corresponding video frame image according to each video frame, and process all the video frame images and the interface screenshots to obtain the interface element images of the software electronic manual to be generated. A processing unit configured to extract the information features in the interface element images, determine the interface level of the interface element images according to the number of the information features, determine whether to use a convolutional neural network model or image processing to perform interface extraction on the interface element images based on the interface level, obtain the interface elements of the interface element images based on the result of the interface extraction. When using the convolutional neural network model to perform the interface extraction, verify whether the interface elements of the interface element images are correct based on historical data. When it is determined that the interface elements of the interface element images are incorrect, use a clustering algorithm to re-determine the interface elements of the interface element images. A generation unit configured to obtain the text information of the interface element images using OCR technology, install and run the software installation package, capture the interaction data during the running process, generate operation steps according to the interface elements, the text information, and the interaction data of the interface element images, and generate a software electronic manual using a large language model.

Images