Excel data analysis method and system based on Transform model
Patent Information
- Application Number
- CN202411307647.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-13
AI Technical Summary
Existing Excel data parsing and importing methods rely on manual operations, leading to inefficiency, errors, lack of intelligent processing, and poor adaptability to data changes, increasing workload and costs.
An Excel data parsing method and system using a Transformer model to preprocess Excel files, analyze data structure and content, and generate intelligent import codes, with real-time monitoring and adaptive learning capabilities to adjust strategies.
Improves automation and development efficiency, reduces errors, and enables the system to adapt to data changes, enhancing the accuracy and efficiency of data import.
Smart Images
Figure CN119129574A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to an Excel data parsing method and system based on a Transformer model. Background Art
[0002] With the rapid development of information technology, data processing and data analysis have become an indispensable part of various industries. As a widely used data processing tool, Excel's data parsing and importing functions play an important role in the data processing process. However, traditional Excel data parsing and importing methods often rely on manual operations, which are inefficient and prone to errors. Therefore, how to help developers generate relevant usable codes based on Excel templates to achieve automatic parsing and importing of Excel data has become an urgent problem to be solved.
[0003] Problems with existing technologies
[0004] 1. Low degree of automation
[0005] Existing technologies usually rely on manual operations to parse Excel data and manually write import codes, which makes the entire data import process inefficient and error-prone. Manual parsing and coding are not only time-consuming, but may also introduce data errors or format inconsistencies due to human factors.
[0006] 2. Lack of intelligent processing
[0007] Traditional Excel data parsing and importing methods lack intelligent processing capabilities. They cannot automatically adjust parsing strategies and import methods according to the structure and characteristics of Excel data, making it difficult to adapt to complex and changing data processing requirements. This results in developers frequently having to manually adjust and process data in actual applications, reducing work efficiency.
[0008] 3. Lack of adaptability
[0009] Existing technologies often lack adaptability when processing Excel data. They cannot automatically adjust parsing strategies and import methods according to data changes, resulting in the need for redevelopment and adjustment when data changes. This not only increases workload and costs, but may also affect the accuracy and efficiency of data import. Summary of the invention
[0010] In view of the above problems, the present invention is proposed to provide an Excel data parsing method and system based on a Transformer model to overcome the above problems or at least partially solve the above problems.
[0011] According to one aspect of the present invention, a method for parsing Excel data based on a Transformer model is provided, the method comprising:
[0012] Receive Excel files uploaded by users;
[0013] Preprocess the input Excel file to obtain the preprocessed Excel template;
[0014] Use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file to obtain the analysis results;
[0015] Generate codes accordingly based on the analysis results.
[0016] Optionally, the preprocessing of the input Excel file specifically includes:
[0017] Perform format check on the input Excel file, give corresponding optimization tips, and output an optimized Excel template, which is the data structure and content of the preprocessed Excel file.
[0018] Optionally, the using of the trained Transformer model to analyze the data structure and content of the preprocessed Excel file to obtain the analysis results specifically includes:
[0019] Use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file and extract key information;
[0020] Based on the parsed Excel data structure and the import requirements set by the user, the corresponding import code is intelligently generated using the Transformer algorithm;
[0021] During the data import process, the Transformer algorithm is used to monitor the import progress and status in real time.
[0022] Optionally, the key information specifically includes: data columns, data rows and data types.
[0023] Optionally, the real-time monitoring of import progress and status using the Transformer algorithm specifically includes:
[0024] Automatically adjust the import strategy when encountering data errors or format inconsistencies;
[0025] Transformer processing analysis has the ability of adaptive learning;
[0026] Optimize your own processing and analysis capabilities based on historical data and imported experience.
[0027] Optionally, generating a code corresponding to the analysis result specifically includes:
[0028] The user chooses to directly import the code or export the generated code into a compressed package to obtain the corresponding generated code.
[0029] Optionally, the Transformer processing analysis has the capability of adaptive learning specifically including:
[0030] Calculate the attention weight, calculate the attention weight between each position and other positions, and obtain the importance of each position to other positions;
[0031] Calculate the weighted sum, multiply each position vector by the attention weight, and add them together to get the weighted sum vector;
[0032] Linear transformation, linear transformation is performed on the weighted sum vector to obtain the final output vector.
[0033] The present invention also provides an Excel data analysis system based on a Transformer model, the data analysis system comprising:
[0034] Excel upload module, used to receive Excel files uploaded by users;
[0035] The preprocessing module is used to preprocess the input Excel file and obtain the Excel template after preprocessing;
[0036] The Transformer processing and analysis module is used to analyze the data structure and content of the preprocessed Excel file using the trained Transformer model to obtain the analysis results;
[0037] The code output module is used to generate codes according to the analysis results.
[0038] The present invention provides an Excel data analysis method and system based on a Transformer model, wherein the data analysis method comprises: receiving an Excel file uploaded by a user; preprocessing the input Excel file to obtain a preprocessed Excel template; using a trained Transformer model to analyze the data structure and content of the preprocessed Excel file to obtain an analysis result; and generating code corresponding to the analysis result. The degree of automation and development efficiency are improved, and the learning and optimization capabilities of the Transformer model enable the system to automatically adapt to changes in data, reducing the complexity and cost of system maintenance.
[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0041] Figure 1 A flowchart of an Excel data parsing method based on a Transformer model provided in an embodiment of the present invention;
[0042] Figure 2 A block diagram of a Transformer model-based Excel data parsing system provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0044] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.
[0045] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0046] like Figure 1 As shown, a method for parsing Excel data based on a Transformer model includes:
[0047] Receive the Excel file uploaded by the user; preprocess the input Excel file to obtain the preprocessed Excel template; use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file to obtain the analysis result; generate code accordingly based on the analysis result.
[0048] like Figure 2As shown, an Excel data parsing system based on a Transformer model includes:
[0049] Excel upload module, preprocessing module, Transformer processing and analysis module, and code output module.
[0050] Excel upload module: responsible for receiving Excel files uploaded by users.
[0051] Preprocessing module: Perform necessary format verification operations on the input Excel file, such as whether there are merged items in the header of the file, whether there is data required verification, etc., and give corresponding optimization tips, and output an optimized Excel template to improve the accuracy of subsequent analysis.
[0052] Transformer processing and analysis module: Use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file and extract key information, such as data columns, data rows, data types, etc. Based on the parsed Excel data structure and the import requirements set by the user, the Transformer algorithm is used to intelligently generate the corresponding import code. During the data import process, the Transformer processing and analysis module can monitor the import progress and status in real time. When encountering problems such as data errors and inconsistent formats, the module can automatically adjust the import strategy, such as skipping erroneous data and converting data formats, to ensure the smooth progress of data import. At the same time, the Transformer processing and analysis module has the ability of adaptive learning. It can continuously optimize its processing and analysis capabilities based on historical data and import experience. Over time, the module will be able to parse Excel files more accurately, generate more efficient import codes, and provide more optimized data import solutions.
[0053] Code output module: Users can choose to directly import the code or export the generated code into a compressed package to obtain the corresponding generated code.
[0054] The Transformer model is a neural network model based on the self-attention mechanism, which is used to process sequence data. Compared with the traditional recurrent neural network model, the Transformer model has better parallel performance and shorter training time, so it has been widely used in the field of natural language processing.
[0055] 1. Model Overview
[0056] As a neural network model based on the self-attention mechanism, the Transformer model can globally model each element in the sequence and establish connections between the elements. Compared with the recurrent neural network model, the Transformer model has better parallel performance and shorter training time.
[0057] The Transformer model contains multiple layers of encoders and decoders, each of which consists of multiple attention mechanism modules and feedforward neural network modules. The encoder is used to encode the input sequence into a high-dimensional feature vector representation, and the decoder is used to decode the vector representation into the target sequence. In the Transformer model, techniques such as residual connections and layer normalization are also used to accelerate model convergence and improve model performance. The core of the Transformer model is the Self-Attention Mechanism, which assigns a weight to each position in each input sequence and then uses these weighted position vectors as output.
[0058] Self-Attention Mechanism: As the core of the Transformer model, it generates an attention weight matrix by calculating the correlation score between any two positions in the input sequence. This matrix determines which positions in the input sequence should be paid attention to when generating the output of a certain position. In this way, the Transformer model can capture long-distance dependencies and process sequences of arbitrary length.
[0059] The calculation process of the self-attention mechanism includes three steps:
[0060] 1. Calculate attention weights: Calculate the attention weights between each position and other positions, that is, the importance of each position to other positions.
[0061] 2. Calculate the weighted sum: multiply each position vector by the attention weight and then add them together to get the weighted sum vector.
[0062] 3. Linear transformation: Perform a linear transformation on the weighted sum vector to obtain the final output vector.
[0063] Application Areas
[0064] Natural Language Processing:
[0065] Natural language processing refers to converting human language into a form that computers can understand so that computers can process and understand the language. Transformer models can classify text, such as classifying an email as spam or not spam. In this case, the Transformer model can take text as input and then output the category label. Transformer models can also perform sentiment analysis on text, such as determining whether an article is positive or negative. In this case, the Transformer model can take text as input and then output the sentiment polarity.
[0066] Voice Recognition:
[0067] Speech recognition refers to converting human speech into a form that computers can understand so that they can process and understand the speech. Some recent studies have shown that Transformer-based speech recognition systems have achieved performance comparable to traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs).
[0068] Reinforcement Learning:
[0069] The application of the Transformer model in the field of reinforcement learning is mainly applied to policy learning and value function approximation. Reinforcement learning refers to allowing a machine to learn the optimal behavior strategy through trial and error in the process of interacting with the environment. In reinforcement learning, the model needs to predict the next state and reward by learning the state transition probability to achieve enhanced learning. The Transformer model can process multiple input sequences through a multi-head attention mechanism and fuse them into an output sequence. In reinforcement learning, the Transformer model takes the current state as input and then outputs an action strategy. Specifically, the Transformer model learns the state transition probability function so that in the current state, the maximum reward can be obtained after selecting an action.
[0070] Transformer model implementation, including:
[0071] When implementing a system that parses Excel and generates code based on the Transformer model, we need to train and infer the model. The following is a simplified implementation step:
[0072] Data preprocessing:
[0073] Parse Excel table data into a sequence format that the model can process. This usually includes converting the cell data in the table into word vector representations. Generate the corresponding data import code as a label for supervised learning.
[0074] Build the model:
[0075] Implement the encoder part of the Transformer model. This includes multiple self-attention layers, feed-forward neural network layers, normalization layers, etc.
[0076] Implement the decoder part of the Transformer model. The decoder is similar to the encoder, but contains an additional attention layer to focus on the output of the encoder.
[0077] Combine the encoder and decoder into a complete Transformer model.
[0078] Model training, including:
[0079] For new Excel spreadsheet data, use the trained Transformer model for inference. Parse the Excel data into a sequence format and input it into the model's encoder. Use the decoder to gradually generate the corresponding data import code. Post-process the generated code (such as formatting, optimization, etc.) for use in the actual environment.
[0080] Optimization and deployment, including:
[0081] In practical applications, more adjustments and optimizations are needed based on specific tasks and data characteristics. In addition, due to the complexity of the Transformer model and the demand for computing resources, high-performance computing devices (such as GPUs) are usually required to accelerate the model training and reasoning process.
[0082] In building a system based on the Transformer model to parse Excel to generate code and implement data import, the Transformer model's ability to process sequence data is fully utilized, especially its advantages in capturing long-distance dependencies and parallel processing. The system first converts Excel table data into a sequence format that the model can understand through data preprocessing, and generates the corresponding data import code as a label. Then, a Transformer model containing an encoder and a decoder is built, and trained using the preprocessed data. During the training process, the model continuously iterates and optimizes parameters to learn the rules and patterns for generating data import code from Excel data.
[0083] After training, the system can receive new Excel data as input, convert it into vector representation through the encoder, and use the decoder to gradually generate the corresponding data import code. Finally, the generated code is post-processed, such as formatting and optimization, so that it can be used in the actual environment.
[0084] It is worth noting that the Transformer model has high complexity and computing resource requirements, so in practical applications, high-performance computing equipment may be needed to accelerate the model training and reasoning process. In addition, depending on the specific tasks and data characteristics, the model may need to be further adjusted and optimized to improve its performance and generalization ability.
[0085] Beneficial effects: The system based on the Transformer model for parsing Excel to generate code and implement data import provides an efficient and accurate method for processing Excel data to generate data import code by taking advantage of the Transformer model. This system has important application value for tasks such as automated data processing and data migration.
[0086] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for analyzing Excel data based on the Transformer model. It is characterized in that The data parsing method comprises: Receive Excel files uploaded by users; Preprocess the input Excel file to obtain the preprocessed Excel template; Use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file to obtain the analysis results; Generate codes accordingly based on the analysis results.
2. According to the Excel data parsing method based on the Transformer model described in claim 1, It is characterized in that The preprocessing of the input Excel file specifically includes: Perform format check on the input Excel file, give corresponding optimization tips, and output an optimized Excel template, which is the data structure and content of the preprocessed Excel file.
3. According to the Excel data analysis method based on the Transformer model described in claim 1, It is characterized in that The trained Transformer model is used to analyze the data structure and content of the preprocessed Excel file, and the analysis results obtained specifically include: Use the trained Transformer model to analyze the data structure and content of the preprocessed Excel file and extract key information; Based on the parsed Excel data structure and the import requirements set by the user, the corresponding import code is intelligently generated using the Transformer algorithm; During the data import process, the Transformer algorithm is used to monitor the import progress and status in real time.
4. According to the Excel data analysis method based on the Transformer model described in claim 3, It is characterized in that The key information specifically includes: data columns, data rows and data types.
5. According to the Excel data parsing method based on the Transformer model as described in claim 3, It is characterized in that The use of the Transformer algorithm to monitor the import progress and status in real time specifically includes: Automatically adjust the import strategy when encountering data errors or format inconsistencies; Transformer processing analysis has the ability of adaptive learning; Optimize your own processing and analysis capabilities based on historical data and imported experience.
6. The Excel data parsing method based on the Transformer model according to claim 1, It is characterized in that The generating code corresponding to the analysis result specifically includes: The user chooses to directly import the code or export the generated code into a compressed package to obtain the corresponding generated code.
7. The Excel data parsing method based on the Transformer model according to claim 5, It is characterized in that The Transformer processing analysis has the ability of adaptive learning, which specifically includes: Calculate the attention weight, calculate the attention weight between each position and other positions, and obtain the importance of each position to other positions; Calculate the weighted sum, multiply each position vector by the attention weight, and add them together to get the weighted sum vector; Linear transformation, linear transformation is performed on the weighted sum vector to obtain the final output vector.
8. An Excel data parsing system based on the Transformer model, It is characterized in that The data analysis system comprises: Excel upload module, used to receive Excel files uploaded by users; The preprocessing module is used to preprocess the input Excel file and obtain the Excel template after preprocessing; The Transformer processing and analysis module is used to analyze the data structure and content of the preprocessed Excel file using the trained Transformer model to obtain the analysis results; The code output module is used to generate codes according to the analysis results.