Variable column Excel file analysis method and system

A file parsing and excel technology, applied in the field of data analysis, can solve problems such as high memory usage, slow parsing speed, OOM memory overflow, etc., to improve parallel processing efficiency and solve the effect of high memory usage

Active Publication Date: 2020-11-13
上海微亿智造科技有限公司
View PDF12 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

It uses traditional POI technology for analysis, loads the entire file at one time, takes up a lot of memory, and when the file is large, the analysis speed is slow, and OOM memory overflow is prone to occur

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Variable column Excel file analysis method and system
  • Variable column Excel file analysis method and system
  • Variable column Excel file analysis method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0046] Such as figure 1 As shown, the Excel file parsing method that supports variable columns includes the following steps:

[0047] Step 1: Verify the header of the Excel file to be analyzed and the abstracted basic template file, determine the type of product detection data in the Excel file, record the path of the Excel file to be analyzed and the type mapping relationship it belongs to, and send the mapping relationship To the message middleware MQ;

[0048] The basic template file includes a process-testing data basic template and a quality-testing data basic template, and the process-testing data basic template includes basic process testing information, and test results for the process, different products, production lines The test items that need to be performed will be different, so the number of test results is variable. This template is compatible with the input of test item information for different products and production lines. It is not necessary to issue a se...

Embodiment 2

[0063] Embodiment 2 can be regarded as a preferred example of Embodiment 1. The variable-column-supporting Excel file parsing system described in Embodiment 2 utilizes the steps of the variable-column-supporting Excel file parsing method described in Embodiment 1.

[0064] An Excel file parsing system that supports variable columns, including:

[0065] Mapping module: Verify the header of the Excel file to be analyzed and the basic template file, determine the type of product detection data in the Excel file, record the mapping relationship between the path of the Excel file to be analyzed and the type, and send the mapping relationship to In the message middleware MQ;

[0066] Loading module: the message middleware MQ consumer listens to the message, obtains the mapping relationship from the message, uses the type in it to load the pre-established parsing object java model according to the mapping relationship, and loads the Excel file corresponding to the file path ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a variable column Excel file analysis method and system, and the method comprises the steps: carrying out the header verification of a to-be-analyzed Excel file and a basic template file, determining the type of product detection data, recording the mapping relation between a file path and the type, and transmitting the mapping relation to message-oriented middleware MQ; after monitoring the message, enabling the message-oriented middleware MQ consumption terminal to acquire a mapping relationship, use the type of the message-oriented middleware MQ consumption terminal for loading a pre-established analysis object java model, and load an Excel file corresponding to the file path into a memory; reading each row of data of the Excel file in the memory, and storing thedata in the corresponding parsing object java model; and serializing the analysis object java model with the stored data into a JSON string, and sending the JSON string to Kafka for library falling. The variable column Excel file analysis method is adopted, the problem that factory product detection data enter a database is solved, and digitization of the factory product detection data is facilitated.

Description

technical field [0001] The invention relates to the technical field of data analysis, in particular to a method and system for analyzing Excel files supporting variable columns. Background technique [0002] Various types of product testing data (such as quality, process) in the factory are kept in the form of Excel files. The fixed part of the table header is the basic information of the product, and the variable part is the specific testing item information. The traditional processing method is to use POI to parse the Excel table. After loading the entire table file at one time with fixed rows and columns, the Java function package is used for parsing. The parsing speed is slow and consumes a lot of memory. [0003] Patent document CN109933769A discloses a method for quickly analyzing report data based on Excel. First, it defines the format standard that needs to be parsed in Excel workbooks; then quickly extracts the formats in different Sheets in the Excel workbook to ob...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F40/12
CPCG06F40/12
Inventor 林大王星宇师文庆旷黎明
Owner 上海微亿智造科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap