Unstructured file analysis and visual display method, system, equipment and medium

Abstract

The invention discloses an unstructured file analysis and visual display method, system and device and a medium, belongs to the technical field of data processing and visual display, and aims to solve the technical problem of how to improve the processing efficiency of unstructured files, enhance the integration and analysis capability of data and improve the data processing efficiency. According to the technical scheme, the method comprises the steps of importing unstructured files into an integrated analyzer, extracting metadata of various types of unstructured files through the integrated analyzer, converting the metadata into structured data, storing the structured data into any target data source, querying by utilizing an SQL (Structured Query Language) or an API (Application Program Interface) interface, and finally obtaining the data-driven decision-making response speed and quality. A data set is generated through various script conversion tools and is called in a large screen and report design, and visualized analysis and display of unstructured file data are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and visual display, and in particular to a method, system, device and medium for analyzing and visualizing an unstructured file. Background Art

[0002] For metadata extraction and processing of different types of unstructured files, specific preprocessing algorithms or processes need to be designed. Especially in the process of data cleaning and conversion, data processing is usually complex and difficult to manage, and the data is not easy to integrate.

[0003] In different application scenarios of unstructured file data, the same metadata often requires additional preprocessing according to needs before it can be used, and the duplication and redundancy of various preprocessing schemes greatly increases the workload. In addition, inconsistent processing methods and the diversity of tools may lead to inconsistency in data processing results, which not only affects the accuracy of data analysis, but also reduces the efficiency of data analysis, and cannot meet the rapid and flexible response to the dynamic changes in user needs.

[0004] Therefore, how to improve the processing efficiency of unstructured files, enhance data integration and analysis capabilities, and ultimately improve the response speed and quality of data-driven decision-making is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical task of the present invention is to provide a method, system, device and medium for parsing and visualizing unstructured files to solve the problem of how to improve the processing efficiency of unstructured files, enhance the integration and analysis capabilities of data, and ultimately improve the response speed and quality of data-driven decision-making.

[0006] The technical task of the present invention is achieved in the following way: a method for parsing and visualizing unstructured files, which imports unstructured files into an integrated parser, extracts metadata of various types of unstructured files through the integrated parser, converts them into structured data and stores them in any target data source, then uses SQL query or API interface query, generates data sets through multiple script conversion tools, and calls them in large screens and report designs to achieve visual analysis and display of unstructured file data.

[0007] Preferably, the unstructured files are imported into the integrated parser, and the metadata of various types of unstructured files are extracted by the integrated parser, and converted into structured data and stored in any target data source as follows:

[0008] The integrated parser integrates ANTLR, OCR and other parsing tools, selects parsing tools according to the file model, and completes metadata extraction;

[0009] Preprocess metadata and select valid metadata information according to requirements;

[0010] Build basic information for structured storage by setting table names, table fields, and field types;

[0011] Select the target data source for metadata to complete the storage of unstructured file data into structured data;

[0012] Select Update Data, re-import the new unstructured file, and select metadata records based on the table field information to update the target table data content;

[0013] During field management, modify and delete table field data types;

[0014] View the metadata records extracted from unstructured files through data preview.

[0015] Preferably, the parsing tool is selected according to the file model as follows:

[0016] For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails;

[0017] For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

[0018] As a preference, the data set is generated as follows:

[0019] Select the data source where the unstructured file data is located;

[0020] Write SQL query statements, add dynamic parameters to the query statements, and provide a preset function for dynamic parameter queries to facilitate data preview;

[0021] View the acquired data through data preview;

[0022] When the data does not meet the requirements, the data is filtered through data conversion. Among them, data conversion provides three methods: Java script, JS script and dictionary item translation. Each method can be used multiple times to achieve serial filtering. After the filtering method is configured, the data is previewed to view the filtering effect.

[0023] As a preferred method, it is called in the large screen and report design to realize the visual analysis and display of unstructured file data as follows:

[0024] When configuring chart data for large-screen design, directly select the data set, match the fields of the data set with the chart to complete the data configuration, and perform data analysis and display;

[0025] When designing a report, select the fields of the data set and drag them to the relevant positions in the report to display the data in the actual data table.

[0026] A non-structured file parsing and visual display system, the system comprising:

[0027] The unstructured file processing module is used to convert unstructured file data into structured data and store it in the data source, and extract valid metadata through the integrated parser and store it in the data source to achieve unified management of structured data; the integrated parser is applicable to unstructured files of various data structures;

[0028] The metadata storage module is used to access different data sources and connect to the required database by configuring basic configuration information;

[0029] The data set module is used to obtain data and reprocess the data. After obtaining the data, the data is further processed and converted according to the needs to meet the data requirements of large-screen design and report analysis;

[0030] The scenario application module allows users to efficiently use and display unstructured file data by directly calling data set information in large-screen analysis and report design.

[0031] Preferably, the unstructured file processing module selects a parsing tool according to the file model, as follows:

[0032] For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails;

[0033] For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

[0034] Preferably, the data set module completes the conversion and integration of unstructured file metadata through SQL queries, API interface calls and data conversion tools.

[0035] An electronic device comprising: a memory and at least one processor;

[0036] Wherein, the memory stores a computer program;

[0037] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the unstructured file parsing and visual display method as described above.

[0038] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the unstructured file parsing and visualization method as described above.

[0039] The unstructured file parsing and visual display method, system, device and medium of the present invention have the following advantages:

[0040] (1) The present invention parses and processes various types of unstructured files through an integrated parser, realizing seamless integration and unified processing of structured and unstructured data by data sets, so that users do not need to pay attention to the complex form of data when performing data analysis applications, and can directly realize efficient use and analysis of data through data sets at the application layer, significantly improving the intelligent level of data analysis;

[0041] (ii) The present invention extracts metadata of various types of unstructured files through an integrated parser and stores them in any target data source to achieve metadata storage. At the same time, by constructing a data set, it provides users with a unified and easy-to-operate data management and call platform, thereby providing faster and more flexible tools and methods for advanced data analysis applications such as large-screen display and report analysis, aiming to significantly improve the processing efficiency of unstructured files, enhance data integration and analysis capabilities, and ultimately improve the response speed and quality of data-driven decision-making;

[0042] (III) The metadata storage data source of the present invention provides a configuration environment for multiple data sources, facilitating the storage of various types of unstructured data;

[0043] (IV) The present invention modifies the information in the form according to the connection information of the database, enters the user password, and then clicks the test. If the information is correct, it will prompt that the connection is successful; and supports 11 commonly used data source types such as MySQL, ClickHouse, DAMO, Shentong, Elasticsearch, etc.;

[0044] (V) The present invention can extract metadata from various unstructured files of different data structures through an integrated parser and convert them into structured data for unified storage;

[0045] (VI) In scenarios such as large-screen analysis and report design, the present invention can flexibly configure data sets according to user needs to achieve efficient use and analysis of unstructured file data;

[0046] (VII) The present invention utilizes an integrated parser to extract valid information from unstructured files, performs data filtering, converts the information into structured data and stores it in a data source. It then uses SQL query or API interface query to generate a final data set through a variety of script conversion tools. Finally, it is called in a large screen and report design to achieve visual analysis and display of unstructured file data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention is further described below in conjunction with the accompanying drawings.

[0048] Attached Figure 1 Schematic diagram of the process of unstructured file parsing and visualization display method. DETAILED DESCRIPTION

[0049] The unstructured document parsing and visual display method, system, device and medium of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0050] Embodiment 1:

[0051] As attached Figure 1 As shown, this embodiment provides a method for parsing and visualizing unstructured files. The method is to import unstructured files into an integrated parser, extract metadata of various types of unstructured files through the integrated parser, convert them into structured data and store them in any target data source, and then use SQL query or API interface query to generate a data set through a variety of script conversion tools, which is called in the large screen and report design to achieve visual analysis and display of unstructured file data.

[0052] In this embodiment, unstructured files are imported into the integrated parser, and metadata of various types of unstructured files are extracted by the integrated parser, converted into structured data and stored in any target data source as follows:

[0053] ① The integrated parser integrates ANTLR, OCR and other parsing tools, selects parsing tools according to the file model, and completes metadata extraction;

[0054] ②Preprocess the metadata and select valid metadata information according to requirements;

[0055] ③ Build basic information for structured storage by setting table names, table fields, and field types;

[0056] ④ Select the target data source of metadata to complete the storage of unstructured file data into structured data;

[0057] ⑤ Select Update Data, re-import the new unstructured file, and select metadata records based on the table field information to update the target table data content;

[0058] ⑥ During the field management process, modify and delete the data type of the table field;

[0059] ⑦ View the metadata records extracted from unstructured files through data preview.

[0060] In this embodiment, the parsing tool is selected according to the file model as follows:

[0061] For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails;

[0062] For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

[0063] In this embodiment, the data set is generated as follows:

[0064] (1) Select the data source where the unstructured file data is located;

[0065] (2) Write SQL query statements, add dynamic parameters to the query statements, and provide a preset function for dynamic parameter queries to facilitate data preview;

[0066] (3) View the acquired data through data preview;

[0067] (4) When the data does not meet the requirements, the data is filtered through data conversion; among them, data conversion provides three methods: Java script, JS script and dictionary item translation. Each method can be used multiple times to achieve serial filtering. After the filtering method is configured, the data is previewed to view the filtering effect.

[0068] In this embodiment, the visualization analysis and display of unstructured file data is realized by calling in the large screen and report design as follows:

[0069] (1) When configuring chart data for large-screen design, directly select the data set, match the fields of the data set with the chart to complete the data configuration, and perform data analysis and display;

[0070] (2) When designing a report, select the fields of the data set and drag them to the relevant locations in the report to display the data in the actual data table.

[0071] Embodiment 2:

[0072] This embodiment provides an unstructured file parsing and visual display system, which includes:

[0073] The unstructured file processing module is used to convert unstructured file data into structured data and store it in the data source, and extract valid metadata through the integrated parser and store it in the data source to achieve unified management of structured data; the integrated parser is applicable to unstructured files of various data structures;

[0074] The metadata storage module is used to access different data sources and connect to the required database by configuring basic configuration information;

[0075] The data set module is used to obtain data and reprocess the data. After obtaining the data, the data is further processed and converted according to the needs to meet the data requirements of large-screen design and report analysis;

[0076] The scenario application module allows users to efficiently use and display unstructured file data by directly calling data set information in large-screen analysis and report design.

[0077] The unstructured file processing module in this embodiment selects a parsing tool according to the file model, as follows:

[0078] For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails;

[0079] For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

[0080] The data set module in this embodiment completes the conversion and integration of unstructured file metadata through SQL query, API interface call and data conversion tools.

[0081] Embodiment 3:

[0082] This embodiment also provides an electronic device, including: a memory and a processor;

[0083] Wherein, the memory stores computer-executable instructions;

[0084] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the unstructured file parsing and visual display method in any embodiment of the present invention.

[0085] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0086] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0087] Embodiment 4:

[0088] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor, so that the processor executes the unstructured file parsing and visualization display method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0089] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0090] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0091] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0092] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for parsing and visualizing unstructured documents, characterized in that: The method is to import unstructured files into an integrated parser, extract metadata of various types of unstructured files through the integrated parser, convert them into structured data and store them in any target data source, then use SQL query or API interface query, generate data sets through various script conversion tools, call them in large screens and report designs, and realize visual analysis and display of unstructured file data.

2. The unstructured document parsing and visual display method according to claim 1, characterized in that: Import unstructured files into the integrated parser, extract metadata of various types of unstructured files through the integrated parser, convert them into structured data and store them in any target data source as follows: The integrated parser integrates ANTLR, OCR and other parsing tools, selects parsing tools according to the file model, and completes metadata extraction; Preprocess metadata and select valid metadata information according to requirements; Build basic information for structured storage by setting table names, table fields, and field types; Select the target data source for metadata to complete the storage of unstructured file data into structured data; Select Update Data, re-import the new unstructured file, and select metadata records based on the table field information to update the target table data content; During field management, modify and delete table field data types; View the metadata records extracted from unstructured files through data preview.

3. The unstructured document parsing and visual display method according to claim 2, characterized in that: The selection of parsing tools according to the file model is as follows: For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails; For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

4. The unstructured document parsing and visual display method according to claim 1, characterized in that: The generated data set is as follows: Select the data source where the unstructured file data is located; Write SQL query statements, add dynamic parameters to the query statements, and provide a preset function for dynamic parameter queries to facilitate data preview; View the acquired data through data preview; When the data does not meet the requirements, the data is filtered through data conversion. Among them, data conversion provides three methods: Java script, JS script and dictionary item translation. Each method can be used multiple times to achieve serial filtering. After the filtering method is configured, the data is previewed to view the filtering effect.

5. The unstructured document parsing and visual display method according to claim 1, characterized in that: Called in the large screen and report design, the visual analysis and display of unstructured file data is as follows: When configuring chart data for large-screen design, directly select the data set, match the fields of the data set with the chart to complete the data configuration, and perform data analysis and display; When designing a report, select the fields of the data set and drag them to the relevant positions in the report to display the data in the actual data table.

6. An unstructured document parsing and visualization system, characterized in that: The system includes: The unstructured file processing module is used to convert unstructured file data into structured data and store it in the data source, and extract valid metadata through the integrated parser and store it in the data source to achieve unified management of structured data; the integrated parser is applicable to unstructured files of various different data structures; The metadata storage module is used to access different data sources and connect to the required database by configuring basic configuration information; The data set module is used to obtain data and reprocess the data. After obtaining the data, the data is further processed and converted according to the needs to meet the data requirements of large-screen design and report analysis; The scenario application module allows users to efficiently use and display unstructured file data by directly calling data set information in large-screen analysis and report design.

7. The unstructured document parsing and visual display system according to claim 6, characterized in that: The unstructured file processing module selects a parsing tool according to the file model, as follows: For log files and emails, select the ANTLR parser to extract keyword information in the logs and the subject and sender information of the emails; For image files, select the OCR tool to extract basic information such as shooting time, location and size, as well as key information such as ticket number and license plate number.

8. The unstructured document parsing and visual display system according to claim 6 or 7, characterized in that: The data set module completes the conversion and integration of unstructured file metadata through SQL query, API interface call and data conversion tools.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the unstructured file parsing and visualization display method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the unstructured file parsing and visualization method according to any one of claims 1 to 5.

Images