Data aggregation tool system and method based on large model driving

Abstract

The invention relates to the technical field of data aggregation, in particular to a data aggregation tool system based on large model driving, which comprises a data acquisition module, a data distribution module, a sub-database, a database, a data arrangement module, an encryption module, a data storage module, a data processing module and a data visualization module, according to the method, firstly, multiple groups of data are divided into a mode of taking single data as a paragraph, and the data set is generated, so that clear management of the data set is realized; in the storage process of the sub-databases, data sets are screened to eliminate repetition; sorting and grouping data sets of the same category through data sorting; the encryption module is used for secondarily eliminating missing repeated data; repeated screening in the data cleaning process is simplified, the data can be repeatedly cleaned in the data arrangement process in a division mode, later processing simplification is facilitated, and the data aggregation rate can be increased.

Description

Technical Field

[0001] The present invention relates to the technical field of data aggregation, and in particular to a data aggregation tool system and method driven by a large model. Background Art

[0002] Data aggregation refers to the process of merging data from multiple data sources into a unified data set. In the era of big data, data aggregation methods are an indispensable part of data processing and analysis. In the application process of existing data aggregation tool systems driven by large models, the pre-processing of data requires data cleaning. During the cleaning process, the overlapping information in the data is a huge project. The system needs to find and remove the matching and duplicate data in the huge data set. This processing method has the problems of slow speed and low efficiency in matching and comparing in the huge data, which greatly affects the subsequent aggregation rate. To this end, a data aggregation tool system and method driven by large models are proposed to solve the above problems. Summary of the invention

[0003] The purpose of the present invention is to provide a data aggregation tool system and method based on large model drive, which solves the problem that the current data aggregation tool system has slow data cleaning and matching duplicate data, low efficiency, and affects the subsequent aggregation rate.

[0004] To achieve the above-mentioned object, the present invention provides the following technical solutions: a data aggregation tool system and method based on large model drive, including a data acquisition module, a data diversion module, a sub-database, a database, a data sorting module, an encryption module, a data storage module, a data processing module and a data visualization module;

[0005] The data acquisition module is used to collect raw data;

[0006] The data distribution module is used to distribute the original data and divide the data into multiple single data information;

[0007] The sub-database is used to store temporarily stored and screened data, and the database is used to store screened data;

[0008] The data sorting module is used to sort the single data into groups;

[0009] The encryption module is used to encrypt the sorted data set;

[0010] The data storage module is used to store data;

[0011] The data processing module is used to process data;

[0012] The data visualization module is used to generate a visualization chart from the data.

[0013] The present invention is achieved by the following steps:

[0014] S1: Obtain raw data through the data acquisition module;

[0015] S2: Divert the acquired raw data through the data diversion module;

[0016] First, separate the single data from the multiple data, and verify the multiple single data to ensure the integrity of the data;

[0017] The verified data is segmented one by one and a data set is generated; each piece of data in the data set is independently transmitted to the sub-database;

[0018] S3: The sub-database receives and stores the data transmitted by the data diversion module. In the process of receiving data, it obtains data information, retrieves the stored data to extract the front-end data information, and compares it with the front-end information of the currently received data, and uses the matching method to filter and eliminate duplicate data; finally, multiple single data are obtained and uploaded to the database for storage;

[0019] S4: The data sorting module has multiple built-in groups for storing sorted data separately;

[0020] The data sorting module extracts multiple single data stored in the database, and groups the data with consistent data front-end information into one group according to the data front-end information; sorts all the data into categories, and generates data sets from the data divided into each group;

[0021] S5: The encryption module performs secondary cleaning on the data collated by the data collation module, and again screens and removes duplicate data by matching; and encrypts the data set collated by the data collation module;

[0022] S6: encrypted data set, stored in the data storage module;

[0023] S7: The data processing module extracts the encrypted data set in the data storage module, decrypts it, and converts, aggregates, and mines each set of data to generate valuable information;

[0024] And analyze each group of data to generate a temporary data report;

[0025] S8: Finally, the data visualization module displays the data analysis results in the form of charts.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention realizes clear management of data sets by first dividing multiple groups of data into sections with single data as one section and generating data sets; by screening and eliminating duplicates from data sets during storage in a sub-database; by sorting and grouping data sets of the same category; by using an encryption module to eliminate missed duplicate data; by simplifying duplicate screening during data cleaning, by using a division method, duplicate cleaning of data can be achieved during the data sorting process, which is convenient for simplifying subsequent processing and is beneficial to accelerating the data aggregation rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the process of the present invention;

[0029] Figure 2 This is a framework diagram of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 , a data aggregation tool system based on large model drive, including a data acquisition module, a data diversion module, a sub-database, a database, a data sorting module, an encryption module, a data storage module, a data processing module and a data visualization module;

[0032] The data acquisition module is used to collect raw data;

[0033] The data distribution module is used to distribute the original data and divide the data into multiple single data information;

[0034] The sub-database is used to store temporarily stored and screened data, and the database is used to store screened data;

[0035] The data sorting module is used to sort the single data into groups;

[0036] The encryption module is used to encrypt the sorted data set;

[0037] The data storage module is used to store data;

[0038] The data processing module is used to process data;

[0039] The data visualization module is used to generate a visualization chart from the data;

[0040] In this embodiment, the data collection module can collect data according to real-time dynamics. The data collected in the early stage is used for subsequent data set division. The data collected in the later stage according to real-time dynamics can be directly summarized on the basis of the divided data set.

[0041] In this embodiment, the sub-database and the database share data, but the database and the sub-database do not share data. The sub-database mainly stores temporary data and can be deleted regularly. The database is used to store data information temporarily stored in the sub-database, which can be found at any time for later tracing.

[0042] In this embodiment, the data sorting module sorts the data, and the data information includes the front end, the terminal and the tail end. The consistency of the front end information of the data is first compared, the consistency of the middle end data is compared, and finally the tail end data is compared. If the front end data and the middle end data are consistent, it means that the data can be classified into a data set with consistent front and middle ends; if the front end data is consistent and the middle end data is inconsistent, it means that the data is classified as a data set with consistent front end data; if the front end and the middle end data are inconsistent and the back end data is consistent, it means that the data can be classified as a data set with consistent back end data;

[0043] In this embodiment, the encryption module encrypts the data set to avoid privacy leakage of the data set;

[0044] By first dividing multiple groups of data into sections with single data as one section and generating data sets, clear management of data sets can be achieved; by screening and eliminating duplicates during the storage process of the sub-database; by sorting data, data sets of the same category are sorted and grouped; the encryption module is used to secondary eliminate missed duplicate data; the repeated screening in the data cleaning process is simplified, and the repeated cleaning of data can be completed in the data sorting process by using the division method, which is convenient for the simplification of subsequent processing and helps to accelerate the data aggregation rate.

[0045] The present invention is achieved by the following steps:

[0046] S1: First, the display platform, search module, supply chain model and analysis model should be interconnected; the procurement data system, inventory data system, logistics data system, sales data system and after-sales data system in the supply chain model should operate independently;

[0047] S2: Make all data in the supply chain transparent through the procurement data system, inventory data system, logistics data system, sales data system and after-sales data system in the supply chain model;

[0048] Data information sharing is achieved through the analysis module. After the search module performs keyword / word search, the analysis module can retrieve data information from various systems in the supply chain model;

[0049] S3: Input keywords and key words through the search module;

[0050] For example, input: purchase product keyword / term year month order query; the analysis module obtains the keyword and can directly retrieve the product order data being queried, and display it through the display platform;

[0051] For example, input; logistics product keyword / word analysis; the analysis module will actively identify the keyword / word and retrieve the logistics data in the logistics data system to use the corresponding algorithm to analyze and predict, and display it through the display platform;

[0052] For example, if you input: inventory product keyword / word analysis, the analysis module will actively identify the keyword / word and retrieve the product data in the inventory data system, and use the corresponding algorithm to analyze and predict. The prediction results will be displayed on the display platform;

[0053] The product information retrieval and analysis process of other systems is carried out in the same way as the above-mentioned keyword / word search, so as to realize the retrieval of product information, the retrieval of logistics information, and the prediction of the corresponding directions of each system;

[0054] The logistics data system also provides the real-time location of the transport vehicles. By inputting: logistics product keywords / words to query, the analysis module identifies the keywords / words, and the analysis module calls the product logistics transportation data information in the logistics data system in the supply chain model, and finds the current location information of the transport vehicle on the road based on the information.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data aggregation tool system driven by a large model, characterized by: It includes data acquisition module, data diversion module, sub-database, database, data sorting module, encryption module, data storage module, data processing module and data visualization module; The data acquisition module is used to collect raw data; The data distribution module is used to distribute the original data and divide the data into multiple single data information; The sub-database is used to store temporarily stored and screened data, and the database is used to store screened data; The data sorting module is used to sort the single data into groups; The encryption module is used to encrypt the sorted data set; The data storage module is used to store data; The data processing module is used to process data; The data visualization module is used to generate a visualization chart from the data.

2. A method for using a data aggregation tool system driven by a large model, characterized in that: S1: Obtain raw data through the data acquisition module; S2: Divert the acquired raw data through the data diversion module; First, separate the single data from the multiple data, and verify the multiple single data to ensure the integrity of the data; Segment the verified data one by one and generate a data set; Each piece of data in the data set is independently transferred to the sub-database; S3: The sub-database receives and stores the data transmitted by the data diversion module. In the process of receiving data, it obtains data information, retrieves the stored data to extract the front-end data information, and compares it with the front-end information of the currently received data, and uses the matching method to filter and eliminate duplicate data; finally, multiple single data are obtained and uploaded to the database for storage; S4: The data sorting module has multiple built-in groups for storing sorted data separately; The data sorting module extracts multiple single data stored in the database, and groups the data with consistent data front-end information into one group according to the data front-end information; sorts all the data into categories, and generates data sets from the data divided into each group; S5: The encryption module performs secondary cleaning on the data collated by the data collation module, and again screens and removes duplicate data by matching; and encrypts the data set collated by the data collation module; S6: encrypted data set, stored in the data storage module; S7: The data processing module extracts the encrypted data set in the data storage module, decrypts it, and converts, aggregates, and mines each set of data to generate valuable information; And analyze each group of data to generate a temporary data report; S8: Finally, the data visualization module displays the data analysis results in the form of charts.

Images