Storage method, system and equipment for big data and medium

Abstract

The invention discloses a storage method, system and device for big data and a medium, and belongs to the technical field of big data management. The storage method comprises the steps that point burying is conducted on applications or products used by a user every day, behavior data generated by the applications or products used by the user every day are collected through the point burying, log data are generated according to the behavior data, and the log data are cached; obtaining the log data in a cache, and storing the log data into a corresponding data warehouse tool hive table; and reading log data of the hive table by calculating an engine spark file, merging the log data into total data of the current partition, and storing the total data into a distributed storage HDFS (Hadoop Distributed File System). According to the application of the invention, the efficiency of storing and reading the log data is improved, and the utilization rate of the HDFS system is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data management technology, and more specifically, to a storage method, system, device and medium for big data. Background Art

[0002] With the development of science and technology and the popularization of the Internet, the amount of data in the world has reached a certain level, and traditional relational databases are simply unable to handle such a large amount of data. Big data has also emerged. In short, big data refers to very large and complex data sets, especially data sets from new data sources, whose scale is so large that traditional data processing software is helpless. Summary of the invention

[0003] In view of the above problems, the present invention proposes a storage method for big data, comprising:

[0004] Tracking the applications or products used by users on a daily basis, collecting behavior data generated by the applications or products used by users on a daily basis, generating log data for the behavior data, and caching the log data;

[0005] Obtain the log data in the cache, and store the log data in a corresponding data warehouse tool hive table;

[0006] The log data of the hive table is read through the computing engine spark file, and the log data is merged into the full data of the current partition, and the full data is stored in the distributed storage HDFS system.

[0007] Optionally, log data is generated for the behavior data, including:

[0008] The behavior data is preliminarily screened to eliminate invalid behavior data in the behavior data, and log data is generated for the behavior data from which the invalid behavior data is eliminated.

[0009] Optionally, caching the log data includes:

[0010] Establish a service background interface for the application or product, and cache the log data to a relational database based on the service background interface, or send the log data to a Kafka platform based on the service background interface, and cache the log data based on the Kafka platform.

[0011] Optionally, after the log data is cached in a relational database, a data table is generated based on the relational database and according to the log data, and the data table is stored.

[0012] Optionally, obtaining the log data in the cache and storing the log data in a corresponding hive table includes:

[0013] Obtain log data based on the kafka platform and store the log data in the corresponding hive table, or obtain the data table corresponding to the log data based on the relational database and import the data table into the corresponding hive table.

[0014] Optional, spark files, including spark computing framework files.

[0015] Optionally, the log data is merged into the full data of the current partition, including:

[0016] Determining the file size of the log data;

[0017] If the log data is a small file, the log data of the small file is merged into the full data of the current partition to generate a file in a table structure;

[0018] If the log data is a large file, the log data of the large file is merged into the full data of the current partition, and the full data is redistributed to generate multiple files in the form of table structures.

[0019] Optionally, redistribute all data, including:

[0020] Split the entire data to redistribute the number of files and the format of the files.

[0021] Optionally, the method further includes:

[0022] After storing the full amount of data in the distributed storage HDFS system, the full amount of data is written into a final table according to the number of files, and the full amount of data stored in the distributed storage HDFS system is read by reading the records of the final table.

[0023] On the other hand, the present invention also provides a storage system for big data, comprising:

[0024] A data collection unit is used to track the applications or products used by users on a daily basis, collect behavior data generated by the applications or products used by users on a daily basis through the tracking points, generate log data for the behavior data, and cache the log data;

[0025] A data processing unit, used to obtain the log data in the cache and store the log data in a corresponding data warehouse tool hive table;

[0026] The data storage unit is used to read the log data of the hive table through the computing engine spark file, merge the log data into the full data of the current partition, and store the full data in the distributed storage HDFS system.

[0027] Optionally, log data is generated for the behavior data, including:

[0028] The behavior data is preliminarily screened to eliminate invalid behavior data in the behavior data, and log data is generated for the behavior data from which the invalid behavior data is eliminated.

[0029] Optionally, caching the log data includes:

[0030] Establish a service background interface for the application or product, and cache the log data to a relational database based on the service background interface, or send the log data to a Kafka platform based on the service background interface, and cache the log data based on the Kafka platform.

[0031] Optionally, after the log data is cached in a relational database, a data table is generated based on the relational database and according to the log data, and the data table is stored.

[0032] Optionally, obtaining the log data in the cache and storing the log data in a corresponding hive table includes:

[0033] Obtain log data based on the kafka platform and store the log data in the corresponding hive table, or obtain the data table corresponding to the log data based on the relational database and import the data table into the corresponding hive table.

[0034] Optional, spark files, including spark computing framework files.

[0035] Optionally, the log data is merged into the full data of the current partition, including:

[0036] Determining the file size of the log data;

[0037] If the log data is a small file, the log data of the small file is merged into the full data of the current partition to generate a file in a table structure;

[0038] If the log data is a large file, the log data of the large file is merged into the full data of the current partition, and the full data is redistributed to generate multiple files in the form of table structures.

[0039] Optionally, redistribute all data, including:

[0040] Split the entire data to redistribute the number of files and the format of the files.

[0041] Optionally, the data storage unit is further used for:

[0042] After storing the full amount of data in the distributed storage HDFS system, the full amount of data is written into a final table according to the number of files, and the full amount of data stored in the distributed storage HDFS system is read by reading the records of the final table.

[0043] In yet another aspect, the present invention further provides a computing device, comprising: one or more processors;

[0044] a processor for executing one or more programs;

[0045] When the one or more programs are executed by the one or more processors, the above-described method is implemented.

[0046] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, the method described above is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention provides a storage method for big data, including: burying points for applications or products used by users daily, collecting behavioral data generated by applications or products used by users daily through the burying points, generating log data for the behavioral data, and caching the log data; obtaining the log data in the cache, and storing the log data in a corresponding data warehouse tool hive table; reading the log data of the hive table through a computing engine spark file, and merging the log data into the full amount of data of the current partition, and storing the full amount of data in a distributed storage HDFS system. The application of the present invention improves the efficiency of storing and reading log data, and improves the utilization rate of the HDFS system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 It is the principle diagram of the method of the present invention;

[0051] Figure 3 It is a structural diagram of the system of the present invention. DETAILED DESCRIPTION

[0052] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.

[0053] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0054] Embodiment 1:

[0055] The present invention proposes a storage method for big data, such as Figure 1 As shown, including:

[0056] Step 1: Tracking the applications or products used by the user on a daily basis, collecting the behavior data generated by the applications or products used by the user on a daily basis, generating log data for the behavior data, and caching the log data;

[0057] Step 2: Obtain the log data in the cache, and store the log data in a corresponding data warehouse tool hive table;

[0058] Step 3: Read the log data of the hive table through the computing engine spark file, merge the log data into the full data of the current partition, and store the full data in the distributed storage HDFS system.

[0059] Wherein, log data is generated for the behavior data, including:

[0060] The behavior data is preliminarily screened to eliminate invalid behavior data in the behavior data, and log data is generated for the behavior data from which the invalid behavior data is eliminated.

[0061] The step of caching the log data includes:

[0062] Establish a service background interface for the application or product, and cache the log data to a relational database based on the service background interface, or send the log data to a Kafka platform based on the service background interface, and cache the log data based on the Kafka platform.

[0063] After the log data is cached in the relational database, a data table is generated based on the relational database and according to the log data, and the data table is stored.

[0064] The process of obtaining the log data in the cache and storing the log data in a corresponding hive table includes:

[0065] Obtain log data based on the kafka platform and store the log data in the corresponding hive table, or obtain the data table corresponding to the log data based on the relational database and import the data table into the corresponding hive table.

[0066] The spark files include the spark computing framework files.

[0067] The log data is merged into the full data of the current partition, including:

[0068] Determining the file size of the log data;

[0069] If the log data is a small file, the log data of the small file is merged into the full data of the current partition to generate a file in a table structure;

[0070] If the log data is a large file, the log data of the large file is merged into the full data of the current partition, and the full data is redistributed to generate multiple files in the form of table structures.

[0071] Among them, the redistribution of all data includes:

[0072] Split the entire data to redistribute the number of files and the format of the files.

[0073] The method further includes:

[0074] After storing the full amount of data in the distributed storage HDFS system, the full amount of data is written into a final table according to the number of files, and the full amount of data stored in the distributed storage HDFS system is read by reading the records of the final table.

[0075] The present invention is described below in conjunction with specific cases:

[0076] First, the platforms or technologies used in the implementation of the case are described as follows:

[0077] Hadoop is an open source distributed computing platform under the Apache Foundation. It is developed based on the Java language, has good cross-platform features, and can be deployed in cheap computer clusters. Users can develop distributed programs without having to understand the underlying details of the distribution, and make full use of the power of the cluster for high-speed computing and storage.

[0078] HDFS is the abbreviation of Hadoop Distributed File System. It has high read and write speed, good fault tolerance and scalability, and provides distributed storage for massive data. Its redundant data storage method ensures data security.

[0079] Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files into a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. The advantage of Hive is that it has a low learning cost and can achieve fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop special MapReduce applications. Hive is very suitable for statistical analysis of data warehouses.

[0080] Apache Spark is a fast and general computing engine designed for large-scale data processing. Spark is a general parallel framework similar to Hadoop MapReduce, which was developed by UC Berkeley AMP lab (AMP Lab of the University of California, Berkeley). Spark has the advantages of Hadoop MapReduce, but it is different from MapReduce in that the intermediate output results of the job can be stored in memory, so there is no need to read and write HDFS. Therefore, Spark is better suited for MapReduce algorithms that require iteration, such as data mining and machine learning.

[0081] The implementation based on the above technology or platform mainly includes: generating logs by collecting daily user behaviors. Transmitting them to the big data cluster through relevant tools. Forming one or more files in the form of table structure. Finally saving them to HDFS, redistributing the format and quantity of files through daily file processing, and finally merging them into one or more files.

[0082] The implementation of the above technology specifically includes:

[0083] 1. Users use the app or related products daily.

[0084] 2. The product generates corresponding behavior log data through embedding points.

[0085] 3. The service backend stores the relevant log data in a relational database or sends it to Kafka through the interface.

[0086] 4. Big data obtains data through Kafka and stores it in related hive tables, or imports tables from relational databases into hive through Sqoop.

[0087] 5. Big data reads data through the spark computing framework and merges the data of the day into the full data of the current partition. During the merging process, the number of files is redistributed according to the data size and stored in the final table.

[0088] 6. Subsequent calculation and statistical program reads the above result table for subsequent demand calculation.

[0089] In the present invention, the latest log file is read through the spark file and merged with all the data of the current partition.

[0090] In the present invention, the number of files finally stored in HDFS is reallocated according to the latest log file size and the total data size of the current partition, and then written to HDFS.

[0091] Among them, HDFS is a distributed file system. Each file will have a corresponding copy. Each time a file is stored, the address of the current file will be stored in the memory, which is convenient for accurately finding the file location when reading the file later. If there are few log files every day, a very small file will be generated every day, but the memory address of each file will also be stored in the memory. In this way, as the file increases, the data in the memory will also increase. If the data is too large and not split, it will be very slow to read the data later. The present invention will re-split the number of files according to the size of the merged log files, merge small files, and split large files. Thereby improving the efficiency of data reading and writing in HDFS, as well as the utilization rate of HDFS.

[0092] Embodiment 2:

[0093] On the other hand, the present invention also provides a storage system 200 for big data, such as Figure 3 As shown, including:

[0094] The data collection unit 201 is used to track the applications or products used by the user every day, collect the behavior data generated by the applications or products used by the user every day, generate log data for the behavior data, and cache the log data;

[0095] The data processing unit 202 is used to obtain the log data in the cache and store the log data in a corresponding data warehouse tool hive table;

[0096] The data storage unit 203 is used to read the log data of the hive table through the computing engine spark file, merge the log data into the full data of the current partition, and store the full data in the distributed storage HDFS system.

[0097] Wherein, log data is generated for the behavior data, including:

[0098] The behavior data is preliminarily screened to eliminate invalid behavior data in the behavior data, and log data is generated for the behavior data from which the invalid behavior data is eliminated.

[0099] The step of caching the log data includes:

[0100] Establish a service background interface for the application or product, and cache the log data to a relational database based on the service background interface, or send the log data to a Kafka platform based on the service background interface, and cache the log data based on the Kafka platform.

[0101] After the log data is cached in the relational database, a data table is generated based on the relational database and according to the log data, and the data table is stored.

[0102] The process of obtaining the log data in the cache and storing the log data in a corresponding hive table includes:

[0103] Obtain log data based on the kafka platform and store the log data in the corresponding hive table, or obtain the data table corresponding to the log data based on the relational database and import the data table into the corresponding hive table.

[0104] The spark files include the spark computing framework files.

[0105] The log data is merged into the full data of the current partition, including:

[0106] Determining the file size of the log data;

[0107] If the log data is a small file, the log data of the small file is merged into the full data of the current partition to generate a file in a table structure;

[0108] If the log data is a large file, the log data of the large file is merged into the full data of the current partition, and the full data is redistributed to generate multiple files in the form of table structures.

[0109] Among them, the redistribution of all data includes:

[0110] Split the entire data to redistribute the number of files and the format of the files.

[0111] The data storage unit 203 is also used for:

[0112] After storing the full amount of data in the distributed storage HDFS system, the full amount of data is written into a final table according to the number of files, and the full amount of data stored in the distributed storage HDFS system is read by reading the records of the final table.

[0113] The application of the present invention improves the efficiency of storing and reading log data and improves the utilization rate of the HDFS system.

[0114] Embodiment 3:

[0115] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of the method in the above embodiment.

[0116] Embodiment 4:

[0117] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both a built-in storage medium in a computer device and an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiment.

[0118] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0123] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A storage method for big data, characterized in that: include: Tracking the applications or products used by users on a daily basis, collecting behavior data generated by the applications or products used by users on a daily basis, generating log data for the behavior data, and caching the log data; Obtain the log data in the cache, and store the log data in a corresponding data warehouse tool hive table; The log data of the hive table is read through the computing engine spark file, and the log data is merged into the full data of the current partition, and the full data is stored in the distributed storage HDFS system.

2. The storage method according to claim 1, characterized in that: The generating log data according to the behavior data includes: The behavior data is preliminarily screened to eliminate invalid behavior data in the behavior data, and log data is generated for the behavior data from which the invalid behavior data is eliminated.

3. The storage method according to claim 1, characterized in that: The caching of the log data comprises: Establish a service background interface for the application or product, and cache the log data to a relational database based on the service background interface, or send the log data to a Kafka platform based on the service background interface, and cache the log data based on the Kafka platform.

4. The storage method according to claim 3, characterized in that: After the log data is cached in the relational database, a data table is generated based on the relational database and according to the log data, and the data table is stored.

5. The storage method according to claim 1, characterized in that: The obtaining the log data in the cache and storing the log data in a corresponding hive table includes: Obtain log data based on the Kafka platform and store the log data in the corresponding hive table, or obtain the data table corresponding to the log data based on the relational database and import the data table into the corresponding hive table.

6. The storage method according to claim 1, characterized in that: The spark file includes: a spark computing framework file.

7. The storage method according to claim 1, characterized in that: The merging of the log data into the full data of the current partition includes: Determining the file size of the log data; If the log data is a small file, the log data of the small file is merged into the full data of the current partition to generate a file in a table structure; If the log data is a large file, the log data of the large file is merged into the full data of the current partition, and the full data is redistributed to generate multiple files in the form of table structures.

8. A storage system for big data, characterized in that: include: A data collection unit is used to track the applications or products used by users on a daily basis, collect behavior data generated by the applications or products used by users on a daily basis through the tracking points, generate log data for the behavior data, and cache the log data; A data processing unit, used to obtain the log data in the cache and store the log data in a corresponding data warehouse tool hive table; The data storage unit is used to read the log data of the hive table through the computing engine spark file, merge the log data into the full data of the current partition, and store the full data in the distributed storage HDFS system.

9. A computer device, characterized in that: include: one or more processors; a processor for executing one or more programs; When the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images