Resource classification system based on big data

Abstract

The invention discloses a resource classification system based on big data. An operation method of the system comprises the steps that server control management of the data resource classification system is carried out; performing data classification and storage management on the data resource classification system; performing classification retrieval optimization processing of the data resource classification system; performing login display processing on the data resource classification system; the server control management of the data resource classification system comprises the steps that data file processing of the data resource classification system is carried out through a Region server, so that the data resource classification system is initially provided with a Region server, and along with addition of data files in a table, when the size of the Region exceeds a set threshold value, the Region server is started; and controlling the Region to perform segmentation so as to complete processing of the data file. The step of carrying out data classification and storage management on the data resource classification system comprises the step of classifying data files in the data resource classification system according to user selection. The method has the characteristics of high management and control efficiency and intelligent optimization processing.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a resource classification system based on big data. Background Art

[0002] In today's big data era, various intelligent terminal devices, network media, sensors, e-commerce platforms, etc. are generating data all the time. A large amount of data is generated every day, and the amount of data is increasing exponentially. Big data has various types, not only containing a large amount of structured data, but also a large amount of unstructured data. The data types are diverse, including audio, video, pictures, emails, logs, website data, location information, etc. With the continuous development of big data and the continuous innovation of business, more types will appear, and the data types will become more diversified and complex. How to mine and manage massive amounts of data is a huge challenge facing enterprises today, and existing technologies lack corresponding optimization processing. Therefore, it is necessary to design a resource classification system based on big data with high management efficiency and intelligent optimization processing. Summary of the invention

[0003] The purpose of the present invention is to provide a resource classification system based on big data to solve the problems raised in the above background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a resource classification method based on big data, comprising: Carry out server control and management of data resource classification system; Carry out data classification and storage management for the data resource classification system; Carry out classification retrieval optimization processing of data resource classification system; Perform login and display processing of the data resource classification system; The server control management of the data resource classification system includes: The data files of the data resource classification system are processed through the Region server, so that the data resource classification system initially sets up a Region server, and as the data files in the table are added, when the size of the Region exceeds the set threshold, the Region is controlled to be split to complete the processing of the data files.

[0005] According to the above technical solution, the data classification and storage management of the data resource classification system includes: Classifying data files in the data resource classification system according to user selection, wherein the data file classification includes data file size, data file type and functional attribute classification; After the user selects the data classification, the data files are further weighted and divided into distributed storage and traditional storage. Then, they are divided into different data file directories according to the type. Each directory represents a corresponding data file of a type.

[0006] According to the above technical solution, the classification retrieval optimization processing of the data resource classification system includes: The database of the data resource classification system is designed so that column storage is performed in columns and multiple column cluster designs are supported.

[0007] According to the above technical solution, the classification retrieval optimization processing of the data resource classification system further includes: Furthermore, the metadata index table is built on a column-oriented distributed database. Two column clusters, dic and file, are added to the metadata table. The full path of the directory is used as the row key to record the directory name and file name contained in each directory. When batch searching for files in the corresponding directory, the full path of the directory is first searched to retrieve the directory set and file set contained in it, and then all subdirectories are traversed, and the subdirectory path is added to the root path, and spliced ​​into a full path for secondary subdirectory search. This is recursive to each contained subdirectory until all subdirectories are located and recursively returned; The file content retrieval of the data resource classification system is optimized by first dividing the data and storing it in the cluster, then distributing the calculation to the nodes that need calculation, and finally returning the calculation results.

[0008] According to the above technical solution, the login display processing of the data resource classification system includes: Before the file is stored and processed, users need to register and log in first. MySQL is used to manage user names and passwords, and MySQL's convenient and efficient user authentication is used to control user login permissions. The control resource list displays the user's stored files in a tree format, which is convenient for subsequent review, analysis and processing.

[0009] According to the above technical solution, a resource classification system based on big data includes: Management control module, used for management and control of data resource classification system; Classification retrieval module, used to optimize classification retrieval of data resource classification system; The login display module is used for user login and data display processing.

[0010] According to the above technical solution, the management control module includes: Server module, used for server management and control of data resource classification system; Data classification module, used for classification control and processing of data information; The storage control module is used to perform storage management and control on data information.

[0011] According to the above technical solution, the classification retrieval module includes: Database module, used for database design optimization; Metadata retrieval module, used for metadata retrieval and analysis; Content retrieval module, used for data content retrieval optimization control.

[0012] According to the above technical solution, the login display module includes: User login module, used for controlling and managing user login; The list display module is used to display data information in a list.

[0013] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention, by providing a management control module, a classification retrieval module and a login display module, can provide users with multiple classification options, effectively meet user needs, and enable the database to be stored in columns. When reading a certain column alone, it is not necessary to extract the entire row of data, only the required column needs to be extracted. At the same time, the data types of the same column are the same, and the data is memory-aligned when reading, so the retrieval efficiency is higher, and the data access pressure can be balanced, solving the scalability requirements of the metadata table. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a resource classification method based on big data provided in Embodiment 1 of the present invention; Figure 2 This is a module structure diagram of a resource classification system based on big data provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1: Figure 1This is a flowchart of a resource classification method based on big data provided by the first embodiment of the present invention. This embodiment can be applied to a data resource classification system. The method can be executed by a resource classification system based on big data provided by the embodiment of the present invention. The system is composed of multiple software and hardware modules, such as Figure 1 As shown, the method specifically comprises the following steps: S101, performing server control management of the data resource classification system; Exemplarily, in an embodiment of the present invention, data files of a data resource classification system are processed through a Region server, so that the data resource classification system initially sets up a Region server, and as data files in the table are added, when the size of the Region exceeds a set threshold, the Region is controlled to be split to complete the processing of the data file; in this step, as the amount of data file processing increases, there will be multiple Regions after a period of time, and these Regions are controlled to be distributed and stored on different nodes in the cluster, so that the data in a Region will only be stored on one machine, not across nodes, and each Region server only maintains one HLog. Through this step, the same data file can be continuously appended, which can reduce the number of disk addressing times compared to multiple data files, thereby speeding up the write performance of the data resource classification system.

[0017] S102, classifying and storing data in the data resource classification system; Exemplarily, in an embodiment of the present invention, data files in a data resource classification system are classified according to user selection, wherein data file classification includes data file size, data file type and functional attribute classification; in this step, classification by data size is to divide data files uniformly into files of set same size, and establish metadata information for each file. When the cluster is started, the metadata information of all nodes in the cluster needs to be loaded into the memory of the cluster management node, wherein the file division cannot be less than the set lower limit value. Too small blocks will result in more disk addressing, affecting the basic efficiency requirements of big data access, and the presence of a large number of small files will overload the memory of the management node; data file type classification is to divide data files into videos, pictures, documents, music, software, etc.; functional attribute classification is to classify according to laboratory classification, office file classification and privacy file classification, etc. Through this step, multiple classification options are provided to users to effectively meet user needs.

[0018] After the user selects the data classification, the data files are further weighted and divided into distributed storage and traditional storage, and then divided into different data file directories according to the type. Each directory represents a corresponding data file of a type. In this step, the data files are weighted according to the set rules and divided into five levels: 5, 4, 3, 2, and 1. The default lowest level is 1 and it is directly transmitted and stored. The remaining high-level data requires higher reliability and efficient read and write rates, so distributed storage is used to ensure security and reliability.

[0019] S103, performing classification retrieval optimization processing on the data resource classification system; Exemplarily, in an embodiment of the present invention, the database of the data resource classification system is designed so that column storage is performed in units of columns and supports multiple column cluster designs; since the row storage method must write an entire row in full at a time, the width of the row is subject to certain limitations, and the table structure cannot be dynamically modified after insertion, and the data must be written in full at one time. Therefore, through this step, the database is stored in units of columns. When reading a column individually, there is no need to extract the data of the entire row, only the required column needs to be extracted. At the same time, the data type of the same column is the same, and the data is memory-aligned when reading, which makes the retrieval efficiency higher.

[0020] Furthermore, the metadata index table is established on a column-oriented distributed database. Two column clusters, dic and file, are added to the metadata table. The full path of the directory is used as the row key, and the directory name and file name contained in each directory are recorded respectively. When batch searching for files under the corresponding directory, the full path of the directory is first retrieved to retrieve the directory set and file set contained in it, and then all subdirectories are traversed, and the subdirectory path is added to the root path, and spliced ​​into a full path for secondary subdirectory retrieval. This is recursive to each contained subdirectory until all subdirectories are located and recursively returned; in this step, as the metadata files increase, the record columns of the column clusters dic and file are added. When the amount of recorded data increases to the set value, the row key is split into two parts and stored on different servers in the cluster, which can balance the data access pressure and solve the scalability requirements of the metadata table.

[0021] The file content retrieval of the data resource classification system is optimized. First, the data is divided and stored in the cluster, and then the calculation is distributed to the nodes that need to be calculated, and finally the calculation results are returned; because the field of big data retrieval is not only the retrieval of metadata information, sometimes the file content needs to be retrieved. For example, in the field of video surveillance, searching for a certain frame in the video, searching for the wrong line in the website log record, searching for the divorce record in the massive transaction records, etc., all require analysis and retrieval of the specific content of the file. Therefore, through this step, the retrieval efficiency of the data file content is effectively improved.

[0022] S104, performing login and display processing of the data resource classification system; Exemplarily, in an embodiment of the present invention, the user needs to register and log in before the file storage process, and MySQL is used to manage the user name and password, and the user login authority is controlled through MySQL's convenient and efficient user authentication; The control resource list displays the user's stored files in a tree format, which is convenient for subsequent review, analysis and processing.

[0023] Embodiment 2: Embodiment 2 of the present invention provides a resource classification system based on big data. Figure 2 A schematic diagram of the module structure of a resource classification system based on big data provided in the second embodiment, such as Figure 2 As shown, the system includes: Management control module, used for management and control of data resource classification system; Classification retrieval module, used to optimize classification retrieval of data resource classification system; The login display module is used for user login and data display processing.

[0024] In some embodiments of the present invention, the management control module includes: Server module, used for server management and control of data resource classification system; Data classification module, used for classification control and processing of data information; The storage control module is used to perform storage management and control on data information.

[0025] In some embodiments of the present invention, the classification retrieval module includes: Database module, used for database design optimization; Metadata retrieval module, used for metadata retrieval and analysis; Content retrieval module, used for data content retrieval optimization control.

[0026] In some embodiments of the present invention, the login display module includes: User login module, used for controlling and managing user login; The list display module is used to display data information in a list.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0028] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A resource classification method based on big data, characterized by: include: Carry out server control and management of data resource classification system; Carry out data classification and storage management for the data resource classification system; Carry out classification retrieval optimization processing of data resource classification system; Perform login and display processing of the data resource classification system; The server control management of the data resource classification system includes: The data files of the data resource classification system are processed through the Region server, so that the data resource classification system initially sets up a Region server, and as the data files in the table are added, when the size of the Region exceeds the set threshold, the Region is controlled to be split to complete the processing of the data files.

2. A resource classification method based on big data according to claim 1, characterized in that: The data classification and storage management of the data resource classification system includes: Classifying data files in the data resource classification system according to user selection, wherein the data file classification includes data file size, data file type and functional attribute classification; After the user selects the data classification, the data files are further weighted and divided into distributed storage and traditional storage. Then, they are divided into different data file directories according to the type. Each directory represents a corresponding data file of a type.

3. The resource classification method based on big data according to claim 1 is characterized in that: The classification retrieval optimization process of the data resource classification system includes: The database of the data resource classification system is designed so that column storage is performed in columns and multiple column cluster designs are supported.

4. The resource classification method based on big data according to claim 1 is characterized in that: The classification retrieval optimization process of the data resource classification system further includes: Furthermore, the metadata index table is built on a column-oriented distributed database. Two column clusters, dic and file, are added to the metadata table. The full path of the directory is used as the row key to record the directory name and file name contained in each directory. When batch searching for files in the corresponding directory, the full path of the directory is first searched to retrieve the directory set and file set contained in it, and then all subdirectories are traversed, and the subdirectory path is added to the root path, and spliced ​​into a full path for secondary subdirectory search. This is recursive to each contained subdirectory until all subdirectories are located and recursively returned; The file content retrieval of the data resource classification system is optimized by first dividing the data and storing it in the cluster, then distributing the calculation to the nodes that need calculation, and finally returning the calculation results.

5. The resource classification method based on big data according to claim 1 is characterized in that: The login display processing of the data resource classification system includes: Before the file is stored and processed, users need to register and log in first. MySQL is used to manage user names and passwords, and MySQL's convenient and efficient user authentication is used to control user login permissions. The control resource list displays the user's stored files in a tree format, which is convenient for subsequent review, analysis and processing.

6. A resource classification system based on big data, characterized by: include: Management control module, used for management and control of data resource classification system; Classification retrieval module, used to optimize classification retrieval of data resource classification system; The login display module is used for user login and data display processing.

7. A resource classification system based on big data according to claim 6, characterized in that: The management control module includes: Server module, used for server management and control of data resource classification system; Data classification module, used for classification control and processing of data information; The storage control module is used to perform storage management and control on data information.

8. The resource classification system based on big data according to claim 6, characterized in that: The classification retrieval module includes: Database module, used for database design optimization; Metadata retrieval module, used for metadata retrieval and analysis; Content retrieval module, used for data content retrieval optimization control.

9. The resource classification system based on big data according to claim 6, characterized in that: The login display module includes: User login module, used for controlling and managing user login; The list display module is used to display data information in a list.

Images