Data storage management method and system based on big data

Abstract

The invention provides a data storage management method and system based on big data, and relates to the technical field of data storage management. The data storage management system based on big data comprises a conversion module, the conversion module is bidirectionally connected with a construction module and an application module, and the construction module is connected with a data traversal module, a data processing module and a model construction module. The application module is connected with a data input module, a data arrangement module and a storage management module. The method comprises the following steps: traversing big data to obtain related contents in the aspect of existing data storage, establishing a related management model through a system, inputting data needing to be managed into the model, carrying out storage management on the data, selecting storage spaces with different functions to carry out storage calling on the data according to different data attributes, and carrying out regular backup. The stability and the safety of the data are ensured, the subsequent calling of the user is facilitated, the use difficulty of the system and the method is reduced, and the overall practicability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage management, and in particular to a data storage management method and system based on big data. Background Art

[0002] The existing patent (publication number: CN116846750A) discloses "a storage management system based on big data, the operation method of the system includes the following steps: performing resource configuration control processing on network nodes; performing synchronous storage and recommendation control analysis on network nodes; performing node clustering analysis processing and query control management; performing backup control processing on faulty nodes. The resource configuration control processing of network nodes includes: establishing a main policy node, initializing configuration processing on network nodes; acquiring network data through cache nodes, and accessing storage processing. The synchronous storage and recommendation control analysis of network nodes includes: performing synchronous caching processing on network cache nodes, controlling and realizing loose coupling between cache nodes; and further performing recommendation analysis processing on cache node query requests. The present invention has the characteristics of intelligent control management and high efficiency."

[0003] In the process of implementing this application, the inventor discovered that the existing technology has the following problems: the existing data management system does not have the function of classifying, processing and storing different data, and different data are stored in the same device, resulting in slow subsequent data search and call speed, low overall work efficiency and poor practicality. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a data storage management method and system based on big data, which solves the problems of low working efficiency and poor practicality of the existing data storage management method and system.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data storage management system based on big data, comprising a conversion module, the conversion module is bidirectionally connected with a construction module and an application module, the construction module is connected with a data traversal module, a data processing module and a model construction module, and the application module is connected with a data input module, a data sorting module and a storage management module; The conversion module includes a model calling module, and the model calling module is connected to a model adaptation module and a target matching module.

[0006] Preferably, the data traversal module includes an iterative traversal module and a parallel computing module, the iterative traversal module is connected to an iterator module and a generation module, and the parallel computing module is connected to a subset segmentation module and a subset traversal module.

[0007] Preferably, the data processing module includes a data cleaning module, and the data cleaning module is connected to a feature selection module, a feature construction module, a feature encoding module, a feature dimension reduction module and a feature scaling module.

[0008] Preferably, the model building module includes a data management module and a model management module, the data management module is connected to a data input and output module and a data storage module, and the model management module is connected to a training management module and an evaluation optimization module.

[0009] Preferably, the data input module includes an interactive interface module, and the interactive interface module is connected to a batch input module and a manual input module.

[0010] Preferably, the data sorting module includes a value assessment module and a data classification module, the value assessment module is connected to a frequency assessment module and a level assessment module, and the data classification module is connected to an attribute classification module and an association determination module.

[0011] Preferably, the storage management module is connected to a region division module and a feedback management module, the region division module is connected to a low frequency management module and a high frequency management module, and the feedback management module is connected to a periodic management module and a call management module.

[0012] A data storage management method based on big data, characterized in that it specifically includes the following steps: S1. Data acquisition The data content related to network data storage management is acquired through the data traversal module under the construction module, the relevant data is iterated through the iterator module under the iterative traversal module, and the required data is acquired through the generation module, the subset under the required data is traversed through the subset traversal module under the parallel computing module, and the acquired data is divided into individual subsets through the subset segmentation module, waiting for subsequent use; S2. Data Preprocessing Input the acquired data into the data processing module, clean the data through the connected data cleaning module, set the required data features through the feature selection module, establish a database of required feature data through the feature construction module, encode the database established by the data with features through the feature encoding module, further enhance the feature strength through the feature dimensionality reduction module and the feature scaling module, screen out data with more complex features, and remove the non-compliant parts from the database; S3. Model building Input the database into the model building module, introduce the database into the model through the data input and output module under the data management module, and temporarily store it through the data storage module. Through the training management module under the model management module, train the model with the data in the established database to enable it to have the function of data storage management. Analyze the advantages and disadvantages of the model through the evaluation and optimization module, and make further optimization and improvement; S4. Model Application Input the improved model into the conversion module, call the required model through the model calling module, adapt the model function to the management system through the model adaptation module, and determine the target data that the model needs to process through the target matching module; S5. Data Entry The user can input the data to be stored and managed through the data input module under the application module, enter the data through the interactive interface module, automatically enter the data to be entered in batches through the batch input module in combination with the form, and supplement the sporadic data through the manual input module; S6. Data storage The input data can be imported into the data sorting module, and the frequency assessment module and level assessment module under the value assessment module are used to assess the usage frequency and confidentiality level of the input data. After the assessment is completed, the data is classified through the data classification module, and the data of different values ​​are classified according to attributes and relevance through the attribute classification module and the association determination module, and classified according to different types under the premise of different value distinctions; S7. Data management Import the classified data into the model, and divide the storage area through the area division module according to the evaluation level and type of the data through the storage management module. The area with faster reading and writing speed is divided into the high-frequency management module for storing data with high usage frequency and similar data. The area with relatively slow reading and writing speed but good stability is divided into the low-frequency management module for storing data with low usage frequency and data that needs to be kept confidential and similar data. After the storage is completed, the stored data can be regularly backed up through the engagement management module under the feedback management module during daily use. When needed, the data can be called up through the call management module for use.

[0013] The present invention provides a data storage management method and system based on big data, which has the following beneficial effects: The present invention provides a data storage management method and system based on big data. Compared with the existing data storage management method and system, when the data storage management method and system are used, they traverse big data to obtain relevant content of existing data storage, establish relevant management models through the system, input data to be managed into the model, store and manage the data, select storage spaces with different functions to store and call the data according to different data attributes, and back up the data regularly to ensure the stability and security of the data, facilitate subsequent calls by users, reduce the difficulty of using the system and method, and improve its overall practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2 is a schematic diagram of the process of the conversion module of the present invention; Figure 3 It is a flow chart of the data traversal module of the present invention; Figure 4 It is a flow chart of the data processing module of the present invention; Figure 5 A schematic diagram of the flow chart of the model construction module of the present invention; Figure 6 It is a flow chart of the data input module of the present invention; Figure 7 It is a flow chart of the data sorting module of the present invention; Figure 8 It is a flowchart of the storage management module of the present invention.

[0015] Among them, 1. Construction module; 2. Conversion module; 3. Application module; 4. Data traversal module; 5. Data processing module; 6. Model construction module; 7. Data input module; 8. Data sorting module; 9. Storage management module; 201. Model call module; 202. Model adaptation module; 203. Target matching module; 401. Iteration traversal module; 402. Iterator module; 403. Generation module; 404. Parallel computing module; 405. Subset segmentation module; 406. Subset traversal module; 501. Data cleaning module; 502. Feature selection module; 503. Feature construction module; 504. Feature encoding module; 505. Feature dimension reduction module; 506. Feature Scaling module; 601, data management module; 602, data input and output module; 603, data storage module; 604, model management module; 605, training management module; 606, evaluation and optimization module; 701, interactive interface module; 702, batch input module; 703, manual input module; 801, value assessment module; 802, frequency assessment module; 803, level assessment module; 804, data classification module; 805, attribute classification module; 806, association determination module; 901, area division module; 902, low frequency management module; 903, high frequency management module; 904, feedback management module; 905, periodic management module; 906, call management module. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] like Figure 1-8 As shown, an embodiment of the present invention provides a data storage management system based on big data, including a conversion module 2, the conversion module 2 is bidirectionally connected with a construction module 1 and an application module 3, the construction module 1 is connected with a data traversal module 4, a data processing module 5 and a model construction module 6, the application module 3 is connected with a data input module 7, a data sorting module 8 and a storage management module 9, the conversion module 2 includes a model calling module 201, and the model calling module 201 is connected with a model adaptation module 202 and a target matching module 203.

[0018] Specifically, by providing a construction module 1, it is convenient to realize the basic function of constructing the model when the system is used. By providing an application module 3, it is convenient to realize the basic application function of the system when it is used. By providing a conversion module 2, it is convenient to realize the function of converting the model when the system is used so that it can be used.

[0019] The data traversal module 4 includes an iterative traversal module 401 and a parallel computing module 404 . The iterative traversal module 401 is connected to an iterator module 402 and a generation module 403 . The parallel computing module 404 is connected to a subset segmentation module 405 and a subset traversal module 406 .

[0020] Specifically, it is convenient to traverse and obtain the basic data required for model training, providing a data basis for subsequent model training.

[0021] The data processing module 5 comprises a data cleaning module 501 , and the data cleaning module 501 is connected to a feature selection module 502 , a feature construction module 503 , a feature encoding module 504 , a feature dimension reduction module 505 and a feature scaling module 506 .

[0022] Specifically, it is convenient to pre-process the acquired data and screen out data suitable for use, so that users can subsequently train the model through these data to ensure the executability of the model.

[0023] The model building module 6 includes a data management module 601 and a model management module 604 . The data management module 601 is connected to a data input and output module 602 and a data storage module 603 . The model management module 604 is connected to a training management module 605 and an evaluation and optimization module 606 .

[0024] Specifically, it is convenient to realize the function of building a model when the system is used, and to conduct subsequent training and optimization of the model to improve the use effect of the system.

[0025] The data input module 7 includes an interactive interface module 701 , and the interactive interface module 701 is connected to a batch input module 702 and a manual input module 703 .

[0026] Specifically, it is conducive to realizing the human-computer interaction function of the system, and is convenient for users to input data that needs to be managed.

[0027] The data sorting module 8 includes a value assessment module 801 and a data classification module 804 . The value assessment module 801 is connected to a frequency assessment module 802 and a level assessment module 803 . The data classification module 804 is connected to an attribute classification module 805 and an association determination module 806 .

[0028] Specifically, it is convenient to classify different data and to classify and store different types of data subsequently, thereby ensuring convenience in use.

[0029] The storage management module 9 is connected to a region division module 901 and a feedback management module 904 . The region division module 901 is connected to a low frequency management module 902 and a high frequency management module 903 . The feedback management module 904 is connected to a periodic management module 905 and a call management module 906 .

[0030] Specifically, it is convenient for users to classify, store and call the managed data in the future to ensure the overall usability of the system.

[0031] A data storage management method based on big data specifically comprises the following steps: S1. Data acquisition The data traversal module 4 under the construction module 1 is used to obtain the data content related to the network data storage management, the iterative traversal module 402 under the iterative traversal module 401 is used to iterate and traverse the related data, and the required data is obtained through the generation module 403, the subset under the required data is traversed through the subset traversal module 406 under the parallel computing module 404, and the obtained data is divided into individual subsets through the subset segmentation module 405, waiting for subsequent use; S2. Data Preprocessing The acquired data is input into the data processing module 5, the data is cleaned by the connected data cleaning module 501, the required data features are set by the feature selection module 502, a database of required feature data is established by the feature construction module 503, the database established by the data with features is encoded by the feature encoding module 504, the feature strength is further improved by the feature dimension reduction module 505 and the feature scaling module 506, the data with more complex features is screened out, and the non-compliant parts are removed from the database; S3. Model building The database is input into the model building module 6, and the database is introduced into the model through the data input and output module 602 under the data management module 601, and temporarily stored through the data storage module 603. The model is trained through the data in the established database through the training management module 605 under the model management module 604, so that it has the function of data storage management. The advantages and disadvantages of the model are analyzed through the evaluation and optimization module 606, and further optimized and improved; S4. Model Application The improved model is input into the conversion module 2, the required model is called through the model calling module 201, the function of the model is adapted to the management system through the model adaptation module 202, and the target data that the model needs to process is determined through the target matching module 203; S5. Data Entry The user can input the data to be stored and managed through the data input module 7 under the application module 4, enter the data through the interactive interface module 701, automatically enter the data to be entered in batches through the batch input module 702 in combination with the form, and supplement the sporadic data through the manual input module 703; S6. Data storage The input data can be imported into the data sorting module 8, and the use frequency and confidentiality level of the input data are evaluated through the frequency evaluation module 802 and the level evaluation module 803 under the value evaluation module 801. After the evaluation is completed, the data is classified through the data classification module 804, and the data of different values ​​are classified according to attributes and relevance through the attribute classification module 805 and the association determination module 806, and classified according to different types under the premise of different value distinctions; S7. Data management The classified data is imported into the model, and the storage management module 9 is combined with the established model to divide the storage area through the area division module 901 according to the evaluation level and type of the data. The area with faster reading and writing speed is divided into the high-frequency management module 903, which is used to store data with high usage frequency and similar data. The area with relatively slow reading and writing data but good stability is divided into the low-frequency management module 902, which is used to store data with low usage frequency and need to be kept confidential and similar data. After the storage is completed, during daily use, the stored data is regularly backed up through the engagement management module 905 under the feedback management module 904, and the data can be called up through the call management module 906 for use when needed.

[0032] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data storage management system based on big data, comprising a conversion module (2), characterized in that: The conversion module (2) is bidirectionally connected to a construction module (1) and an application module (3); the construction module (1) is connected to a data traversal module (4), a data processing module (5) and a model construction module (6); and the application module (3) is connected to a data input module (7), a data sorting module (8) and a storage management module (9); The conversion module (2) comprises a model calling module (201), and the model calling module (201) is connected to a model adaptation module (202) and a target matching module (203).

2. A data storage management system based on big data according to claim 1, characterized in that: The data traversal module (4) comprises an iterative traversal module (401) and a parallel computing module (404); the iterative traversal module (401) is connected to an iterator module (402) and a generation module (403); and the parallel computing module (404) is connected to a subset segmentation module (405) and a subset traversal module (406).

3. A data storage management system based on big data according to claim 1, characterized in that: The data processing module (5) comprises a data cleaning module (501), wherein the data cleaning module (501) is connected to a feature selection module (502), a feature construction module (503), a feature encoding module (504), a feature dimensionality reduction module (505) and a feature scaling module (506).

4. The data storage management system based on big data according to claim 1, characterized in that: The model building module (6) comprises a data management module (601) and a model management module (604), wherein the data management module (601) is connected to a data input and output module (602) and a data storage module (603), and the model management module (604) is connected to a training management module (605) and an evaluation optimization module (606).

5. The data storage management system based on big data according to claim 1, characterized in that: The data input module (7) comprises an interactive interface module (701), and the interactive interface module (701) is connected to a batch input module (702) and a manual input module (703).

6. A data storage management system based on big data according to claim 1, characterized in that: The data sorting module (8) comprises a value assessment module (801) and a data classification module (804); the value assessment module (801) is connected to a frequency assessment module (802) and a level assessment module (803); and the data classification module (804) is connected to an attribute classification module (805) and an association determination module (806).

7. The data storage management system based on big data according to claim 1, characterized in that: The storage management module (9) is connected to a region division module (901) and a feedback management module (904); the region division module (901) is connected to a low frequency management module (902) and a high frequency management module (903); and the feedback management module (904) is connected to a periodic management module (905) and a call management module (906).

8. A data storage management method based on big data, characterized in that: The specific steps include: S1. Data acquisition The data content related to the network data storage management is obtained through the data traversal module (4) under the construction module (1), the relevant data is iteratively traversed through the iterative traversal module (402) under the iterative traversal module (401), and the required data is obtained through the generation module (403), the subset under the required data is traversed through the subset traversal module (406) under the parallel computing module (404), and the obtained data is divided into individual subsets through the subset segmentation module (405) to wait for subsequent use; S2. Data Preprocessing Input the acquired data into the data processing module (5), clean the data through the connected data cleaning module (501), set the required data features through the feature selection module (502), establish a database of required feature data through the feature construction module (503), encode the database established by the data with features through the feature encoding module (504), further enhance the feature strength through the feature dimension reduction module (505) and the feature scaling module (506), screen out data with more complex features, and remove the non-compliant parts from the database; S3. Model building Input the database into the model building module (6), introduce the database into the model through the data input and output module (602) under the data management module (601), and temporarily store it through the data storage module (603), train the model with the data in the established database through the training management module (605) under the model management module (604), so that it has the function of data storage management, analyze the advantages and disadvantages of the model through the evaluation and optimization module (606), and further optimize and improve it; S4. Model Application The improved model is input into the conversion module (2), the required model is called through the model calling module (201), the function of the model is adapted to the management system through the model adaptation module (202), and the target data to be processed by the model is determined through the target matching module (203); S5. Data Entry The user can input the data to be stored and managed through the data input module (7) under the application module (4), enter the data through the interactive interface module (701), automatically enter the data to be entered in batches in combination with a table format through the batch input module (702), and supplement the entry of sporadic data through the manual input module (703); S6. Data storage The input data can be imported into the data sorting module (8), and the frequency evaluation module (802) and the level evaluation module (803) under the value evaluation module (801) are used to evaluate the usage frequency and confidentiality level of the input data. After the evaluation is completed, the data is classified by the data classification module (804), and the data of different values ​​are classified according to attributes and relevance through the attribute classification module (805) and the association determination module (806), and classified according to different types based on the premise of different value distinctions; S7. Data management The classified data is imported into the model, and the storage management module (9) is combined with the established model to divide the storage area according to the evaluation level and type of the data through the area division module (901). The area with faster reading and writing speed is divided into a high-frequency management module (903) for storing data with high usage frequency and similar data. The area with relatively slow reading and writing speed but good stability is divided into a low-frequency management module (902) for storing data with low usage frequency and data that needs to be kept confidential and similar data. After storage is completed, during daily use, the stored data is regularly backed up through the engagement management module (905) under the feedback management module (904). When it is needed, the data can be called up through the call management module (906) for use.

Images