Cloud big data storage and computing resource scheduling system

Abstract

The application discloses a cloud big data storage and computing resource scheduling system, and relates to the technical field of data storage and scheduling.The cloud big data storage and computing resource scheduling system comprises a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module and a leakage early warning module.The application has the advantages that the data classification module can classify data, and divide the result data into conclusion data and the original material into original data, so that the data is classified and divided;the achievement storage end module can separately store the data used to draw the conclusion and the data used to draw the conclusion of the achievement, and uniformly collects the data used to draw the conclusion;when the data in the achievement storage end module is called, the use of the original data storage module is closed, so that the starting time of the achievement storage end module and the original data storage module is separated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage and scheduling, in particular to a cloud big data storage and computing resource scheduling system. BACKGROUND

[0002] The cloud big data storage and computing resource scheduling system is a system for managing and optimizing big data storage and computing resources, and is a comprehensive system that needs to have functions such as storage management, computing resource scheduling, data security and privacy protection, elastic expansion, resource monitoring and performance optimization to meet the needs of big data processing.

[0003] However, the general cloud big data storage and computing resource scheduling system on the market directly uses the obtained data when in use, but this will cause a large amount of data to run synchronously, and the data cannot be classified and stored, which causes useless data to also run, reduces the running memory, and affects the smoothness of the system when running. SUMMARY

[0004] The present application relates to the technical field of data storage and scheduling, in particular to a cloud big data storage and computing resource scheduling system.

[0005] To solve the problems in the above background art, the present application provides the following technical solution: a cloud big data storage and computing resource scheduling system, comprising a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module, and a leakage early warning module.

[0006] The database module is in communication connection with the data classification module and the data scheduling module, the data classification module is in communication connection with the achievement storage end module and the original data storage module, the data scheduling module is in communication connection with the temporary database module, and the temporary database module is in communication connection with the leakage early warning module, the achievement storage end module, and the original data storage module.

[0007] The database module is used for obtaining data, manually transmitting and writing data into the database module, perfecting the data, and transmitting the perfected data to the data classification module.

[0008] The data classification module is used for classifying the received data, classifying the results and original data, and transmitting the classified data to the achievement storage end module and the original data storage module, respectively.

[0009] The achievement storage end module is used for storing the classified achievement data and transmitting the achievement data to the temporary database module.

[0010] The original data storage module is used for storing the classified original data and can transmit the original data module to the temporary database module;

[0011] The data scheduling module is used for judging the usefulness of data scheduling, dividing the data scheduling into resource scheduling, data calculation and other aspects, and then transmitting the judging result to the temporary database module;

[0012] The temporary database module is used for processing data and can extract relevant data from the achievement storage end module and the original data storage module after receiving the data of the data scheduling module;

[0013] The leakage warning module can perform early warning to the outside world when the data is leaked during operation, so that internal staff can timely process.

[0014] As a further scheme of the present application, the database module can acquire data on the Internet of Things, and when acquiring data, can manually transmit data to the database module, and when it is found that the data is not updated in time, the data in the database module can be modified by internal professional personnel, so that the data in the database module is more perfect, and then the perfect data is transmitted to the data classification module.

[0015] As a further scheme of the present application, the data classification module can receive the data transmitted by the database module and classify the data, wherein;

[0016]

[0017] The scheduling formula of the whole data is the above formula, is the original data, is the provided data problem, is the conclusion, at this time, the obtained data can be classified into original data and achievement conclusion, wherein;

[0018] =

[0019] is obtained and is the original data, is the achievement data, and then the original material and the achievement data are respectively transmitted to the original data storage module and the achievement storage end module.

[0020] As a further scheme of the present application, the achievement storage end module can store the result material , and at the same time, the original material of the result is divided into a subsequent subset, and the subset is set as a class set;

[0021]

[0022] When the scheduling data belongs to the original data storage module, the data storage module will directly obtain the data that will contain the original data storage module; At the same time, when the result storage module is running, the original data storage module will be closed, thereby increasing the running space that can be used when the result storage module is scheduled.

[0023] As a further scheme of the present application, the original data storage module can store the raw materials and obtained by the data classification module, and can store the raw materials and obtained by the corresponding results, and when the original data storage module is running, the result storage module will be closed at the same time, thereby increasing the space and the process degree when the original data storage module is running.

[0024] As a further scheme of the present application, the data scheduling module can receive the input information, judge the information, and obtain whether the information is resource scheduling or data calculation, and then transmit the judgment result to the temporary database module.

[0025] As a further scheme of the present application, the data scheduling module compares the formula information belonging to the resource scheduling template with the input information, wherein;

[0026] =

[0027] is resource scheduling, and the input information is;

[0028] The formula information belonging to the data calculation is compared with the input data, wherein;

[0029] … =

[0030] is data calculation, and m is an unknown number.

[0031] As a further scheme of the present application: the temporary database module can receive the conclusion judged by the data scheduling module, extract the relevant data about the conclusion of the data scheduling module in the achievement storage end module and the original data storage module, and stop the operation of the other module when the relevant data in one of the achievement storage end module or the original data storage module is called, so that there is a time difference when the information in the achievement storage end module or the original data storage module is called, and the achievement storage end module and the original data storage module are not called simultaneously.

[0032] As a further scheme of the present application: the leakage warning module will alert the internal staff when the data in the temporary database module leaks, so that the internal staff can timely process the relevant data and search for the leaked data, thereby preventing data loss, and when the data in the temporary database module leaks, the data will not be lost too much, reducing the impact of data leakage.

[0033] By adopting the above technical scheme, compared with the prior art, the present application has the following beneficial effects:

[0034] The data classification module can classify the data, and divide the result data into conclusion data and the original material into original data, thereby classifying the data, the achievement storage end module can separately store the data and the material used to derive the conclusion belonging to the achievement, and the data used to derive the conclusion is uniformly collected, when the input question belongs to the collection, the corresponding conclusion is derived, and when the data in the achievement storage end module is called, the use of the original data storage module is closed, so that the start time of the achievement storage end module and the original data storage module is separated, the running memory is reduced by too much useless data, and the smoothness of the system running is increased.

[0035] The present application increases the space of the memory when the data in the achievement storage end module or the original data storage module is called, so that the data calling is more smooth, the data scheduling module can judge whether the input question belongs to data calculation or resource scheduling, and the data scheduling module can be divided into two options of data calculation and resource scheduling, and the usefulness of the data is selected by artificial selection.

[0036] The present application uses the temporary database module, when the data leaks during running, only the temporary data in the temporary database module leaks, thereby reducing the impact of data leakage, and the leakage warning module can timely alert the internal staff when the data in the temporary database module leaks, so that the staff can timely process the leaked data and prevent data loss. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The overall flowchart of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The specific embodiments of the present application will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0039] Embodiment one, please refer to the attached Figure 1 The present application provides a technical solution: a cloud big data storage and computing resource scheduling system, the cloud big data storage and computing resource scheduling system includes a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module and a leakage early warning module;

[0040] The database module is in communication connection with the data classification module and the data scheduling module, the data classification module is in communication connection with the achievement storage end module and the original data storage module, the data scheduling module is in communication connection with the temporary database module, and the temporary database module is in communication connection with the leakage early warning module, the achievement storage end module and the original data storage module;

[0041] The database module is used for obtaining data, and can also manually transmit and write data into the database module, perfect the data, and transmit the perfect data to the data classification module;

[0042] The data classification module is used for classifying the received data, classifying the results and the original data, and transmitting the classified data to the achievement storage end module and the original data storage module respectively;

[0043] The achievement storage end module is used for storing the classified achievement data, and can transmit the achievement data to the temporary database module;

[0044] The original data storage module is used for storing the classified original data, and can transmit the original data module to the temporary database module;

[0045] The data scheduling module is used for judging the usefulness of data scheduling, dividing the data scheduling into resource scheduling, data calculation and other aspects, and then transmitting the judgment result to the temporary database module;

[0046] The temporary database module is used for processing data, and can extract related data from the achievement storage end module and the original data storage module after receiving the data from the data scheduling module;

[0047] The leakage early warning module, when the data is running, the data is leaked once, and the outside is warned, so that the internal staff can handle it in time.

[0048] Please refer to the attached Figure 1 The database module can acquire data on the Internet of Things, and when acquiring data, artificial data can be transmitted to the database module, and when it is found that the data is not updated in time, the data in the database module can be modified by internal professionals, so that the data in the database module is more perfect, and then the perfect data is transmitted to the data classification module. The data classification module can receive the data transmitted by the database module and classify the data, wherein;

[0049]

[0050] The scheduling formula of the whole data is the above formula, is the original data, is the data provided by the data problem, is the conclusion, and the obtained data can be classified as original data and achievement conclusion, wherein;

[0051] =

[0052] It is concluded that and is the original data, is the achievement data, and then the original material and the achievement material are transmitted to the original data storage module and the achievement storage end module, respectively. The achievement storage end module can store the result material , and at the same time, the original material of the result is divided into a subsequent subset, which is set as class set;

[0053]

[0054] When the scheduling data belongs to set, it is directly concluded that set containing ; At the same time, when the achievement storage end module is running, the original data storage module is closed, so as to increase the running space that can be used when the achievement storage end module is scheduled;

[0055] In this embodiment, the data on the Internet can be acquired through the database module, and when the internal staff conducts experiments, new data can be obtained. At this time, the new data can be artificially transmitted to the database module to update the data, and the data can be modified internally to perfect the data.

[0056] In use, the data can be classified by the data classification module, and the result data is divided into conclusion data, and the original material is divided into original data, so as to classify the data, and the achievement storage end module can store the data and the data used to obtain the conclusion separately, and the data used to obtain the conclusion is uniformly collected, when the input belongs to the set question, the corresponding conclusion is obtained, and when the data in the achievement storage end module is called, the use of the original data storage module is closed, so that the starting time of the achievement storage end module and the original data storage module is interval.

[0057] Embodiment two, please refer to the attached Figure 1 The application provides a technical scheme: a cloud big data storage and computing resource scheduling system, the cloud big data storage and computing resource scheduling system comprises a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module and a leakage early warning module.

[0058] The database module is in communication connection with the data classification module and the data scheduling module, the data classification module is in communication connection with the achievement storage end module and the original data storage module, the data scheduling module is in communication connection with the temporary database module, and the temporary database module is in communication connection with the leakage early warning module, the achievement storage end module and the original data storage module.

[0059] The database module is used for acquiring data, and data can be manually transmitted and written into the database module, the data is perfected, and the perfected data is transmitted to the data classification module.

[0060] The data classification module is used for classifying the received data, classifying the data results and the original data, and transmitting the classified data to the achievement storage end module and the original data storage module respectively.

[0061] The achievement storage end module is used for storing the classified achievement data, and can transmit the achievement data to the temporary database module.

[0062] The original data storage module is used for storing the classified original data, and can transmit the original data module to the temporary database module.

[0063] The data scheduling module is used for judging the usefulness of data scheduling, dividing the data scheduling into resource scheduling, data calculation and the like, and then transmitting the judgment result to the temporary database module.

[0064] The temporary database module is used for processing data, and can extract relevant data from the achievement storage end module and the original data storage module after receiving the data of the data scheduling module.

[0065] The leakage early warning module, when the data is running, once the data is leaked, it will give an early warning to the outside world, so that the internal staff can handle it in time.

[0066] Please refer to the attached Figure 1 The original data storage module can store the raw materials and transmitted by the data classification module, and can store the raw materials and that derive the corresponding results. When the original data storage module is running, the operation of the achievement storage end module will be closed synchronously, thereby increasing the space when the original data storage module is running, increasing the process degree when the original data storage module is running, and the data scheduling module can receive the input information and judge the information. Whether it is resource scheduling or data calculation, then the result of the judgment is transmitted to the temporary database module. The data scheduling module compares the formula information belonging to the resource scheduling template with the input information, wherein;

[0067] =

[0068] is the resource scheduling, is the input information;

[0069] The formula information belonging to the data calculation is compared with the input data, wherein;

[0070] ……+ =

[0071] is the data calculation, m is an unknown number in ; and

[0072] In this embodiment, the original data can be stored by the original data storage module. When the regional original data storage module is running, the operation of the achievement storage end module is stopped, so that the achievement storage end module and the original data storage module can be used alternately. When one of them is used, the other module is closed, reducing the data occupying space when retrieving information.

[0073] When in use, by increasing the memory space of the data retrieved from the achievement storage end module or the original data storage module, the data retrieval is more smooth. The data scheduling module can judge whether the input problem belongs to data calculation or resource scheduling, and can divide the data scheduling module into two options of data calculation and resource scheduling. The use of data is selected by artificial.

[0074] Example three, please refer to the attached Figure 1The application provides a technical scheme: a cloud big data storage and computing resource scheduling system, which comprises a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module and a leakage early warning module.

[0075] The database module is in communication connection with the data classification module and the data scheduling module, the data classification module is in communication connection with the achievement storage end module and the original data storage module, the data scheduling module is in communication connection with the temporary database module, and the temporary database module is in communication connection with the leakage early warning module, the achievement storage end module and the original data storage module.

[0076] The database module is used for acquiring data, manually transmitting and writing data into the database module, perfecting the data, and transmitting the perfected data to the data classification module.

[0077] The data classification module is used for classifying the received data, classifying the data results and the original data, and respectively transmitting the classified data to the achievement storage end module and the original data storage module.

[0078] The achievement storage end module is used for storing the classified achievement data and transmitting the achievement data to the temporary database module.

[0079] The original data storage module is used for storing the classified original data and transmitting the original data to the temporary database module.

[0080] The data scheduling module is used for judging the usefulness of data scheduling, dividing the data scheduling into resource scheduling, data calculation and the like, and then transmitting the judged result to the temporary database module.

[0081] The temporary database module is used for processing data and extracting related data from the achievement storage end module and the original data storage module after accepting the data of the data scheduling module.

[0082] The leakage early warning module is used for early warning the outside world when the data leaks during operation, so that the internal staff can timely process.

[0083] Please refer to the accompanying drawings Figure 1, temporary database module can receive the conclusion of data scheduling module, through the conclusion of the relevant data extraction results storage module and the original data storage module about data scheduling module judgment conclusion, and in the results storage module or original data storage module one of the module will stop running when the other module, so that the retrieval results storage module or original data storage module information exists time difference, make results storage module and original data storage module will not be retrieved at the same time running, leakage warning module when the temporary database module data leakage, will alert the internal staff, so that the internal staff in a timely manner to deal with the relevant data, and search for the leaked data, so as to prevent data loss, at the same time when the temporary database module data leakage, resulting in data will not lose too much, reduce the impact of data leakage;

[0084] In this embodiment, through the temporary database module can be interval retrieval results storage module and the original data storage module inside the data about input problem, make data in temporary database module for use, and when the temporary database module complete the data retrieval results storage module or original data storage module, will close the temporary database module and the retrieval relationship between the results storage module and the original data storage module.

[0085] In use, through the use of temporary database module, when the data leakage occurs during running, only the temporary data belonging to the temporary database module will be leaked, thereby reducing the influence degree of data leakage, through the leakage warning module when the data leakage occurs in the temporary database module, can timely alert the internal staff, so that the staff in a timely manner to deal with the leaked data, prevent data loss.

[0086] Working principle:

[0087] The first step, first, through the database module can get the data on the Internet, and complete the acquisition can be through artificial internal data to improve, then the improved data transmission to data classification module, using data classification module to classify data, so that the data classification results class data and raw material class data, and will be classified data transmission respectively transmission to results storage module and original data storage module, through the original data storage module to store data;

[0088] Second step, secondly, the required input question is input in the data scheduling module, the input data is judged by using the data scheduling module, so that the input question belongs to resource scheduling or data calculation is obtained, and manual selection is required to be carried out resource scheduling or data calculation, then the data on the achievement storage module and the original data storage module are retrieved by using the temporary database module, and when the data in one of the modules is retrieved, the remaining modules will be closed, so as to prevent the remaining data from occupying the space;

[0089] Third step, finally, when the data in the original data storage module leaks, only the data in the original data storage module leaks, so that when data leakage occurs, the leakage range is limited, and when data leakage occurs, the internal staff can be warned in time through the leakage warning module, so that the staff can process the leaked data, thereby reducing the adverse effects caused by data leakage.

[0090] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by using a general computer device, alternatively, they can be realized by using program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0091] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0092] The above describes the specific embodiments of the present application in combination with the drawings, but is not limited to the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A cloud big data storage and computing resource scheduling system, characterized in that: The cloud big data storage and computing resource scheduling system comprises a database module, a data classification module, an achievement storage end module, an original data storage module, a data scheduling module, a temporary database module and a leakage early warning module. The database module is in communication connection with the data classification module and the data scheduling module, the data classification module is in communication connection with the achievement storage end module and the original data storage module, the data scheduling module is in communication connection with the temporary database module, and the temporary database module is in communication connection with the leakage early warning module, the achievement storage end module and the original data storage module. The database module is used for acquiring data, manually transmitting and writing data into the database module, perfecting the data, and transmitting the perfected data to the data classification module. The data classification module is used for classifying the received data, classifying the data results and the original data, and transmitting the classified data to the achievement storage end module and the original data storage module respectively. The achievement storage end module is used for storing the classified achievement data and transmitting the achievement data to the temporary database module. The original data storage module is used for storing the classified original data and transmitting the original data to the temporary database module. The data scheduling module is used for judging the usefulness of data scheduling, dividing the data scheduling into resource scheduling and data calculation, and then transmitting the judgment result to the temporary database module. The temporary database module is used for processing data and extracting relevant data from the achievement storage end module and the original data storage module after receiving the data from the data scheduling module. The temporary database module can receive the conclusion of the data scheduling module, extract relevant data about the conclusion of the data scheduling module from the achievement storage end module and the original data storage module, and stop the operation of the other module when relevant data is retrieved from one of the achievement storage end module and the original data storage module, so that there is a time difference when retrieving information from the achievement storage end module and the original data storage module, and the achievement storage end module and the original data storage module are not operated simultaneously. The leakage early warning module can perform early warning to the outside world when the data is running, so that internal staff can timely process the data.

2. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The database module can acquire data on the Internet of Things, manually transmit data to the database module when acquiring data, modify the data in the database module through internal professional personnel when finding that the data is not updated in time, so that the data in the database module is more perfect, and then transmit the perfected data to the data classification module.

3. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The data classification module can receive the data transmitted by the database module and classify the data. The scheduling formula of the whole data is the above formula, The original data, The data problem provided, The conclusion obtained, at this time, the data obtained can be classified as original data and achievement conclusion, wherein; = derived and as raw data, as result data, and then the raw data and the result data are transmitted to a raw data storage module and a result storage module, respectively.

4. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The result storage module is capable of storing the result material and simultaneously dividing the raw material from which the result is derived into a subsequent subset, which is set as a class set; When the scheduling data belongs to the set, it is directly concluded that the data to be contained in the set ; meanwhile, when the achievement storage end module is running, the original data storage module is closed, thereby increasing the running space that can be used when the achievement storage end module is scheduled.

5. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The original data storage module can store the raw materials transmitted by the data classification module And The original data storage module can store the raw materials transmitted by the data classification module And The original data storage module can store the raw materials transmitted by the data classification module 6. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The data scheduling module can receive the input information and judge whether the information is resource scheduling or data calculation, and then transmit the judgment result to the temporary database module.

7. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The data scheduling module compares the formula information belonging to the resource scheduling template with the input information. = for resource scheduling, for input information; Compare the formula information belonging to data calculation with input data, wherein; ……+ = m is an unknown number in the data, m is an unknown number in the data, 8. The cloud big data storage and computing resource scheduling system of claim 1, wherein: The leakage early warning module can warn the internal staff when data in the temporary database module is leaked, so that the internal staff can process the relevant data in time and search the leaked data, thereby preventing data loss.

Images