Data center storage and reading efficiency optimization system supporting diverse hard disks

Abstract

The present invention discloses a data center storage and reading efficiency optimization system supporting diversified hard disks, which relates to the technical field of data reading efficiency optimization, including a solid-state hard disk module, a mechanical hard disk module, a program acquisition module, a file acquisition module, a program and file association module, a program prediction module, a file prediction module, and a storage execution module; the data center storage and reading efficiency optimization system supporting diversified hard disks, by setting the solid-state hard disk module, the mechanical hard disk module, the program acquisition module, the file acquisition module, the program and file association module, and the storage execution module, realizes statistics on the frequency of users using each program and file, and automatically stores the corresponding data in the solid-state hard disk or the mechanical hard disk according to the user's usage frequency, thereby ensuring the reading speed of commonly used data in the solid-state hard disk, and using the mechanical hard disk to store infrequently used data, thereby reducing storage costs.

Description

Technical Field

[0001] The present invention relates to the technical field of data reading efficiency optimization, and in particular to a data center storage reading efficiency optimization system supporting diversified hard disks. Background Art

[0002] In order to meet the different storage needs of different users, different storage hard drives or storage media are usually selected to store user data. For example, data that needs to be read at high speed can be stored in a solid-state drive, that is, SSD media is used to store data. Mechanical hard drives can be used to store data at low cost, that is, data is stored in HDD media.

[0003] However, SSDs are expensive and have limited storage space. How to select data storage media based on users' actual data usage needs to ensure data reading efficiency while reducing data storage costs is a problem that needs to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide a data center storage and reading efficiency optimization system supporting diversified hard disks to solve the above-mentioned deficiencies in the prior art.

[0005] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a data center storage and reading efficiency optimization system supporting diversified hard disks, including a solid state hard disk module, a mechanical hard disk module, a program acquisition module, a file acquisition module, a program and file association module, a program prediction module, a file prediction module, and a storage execution module;

[0006] The solid state drive module and the mechanical hard drive module are used to store data, and the data includes program data and file data. The program data is the code data of the program, and the file data is the data for storing file information, such as text, pictures, audio, video and other data;

[0007] The program acquisition module is used to acquire the usage records of each program and obtain the usage frequency of each program;

[0008] The file acquisition module is used to acquire the reading records of each file and obtain the reading frequency of each file;

[0009] The program and file association module is used to obtain the file records read when each program is executed, and associate the file with the program that reads the file. One file can be associated with multiple programs, and one program can also be associated with multiple files, that is, one file can be read by multiple programs, and one program can also read multiple files;

[0010] The program prediction module is used to train a hidden Markov model based on the usage records of each program, and obtain a program prediction model for predicting the programs to be used within a set number of times in the future based on the usage records of each program, including the following steps:

[0011] A1. Obtain the usage records of each program, and arrange the programs in the chronological order of the program usage records to obtain a program usage sequence, wherein the program usage sequence is an observation sequence composed of multiple program observation values, and each program observation value represents a program used by a user at a certain moment;

[0012] A2. Encode each program in the program usage sequence and map the program to a numerical value, that is, encode the program name to a number, for example, program A is encoded as 0, that is, 0 is used to represent program A;

[0013] A3. Determine the number of hidden states, use the program usage sequence to train the hidden Markov model, obtain state transition probability and observation probability, and learn the transition probability between hidden states and the observation probability under each hidden state to obtain a program prediction model;

[0014] The file prediction module is used to train a hidden Markov model based on the reading records of each program for each file, and obtain a file prediction model for predicting the files read by the program within a set number of times in the future based on the reading records of the current program and the corresponding files, including the following steps:

[0015] D1. Obtain the reading records of each program for each file, and classify the reading records of the files according to the different programs to obtain multiple file sample subsets;

[0016] D2. Arrange the files in each file sample subset in chronological order according to the file reading records to obtain a file reading subsequence;

[0017] D3, encoding the files in the file reading subsequence and mapping the files into numerical values;

[0018] D4. Determine the number of hidden states, use the file reading subsequence to train the hidden Markov model, obtain the state transition probability and observation probability, and learn the transition probability between hidden states and the observation probability under each hidden state to obtain the file prediction submodel;

[0019] D5. Integrate the file prediction sub-models corresponding to the file sample subsets to obtain a file prediction model, wherein the file prediction model is used to select a file prediction sub-model corresponding to the file sample subset corresponding to the program according to the program for reading the file, and predict the files to be read within a set number of times in the future;

[0020] The storage execution module is used to select a solid state hard disk module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency;

[0021] The storage execution module is also used to temporarily copy the programs and files stored in the mechanical hard disk module and output by the program prediction model and the file prediction model to the solid state disk module, so that when the program is executed and the file is read, the program data corresponding to the program and the file data corresponding to the file are read from the solid state disk module, wherein the temporary copying is performed while changing the path of the program reading the file to the location where the file is temporarily copied in the solid state disk, so that the program no longer reads the file from the mechanical hard disk but reads the file from the solid state disk.

[0022] Furthermore, the program prediction module is also used to screen out programs whose probabilities output by the program prediction model are lower than the set first probability, including the following steps:

[0023] B1. Set the first probability;

[0024] B2. Obtaining the hidden state probability corresponding to the program output by the program prediction model;

[0025] B3. Determine whether the hidden state probability is greater than the first probability;

[0026] B4. If yes, output the program corresponding to the hidden state probability; if no, filter out the program corresponding to the hidden state probability.

[0027] Furthermore, the program prediction module is also used to construct a program execution network diagram based on the program usage record and the predicted program, including the following steps:

[0028] C1. Build a program execution network diagram from left to right according to the order of use of each program in the program usage record;

[0029] C2. Name the leftmost program in each program execution network diagram as the initial program;

[0030] C3. Extracting the program data of the initial program and storing the program data of the initial program in the solid state drive, thereby ensuring the reading speed of the program data of the initial program and the program startup speed.

[0031] Furthermore, the storage execution module is used to select a solid state hard disk module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency, including the following steps:

[0032] E1. Set the program usage frequency threshold and file reading frequency threshold;

[0033] E2. Determine whether the program usage frequency is greater than the program usage frequency threshold;

[0034] E3. If yes, store the program to the solid state drive; if no, store the program to the mechanical hard drive;

[0035] E4. Determine whether the file reading frequency is greater than the file reading frequency threshold:

[0036] E5. If yes, store the file in the solid state drive; if no, store the file in the mechanical hard drive.

[0037] Furthermore, the storage execution module further includes the following steps:

[0038] F1, set the timer and reset it to zero;

[0039] F2: Determine whether the timer reaches the set time;

[0040] F3. If yes, update the program usage frequency and file reading frequency to the program usage frequency and file reading frequency in the current timing cycle of the timer, reset the timer timing to zero, and jump to E2; if no, return to F2.

[0041] Furthermore, the storage execution module is also used to delete the files temporarily copied to the solid state drive module after the program ends, including the following steps:

[0042] G1. Get the currently executed program and the program predicted by the program prediction model;

[0043] G2. Determine whether there is a program associated with the temporarily copied file among the acquired programs;

[0044] G3. If yes, return to G2; if no, delete the temporarily copied file from the SSD.

[0045] Furthermore, the storage execution module is also used to determine whether to perform distributed storage on the file according to the file size read by the program, including the following steps:

[0046] Get the volume of a single file that the program reads simultaneously;

[0047] Determine whether the file size exceeds the set file size threshold;

[0048] If so, the file is distributedly stored, the file is divided into multiple data packets, and the data packets are evenly stored in multiple preset hard disks. When the program reads the file, the file is read from multiple hard disks at the same time, which improves the file reading speed and shortens the loading time of opening the file. Among them, if there are multiple solid-state drives and mechanical hard disks, if the file should originally be stored in the solid-state drive, the file is distributedly stored in the solid-state drive; if the file should originally be stored in the mechanical hard disk, the file is distributedly stored in the mechanical hard disk; if the file distributedly stored in the mechanical hard disk is temporarily copied to the solid-state drive, the distributed storage method is also adopted.

[0049] 1. Compared with the prior art, the data center storage and reading efficiency optimization system supporting diversified hard disks provided by the present invention can realize statistics on the frequency of users using various programs and files by setting a solid-state hard disk module, a mechanical hard disk module, a program acquisition module, a file acquisition module, a program and file association module, and a storage execution module, and automatically stores the corresponding data in the solid-state hard disk or the mechanical hard disk according to the user's usage frequency, thereby ensuring the reading speed of commonly used data in the solid-state hard disk, and using the mechanical hard disk to store infrequently used data, thereby reducing storage costs.

[0050] 2. Compared with the prior art, the data center storage and reading efficiency optimization system supporting diversified hard disks provided by the present invention realizes the prediction of programs to be used and files to be read in the future by setting a program prediction module, a file prediction module, and a storage execution module. If the data corresponding to the predicted program or file is stored in a mechanical hard disk, the data is temporarily copied to the solid-state hard disk in advance, thereby improving the data reading speed when using the program and reading the file. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0052] Figure 1 A system structure block diagram provided for an embodiment of the present invention;

[0053] Figure 2 A step diagram of a storage execution module provided in an embodiment of the present invention selecting a solid state drive module or a mechanical hard drive module for storing program data and file data. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0057] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0058] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0059] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0060] See also Figure 1-Figure 2 , a data center storage and reading efficiency optimization system that supports diversified hard disks, including a solid-state hard disk module, a mechanical hard disk module, a program acquisition module, a file acquisition module, a program and file association module, a program prediction module, a file prediction module, and a storage execution module;

[0061] The solid state drive module and the mechanical hard drive module are used to store data, including program data and file data. The program data is the code data of the program, and the file data is the data for storing file information, such as text, pictures, audio, video and other data;

[0062] The program acquisition module is used to obtain the usage records of each program and obtain the usage frequency of each program;

[0063] The file acquisition module is used to obtain the reading records of each file and obtain the reading frequency of each file;

[0064] The program and file association module is used to obtain the file records read when each program is executed, and associate the file with the program that reads the file. One file can be associated with multiple programs, and one program can also be associated with multiple files, that is, one file can be read by multiple programs, and one program can also read multiple files;

[0065] The program prediction module is used to train a hidden Markov model based on the usage records of each program, and obtain a program prediction model for predicting the program to be used within a set number of times in the future based on the usage records of each program, including the following steps:

[0066] A1. Obtain the usage records of each program, and arrange the programs in the chronological order of the program usage records to obtain a program usage sequence, wherein the program usage sequence is an observation sequence composed of multiple program observation values, and each program observation value represents a program used by a user at a certain moment;

[0067] A2. Encode each program in the program usage sequence and map the program to a numerical value, that is, encode the program name to a number, for example, program A is encoded as 0, that is, 0 is used to represent program A;

[0068] A3. Determine the number of hidden states, use the program sequence to train the hidden Markov model, obtain the state transition probability and observation probability, and learn the transition probability between hidden states and the observation probability under each hidden state to obtain the program prediction model.

[0069] In one embodiment, when predicting the possible next program, the probability distribution under each hidden state is calculated; the hidden state with the maximum probability is obtained to infer the program to be used in the next step; the program corresponding to the hidden state is obtained to obtain the program that may be used in the next step.

[0070] The program prediction module is also used to screen out programs whose probability output by the program prediction model is lower than the set first probability, including the following steps:

[0071] B1. Set the first probability;

[0072] B2. Obtain the hidden state probability corresponding to the program output by the program prediction model;

[0073] B3. Determine whether the hidden state probability is greater than the first probability;

[0074] B4. If yes, then output the program corresponding to the hidden state probability; if no, then filter out the program corresponding to the hidden state probability.

[0075] The program prediction module is also used to construct a program execution network diagram based on the program usage records and the predicted programs, including the following steps:

[0076] C1. Build a program execution network diagram from left to right according to the order of use of each program in the program usage record;

[0077] C2. Name the leftmost program in each program execution network diagram as the initial program;

[0078] C3. Extracting the program data of the initial program and storing the program data of the initial program in the solid state drive, thereby ensuring the reading speed of the program data of the initial program and the program startup speed.

[0079] The file prediction module is used to train a hidden Markov model based on the reading records of each program for each file, and obtain a file prediction model for predicting the files to be read by the program within a set number of times in the future based on the reading records of the current program and the corresponding files, including the following steps:

[0080] D1. Obtain the reading records of each program for each file, and classify the reading records of the files according to the different programs to obtain multiple file sample subsets;

[0081] D2. Arrange the files in each file sample subset in chronological order according to the file reading records to obtain a file reading subsequence;

[0082] D3, encode the files in the file reading subsequence and map the files into numerical values;

[0083] D4. Determine the number of hidden states, use the file reading subsequence to train the hidden Markov model, obtain the state transition probability and observation probability, and learn the transition probability between hidden states and the observation probability under each hidden state to obtain the file prediction submodel;

[0084] D5. Integrate the file prediction sub-models corresponding to the file sample subsets to obtain a file prediction model. The file prediction model is used to select a file prediction sub-model corresponding to the file sample subset corresponding to the program according to the program that reads the file, and predict the files to be read within a set number of times in the future.

[0085] The storage execution module is used to select a solid state hard disk module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency;

[0086] The storage execution module is also used to temporarily copy the programs and files stored in the mechanical hard disk module and output by the program prediction model and the file prediction model to the solid-state hard disk module, so that when the program is executed and the file is read, the program data corresponding to the program and the file data corresponding to the file are read from the solid-state hard disk module, wherein the temporary copying simultaneously changes the path of the program reading the file to the location where the file is temporarily copied in the solid-state hard disk, so that the program no longer reads the file from the mechanical hard disk but reads the file from the solid-state hard disk.

[0087] The storage execution module is used to select a solid state drive module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency, including the following steps:

[0088] E1. Set the program usage frequency threshold and file reading frequency threshold;

[0089] E2. Determine whether the program usage frequency is greater than the program usage frequency threshold;

[0090] E3. If yes, store the program to the solid state drive; if no, store the program to the mechanical hard drive;

[0091] E4. Determine whether the file reading frequency is greater than the file reading frequency threshold:

[0092] E5. If yes, store the file in the solid state drive; if no, store the file in the mechanical hard drive.

[0093] The storage execution module also includes the following steps:

[0094] F1, set the timer and reset it to zero;

[0095] F2: Determine whether the timer reaches the set time;

[0096] F3. If yes, update the program usage frequency and file reading frequency to the program usage frequency and file reading frequency in the current timing cycle of the timer, reset the timer to zero, and jump to E2; if no, return to F2.

[0097] The storage execution module is also used to delete the files temporarily copied to the solid state drive module after the program ends, including the following steps:

[0098] G1. Get the currently executed program and the program predicted by the program prediction model;

[0099] G2. Determine whether there is a program associated with the temporarily copied file among the acquired programs;

[0100] G3. If yes, return to G2; if no, delete the temporarily copied file from the SSD.

[0101] The storage execution module is also used to determine whether to perform distributed storage on the file according to the size of the file read by the program, including the following steps:

[0102] Get the volume of a single file that the program reads simultaneously;

[0103] Determine whether the file size exceeds the set file size threshold;

[0104] If so, the file is distributedly stored, the file is divided into multiple data packets, and the data packets are evenly stored in multiple preset hard disks. When the program reads the file, the file is read from multiple hard disks at the same time, which improves the file reading speed and shortens the loading time of opening the file. Among them, if there are multiple solid-state drives and mechanical hard disks, if the file should originally be stored in the solid-state drive, the file is distributedly stored in the solid-state drive; if the file should originally be stored in the mechanical hard disk, the file is distributedly stored in the mechanical hard disk; if the file distributedly stored in the mechanical hard disk is temporarily copied to the solid-state drive, the distributed storage method is also adopted.

[0105] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A data center storage and reading efficiency optimization system that supports a variety of hard disks, characterized by: It includes a solid state drive module, a mechanical hard drive module, a program acquisition module, a file acquisition module, a program and file association module, a program prediction module, a file prediction module, and a storage execution module; The solid state drive module and the mechanical hard drive module are used to store data, and the data includes program data and file data. The program data is the code data of the program, and the file data is the data for storing file information. The program acquisition module is used to acquire the usage records of each program and obtain the usage frequency of each program; The file acquisition module is used to acquire the reading records of each file and obtain the reading frequency of each file; The program and file association module is used to obtain the file records read when each program is executed, and associate the file with the program that reads the file; The program prediction module is used to train a hidden Markov model based on the usage records of each program, and obtain a program prediction model for predicting the programs to be used within a set number of times in the future based on the usage records of each program; The file prediction module is used to train a hidden Markov model based on the reading records of each program for each file, and obtain a file prediction model for predicting the files read by the program within a set number of times in the future based on the reading records of the current program and the corresponding files; The storage execution module is used to select a solid state hard disk module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency; The storage execution module is also used to temporarily copy the programs and files stored in the mechanical hard disk module and output by the program prediction model and the file prediction model to the solid state disk module, so that when the program is executed and the file is read, the program data corresponding to the program and the file data corresponding to the file are read from the solid state disk module.

2. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 1, characterized in that: The program prediction module is also used to screen out programs whose probabilities output by the program prediction model are lower than the set first probability, comprising the following steps: B1. Set the first probability; B2. Obtaining the hidden state probability corresponding to the program output by the program prediction model; B3. Determine whether the hidden state probability is greater than the first probability; B4. If yes, output the program corresponding to the hidden state probability; if no, filter out the program corresponding to the hidden state probability.

3. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 1, characterized in that: The program prediction module is also used to construct a program execution network diagram based on the program usage record and the predicted program, including the following steps: C1. Build a program execution network diagram from left to right according to the order of use of each program in the program usage record; C2. Name the leftmost program in each program execution network diagram as the initial program; C3. Extracting program data of the initial program, and storing the program data of the initial program in the solid state drive.

4. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 1, characterized in that: The storage execution module is used to select a solid state hard disk module or a mechanical hard disk module for storing program data and file data according to the program usage frequency and file reading frequency, and includes the following steps: E1. Set the program usage frequency threshold and file reading frequency threshold; E2. Determine whether the program usage frequency is greater than the program usage frequency threshold; E3. If yes, store the program to the solid state drive; if no, store the program to the mechanical hard drive; E4. Determine whether the file reading frequency is greater than the file reading frequency threshold: E5. If yes, store the file in the solid state drive; if no, store the file in the mechanical hard drive.

5. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 4, characterized in that: The storage execution module also includes the following steps: F1, set the timer and reset it to zero; F2: Determine whether the timer reaches the set time; F3. If yes, update the program usage frequency and file reading frequency to the program usage frequency and file reading frequency in the current timing cycle of the timer, reset the timer timing to zero, and jump to E2; if no, return to F2.

6. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 1, characterized in that: The storage execution module is also used to delete the files temporarily copied to the solid state drive module after the program ends, including the following steps: G1. Get the currently executed program and the program predicted by the program prediction model; G2. Determine whether there is a program associated with the temporarily copied file among the acquired programs; G3. If yes, return to G2; if no, delete the temporarily copied file from the SSD.

7. The data center storage and reading efficiency optimization system supporting diversified hard disks according to claim 4, characterized in that: The storage execution module is also used to determine whether to perform distributed storage on the file according to the file size read by the program, including the following steps: Get the volume of a single file that the program reads simultaneously; Determine whether the file size exceeds the set file size threshold; If yes, the file is stored in a distributed manner.

Images