Data scheduling method and system based on prediction

Through the prediction-based data scheduling method, the affinity prediction model is constructed using data affinity parameters, and the problem of data cross-pool and cross-layer scheduling in intelligent computing scenarios is solved, efficient data scheduling is achieved, meeting the needs of intelligent computing storage, and improving the The performance and efficiency of the storage system.

CN119002807APending Publication Date: 2024-11-22CHINA TELECOM DIGITAL INTELLIGENCE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411027865.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing technology is difficult to achieve efficient data cross-pooling and cross-layer scheduling in intelligent computing scenarios, and cannot meet the needs of intelligent computing for extremely high performance, low latency, massive storage, and low-cost storage of storage, and lacks predictability and coordination. Understanding of business logic.

Method used

Using prediction-based data scheduling method, affinity prediction model is constructed by obtaining data affinity parameters, the reading request of data blocks is judged, and the memory partition is scheduled, upgraded or downgraded based on the predicted affinity data blocks to optimize Data access.

Benefits of technology

By predicting data affinity, identifying data blocks that need to be accessed together in advance, reducing data access latency, improving data scheduling efficiency, and optimizing storage system performance and cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002807A_ABST
    Figure CN119002807A_ABST
Patent Text Reader

Abstract

The invention relates to a prediction-based data scheduling method and system, and belongs to the technical field of data processing, and the method comprises the steps: obtaining an affinity prediction model constructed based on data affinity parameters, and constructing corresponding metadata for a storage partition based on the affinity prediction model; judging whether a reading request of a target data block is received within a preset time or not, and inputting the target data block into the affinity prediction model based on a judgment result to obtain one or more predicted affinity data blocks; and judging whether the predicted affinity data block is correct or not, determining a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data block under the condition that the predicted affinity data block is determined to be correct, and scheduling the first partition and the second partition. According to the method, the data access delay is reduced, and the data scheduling efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a prediction-based data scheduling method and system. Background Art

[0002] Intelligent computing has become the hottest trend. The old infrastructure structure can no longer meet the needs of intelligent computing. It is urgent to propose a new infrastructure architecture paradigm to build a computing resource pool. Storage is an important part of the computing resource pool. Intelligent computing requires storage with extremely high performance, extremely low latency, massive storage, and low-cost storage. In order to meet the storage needs of intelligent computing, the storage architecture is usually designed as two-level storage. High-speed storage is implemented using a parallel file system, and massive storage is implemented using a data lake. The data lake usually includes multiple storage resource pools with different performances. According to the performance, they can be divided into hot pools, warm pools, cold pools, and ice pools. Cross-pool data scheduling has become a difficult problem. In addition, usually in an independent storage system, there will also be a cache layer and a persistent storage layer, which also face the same problem of cross-layer data scheduling. Therefore, how to better implement data scheduling has become a problem to be solved. Summary of the invention

[0003] In view of the above deficiencies in the prior art, an object of the present invention is to provide a prediction-based data scheduling method and system, which improves the efficiency of data scheduling.

[0004] According to a first aspect of the present invention, a prediction-based data scheduling method is proposed, comprising: S1, obtaining an affinity prediction model constructed based on data affinity parameters, and constructing corresponding metadata for a storage partition based on the affinity prediction model; S2, judging whether a read request for a target data block is received within a preset time, and based on a judgment result, inputting the target data block into the affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block may be one or more; S3, judging whether the predicted affinity data block is correct, and if it is determined that the predicted affinity data block is correct, determining a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data, scheduling the first partition and the second partition, and changing the metadata of the first partition and the second partition at the same time.

[0005] Furthermore, the data affinity parameters include storage partition parameters, user affinity parameters and time affinity parameters, wherein the storage partition parameters are used to describe the physical storage structure, the user affinity parameters are used to describe the probability of data being read by the same user, and the time affinity parameters are used to describe the probability of data being read within a specified time.

[0006] Furthermore, the target data block is input into the affinity prediction model to obtain a predicted affinity data block, including: inputting the target data block into the affinity prediction model to determine a target user corresponding to accessing the target data block; obtaining a data block accessed by the target user, and using the data block accessed by the target user as the predicted affinity data block; and / or inputting the target data block into the affinity prediction model to determine a target reading time period corresponding to the target data block; obtaining a data block accessed within the target reading time period, and using the data block accessed within the target reading time period as the predicted affinity data block.

[0007] Further, judging whether the predicted affinity data block is correct includes: judging whether the target data block and the predicted affinity data block are both read by different users or whether they are both read by the same user within a target time; wherein, determining that the predicted affinity data block is correct includes: determining that the predicted affinity data block is correct when determining that the target data block and the predicted affinity data block are both read by different users or both read by the same user.

[0008] Further, it is determined that a read request for the target data block is received within a preset time, wherein a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data are determined, and the first partition and the second partition are scheduled, including: determining the corresponding first partition according to the attribute information of the target data block, and determining the corresponding second partition according to the attribute information of the predicted affinity data block; and upgrading the first partition and the second partition.

[0009] Further, it is determined that no read request for the target data block is received within a preset time, wherein a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data are determined, and the first partition and the second partition are scheduled, including: determining the corresponding first partition according to the attribute information of the target data block, and determining the corresponding second partition according to the attribute information of the predicted affinity data block; and downgrading the first partition and the second partition.

[0010] Furthermore, the method also includes: determining a scheduling result, and feeding the scheduling result back to the affinity prediction model to optimize the affinity prediction model; wherein, determining the scheduling result includes: when it is determined that the predicted affinity data block is correct, the scheduling result is reasonable; when it is determined that the predicted affinity data block is incorrect, the scheduling result is unreasonable.

[0011] According to a second aspect of the present invention, a prediction-based data scheduling system is proposed, comprising: an acquisition module, used to acquire an affinity prediction model constructed based on data affinity parameters, and to construct corresponding metadata for a storage partition based on the affinity prediction model; an acquisition module, used to determine whether a read request for a target data block is received within a preset time, and based on the determination result, input the target data block into the affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more; a scheduling module, used to determine whether the predicted affinity data block is correct, and when it is determined that the predicted affinity data block is correct, determine a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data, schedule the first partition and the second partition, and simultaneously change the metadata of the first partition and the second partition.

[0012] According to a third aspect of the present invention, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described in the first aspect of the present invention.

[0013] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to the first aspect of the present invention.

[0014] The beneficial effects of the present invention are as follows:

[0015] The prediction-based data scheduling method of the present invention obtains an affinity prediction model constructed based on data affinity parameters, and constructs corresponding metadata for the storage partition based on the affinity prediction model; determines whether a read request for a target data block is received within a preset time, and based on the judgment result, inputs the target data block into the affinity prediction model to obtain a predicted affinity data block, and the predicted affinity data block is one or more; determines whether the predicted affinity data block is correct, and when it is determined that the predicted affinity data block is correct, determines the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data, and schedules the first partition and the second partition. By predicting data affinity, the method can predict in advance the data blocks that may need to be accessed together, so that when scheduling data, the associated data blocks can be scheduled together, reducing the delay of data access and improving the efficiency of data scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a flow chart of a prediction-based data scheduling method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a prediction-based data scheduling method according to a specific embodiment of the present invention;

[0019] Figure 3 is a structural block diagram of a prediction-based data scheduling system according to an embodiment of the present invention;

[0020] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0022] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0023] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0025] Intelligent computing has become the hottest trend. The old infrastructure structure can no longer meet the needs of intelligent computing. It is urgent to propose a new infrastructure architecture paradigm to build a computing resource pool. Storage is an important part of the computing resource pool. Intelligent computing requires storage with extremely high performance, extremely low latency, massive storage, and low-cost storage. In order to meet the storage needs of intelligent computing, the storage architecture is usually designed as two-level storage. High-speed storage is implemented using a parallel file system, and massive storage is implemented using a data lake. The data lake usually includes multiple storage resource pools with different performances. According to the performance, they can be divided into hot pools, warm pools, cold pools, and ice pools. Cross-pool data scheduling has become a difficult problem. In addition, usually in an independent storage system, there will also be a cache layer and a persistent storage layer, which also face the same problem of cross-layer data scheduling.

[0026] Existing data cross-pool / cross-layer scheduling methods usually perform data scheduling based on the business level or the storage heat level. Scheduling strategies based on the business level require the storage system to understand or obtain business behaviors, and then schedule according to the set strategies. The strategies are actually determined by the business and have nothing to do with the storage itself. Strategies based on the storage heat level simply schedule based on the access frequency of data blocks on the storage system side. The advantage of this strategy is that the storage layer inherently does not know the content of the data it stores, so this strategy is purely data scheduling from a storage perspective. At the same time, data with higher access heat is usually indeed hot data, and its hit rate is better. The disadvantage of this strategy is that since the storage layer cannot understand the scheduling requirements at the business level, this strategy lacks reasoning about business logic and cannot further improve the hit rate.

[0027] Therefore, existing methods cannot solve the following problems in intelligent computing scenarios and massive scenarios: 1) Data scheduling can only be done in units of data blocks. In the context of massive data, the data blocks that need to be monitored are also massive, making any strategy ineffective; 2) Data scheduling can only be done based on hot and cold data, and its scheduling results are difficult to match the actual business behavior, making it difficult to further increase the hit rate; 3) Data scheduling lacks predictability. Existing methods all schedule data blocks, and the pre-reading effect / hit rate reaches a bottleneck and cannot be increased. 4) Data scheduling does not take into account the underlying storage medium type.

[0028] To this end, the present invention proposes a prediction-based data scheduling method, system and related equipment. Specifically, the prediction-based data scheduling method, system and related equipment of an embodiment of the present invention are described below with reference to the accompanying drawings.

[0029] Figure 1 1 is a flow chart of a prediction-based data scheduling method according to an embodiment of the present invention. It should be noted that the prediction-based data scheduling method of the embodiment of the present invention can be applied to the prediction-based data scheduling system of the embodiment of the present invention. The prediction-based data scheduling system can be configured on an electronic device or in a server. The present application embodiment does not limit this.

[0030] like Figure 1 As shown, the prediction-based data scheduling method includes:

[0031] S110, obtaining an affinity prediction model constructed based on data affinity parameters, and constructing corresponding metadata for the storage partition based on the affinity prediction model.

[0032] In an embodiment of the present invention, data affinity parameters include storage partition parameters, user affinity parameters and time affinity parameters, wherein the storage partition parameters are used to describe the physical storage structure, the user affinity parameters are used to describe the probability that the data is read by the same user, and the time affinity parameters are used to describe the probability that the data is read within a specified time.

[0033] Among them, storage partition parameters include but are not limited to physical location, capacity, speed, etc.; user affinity parameters include but are not limited to users' historical access data and access patterns; time affinity parameters include but are not limited to data access frequency, access time period, etc.

[0034] Among them, due to the huge scale of the storage pool, it is necessary to set the storage partitions in a hierarchical manner. For mechanical hard disks, partitioning is no longer performed by head or sector. In order to reduce the amount of metadata, cylinders are used as the basic storage partition unit, and metadata is established for each cylinder. For solid-state drives, partitioning is no longer performed by particles or pages. In order to reduce the amount of metadata, blocks are used as the basic storage partition unit. For the above basic storage partition units, the corresponding metadata stores various parameters in the form of vectors.

[0035] In an embodiment of the present invention, the implementation method of building an affinity prediction model based on data affinity parameters is as follows: after obtaining the data affinity parameters, the data affinity parameters can be preprocessed, wherein the preprocessing of the data affinity parameters includes cleaning the data affinity parameters (i.e., removing outliers, filling missing values, etc.), converting (i.e., converting non-numerical data into numerical data, etc.) and standardizing (i.e., converting data into a unified metric, etc.); then, according to business needs and the results of data analysis, select features that have an impact on affinity prediction. For example, if it is found that the user's access pattern has a significant impact on the affinity of the data, the user affinity parameter can be used as an important feature; then, based on the selected features and the collected data, the model is trained using a training set; the prediction performance of the model is evaluated using data that is not involved in the training (test set), and commonly used evaluation indicators include accuracy, recall, F1 score, AUC value, etc.

[0036] Among them, metadata can be understood as labels, and building corresponding metadata for storage partitions can be understood as labeling storage partitions. For example, when it is detected that storage partition A is often read by the same user, the label corresponding to storage partition A has a 60% probability of being read by storage partition A.

[0037] S120, determining whether a read request for a target data block is received within a preset time, and based on the determination result, inputting the target data block into an affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more.

[0038] In an embodiment of the present invention, when a read request for a target data block is received within a preset time, the target data block may be input into an affinity prediction model to obtain a predicted affinity data block.

[0039] There are one or more predicted affinity data blocks.

[0040] In an embodiment of the present invention, when no read request for a data block is received within a preset time, a target data block may be customized and input into an affinity prediction model to obtain a predicted affinity data block.

[0041] S130, determining whether the predicted affinity data block is correct, and if it is determined that the predicted affinity data block is correct, determining a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data, and scheduling the first partition and the second partition.

[0042] The corresponding first partition is determined according to the attribute information of the target data block, and the corresponding second partition is determined according to the attribute information of the predicted affinity data block.

[0043] In an embodiment of the present invention, when a predicted affinity data block is obtained, it is possible to determine whether the predicted affinity data block is correct, and based on the determination result, the first partition and the second partition are scheduled, and the metadata of the first partition and the second partition are changed. For specific implementation methods, reference may be made to subsequent embodiments.

[0044] According to the prediction-based data scheduling method of an embodiment of the present invention, an affinity prediction model constructed based on data affinity parameters is obtained, and corresponding metadata is constructed for the storage partition based on the affinity prediction model; it is determined whether a read request for a target data block is received within a preset time, and based on the determination result, the target data block is input into the affinity prediction model to obtain a predicted affinity data block, and the predicted affinity data block is one or more; it is determined whether the predicted affinity data block is correct, and when it is determined that the predicted affinity data block is correct, the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data are determined, and the first partition and the second partition are scheduled. By predicting data affinity, this method can predict in advance the data blocks that may need to be accessed together, so that when scheduling data, the associated data blocks can be scheduled together, reducing the delay of data access and improving the efficiency of data scheduling.

[0045] In order to make it easier for those skilled in the art to understand the present invention, Figure 2 is a prediction-based data scheduling method according to a specific embodiment of the present invention, such as Figure 2 As shown, the prediction-based data scheduling method includes:

[0046] S210, obtaining an affinity prediction model constructed based on data affinity parameters, and constructing corresponding metadata for the storage partition based on the affinity prediction model.

[0047] In the embodiment of the present invention, the implementation of step S210 may refer to the implementation of step S110 described above, and the present invention will not elaborate on this any more.

[0048] S220, determining whether a read request for a target data block is received within a preset time, and based on the determination result, inputting the target data block into an affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more.

[0049] In an embodiment of the present invention, a target data block is input into an affinity prediction model to determine a target user corresponding to the target data block; a data block accessed by the target user is obtained, and the data block accessed by the target user is used as a predicted affinity data block; and / or a target data block is input into an affinity prediction model to determine a target reading time period corresponding to the target data block; a data block accessed within the target reading time period is obtained, and the data block accessed within the target reading time period is used as a predicted affinity data block.

[0050] That is, for user-level affinity prediction: First, the target data block is input into the affinity prediction model to determine which users are most likely to access the target data block. This involves analyzing user access patterns, including but not limited to user access history, preferences, behavioral characteristics, etc. Then, the data blocks visited by the target user are obtained and regarded as predicted affinity data blocks. This means that there is a potential correlation between this predicted affinity data block and the target data block, which may be because they are usually visited by the same user or user group.

[0051] For affinity prediction at the time level: the target data block is input into the affinity prediction model to determine the time period in which the target data block is usually read. This may involve analyzing the periodic pattern of data access, such as some data may be frequently accessed during a specific period of time on weekdays. Then the data blocks that have been accessed during this target reading time period are obtained and used as the predicted affinity data blocks. This indicates that the predicted affinity data blocks are temporally associated with the target data block, perhaps because they are accessed multiple times in the same time period.

[0052] Therefore, the storage and access of data can be optimized by analyzing and predicting user access patterns and time characteristics, thereby improving the performance and efficiency of the storage system.

[0053] S230: Determine whether the predicted affinity data block is correct.

[0054] In an embodiment of the present invention, within a target time, it is determined whether the target data block and the predicted affinity data block are both read by different users or are both read by the same user, so as to determine whether the predicted affinity data block is correct.

[0055] S240 , when it is determined that the predicted affinity data block is correct, determine a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data, and schedule the first partition and the second partition.

[0056] In an embodiment of the present invention, when it is determined that the target data block and the predicted affinity data block are both read by different users or are both read by the same user, it is determined that the predicted affinity data block is correct.

[0057] In one embodiment of the present invention, when it is determined that a read request for a target data block is received within a preset time, a corresponding first partition is determined based on attribute information of the target data block, and a corresponding second partition is determined based on attribute information of a predicted affinity data block; the first partition and the second partition are scheduled for upgrade.

[0058] In another embodiment of the present invention, when it is determined that no read request for the target data block is received within a preset time, a corresponding first partition is determined based on attribute information of the target data block, and a corresponding second partition is determined based on attribute information of the predicted affinity data block; the first partition and the second partition are downgraded.

[0059] For example, in a massive distributed storage system, there are two levels, where the hot storage pool is an all-flash storage pool consisting of 1,000 flash memories, and the cold storage pool is an all-mechanical hard disk storage pool consisting of 5,000 large-capacity mechanical hard disks. For example, there are 10 data blocks, of which A, B, C, D, and E are stored in the hot storage pool, and f, g, h, i, and j are stored in the cold storage pool.

[0060] When the storage system operates normally and receives a read request for the target data block g within a preset time, since the target data block g and the data block j are read by the first user within the target time or the target data block g and the data block j are read by different users within the target time, that is, the affinity data block of the target data block g is the data block j, the first partition of the target data block g is determined, and the second partition of the data block j is determined, and the first partition and the second partition are upgraded and scheduled (that is, the first partition and the second partition are scheduled from the cold storage pool to the hot storage pool), and the metadata of the first partition and the second partition are changed.

[0061] Similarly, if no read request for a data block is received within the preset time, a target data block A can be customized. Since the target data block A, data block C, and data block D are read by the first user within the target time or the target data block A, data block C, and data block D are read by different users within the target time, that is, the affinity data blocks of the target data block A are data block C and data block D, the first partition of the target data block A is determined, and the second partition corresponding to data block C and the second partition corresponding to data block D are determined. The first partition and the second partition are upgraded and scheduled (that is, the first partition and the second partition are scheduled from the hot storage pool to the cold storage pool), and the metadata of the first partition and the second partition are changed at the same time. This improves access efficiency.

[0062] S250, determining a scheduling result, and feeding the scheduling result back to the affinity prediction model to optimize the affinity prediction model.

[0063] In the embodiment of the present invention, when it is determined that the predicted affinity data block is correct, the scheduling result is reasonable; when it is determined that the predicted affinity data block is incorrect, the scheduling result is unreasonable.

[0064] According to the prediction-based data scheduling method of the embodiment of the present invention, by using the affinity prediction model to analyze and predict the access pattern between data blocks (including the correlation at the user level and the time level), it is possible to effectively identify which data blocks should be scheduled together. This data layout optimization based on the access pattern helps to reduce the delay of data access and improve the response speed and overall performance of the storage system; the storage partition is dynamically scheduled (upgraded or downgraded) according to the access situation of the data block, so that frequently accessed data can be read faster, and infrequently accessed data can be migrated to a storage medium with a lower cost, thereby achieving cost-effectiveness optimization; by feeding the scheduling results back to the affinity prediction model, the prediction accuracy of the model can be continuously optimized. This means that the system can self-learn and adapt to the ever-changing data access pattern, and the accuracy of the prediction and the effectiveness of the scheduling decision will continue to improve over time; through accurate data analysis and prediction, storage resources can be allocated and used more reasonably. This includes not only the effective use of physical storage space, but also the reduction of energy consumption and system operation and maintenance costs, because more efficient data processing and access can reduce unnecessary hardware operations and energy consumption.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0066] According to one aspect of an embodiment of the present invention, a prediction-based data scheduling system is also proposed. Figure 3 is a structural block diagram of a prediction-based data scheduling system according to an embodiment of the present invention; Figure 3 As shown, including:

[0067] An acquisition module 310 is used to acquire an affinity prediction model constructed based on data affinity parameters, and to construct corresponding metadata for a storage partition based on the affinity prediction model;

[0068] The obtaining module 320 is used to determine whether a read request of a target data block is received within a preset time, and based on the determination result, input the target data block into the affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more;

[0069] The scheduling module 330 is used to determine whether the predicted affinity data block is correct. When it is determined that the predicted affinity data block is correct, the scheduling module 330 determines the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data, schedules the first partition and the second partition, and changes the metadata of the first partition and the second partition.

[0070] According to the prediction-based data scheduling system of the embodiment of the present invention, an affinity prediction model constructed based on data affinity parameters is obtained, and corresponding metadata is constructed for the storage partition based on the affinity prediction model; it is determined whether a read request for a target data block is received within a preset time, and based on the determination result, the target data block is input into the affinity prediction model to obtain a predicted affinity data block, and the predicted affinity data block is one or more; it is determined whether the predicted affinity data block is correct, and when it is determined that the predicted affinity data block is correct, the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data are determined, and the first partition and the second partition are scheduled. Thus, by predicting data affinity, data blocks that may need to be accessed together can be predicted in advance, so that when scheduling data, related data blocks can be scheduled together, reducing the delay of data access and improving the efficiency of data scheduling.

[0071] Optionally, the data affinity parameters include storage partition parameters, user affinity parameters and time affinity parameters, wherein the storage partition parameters are used to describe the physical storage structure, the user affinity parameters are used to describe the probability of data being read by the same user, and the time affinity parameters are used to describe the probability of data being read within a specified time.

[0072] Optionally, a module 320 is obtained, which is specifically used to input the target data block into the affinity prediction model to determine the target user corresponding to the target data block; obtain the data block accessed by the target user, and use the data block accessed by the target user as the predicted affinity data block; and / or input the target data block into the affinity prediction model to determine the target reading time period corresponding to the target data block; obtain the data block accessed within the target reading time period, and use the data block accessed within the target reading time period as the predicted affinity data block.

[0073] Optionally, the scheduling module 330 is specifically used to determine whether the target data block and the predicted affinity data block are both read by different users or whether they are both read by the same user within the target time; wherein, when it is determined that the target data block and the predicted affinity data block are both read by different users or both read by the same user, it is determined that the predicted affinity data block is correct.

[0074] Optionally, it is determined that a read request for the target data block is received within a preset time, wherein the scheduling module 330 is specifically used to determine the corresponding first partition according to the attribute information of the target data block, and to determine the corresponding second partition according to the attribute information of the predicted affinity data block; and to perform upgrade scheduling on the first partition and the second partition.

[0075] Optionally, it is determined that no read request for the target data block is received within a preset time, wherein the corresponding first partition is determined based on attribute information of the target data block, and the corresponding second partition is determined based on attribute information of the predicted affinity data block; and the first partition and the second partition are downgraded.

[0076] Optionally, the system also includes an optimization module, which is used to determine a scheduling result and feed the scheduling result back to the affinity prediction model so as to optimize the affinity prediction model; wherein, determining the scheduling result includes: when it is determined that the predicted affinity data block is correct, the scheduling result is reasonable; when it is determined that the predicted affinity data block is incorrect, the scheduling result is unreasonable.

[0077] According to one aspect of an embodiment of the present invention, an electronic device is provided.

[0078] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 4 As shown, the electronic device may include one or more ( Figure 4 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the electronic device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only and does not limit the structure of the above terminal device. Figure 4 More or fewer components as shown, or with Figure 4 Equivalent functions or comparisons shown Figure 4 A different configuration with more features is shown.

[0079] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the prediction-based data scheduling method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the switching device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0081] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a prediction-based data scheduling method.

[0082] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0083] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0084] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A prediction-based data scheduling method, characterized in that: include: S1, obtaining an affinity prediction model constructed based on data affinity parameters, and constructing corresponding metadata for the storage partition based on the affinity prediction model; S2, determining whether a read request for a target data block is received within a preset time, and based on the determination result, inputting the target data block into the affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more; S3, judging whether the predicted affinity data block is correct. If it is determined that the predicted affinity data block is correct, determining the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data, scheduling the first partition and the second partition, and changing the metadata of the first partition and the second partition.

2. The prediction-based data scheduling method according to claim 1, characterized in that: The data affinity parameters include storage partition parameters, user affinity parameters and time affinity parameters, wherein the storage partition parameters are used to describe the physical storage structure, the user affinity parameters are used to describe the probability of data being read by the same user, and the time affinity parameters are used to describe the probability of data being read within a specified time.

3. The prediction-based data scheduling method according to claim 1, characterized in that: Inputting the target data block into the affinity prediction model to obtain a predicted affinity data block includes: Inputting the target data block into the affinity prediction model to determine a target user corresponding to accessing the target data block; Acquire the data block accessed by the target user, and use the data block accessed by the target user as the predicted affinity data block; and / or Inputting the target data block into the affinity prediction model to determine a target reading time period corresponding to the target data block; The data blocks accessed within the target reading time period are acquired, and the data blocks accessed within the target reading time period are used as the predicted affinity data blocks.

4. The prediction-based data scheduling method according to claim 1, characterized in that: Determining whether the predicted affinity data block is correct includes: Within a target time, determining whether the target data block and the predicted affinity data block are both read by different users or are both read by the same user; Wherein, determining that the predicted affinity data block is correct includes: In the case that it is determined that the target data block and the predicted affinity data block are both read by different users or are both read by the same user, it is determined that the predicted affinity data block is correct.

5. The prediction-based data scheduling method according to claim 1, characterized in that: Determining that a read request for the target data block is received within a preset time, wherein a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data are determined, and the first partition and the second partition are scheduled, including: Determine the first partition corresponding to the target data block according to the attribute information, and determine the second partition corresponding to the predicted affinity data block according to the attribute information; The first partition and the second partition are scheduled for upgrade.

6. The prediction-based data scheduling method according to claim 1, characterized in that: Determining that no read request for the target data block is received within a preset time, wherein determining a first partition corresponding to the target data block and a second partition corresponding to the predicted affinity data, and scheduling the first partition and the second partition, comprises: Determine the first partition corresponding to the target data block according to the attribute information, and determine the second partition corresponding to the predicted affinity data block according to the attribute information; The first partition and the second partition are downgraded and scheduled.

7. The prediction-based data scheduling method according to claim 1, characterized in that: The method further comprises: Determining a scheduling result, and feeding the scheduling result back to the affinity prediction model so as to optimize the affinity prediction model; Determining the scheduling result includes: In the case where it is determined that the predicted affinity data block is correct, the scheduling result is reasonable; In the case where it is determined that the predicted affinity data block is incorrect, the scheduling result is unreasonable.

8. A prediction-based data scheduling system, characterized in that: include: An acquisition module, used to acquire an affinity prediction model constructed based on data affinity parameters, and construct corresponding metadata for the storage partition based on the affinity prediction model; an obtaining module, used for judging whether a read request of a target data block is received within a preset time, and based on the judgment result, inputting the target data block into the affinity prediction model to obtain a predicted affinity data block, wherein the predicted affinity data block is one or more; A scheduling module is used to determine whether the predicted affinity data block is correct. When it is determined that the predicted affinity data block is correct, the scheduling module determines the first partition corresponding to the target data block and the second partition corresponding to the predicted affinity data, schedules the first partition and the second partition, and changes the metadata of the first partition and the second partition.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.