A data processing method and apparatus

Abstract

The application discloses a data processing method and device, comprising: obtaining to-be-processed data; determining a first storage group mapped with the to-be-processed data from each storage group based on a first model; wherein any storage group has a mapping relationship between the storage group and a storage medium; the mapping relationship of any storage group is periodically determined based on a second model and state information of each storage medium; determining a first storage medium from each storage medium according to the mapping relationship of the first storage group; and mapping the to-be-processed data to the first storage medium. Based on the first model, random factors are reduced, and high-frequency small data distribution is realized; based on the second model, the mapping relationship of the storage group is updated through a low-frequency large data adjustment mode, and the balance, flexibility and efficiency of data processing are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] With the development of computer network technology, the amount of data is gradually increasing. In order to better manage data, it is necessary to store data on storage media such as physical hard drives, distributed databases, and cloud servers.

[0003] Currently, data processing typically involves calculating the data to be stored using a pre-defined algorithm to obtain the calculation results; determining the target placement group based on the calculation results; mapping the data to be stored to the target placement group; and then determining the target storage medium based on the mapping rules of the target placement group, and finally mapping the data to be stored to the target storage medium.

[0004] In the above methods, the calculation results obtained according to the preset algorithm have random factors; for example, if the preset algorithm is a modulo algorithm, multiple data to be stored may have the same calculation result; or different calculation results may point to the same placement group; thus, multiple data may point to the same placement group, causing uneven data distribution and resulting in uneven data processing. The method of determining the target storage medium based on the mapping rules of the placement group has poor flexibility and cannot flexibly update the mapping rules of the placement group, resulting in poor load balancing of the storage medium and poor data processing balance, which in turn affects the efficiency of data processing. Summary of the Invention

[0005] This invention provides a data processing method and apparatus to improve the balance, efficiency, and flexibility of data processing.

[0006] In a first aspect, embodiments of the present invention provide a data processing method, comprising:

[0007] Obtain the data to be processed;

[0008] Based on the first model, a first placement group is determined from each placement group and mapped to the data to be processed; each placement group has a mapping relationship between the placement group and the storage medium; the mapping relationship of each placement group is determined periodically based on the second model and the state information of each storage medium;

[0009] Based on the mapping relationship of the first placement group, the first storage medium is determined from each of the storage media;

[0010] The data to be processed is mapped to the first storage medium.

[0011] In the above technical solution, the first model is a reinforcement learning model; the first model is trained based on historical storage data and preset optimization objectives; the preset optimization objectives include, but are not limited to: balancing the load of each placement group, balancing the load of each storage medium, reducing the response time of data processing, data balancing caused by the update of the mapping relationship of the placement group, and custom objectives.

[0012] By inputting the data to be processed and the status information of each storage medium into a first model, the first model outputs a first placement group mapped to the data to be processed, and then the first placement group mapped to the data to be processed is determined by the first model. The status information of each storage medium can be collected in real time or is historical status data collected periodically.

[0013] The first model eliminates the need to determine the placement group mapped to the data to be processed based on a preset algorithm. Since the input of the first model includes the state information of each storage medium, the high-frequency, low-volume data distribution method based on the first model can reduce random factors, improve the uniformity and efficiency of data distribution, and thus improve the balance and efficiency of data processing.

[0014] In addition, the mapping relationship of any placement group is determined periodically based on the second model and the status information of each storage medium, so as to realize flexible updating of the mapping relationship of the placement group, and ensure the load balance of the storage medium and the balance of data processing of the storage medium through low-frequency large-volume adjustment, thereby improving the balance and efficiency of data processing.

[0015] Optionally, the method further includes:

[0016] Collect the status information of each storage medium at the current moment;

[0017] The state information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0018] The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0019] In the above technical solution, the second model is a reinforcement learning model. The second model is obtained after training based on the mapping relationship between the placement group and the storage medium, and a preset optimization objective. The second model is used to output the probability that the mapping relationship of the placement group needs to be updated based on the current state information of each storage medium, thereby achieving flexible updates to the mapping relationship of the placement group. By adjusting the data volume in a low-frequency, high-frequency manner, the load balance of the storage medium and the balance of data processing by the storage medium are ensured, thereby improving the balance and efficiency of data processing.

[0020] Optionally, the mapping relationship of the placement groups is updated according to the update probability of each placement group, the mapping probability of each storage medium, and the migration probability, including:

[0021] When the migration probability is greater than the migration threshold, a second placement group is determined based on the update probability of each placement group;

[0022] The mapping relationship of the second placement group is updated according to the mapping probability of each storage medium.

[0023] Optionally, a second placement group is determined based on the update probability of each placement group, including:

[0024] The placement group with the highest update probability is determined as the second placement group;

[0025] Updating the mapping relationship of the second placement group according to the mapping probability of each storage medium includes:

[0026] The second storage medium is determined according to the mapping probability of each storage medium, following the rule from smallest to largest.

[0027] The updated mapping relationship of the second placement group is determined based on the second storage medium.

[0028] In a second aspect, embodiments of the present invention provide a data processing apparatus, comprising:

[0029] The acquisition module is used to acquire data to be processed.

[0030] The processing module is used to determine a first placement group that maps to the data to be processed from each placement group based on a first model; each placement group has a mapping relationship between the placement group and the storage medium; the mapping relationship of each placement group is determined periodically based on a second model and the status information of each storage medium;

[0031] Based on the mapping relationship of the first placement group, the first storage medium is determined from each of the storage media;

[0032] The data to be processed is mapped to the first storage medium.

[0033] Optionally, the processing module is further configured to:

[0034] Collect the status information of each storage medium at the current moment;

[0035] The state information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability; wherein, the mapping probability of the storage medium is inversely proportional to the load of the storage medium;

[0036] The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0037] Optionally, the processing module is specifically used for:

[0038] When the migration probability is greater than the migration threshold, a second placement group is determined based on the update probability of each placement group;

[0039] The mapping relationship of the second placement group is updated according to the mapping probability of each storage medium.

[0040] Optionally, the processing module is specifically used for:

[0041] The placement group with the highest update probability is determined as the second placement group;

[0042] Updating the mapping relationship of the second placement group according to the mapping probability of each storage medium includes:

[0043] The second storage medium is determined according to the mapping probability of each storage medium, following the rule from smallest to largest.

[0044] The updated mapping relationship of the second placement group is determined based on the second storage medium.

[0045] Thirdly, embodiments of the present invention also provide a computer device, comprising:

[0046] Memory, used to store program instructions;

[0047] The processor is used to call the program instructions stored in the memory and execute the above data processing method according to the obtained program.

[0048] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described data processing method. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present invention;

[0051] Figure 2A flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram illustrating a data processing method provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram illustrating an updated placement group mapping relationship provided in an embodiment of the present invention;

[0054] Figure 5 A schematic diagram illustrating a data processing method provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0057] In a big data environment, the scale and complexity of data typically increase rapidly. To meet the needs of large-scale data storage and ensure data security, data needs to be distributed across multiple storage media.

[0058] Currently, distributed storage systems enable the decentralized storage of data, distributing it across multiple physical hard drives, storage servers, cloud storage spaces, and other storage media. This allows each hard drive and storage server to handle the cluster load without becoming a bottleneck due to excessive pressure on a single drive or server, and also enables cluster expansion and contraction.

[0059] To ensure data processing efficiency and balance, it is necessary to design data processing methods that allow data to be evenly mapped across multiple storage media.

[0060] In some embodiments, the data to be stored is calculated using a preset algorithm to obtain the calculation result; a target placement group is determined based on the calculation result, and the data to be stored is then mapped to the target placement group; then, the target storage medium is determined according to the mapping rules of the target placement group, and the data to be stored is then mapped to the target storage medium.

[0061] For example, the default algorithm is consistent hashing. The storage space in consistent hashing can be abstracted as a ring, evenly divided into multiple intervals; these intervals are equivalent to multiple placement groups. For any given interval, a mapping rule can be set to represent the storage medium corresponding to that interval.

[0062] After acquiring the data to be processed, a hash operation is performed on the data to obtain a hash value. The hash value is then mapped to a specific interval on the interval in a clockwise direction, and this interval is determined as the target storage group for the data to be processed. The storage medium corresponding to this interval is then used as the target storage medium for the data to be processed, thus mapping the data to the target storage medium.

[0063] To illustrate again, the default algorithm is the modulo algorithm. Multiple placement groups are pre-defined, and each placement group has a mapping rule. After acquiring the data to be processed, a hash operation is performed on the data to obtain a hash value. The hash value is then moduloed according to the number of placement groups to obtain the modulo result. Based on the modulo result, the target placement group is determined from among the multiple placement groups. Then, the target storage medium is determined according to the mapping rule of the target placement group, and the data to be stored is mapped to the target storage medium.

[0064] However, the calculation results obtained according to the preset algorithm in the above method have random factors. Based on the above embodiments, assuming that multiple data points to the same interval after being processed by the consistent hash algorithm, it means that all the data are mapped to the storage medium corresponding to that interval. Assuming that the modulo results of multiple data points are the same, it means that the multiple data points point to the same placement group, and all the data need to be mapped to the storage medium corresponding to that placement group.

[0065] In other words, because the preset algorithm contains random factors, the distribution of data to be processed among the placement groups is uneven, which in turn leads to uneven distribution of the storage media mapped to the data, resulting in unbalanced data distribution and processing. Furthermore, during expansion or contraction, a large amount of unnecessary data migration occurs, impacting data processing efficiency.

[0066] Furthermore, the mapping rules for any placement group are pre-defined and cannot be flexibly updated during the data mapping process. This can easily lead to some storage media having a high load while others have a low load, resulting in poor load balancing and uneven data processing across the storage media, which in turn affects the overall balance and efficiency of data processing.

[0067] Therefore, there is an urgent need for a data processing method to improve the balance, efficiency, and flexibility of data processing.

[0068] Figure 1An exemplary system architecture applicable to an embodiment of the present invention is shown. The system architecture includes a server 100, which may include a processor 110, a communication interface 120, and a memory 130.

[0069] The communication interface 120 is used to obtain the data to be processed written by the user.

[0070] The processor 110 is the control center of the server 100, connecting various parts of the server 100 through various interfaces and routes. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 130, and by calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units.

[0071] The memory 130 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 130. The memory 130 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to business processing, etc. In addition, the memory 130 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0072] It should be noted that the above Figure 1 The structure shown is merely an example, and the embodiments of the present invention are not limited thereto.

[0073] Based on the above description Figure 2 An exemplary flowchart of a data processing method provided by an embodiment of the present invention is shown, which can be executed by a data processing device.

[0074] like Figure 2 As shown, the process specifically includes:

[0075] Step 210: Obtain the data to be processed.

[0076] In this embodiment of the invention, the data to be processed can be in the form of data blocks, data streams, bytes, etc.; specifically, the data to be processed can be image data, sound data, text, messages, tables, etc., without being specifically limited here.

[0077] Step 220: Based on the first model, determine the first placement group that maps to the data to be processed from each placement group.

[0078] In this embodiment of the invention, any placement group has a mapping relationship between the placement group and the storage medium; for example, placement group g1 has a mapping relationship with storage medium c1, storage medium c2 and storage medium c3.

[0079] In some embodiments, the input parameters of the first model include the data to be processed and the status data of each storage medium. The status information of the storage media includes the number of storage media, the load status of each storage media, the type of each storage media, the capacity of each storage media, and the number of CPU (central processing unit) cores of each storage media.

[0080] Because any placement group has a mapping relationship between the placement group and the storage medium, when the input parameters of the first model include the status information of each storage medium, the first model will take the placement group that has a better mapping relationship with the storage medium with better status information (such as low load, high efficiency, etc.) as the output result.

[0081] For example, there is a mapping relationship between placement group g1 and storage media c1, c2 and c3; there is a mapping relationship between placement group g2 and storage media c2, c3 and c4; wherein, the status information of storage media c4 is better than the status information of storage media c1 (e.g., the load of storage media c4 is less than that of storage media c1).

[0082] After inputting the data to be processed and the status information of each storage medium (storage medium c1, storage medium c2, storage medium c3 and storage medium c4) into the first model, the output of the first model is the placement group g2, that is, the placement group g2 is the first placement group mapped to the data to be processed.

[0083] It should be noted that the comparison of the status information of various storage media is not specifically limited here, and may include a comparison method that combines one or more data such as load, memory, processing speed, and bandwidth.

[0084] Based on the above description, by inputting the data to be processed and the status information of each storage medium into the first model, the placement group identifier output by the first model is obtained; then, the first placement group mapped to the data to be processed is determined by the placement group identifier, thus eliminating the need to determine the placement group mapped to the data to be processed according to a preset algorithm. Based on the first model, a high-frequency, low-volume data distribution method is implemented to reduce random factors, improve the uniformity and efficiency of data allocation, and thereby improve the balance and efficiency of data processing.

[0085] In some embodiments, after determining a first placement group that maps to the data to be processed, the data to be processed is mapped to the first placement group.

[0086] Step 230: Determine the first storage medium from each storage medium according to the mapping relationship of the first placement group.

[0087] In this embodiment of the invention, the mapping relationship of the first placement group is updated based on the second model, thereby enabling flexible changes to the mapping relationship between the first placement group and each storage medium. For example, before the mapping relationship of the first placement group is updated, there are mapping relationships between the first placement group and storage media c2, c3, and c4. After the mapping relationship of the first placement group is updated, there are mapping relationships between the first placement group and storage media c6, c7, and c8.

[0088] Step 240: Map the data to be processed to the first storage medium.

[0089] In this embodiment of the invention, since the data to be processed in step 220 is mapped to the first placement group, the data to be processed is mapped to the first storage medium by mapping the data in the first placement group to the first storage medium.

[0090] In some embodiments, the first model and the second model include, but are not limited to, having the following parameter information; wherein, the following parameter information includes the status information of the storage medium.

[0091] Hardware parameters: The hardware parameters of each storage medium. These include, but are not limited to, the number of CPU cores, CPU frequency, memory size, bandwidth, hardware type, and capacity of the storage medium.

[0092] Load parameters: Load information for each storage medium. This includes, but is not limited to, CPU utilization, memory usage, average traffic used over a preset time period (e.g., 24 hours), remaining memory capacity, IOPS (Input / Output Operations Per Second), average bandwidth, and processing latency over a preset time period (e.g., 24 hours).

[0093] Placement group parameters include, but are not limited to, the preset number of placement groups, the number of placement groups currently in use, the mapping relationship between each placement group and the storage medium, and the capacity of any placement group.

[0094] Historical parameters: Set historical data processing records for a historical period (such as the week or month before the current time) to be used as training samples for the first and second models.

[0095] In this embodiment of the invention, both the first model and the second model are reinforcement learning models; wherein, reinforcement learning refers to taking actions based on feedback from the environment, and through continuous interaction and trial and error with the environment, to achieve the learning objective or to make the action achieve the maximum optimization objective among the preset optimization objectives.

[0096] Furthermore, the main body of reinforcement learning training is the agent (intelligent agent). Reinforcement learning training also includes the training environment, the current state of the training environment and the agent, the actions that the agent can perform in the current state, and the feedback obtained by the agent when it takes any action in the current state.

[0097] In this embodiment of the invention, the first model and the second model serve as agents of the reinforcement learning model.

[0098] For the first model, the training environment of the first model is the first placement group corresponding to the data to be processed; the current state of the training environment and the agent is: whether the first model is input with the data to be processed and the state information of each storage medium.

[0099] Furthermore, assuming the current state is: the first model is input with the data to be processed and the state information of each storage medium, then the actions that the agent can perform in the current state are: the first model selects each placement group as the first placement group; the feedback obtained by the agent taking any action in the current state is: the first model selects each placement group as the first placement group, and obtains the first feedback results corresponding to the feedback from the training environment. The first feedback results represent information such as the state and load of each storage medium and each placement group after the first placement group is determined.

[0100] For the second model, the training environment is to determine the mapping relationship between any placement group and the storage medium; the current training environment and the state of the agent are: the second model determines whether to input the state information of each storage medium.

[0101] Furthermore, assuming the current state is: the second model receives the state information of each storage medium as input, then the actions that the agent can perform in the current state are: the second model selects each placement group as the second placement group whose mapping relationship needs to be updated, and determines the updated mapping relationship of the second placement group; the feedback obtained by the agent taking any action in the current state is: the second model selects each placement group as the second placement group, and obtains the corresponding second feedback results from the training environment. The second feedback results represent the state, load, and other information of each storage medium and each placement group after determining the second placement group and updating the mapping relationship of the second placement group.

[0102] Based on the above description, the first model is trained using the parameter information described above and historical data as training samples, and converges based on a preset optimization objective. The second model is trained using the above parameter information (such as the number of placement groups and the mapping relationship between each placement group) as training samples, and converges based on a preset optimization objective.

[0103] In this embodiment of the invention, the preset optimization objectives of the first model and the second model are the same, including but not limited to the following preset optimization objectives:

[0104] Balance the load of each placement group, balance the load of each storage medium, reduce the response time of data processing, balance data due to the update of the mapping relationship of placement groups, and customize targets.

[0105] The load balancing of each placement group can be represented by the variance of the number of data mapped to each placement group. Variance is inversely proportional to optimization performance; that is, the larger the variance, the worse the optimization performance (i.e., the smaller the reward).

[0106] Balancing the load across storage media can be represented by the variance of preset metrics for each placement group and each storage media; for example, the variance of CPU utilization for each storage media, the variance of the proportion of used memory to available memory for each storage media, etc.; where variance is inversely proportional to optimization performance. That is, the larger the variance, the worse the optimization performance (i.e., the less reward).

[0107] Reducing data processing response time can be represented by an aggregated value of the data processing response time. The aggregated value is inversely proportional to the optimization effect; that is, the smaller the aggregated value, the better the optimization effect (i.e., the greater the reward).

[0108] Custom goals can include the quantity of data used to help recover data from a failed server. The quantity is directly proportional to the optimization effect; that is, the larger the quantity, the better the optimization effect (i.e., the greater the reward).

[0109] In this embodiment of the invention, when the first model and the second model converge, weights can be set for a preset optimization target, thereby increasing the training flexibility and accuracy of the first model and the second model. For example, the weight for balancing the load of each placement group can be set to 1.3, and the weight for balancing the load of each storage medium can be set to 1.5, etc.

[0110] In this embodiment of the invention, the first model and the second model employ imitation learning during model training, such as using a data placement method for training, thereby avoiding unnecessary agent training and saving training time.

[0111] In some real-time methods, training samples can be generated by collecting data from other storage clusters as training samples or by using tools such as DaDiSi to simulate the workload of storage clusters, thereby increasing the amount of training sample data and improving the training accuracy of the first and second models.

[0112] In other real-time methods, selective pruning of agent actions is performed during the training of the first and second models. This reduces poorly performing decisions, improves the training efficiency of both models, and reduces their training error rates. For example, data replicas are prevented from being placed in large placement groups to avoid storage cluster failures.

[0113] In some real-time methods, the number of training samples for the first model is generally greater than that for the second model. Therefore, during model training, the network layers that extract parameter features in the second model should be kept consistent with those in the first model. This reduces the training load of the second model and increases its training accuracy.

[0114] To better illustrate the above technical solution, Figure 3 An exemplary schematic diagram of a data processing method is shown, such as... Figure 3 As shown, the placement groups include placement group g1, placement group g2, and placement group g3. The storage media include storage media c1, storage media c2, storage media c3, and storage media c4.

[0115] After obtaining the data block k to be processed, input the data block k into the first model to obtain the placement group identifier; if the placement group identifier is g1, then the placement group g1 is determined as the first placement group mapped to the data block k. Then the data block k is mapped to the placement group g1.

[0116] like Figure 3 As shown, the storage media that have a mapping relationship with the placement group g1 include storage media c1, storage media c2, and storage media c3. Therefore, storage media c1, storage media c2, and storage media c3 are determined as the first storage media. Then, the data block k in the placement group g1 is mapped to storage media c1, storage media c2, and storage media c3.

[0117] It should be noted that, Figure 3 As an example only, this embodiment of the invention does not limit the number of placement groups, the number of storage media, or the number of storage media that have a mapping relationship with placement groups.

[0118] In some embodiments, the mapping relationship of any placement group is periodically determined based on the second model and the state information of each storage medium. The mapping relationship of the placement groups can be periodically updated through the second model, thereby ensuring the balance and flexibility of data processing.

[0119] Specifically, the status information of each storage medium at the current moment is collected; the status information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability; the mapping relationship of the placement group is updated according to the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0120] The update probability of each placement group is used to determine the second placement group to be updated in the placement group; the mapping probability of each storage medium is used as the mapping relationship after the second placement group is updated; and the migration probability is used to determine whether to update the mapping relationship of the second placement group.

[0121] Furthermore, when the migration probability is greater than the migration threshold, the second placement group is determined based on the update probability of each placement group; the mapping relationship of the second placement group is updated based on the mapping probability of each storage medium.

[0122] For example, assuming the migration threshold is 90%, if the migration probability is greater than 90%, it means that the mapping relationship of the placement group needs to be updated, thereby determining the second placement group to be updated, and then updating the mapping relationship of the second placement group according to the mapping probability of each storage medium. It should be noted that the migration threshold can be a value preset based on experience, such as 90%, 85%, etc., and is not specifically limited here.

[0123] In some feasible implementations, the second placement group is determined according to a preset rule based on the update probability of each placement group. For example, if the preset rule is an order from largest to smallest, then the three placement groups with the highest, second highest, and third highest update probabilities are determined as the second placement group, following the order of update probabilities from largest to smallest. In this embodiment of the invention, the placement group with the highest update probability is determined as the second placement group.

[0124] In some feasible implementations, the second storage medium can be determined according to a preset rule based on the mapping probability of each storage medium. In this invention, the second storage medium is determined according to a rule from smallest to largest mapping probability. For example, the three placement groups with the highest, second highest, and third highest mapping probabilities are determined as the second placement group. In some embodiments, the mapping probability of a storage medium can reflect its performance to some extent, such as the storage medium with a higher mapping probability having a lower load or larger memory capacity.

[0125] Based on the above Figure 3 , Figure 4 An exemplary diagram illustrates an update of the placement group mapping relationship, such as... Figure 4As shown, the status information of storage media c1, c2, c3, and c4 at the current moment is collected; this status information is used to indicate the load of each storage medium.

[0126] Input the status information of storage media c1, c2, c3, and c4 into the second model to obtain the mapping parameters.

[0127] The mapping parameters include the update probability of each placement group, the mapping probability of each storage medium, and the migration probability. Assume the update probability of placement group g1 is 87%, placement group g2 is 68%, and placement group g3 is 57%; the mapping probability of storage medium c1 is 91%, storage medium c2 is 62%, storage medium c3 is 78%, and storage medium c4 is 87%; and the migration probability is 91%.

[0128] If the migration probability (91%) is greater than 90% (migration threshold), the placement group g1 with the highest update probability is selected as the first placement group to be updated.

[0129] The storage medium c1 with the highest mapping probability, c3 with the second highest mapping probability, and c4 with the third highest mapping probability are selected as the second storage mediums. The updated placement group g1 is then determined to have a mapping relationship with storage media c1, c3, and c4.

[0130] Based on the above Figure 3 , Figure 5 An exemplary schematic diagram of a data processing method is shown. For example... Figure 5 As shown, after obtaining the data block k+1 to be processed, the data block k+1 is input into the first model to obtain the placement group identifier; if the placement group identifier is g1, then the placement group g1 is determined as the first placement group mapped to the data block k+1, and then the data block k+1 is mapped to the placement group g1.

[0131] like Figure 5 As shown, the storage media that have a mapping relationship with the placement group g1 at this time include storage media c1, storage media c3, and storage media c4. Therefore, storage media c1, storage media c3, and storage media c4 are determined as the first storage media. Then, data block k+1 in the placement group g1 is mapped to storage media c1, storage media c3, and storage media c4.

[0132] From the above Figure 3 It can be seen that the original placement group g1 has a mapping relationship with storage media c1, c2, and c3. (From...) Figure 5 It can be seen that the updated placement group g1 has a mapping relationship with storage media c1, storage media c3, and storage media c4.

[0133] In some embodiments, the mapping probability can reflect the performance of the storage medium to a certain extent; that is, the higher the mapping probability, the higher the performance of the storage medium. The performance of the storage medium includes various factors, such as load, memory size, occupied memory size, and free memory size, which are not specifically limited here.

[0134] Therefore, by flexibly updating the mapping relationship of the placement group based on the mapping probability, the load balance of the storage medium and the balance of data processing of the storage medium are guaranteed, thereby improving the balance and efficiency of data processing.

[0135] Based on the same technological concept Figure 6 An exemplary schematic diagram of a data processing apparatus provided in an embodiment of the present invention is shown, which can execute the flow of a data processing method.

[0136] like Figure 6 As shown, the device specifically includes:

[0137] The acquisition module 610 is used to acquire data to be processed;

[0138] The processing module 620 is used to determine a first placement group that maps to the data to be processed from each placement group based on a first model; each placement group has a mapping relationship between the placement group and the storage medium; the mapping relationship of each placement group is determined periodically based on a second model and the state information of each storage medium;

[0139] Based on the mapping relationship of the first placement group, the first storage medium is determined from each of the storage media;

[0140] The data to be processed is mapped to the first storage medium.

[0141] Optionally, the processing module 620 is further configured to:

[0142] Collect the status information of each storage medium at the current moment;

[0143] The state information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0144] The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

[0145] Optionally, the processing module 620 is specifically used for:

[0146] When the migration probability is greater than the migration threshold, a second placement group is determined based on the update probability of each placement group;

[0147] The mapping relationship of the second placement group is updated according to the mapping probability of each storage medium.

[0148] Optionally, the processing module 620 is specifically used for:

[0149] The placement group with the highest update probability is determined as the second placement group;

[0150] Updating the mapping relationship of the second placement group according to the mapping probability of each storage medium includes:

[0151] The second storage medium is determined according to the mapping probability of each storage medium, following the rule from smallest to largest.

[0152] The updated mapping relationship of the second placement group is determined based on the second storage medium.

[0153] Based on the same technical concept, embodiments of the present invention also provide a computer device, including:

[0154] Memory, used to store program instructions;

[0155] The processor is used to call the program instructions stored in the memory and execute the above data processing method according to the obtained program.

[0156] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-described data processing method.

[0157] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, include: Obtain the data to be processed; The data to be processed and the status information of each storage medium collected in real time are input into the first model, and the first model outputs the first placement group that is mapped to the data to be processed. Each placement group has a mapping relationship between the placement group and the storage medium; The mapping relationship of any placement group is determined periodically based on the second model and the state information of each storage medium; The first model and the second model are reinforcement learning models; The first model and the second model have the same preset optimization goal, which includes: balancing the load of each placement group and balancing the load of each storage medium; Based on the mapping relationship of the first placement group, the first storage medium is determined from each of the storage media; The data to be processed is mapped to the first storage medium.

2. The method as described in claim 1, characterized in that, The method further includes: Collect the status information of each storage medium at the current moment; The state information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability. The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

3. The method as described in claim 2, characterized in that, The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability, including: When the migration probability is greater than the migration threshold, a second placement group is determined based on the update probability of each placement group; The mapping relationship of the second placement group is updated according to the mapping probability of each storage medium.

4. The method as described in claim 3, characterized in that, The second placement group is determined based on the update probability of each placement group, including: The placement group with the highest update probability is determined as the second placement group; Updating the mapping relationship of the second placement group according to the mapping probability of each storage medium includes: The second storage medium is determined according to the mapping probability of each storage medium, following the rule from smallest to largest. The updated mapping relationship of the second placement group is determined based on the second storage medium.

5. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire data to be processed. The processing module is used to input the data to be processed and the status information of each storage medium collected in real time into the first model, and the first model outputs a first placement group that is mapped to the data to be processed. Each placement group has a mapping relationship between the placement group and the storage medium; The mapping relationship of any placement group is determined periodically based on the second model and the state information of each storage medium; The first model and the second model are reinforcement learning models; The first model and the second model have the same preset optimization goal, which includes: balancing the load of each placement group and balancing the load of each storage medium; Based on the mapping relationship of the first placement group, the first storage medium is determined from each of the storage media; The data to be processed is mapped to the first storage medium.

6. The apparatus as claimed in claim 5, characterized in that, The processing module is also used for: Collect the status information of each storage medium at the current moment; The state information of each storage medium at the current moment is input into the second model to obtain the update probability of each placement group, the mapping probability of each storage medium, and the migration probability. The mapping relationship of the placement groups is updated based on the update probability of each placement group, the mapping probability of each storage medium, and the migration probability.

7. The apparatus as claimed in claim 6, characterized in that, The processing module is specifically used for: When the migration probability is greater than the migration threshold, a second placement group is determined based on the update probability of each placement group; The mapping relationship of the second placement group is updated according to the mapping probability of each storage medium.

8. The apparatus as claimed in claim 7, characterized in that, The processing module is specifically used for: The placement group with the highest update probability is determined as the second placement group; Updating the mapping relationship of the second placement group according to the mapping probability of each storage medium includes: The second storage medium is determined according to the mapping probability of each storage medium, following the rule from smallest to largest. The updated mapping relationship of the second placement group is determined based on the second storage medium.

9. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method according to any one of claims 1 to 4.

Images