Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium. The method comprises the following steps: after a data processing event for target log data in a log storage node is detected, screening out associated log data of the target log data from log data stored in the log storage node according to a data storage position corresponding to the target log data; wherein the data storage position corresponding to each log data is the data storage position of the to-be-stored data contained in the log data in the data storage node; merging the data storage position corresponding to the target log data and the data storage position corresponding to the associated log data to obtain a merged storage position; and according to the merged storage position, merging the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data, and migrating to the data storage node. The method can improve the utilization rate of storage resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] Block storage is a storage mode of providing a user with a bare disk space, and the user can directly perform read and write operations on the disk or format the disk into a file system to perform file read and write operations through a network connection to a block storage server. The block storage needs to ensure high reliability of user data storage on the server, and cannot cause user data loss due to node or disk failure.

[0003] At present, the most commonly used data guarantee mechanism is a three-copy mode, which copies user data three times and stores them in different fault domain machine nodes. In order to keep each copy data consistent and reliably recover after a breakdown, a Write-Ahead Log (WAL) mode can be used for auxiliary protection, that is, the user's data operation log is first written into a WAL partition, and a copy of the data is saved in the memory. When the amount of data in the memory reaches a certain level, the data in the memory is updated to a data block in a non-modifiable state, and then the data block is persisted to the disk.

[0004] However, during the data block persistence process, the data block may contain data that has been overwritten or deleted by subsequent operations, resulting in the saving of redundant data to the disk, occupying a lot of storage space, and low storage resource utilization. SUMMARY

[0005] Therefore, it is necessary to provide a data processing method, device, computer equipment and storage medium capable of improving storage resource utilization to solve the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] After detecting a data processing event for target log data in a log storage node, filtering out associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is a data storage location of the data to be stored in the data storage node included in the log data;

[0008] Merging the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location;

[0009] According to the merging storage location, the target log data corresponding to-be-stored data and the associated log data corresponding to-be-stored data are merged and then migrated to the data storage node.

[0010] In one of the embodiments, the method further comprises:

[0011] For each to-be-stored data, according to the storage condition of the log storage node, a log storage location corresponding to the to-be-stored data is determined;

[0012] According to the data storage location corresponding to the to-be-stored data and the log storage location, metadata of the to-be-stored data is generated;

[0013] The metadata and the to-be-stored data are spliced to obtain log data of the to-be-stored data.

[0014] In one of the embodiments, the data storage location corresponding to the to-be-stored data comprises a first starting storage location; after the metadata and the to-be-stored data are spliced to obtain the log data of the to-be-stored data, the method further comprises:

[0015] According to the position sequence number of the first starting storage location, a log index of the log data is determined;

[0016] The log index is saved in a log index table corresponding to the first starting storage location; wherein the log index table is used to save log indexes from a first storage location to a second storage location, and the first starting storage location is located between the first storage location and the second storage location.

[0017] In one of the embodiments, the log indexes in the log index table are arranged in size order; and the saving of the log index in the log index table corresponding to the starting storage location comprises:

[0018] Taking the log index as a target log index, a left index and a right index of the target log index are searched from the log index table corresponding to the starting storage location; wherein the left index is a log index among the log indexes smaller than the target log index, and the difference between the target log index and the left index is the smallest; and the right index is a log index among the log indexes larger than the target log index, and the difference between the target log index and the right index is the smallest;

[0019] The target log index is inserted between the left index and the right index.

[0020] In one of the embodiments, after the merging of the target log data corresponding to-be-stored data and the associated log data corresponding to-be-stored data, the method further comprises:

[0021] deleting the target log data and the associated log data from the log storage node; and

[0022] deleting target log indexes of the target log data and target log indexes of the associated log data from corresponding log index tables.

[0023] In one of the embodiments, the detecting the data processing event for the target log data in the log storage node comprises:

[0024] In a case where a number of log indexes in any log index table is greater than a number threshold, determining that log data corresponding to a first log index in the log index table is the target log data, and determining that the data processing event for the target log data in the log storage node is detected.

[0025] In one of the embodiments, the method further comprises:

[0026] saving the metadata to a metadata list;

[0027] The detecting the data processing event for the target log data in the log storage node comprises:

[0028] In a case where a saving duration of a first metadata in the metadata list is greater than a duration threshold, determining that log data corresponding to the first metadata is the target log data, and determining that the data processing event for the target log data in the log storage node is detected.

[0029] In one of the embodiments, the method further comprises:

[0030] receiving a data query request of a first object; wherein the data query request comprises a second starting storage location of to-be-queried data in the data storage node;

[0031] determining a log index of the to-be-queried data according to the second starting storage location;

[0032] querying from a corresponding log index table according to the log index of the to-be-queried data to obtain a query result;

[0033] obtaining the to-be-queried data according to the query result, and sending the to-be-queried data to the first object.

[0034] In one of the embodiments, the data query request further comprises a storage location number occupied by the to-be-queried data; and the determining the log index corresponding to the to-be-queried data according to the second starting storage location comprises:

[0035] determining each storage location occupied by the to-be-queried data according to the second starting storage location and the storage location quantity;

[0036] determining the log index corresponding to the to-be-queried data according to the position sequence number of each storage location.

[0037] In one of the embodiments, the data query request further includes the storage location quantity occupied by the to-be-queried data; and the obtaining the to-be-queried data according to the query result includes:

[0038] in the case that the query result does not include the log index of the to-be-queried data in the corresponding log index table, obtaining the to-be-queried data from the data storage node according to the second starting storage location and the storage location quantity;

[0039] in the case that the query result includes the log index of the to-be-queried data in the corresponding log index table, obtaining the to-be-queried data from the log storage node according to the log index of the to-be-queried data.

[0040] In one of the embodiments, the filtering the associated log data of the target log data from each log data stored in the log storage node according to the data storage location of the target log data includes:

[0041] the log data corresponding to the data storage location of the target log data as the associated log data of the target log data.

[0042] In one of the embodiments, the method further includes:

[0043] receiving a data storage request sent by a second object; wherein the data storage request includes first data and a data storage location of the first data in a data storage node;

[0044] generating target operation information of the first data according to the first data and the data storage location corresponding to the first data;

[0045] in the case that the data storage location corresponding to the target operation information coincides with the data storage location corresponding to the candidate operation information in the operation information queue, determining to-be-stored data according to the first data and the data corresponding to the candidate operation information;

[0046] determining the data storage location of the to-be-stored data in the data storage node according to the data storage location corresponding to the target operation information and the data storage location corresponding to the candidate operation information.

[0047] In a second aspect, the present application provides a data processing apparatus, comprising:

[0048] a detecting module configured to, after detecting a data processing event directed to target log data in a log storage node, filter out associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is a data storage location of to-be-stored data contained in the log data in the data storage node;

[0049] a merging module configured to merge the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location;

[0050] a migrating module configured to, after merging the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data according to the merged storage location, migrate the to-be-stored data to the data storage node.

[0051] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of each method embodiment provided in the first aspect when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of each method embodiment provided in the first aspect when executed by a processor.

[0053] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program implements the steps of each method embodiment provided in the first aspect when executed by a processor.

[0054] The data processing method, device, computer device and storage medium detect the data processing event of the target log data in the log storage node, filter the associated log data of the target log data from each log data stored in the log storage node according to the data storage position corresponding to the target log data, merge the data storage position corresponding to the target log data and the data storage position corresponding to the associated log data, obtain a merged storage position, and migrate the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data to the data storage node after the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data are merged according to the merged storage position. In the method, after the data processing event is detected, data migration is not directly performed between the log storage node and the data storage node, but the associated log data capable of being migrated in combination with the target log data is found. In this way, the migration times can be reduced and the migration efficiency can be improved by migrating the target log data and the associated log data in combination. In addition, the redundant data covered between the target log data and the associated log data can be eliminated, the redundant data can be avoided from being migrated to the data storage node, and the storage resource utilization rate can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without any creative effort.

[0056] Figure 1 An application environment diagram of the data processing method in one embodiment;

[0057] Figure 2 A flowchart of the data processing method in one embodiment;

[0058] Figure 3 A flowchart of the step of obtaining to-be-stored data in one embodiment;

[0059] Figure 4 A flowchart of the step of generating log data in one embodiment;

[0060] Figure 5 A correspondence diagram of a log index, a log volume and a user logical volume in one embodiment;

[0061] Figure 6 A flowchart of the data query step in one embodiment;

[0062] Figure 7 A flowchart of the data processing method in another embodiment;

[0063] Figure 8 is a structural block diagram of a data processing device in an embodiment;

[0064] Figure 9 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0066] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" used in the present application and any variations thereof are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application means two and more than two. The term "and / or" used in the present application means one of the options or any combination of multiple options.

[0067] Block storage is a storage method that provides raw disk space to users. Users usually connect to the block storage server through a network and can directly read and write to the disk or format the disk into a file system to read and write files. Block storage needs to ensure the high reliability of user data storage on the server side, and cannot cause user data loss due to node or disk failure.

[0068] Currently, the most commonly used data guarantee mechanism is the three-copy method, which copies the user's data three times and stores them in different fault domains of machine nodes. In order to keep each copy of data consistent and reliably recover after a crash, a Write-Ahead Log (WAL) method can be used for auxiliary protection. The WAL mechanism is a technology that ensures data is not lost in the event of a failure. It requires that any data modification be recorded in a log before it is made. In this way, even if the storage system fails, data can be recovered by reading the WAL log, thereby ensuring data persistence and consistency. Specifically, the user's data operation log is written to the WAL partition, and a copy of the data is saved in memory. When the amount of data in memory reaches a certain level, the data in memory is updated to a data block that cannot be modified, and then the data block is persisted to disk.

[0069] However, during the data block persistence process, the data block may contain data that has been overwritten or deleted by subsequent operations, resulting in the saving of redundant data to the disk, occupying a lot of storage space, and low storage resource utilization.

[0070] The data processing method provided by the embodiments of the present application can be applied to a data storage system as shown in Figure 1 The storage gateway, the log storage node and the data storage node communicate with each other through a network. After detecting a data processing event for target log data in the log storage node, the storage gateway screens associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; merges the data storage location corresponding to the target log data and a data storage location corresponding to the associated log data to obtain a merged storage location; and migrates, to the data storage node, data to be stored corresponding to the target log data and data to be stored corresponding to the associated log data after merging the data to be stored. The storage gateway, the log storage node and the data storage node can each be a server, which can be a physical server, a server cluster or a distributed system formed by multiple physical servers, or a cloud server providing cloud computing services.

[0071] In an exemplary embodiment, as shown in Figure 2 A data processing method is provided, and the method is described by taking a storage gateway in Figure 1 as an example, and includes the following steps.

[0072] S201: After detecting a data processing event for target log data in the log storage node, screening, from each log data stored in the log storage node, associated log data of the target log data according to a data storage location corresponding to the target log data.

[0073] The data storage location corresponding to each log data is a data storage location of data to be stored in the data storage node, which is contained in the log data. It can be understood that the data to be stored is data required to be saved by a user, and the user can send the data to be stored to the storage gateway through a data storage request. The data storage request can contain a data storage location, which is used to indicate a storage location of the data to be stored in the data storage node. The data storage location can be a logical block address (LBA). LBA is a standardized method for managing data storage and addressing on storage devices such as hard disks. In the LBA mode, each data block on the hard disk is assigned a unique logical block address. These addresses are consecutive, starting from 0 and increasing. Each LBA corresponds to a data block on the hard disk. For example, the data storage address of the data to be stored 1 can be LBA0-LBA5, and the data storage address of the data to be stored 2 can be LBA4-LBA8.

[0074] The data processing event is used to represent that the target log data needs to be migrated to the data storage node for storage. The data processing event can be that the data storage amount of the log storage node is greater than a storage amount threshold, or that the interval time from the last data migration is greater than a time length threshold, etc. The specific setting can be made according to actual needs.

[0075] The associated log data can be log data whose corresponding data storage location is associated with the data storage location corresponding to the target log data. Optionally, among the log data stored by the log storage node, the log data whose corresponding data storage location is coincident with the data storage location corresponding to the target log data is taken as the associated log data of the target log data.

[0076] For example, the data storage location corresponding to the target log data is LBA0-LBA5, the data storage location corresponding to the log data 1 is LBA7-LBA8, and the data storage location corresponding to the log data 2 is LBA3-LBA6. The data storage location corresponding to the log data 2 is coincident with the data storage location corresponding to the target log data, and the log data 2 is taken as the associated log data of the target log data. In addition, if the data storage location corresponding to the log data is adjacent to the data storage location corresponding to the target log data, it can also be considered that the data storage locations are coincident. For example, the data storage location corresponding to the log data 3 is LBA5-LBA6, and the log data 3 is also the associated log data of the target log data.

[0077] S202: merging the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location.

[0078] For example, the data storage location corresponding to the target log data is LBA5-LBA6, and the data storage location corresponding to the associated log data is LBA7-LBA8. The merged storage location is LBA5-LBA8. For another example, the data storage location corresponding to the target log data is LBA11-LBA15, and the data storage location corresponding to the associated log data is LBA13-LBA17. The merged storage location is LBA11-LBA17.

[0079] S203: migrating the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data to the data storage node after merging according to the merged storage location.

[0080] The merging of the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data can eliminate the data covered between the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data, avoid writing redundant data to the data storage node, and thus reduce the amount of data written to the data storage node. After the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data are merged, the to-be-stored data is migrated to the merged storage location in the data storage node. Optionally, to ensure data reliability, the merged to-be-stored data can be stored in data storage nodes in different fault domains.

[0081] In the method, after detecting the data processing event for the target log data in the log storage node, the associated log data of the target log data is filtered from each log data stored in the log storage node according to the data storage location corresponding to the target log data, the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data are merged to obtain a merged storage location, and the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data are merged and then migrated to the data storage node according to the merged storage location. In the method, after detecting the data processing event, the data is not directly migrated between the log storage node and the data storage node, but the associated log data that can be merged and migrated with the target log data is found. In this way, by merging and migrating the target log data and the associated log data, the number of migration times can be reduced, and the migration efficiency can be improved. In addition, redundant data covered between the target log data and the associated log data can be eliminated, the redundant data can be avoided from being migrated to the data storage node, and the storage resource utilization rate can be improved.

[0082] In the above embodiments, the method for migrating data for the target log data in the log storage node is introduced. The generation process of the log data in the log storage node is introduced below. As described above, the log data includes to-be-stored data. Therefore, the source of the to-be-stored data is first introduced. In an exemplary embodiment, as shown in Figure 3 The process of obtaining the to-be-stored data is introduced, including:

[0083] S301: receiving a data storage request sent by a second object.

[0084] The data storage request includes first data and a data storage location of the first data in the data storage node. The first data is the to-be-stored data of the second object, and the data storage location is a storage location in the data storage node, for example, LBA11-LBA17.

[0085] S302: generating target operation information of the first data according to the first data and the data storage location corresponding to the first data.

[0086] The target operation information may include first data and data storage location, so as to store the first data in the corresponding data storage location.

[0087] S303: When the data storage location corresponding to the target operation information overlaps with the data storage location corresponding to the candidate operation information in the operation information queue, determine the data to be stored based on the first data and the data corresponding to the candidate operation information.

[0088] Specifically, when a user writes data, operation information (also known as input / output (I / O) operations) is first cached in an operation information queue. Each operation information includes the data the user needs to write and the data storage location. The storage gateway retrieves the operation information from the queue and performs a log write operation. To improve efficiency, when retrieving operation information, the storage gateway merges the operation information in the queue, combining operation information with overlapping data storage locations. This improves write performance and avoids multiple writes. Optionally, high-speed hardware can be used as the log storage node to convert user random I / O into sequential I / O.

[0089] S304: Determine the data storage location of the data to be stored in the data storage node based on the data storage location corresponding to the target operation information and the data storage location corresponding to the candidate operation information.

[0090] Specifically, the data storage locations corresponding to the target operation information and the candidate operation information are merged to obtain the data storage location of the data to be stored in the data storage node. For example, if the data storage location corresponding to the target operation information is LBA5-LBA6 and the data storage location corresponding to the candidate operation information is LBA7-LBA8, then the data storage location of the data to be stored in the data storage node is LBA5-LBA8.

[0091] In the above embodiments, by merging multiple operation information, the number of times log data is written to the log storage node can be reduced, data storage efficiency can be improved, and bandwidth consumption can be reduced.

[0092] After obtaining the data to be stored and its data storage location, in an exemplary embodiment, such as Figure 4 As shown, the steps for generating log data to be stored are detailed, including:

[0093] S401: For each piece of data to be stored, determine the log storage location corresponding to the data to be stored based on the storage status of the log storage node.

[0094] The log storage node can include at least one log volume, log data is written into the log volume in time sequence, and the storage condition of the log storage node can include a writing condition of the log volume. For example, the last time log data is written to position 2 of the log volume 1, and the log storage position corresponding to the to-be-stored data is position 3 of the log volume 1. The effective interval of the log volume is completed by using two offsets. The "log start" represents a start offset flag bit of the effective log interval, and the "log end" represents a terminal offset flag bit of the effective log interval. The effective log data is between the two offset flag bits. When log data is written, the log end moves forward, and the effective interval becomes larger. When log data is eliminated, the log start moves forward, and the effective interval becomes smaller. The log volume is sequentially written with data. When the flag bit reaches the tail, data will be written from the head.

[0095] S402: According to the data storage position corresponding to the to-be-stored data and the log storage position, the metadata of the to-be-stored data is generated.

[0096] The metadata can include data storage positions and log storage positions. Optionally, the metadata is saved to a metadata list, and each metadata is arranged in the metadata list in time sequence.

[0097] S403: The metadata and the to-be-stored data are spliced to obtain log data of the to-be-stored data.

[0098] The metadata and the to-be-stored data are spliced to obtain the log data. The metadata is usually placed at the head of the log data. Optionally, after the log data is written to the log storage node, in order to ensure the reliability of the log data, multiple copies of the log data are saved in different fault domains to avoid loss of log data due to node failure or underlying disk failure, and the execution result is returned to the user after successful writing.

[0099] In the above embodiment, according to the data storage position corresponding to the to-be-stored data and the log storage position, the log data of the to-be-stored data is generated, the integrity of the log data is ensured, the to-be-stored data of the user can be recovered through the log data, and the reliability of the storage system is ensured.

[0100] In an exemplary embodiment, the data storage position corresponding to the to-be-stored data includes a first start storage position. The data storage node can include at least one user logical volume, the data storage position is a position on the user logical volume, and the first start storage position is used to represent the start offset of the to-be-stored data on the user logical volume. After obtaining the log data of the to-be-stored data, the following steps are further included:

[0101] According to the position sequence number of the first start storage position, a log index of the log data is determined; and the log index is saved into a log index table corresponding to the first start storage position.

[0102] The log index table is used to save log indexes from the first storage position to the second storage position, and the first start storage position is between the first storage position and the second storage position. Specifically, each LBA can be divided into multiple storage positions, and the log indexes of the storage positions contained in each LBA are saved into a log index table. For example, LBA1 is divided into LBA0-LBA5, log index table 1 is used to save log indexes from LBA0-LBA5, LBA2 is divided into LBA6-LBA10, and log index table 2 is used to save log indexes from LBA6-LBA10. Taking the position sequence number of the first start storage position as LBA7 as an example, the log index is LBA7, the corresponding log index table is log index table 2, and the log index is saved into log index table 2. Different log index tables can be allocated to different CPU cores, avoiding the performance bottleneck of a single index structure. The user logical volume is divided into multiple smaller segments, each segment constructs a log index, and different index tables are placed on different CPU cores for isolation, so that the index can be operated concurrently without locking.

[0103] In the above embodiments, by constructing the log index of the log data, the log data can be quickly queried according to the log index, and the data query efficiency is improved.

[0104] In an exemplary embodiment, the step of saving the log index is further refined, including:

[0105] Taking the log index as a target log index, the left index and the right index of the target log index are found from the log index table corresponding to the start storage position; and the target log index is inserted between the left index and the right index.

[0106] Optionally, the log indexes in the log index table are arranged in size order, for example, the data structure of the log index table is a skip list, and each log index is arranged in the bottom layer of the skip list in ascending order, so that, taking a log index as a target log index, the left index and the right index of the target log index can be found from the bottom layer elements of the log index table, the left index is the log index among the log indexes smaller than the target log index, and the difference between the target log index and the left index is the smallest, the right index is the log index among the log indexes greater than the target log index, and the difference between the target log index and the right index is the smallest, and the target log index is inserted between the left index and the right index. Since each log index is arranged in ascending order in the bottom layer of the skip list, the left index of the target log index can also be found, and the target log index can be inserted next to the left index on the right side of the left index.

[0107] As shown in Figure 5 , a correspondence relationship among the log indexes, the log volumes, and the user logical volumes is shown, wherein all the log indexes are managed by the log index table, wherein the log index table 1 is used to manage the log indexes "LOG1, LOG3, …" of the to-be-stored data (indicated by data in the figure) in the user logical volume with the storage positions of LBA0~LBAk, the log index table 2 is used to manage the log indexes of the to-be-stored data in the user logical volume with the storage positions of LBAk+1~LBAm, and so on. It should be noted that the log data is written into the log volume in sequence, that is, the sequence of "LOG1, LOG2, LOG3, LOG4, …", and the storage positions of the to-be-stored data in the user logical volume are specified by the user, that is, the sequence of "data1, data2, …data3, …data4, …". Therefore, in the log volume in Figure 5 , the log data 2 is located before the log data 3, and the to-be-stored data 3 corresponding to the log data 2 in the user logical volume is located after the to-be-stored data 2 corresponding to the log data 3.

[0108] In the above embodiments, by arranging the log indexes in the log index table in size order, the time required for inserting the log indexes and finding the log indexes can be reduced, the index insertion efficiency and the finding efficiency are improved, and the data storage efficiency is further improved.

[0109] On the basis of maintaining the metadata list and the log index table in each of the above embodiments, in an exemplary embodiment, when any of the following conditions is detected, it is determined that a data processing event for the target log data in the log storage node is detected, including:

[0110] (1) In the case where the number of log indexes in any log index table is greater than the number threshold, it is determined that the log data corresponding to the first log index in the log index table is the target log data, and it is determined that the data processing event for the target log data in the log storage node is detected.

[0111] The quantity threshold can be set based on experience, multiple trials, and actual needs, and is not specifically limited here. When the number of log indexes exceeds the quantity threshold, it indicates that the data volume of the log storage node has reached a certain scale. Therefore, data migration of the log storage node can be performed. In this case, the log data corresponding to the first log index in the log index table can be used as the target log data for data migration (also known as data eviction). When performing log data eviction, the data to be stored is read from the corresponding log volume and written to the user logical volume, realizing the log flushing operation.

[0112] (2) If the storage duration of the first metadata in the metadata list is detected to be greater than the duration threshold, the log data corresponding to the first metadata is determined to be the target log data, and a data processing event for the target log data in the log storage node is detected.

[0113] The duration threshold can be set based on experience, multiple trials, and actual needs, and is not specifically limited here. If the storage duration of the first metadata exceeds the duration threshold, it indicates that the log data corresponding to the metadata has been stored in the log storage node for too long. Therefore, the log data corresponding to the first metadata can be used as the target log data for data eviction.

[0114] If the retention period of the first metadata in the metadata list exceeds a threshold, the log index of the corresponding log data (i.e., the target log data) can be found first. Then, in the log index table, related log data of the target log data is searched. Specifically, a greedy approach is used to look for log indexes that can be merged on both sides of the target log data. That is, log indexes that represent overlapping data storage addresses can be merged into one data storage address. This merges overlapping data storage addresses of the evicted log indexes into one data storage address, reducing the number of data evictions and improving efficiency and bandwidth. Finally, based on the data offset and length contained in the merged data storage address, the corresponding data is read from the log volume and written to the data volume. After completion, the corresponding index node is removed from the index table.

[0115] In the above embodiments, by maintaining the metadata list and the log index table, data processing events targeting target log data in the log storage node can be accurately detected, and the target log data can be migrated in a timely manner to ensure that new log data can be continuously written to the log storage node.

[0116] In an exemplary embodiment, after merging the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data, and migrating them to the data storage node, the method further includes:

[0117] The target log data and the associated log data are deleted from the log storage node, and the target log index of the target log data and the target log index of the associated log data are deleted from the corresponding log index table.

[0118] Specifically, the target log data and the associated log data are discarded to the data storage node, the target log data and the associated log data are deleted from the log storage node, and new log data is written to the log storage node. Accordingly, since the target log data and the associated log data have been deleted from the log storage node, the target log index of the target log data and the target log index of the associated log data also do not need to be maintained, and are deleted from the corresponding log index table. Optionally, the metadata of the target log data and the metadata of the associated log data are deleted from the metadata list.

[0119] In the above embodiment, after the target log data and the associated log data are discarded to the data storage node, the corresponding log index is also deleted from the log index table, reducing the storage space occupied by redundant data and improving resource utilization.

[0120] The above embodiment mainly introduces the process of data storage in the data storage system. In an exemplary embodiment, as shown in FIG. 6, data query can also be performed in the following manner: Figure 6

[0121] S601: Receive a data query request for a first object.

[0122] The data query request includes a second starting storage location of the data to be queried in the data storage node, that is, the data query request needs to include the starting offset of the data to be queried in the data storage node, so as to find the data to be queried.

[0123] S602: Determine the log index of the data to be queried according to the second starting storage location.

[0124] As described above, the data is first saved in the form of log data in the log storage node and then discarded to the data storage node. Therefore, first, the log index of the data to be queried is determined according to the second starting storage location, so as to find the data to be queried from the log storage node. In a specific implementation, the second starting storage location can be used as the log index.

[0125] S603: Query from the corresponding log index table according to the log index of the data to be queried, and obtain a query result.

[0126] The query result is used to represent whether the log index table contains the log index of the data to be queried.

[0127] ​S604: According to the query result, the to-be-queried data is obtained, and the to-be-queried data is sent to the first object.

[0128] If the log index table contains the log index of the to-be-queried data, it indicates that the to-be-queried data is still saved in the form of log data in the log storage node, and then the to-be-queried data can be obtained from the log storage node. If the log index table does not contain the log index of the to-be-queried data, it indicates that the log data of the to-be-queried data has been eliminated to the data storage node, and then the to-be-queried data is obtained from the data storage node.

[0129] In the above embodiment, by searching for the to-be-queried data according to the log index of the to-be-queried data, the data query efficiency can be improved.

[0130] In an exemplary embodiment, the step of determining the log index corresponding to the to-be-queried data in S602 is further refined, comprising:

[0131] According to the second starting storage location and the number of storage locations, each storage location occupied by the to-be-queried data is determined; and according to the position sequence number of each storage location, the log index corresponding to the to-be-queried data is determined.

[0132] In the above embodiment, according to the second starting storage location and the number of storage locations, the log index corresponding to the to-be-queried data is determined, and then the to-be-queried data is searched according to the log index, which can improve the data query efficiency.

[0133] In the above embodiment, according to the second starting storage location and the number of storage locations, the log index corresponding to the to-be-queried data is determined, and then the to-be-queried data is searched according to the log index, which can improve the data query efficiency.

[0134] In an exemplary embodiment, the step of obtaining the to-be-queried data according to the query result in S604 is further refined, comprising:

[0135] In the case where the query result indicates that the corresponding log index table does not include the log index of the to-be-queried data, the to-be-queried data is obtained from the data storage node according to the second starting storage location and the number of storage locations; and in the case where the query result indicates that the corresponding log index table includes the log index of the to-be-queried data, the to-be-queried data is obtained from the log storage node according to the log index of the to-be-queried data.

[0136] In a case where the log index of the data to be queried is not included in the log index table corresponding to the query result representation, the data storage location of the data to be queried in the data storage node is determined according to the second starting storage location and the number of storage locations, and the data to be queried is obtained from the data storage node according to the determined data storage location. In a case where the log index of the data to be queried is included in the log index table corresponding to the query result representation, the log index of the data to be queried can be taken as a key to query the metadata of the log data, and the data to be queried is obtained from the log storage node according to the log storage location contained in the metadata.

[0137] In addition, there can be a case where part of the data to be queried is in the log storage node and part of the data to be queried is in the data storage node, and then the data to be queried is combined into complete data to be queried and returned to the first object according to the determined log index corresponding to the data to be queried and read from the log storage node and the data storage node respectively.

[0138] In the above embodiments, the data is queried by judging whether the log index of the data to be queried is included in the log index table, which can ensure fast searching of the data to be queried and improve the data query efficiency.

[0139] On the basis of the above embodiments, in an exemplary embodiment, as shown in Figure 7 The data processing method in the embodiments of the present application can include the following steps:

[0140] S701: receiving a data storage request sent by a second object; generating target operation information of first data according to the first data and the data storage location corresponding to the first data; in a case where the data storage location corresponding to the target operation information coincides with the data storage location corresponding to the candidate operation information in the operation information queue, determining data to be stored according to the first data and the data corresponding to the candidate operation information; and determining the data storage location of the data to be stored in the data storage node according to the data storage location corresponding to the operation information and the data storage location corresponding to the candidate operation information.

[0141] The data storage request includes the first data and the data storage location of the first data in the data storage node.

[0142] S702: for each data to be stored, determining the log storage location corresponding to the data to be stored according to the storage condition of the log storage node; generating the metadata of the data to be stored according to the data storage location and the log storage location corresponding to the data to be stored; splicing the metadata and the data to be stored to obtain the log data of the data to be stored, and saving the metadata to a metadata list.

[0143] The data storage location corresponding to the data to be stored includes a first starting storage location.

[0144] S703: determining a log index of the log data according to a position sequence number of the first start storage position; and storing the log index into a log index table corresponding to the first start storage position.

[0145] The log index table is used to store log indexes from the first storage position to the second storage position, and the first start storage position is located between the first storage position and the second storage position.

[0146] S704: determining that a data processing event for target log data in the log storage node is detected, in a case where a quantity of log indexes in any log index table is greater than a quantity threshold, or a storage duration of a first metadata in the metadata list is greater than a duration threshold.

[0147] The target log data is log data corresponding to a first log index in the log index table, or log data corresponding to the first metadata.

[0148] S705: taking, as associated log data of the target log data, log data of which corresponding data storage position overlaps with the data storage position corresponding to the target log data, from the log data stored in the log storage node.

[0149] S706: merging the data storage position corresponding to the target log data and the data storage position corresponding to the associated log data to obtain a merged storage position.

[0150] S707: migrating, to the data storage node, the to-be-stored data corresponding to the target log data and the to-be-stored data corresponding to the associated log data after the to-be-stored data is merged according to the merged storage position.

[0151] S708: deleting the target log data and the associated log data from the log storage node; and deleting a target log index of the target log data and a target log index of the associated log data from corresponding log index tables.

[0152] The specific implementation manners of S701-S708 are the same as those in the specific implementation manners of the method embodiments, and will not be described herein again.

[0153] It should be understood that although each step in the flowchart involved in the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0154] Based on the same inventive concept, the embodiments of the present application also provide a data processing device for implementing the above-mentioned data processing method. The implementation scheme of the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing device embodiments provided below can refer to the limitations of the data processing method in the above text, which will not be repeated here.

[0155] In one exemplary embodiment, as shown in Figure 8 A data processing device is provided, comprising: a detection module 801, a merging module 802 and a migration module 803, wherein:

[0156] The detection module 801 is configured to, after detecting a data processing event for target log data in a log storage node, filter out associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is the data storage location of the data to be stored in the data storage node contained in the log data.

[0157] The merging module 802 is configured to merge the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location.

[0158] The migration module 803 is configured to, after merging the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data according to the merged storage location, migrate to the data storage node.

[0159] In one embodiment, the device further comprises a generation module configured to:

[0160] For each data to be stored, according to the storage condition of the log storage node, a log storage position corresponding to the data to be stored is determined;

[0161] According to the data storage position and the log storage position corresponding to the data to be stored, the metadata of the data to be stored is generated;

[0162] The metadata and the data to be stored are spliced to obtain log data of the data to be stored.

[0163] In one of the embodiments, the data storage position corresponding to the data to be stored includes a first starting storage position; the generating module further includes:

[0164] The determining unit is configured to determine a log index of the log data according to a position sequence number of the first starting storage position;

[0165] The saving unit is configured to save the log index into a log index table corresponding to the first starting storage position; wherein, the log index table is configured to save log indexes from the first storage position to the second storage position, and the first starting storage position is located between the first storage position and the second storage position.

[0166] In one of the embodiments, the log indexes in the log index table are arranged in size order; and the saving unit is specifically configured to:

[0167] Taking the log index as a target log index, the left index and the right index of the target log index are searched from the log index table corresponding to the starting storage position; wherein, the left index is a log index that is smaller than the target log index and has the smallest difference with the target log index among the log indexes smaller than the target log index; and the right index is a log index that is greater than the target log index and has the smallest difference with the target log index among the log indexes greater than the target log index;

[0168] The target log index is inserted between the left index and the right index.

[0169] In one of the embodiments, the saving unit is further configured to:

[0170] The target log data and the associated log data are deleted from the log storage node; and,

[0171] The target log index of the target log data and the target log index of the associated log data are deleted from the corresponding log index table.

[0172] In one of the embodiments, the detecting module 801 is specifically configured to:

[0173] In a case where the number of log indexes in any log index table is greater than the number threshold, the log data corresponding to the first log index in the log index table is determined as the target log data, and it is determined that the data processing event for the target log data in the log storage node is detected.

[0174] In one of the embodiments, the generation model is further used for:

[0175] saving the metadata into a metadata list;

[0176] The detection module 801 is specifically used for:

[0177] In a case where the duration of saving the first metadata in the metadata list is greater than the duration threshold, determining that the log data corresponding to the first metadata is target log data, and determining that the data processing event for the target log data in the log storage node is detected.

[0178] In one of the embodiments, the generation module is further used for:

[0179] receiving a data query request of the first object; wherein the data query request includes a second starting storage location of the to-be-queried data in the data storage node;

[0180] determining the log index of the to-be-queried data according to the second starting storage location;

[0181] querying from the corresponding log index table according to the log index of the to-be-queried data to obtain a query result;

[0182] obtaining the to-be-queried data according to the query result, and sending the to-be-queried data to the first object.

[0183] In one of the embodiments, the data query request further includes a storage location quantity occupied by the to-be-queried data; and the generation module is specifically used for:

[0184] determining each storage location occupied by the to-be-queried data according to the second starting storage location and the storage location quantity;

[0185] determining the log index corresponding to the to-be-queried data according to the position serial number of each storage location.

[0186] In one of the embodiments, the data query request further includes a storage location quantity occupied by the to-be-queried data; and the generation module is specifically used for:

[0187] in a case where the query result represents that the log index of the to-be-queried data is not included in the corresponding log index table, obtaining the to-be-queried data from the data storage node according to the second starting storage location and the storage location quantity;

[0188] in a case where the query result represents that the log index of the to-be-queried data is included in the corresponding log index table, obtaining the to-be-queried data from the log storage node according to the log index of the to-be-queried data.

[0189] In one of the embodiments, the detection module 801 is specifically used for:

[0190] Among the log data stored by the log storage node, log data corresponding to the data storage location of the target log data and the data storage location corresponding to the target log data coincide as associated log data of the target log data.

[0191] In one of the embodiments, the apparatus further comprises a receiving module configured to:

[0192] receive a data storage request sent by a second object; wherein the data storage request comprises first data, and a data storage location of the first data in the data storage node;

[0193] generate target operation information of the first data according to the first data and the data storage location corresponding to the first data;

[0194] in a case where the data storage location corresponding to the target operation information and the data storage location corresponding to the candidate operation information in the operation information queue coincide, determine the data to be stored according to the first data and the data corresponding to the candidate operation information;

[0195] determine the data storage location of the data to be stored in the data storage node according to the data storage location corresponding to the target operation information and the data storage location corresponding to the candidate operation information.

[0196] Each module in the above data processing apparatus can be realized by software, hardware and combinations thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0197] In an exemplary embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 9 The computer device comprises a processor, a memory, an I / O interface and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store log data and the like. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data processing method.

[0198] Those skilled in the art can understand that Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0199] In one exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following when executing the computer program: after detecting a data processing event for target log data in a log storage node, filtering out associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is a data storage location of data to be stored in the data storage node included in the log data; merging the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location; and migrating the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data to the data storage node after merging according to the merged storage location.

[0200] In one embodiment, a computer readable storage medium is provided, storing a computer program thereon, and the computer program is executed by a processor to implement: after detecting a data processing event for target log data in a log storage node, filtering out associated log data of the target log data from each log data stored in the log storage node according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is a data storage location of data to be stored in the data storage node included in the log data; merging the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location; and migrating the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data to the data storage node after merging according to the merged storage location.

[0201] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements: after detecting a data processing event for target log data in a log storage node, screening, from each log data stored in the log storage node, associated log data of the target log data according to a data storage location corresponding to the target log data; wherein the data storage location corresponding to each log data is a data storage location in the data storage node of data to be stored contained in the log data; merging the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location; and migrating, to the data storage node, the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data after merging according to the merged storage location.

[0202] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0203] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0204] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0205] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: After detecting a data processing event targeting target log data in a log storage node, the associated log data of the target log data is filtered out from the log data stored in the log storage node according to the data storage location corresponding to the target log data; wherein, the data storage location corresponding to each log data is the data storage location of the data to be stored contained in the log data in the data storage node; The data storage location corresponding to the target log data and the data storage location corresponding to the associated log data are merged to obtain a merged storage location; Based on the merged storage location, the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data are merged and then migrated to the data storage node.

2. The method according to claim 1, characterized in that, The method further includes: For each piece of data to be stored, the log storage location corresponding to the data to be stored is determined based on the storage status of the log storage node. Based on the data storage location corresponding to the data to be stored and the log storage location, generate the metadata of the data to be stored; The metadata and the data to be stored are concatenated to obtain the log data of the data to be stored.

3. The method according to claim 2, characterized in that, The data storage location corresponding to the data to be stored includes a first starting storage location; after concatenating the metadata and the data to be stored to obtain the log data of the data to be stored, the method further includes: The log index of the log data is determined based on the location number of the first starting storage location; The log index is saved to the log index table corresponding to the first starting storage location; wherein, the log index table is used to save the log index from the first storage location to the second storage location, and the first starting storage location is located between the first storage location and the second storage location.

4. The method according to claim 3, characterized in that, The log indexes in the log index table are arranged in ascending order; Saving the log index to the log index table corresponding to the starting storage location includes: Using the log index as the target log index, the left index and right index of the target log index are searched in the log index table corresponding to the starting storage location; wherein, the left index is the log index with the smallest difference between itself and the target log index among the log indexes that are less than the target log index; and the right index is the log index with the smallest difference between itself and the target log index among the log indexes that are greater than the target log index. Insert the target log index between the left index and the right index.

5. The method according to claim 3, characterized in that, After merging the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data, and migrating them to the data storage node, the process further includes: Delete the target log data and the associated log data from the log storage node; and, Delete the target log index of the target log data and the target log index of the associated log data from the corresponding log index table.

6. The method according to claim 3, characterized in that, The detected data processing event targeting target log data in the log storage node includes: If the number of log indexes in any log index table is greater than the number threshold, the log data corresponding to the first log index in the log index table is determined to be the target log data, and a data processing event targeting the target log data in the log storage node is determined to have been detected.

7. The method according to claim 2, characterized in that, The method further includes: Save the metadata to a metadata list; The detected data processing event targeting target log data in the log storage node includes: If the storage duration of the first metadata in the metadata list is detected to be greater than the duration threshold, the log data corresponding to the first metadata is determined to be the target log data, and a data processing event targeting the target log data in the log storage node is detected.

8. The method according to claim 3, characterized in that, The method further includes: Receive a data query request from a first object; wherein the data query request includes the second starting storage location of the data to be queried in the data storage node; Determine the log index of the data to be queried based on the second starting storage location; Based on the log index of the data to be queried, query from the corresponding log index table to obtain the query result; Based on the query results, the data to be queried is obtained and sent to the first object.

9. The method according to claim 8, characterized in that, The data query request also includes the number of storage locations occupied by the data to be queried; determining the log index corresponding to the data to be queried based on the second starting storage location includes: Based on the second starting storage location and the number of storage locations, determine each storage location occupied by the data to be queried; The log index corresponding to the data to be queried is determined based on the location number of each storage location.

10. The method according to claim 8, characterized in that, The data query request also includes the number of storage locations occupied by the data to be queried; obtaining the data to be queried based on the query result includes: If the log index of the data to be queried is not included in the log index table corresponding to the query result representation, the data to be queried is obtained from the data storage node according to the second starting storage location and the number of storage locations; If the query result indicates that the log index of the data to be queried is included in the corresponding log index table, the data to be queried is obtained from the log storage node according to the log index of the data to be queried.

11. The method according to any one of claims 1 to 10, characterized in that, The step of filtering out associated log data of the target log data from the log data stored in the log storage node according to the data storage location of the target log data includes: Log data whose data storage location overlaps with the data storage location of the target log data are considered as associated log data of the target log data.

12. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Receive a data storage request sent by a second object; wherein the data storage request includes first data and the data storage location of the first data in the data storage node; Based on the first data and the data storage location corresponding to the first data, generate the target operation information of the first data; If the data storage location corresponding to the target operation information overlaps with the data storage location corresponding to the candidate operation information in the operation information queue, the data to be stored is determined based on the first data and the data corresponding to the candidate operation information. Based on the data storage location corresponding to the target operation information and the data storage location corresponding to the candidate operation information, the data storage location of the data to be stored in the data storage node is determined.

13. A data processing apparatus, characterized in that, The device includes: The detection module is used to detect a data processing event targeting target log data in the log storage node, and then, based on the data storage location corresponding to the target log data, filter out the associated log data of the target log data from the log data stored in the log storage node; wherein, the data storage location corresponding to each log data is the data storage location of the data to be stored contained in the log data in the data storage node; The merging module is used to merge the data storage location corresponding to the target log data and the data storage location corresponding to the associated log data to obtain a merged storage location; The migration module is used to merge the data to be stored corresponding to the target log data and the data to be stored corresponding to the associated log data according to the merged storage location, and then migrate them to the data storage node.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images