Data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, equipment and a medium, and relates to the technical field of multi-controller storage. In the multi-controller storage system, when a current controller reads data in a corresponding rear-end storage area and determines that the data belongs to other controllers according to the attribute of an owner of the data, the address of the data is not directly sent to the corresponding controller, but is stored in a temporary data structure of the current controller; storing data information corresponding to other controllers through a temporary data structure; and after all the data in the rear-end storage area is read, uniformly sending the data information in the temporary data structure to the corresponding controller. Through the one-time data transmission mode, the consumption of message communication resources between the controllers is effectively reduced, and the data processing efficiency and the system performance are improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-controller storage technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] In order to achieve high reliability, storage systems from mainstream manufacturers generally use a multi-controller architecture for data backup and management. Through redundant design, load balancing, fault isolation, and high availability technologies, efficient, reliable, and scalable data management is achieved, and it is widely used in data centers, cloud storage, and large enterprises.

[0003] Currently, when accessing backend storage data, multi-controller storage systems usually divide the backend storage data area to be accessed according to the number of controllers, and each controller accesses the backend storage data in multiple threads according to the allocated area. Since the backend accessed data has owner attributes, each non-owner node data needs to send the data address to the corresponding owner node (i.e. the corresponding controller) through message resources for processing. This causes each controller to consume a huge amount of message resources to process non-owner data when reading all data within the scope of the controller area, which is inefficient and seriously affects system performance.

[0004] In view of the above problems, how to solve the current problem that in the back-end storage data access of the multi-controller storage system, each non-owner node data needs to be sent to the corresponding owner node for processing through message resources, which consumes huge message resources and seriously affects the system performance is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a data processing method, device, equipment and medium to solve the problem that in the current back-end storage data access of a multi-controller storage system, each non-owner node data needs to be sent to the corresponding owner node for processing through message resources, which consumes huge message resources and seriously affects system performance.

[0006] In order to solve the above technical problems, the present invention provides a data processing method, comprising:

[0007] Read the data in the corresponding backend storage area in sequence;

[0008] According to the owner attribute of the data, determining whether the data belongs to other controllers;

[0009] If it is confirmed that the data belongs to the other controllers, the data information corresponding to the data is stored in a temporary data structure;

[0010] After all the data in the backend storage area are read, the data information in the temporary data structure is uniformly sent to the corresponding controller.

[0011] On the one hand, storing data information corresponding to the data into a temporary data structure includes:

[0012] Storing the data address of the data in the temporary data structure, or storing the data in the temporary data structure;

[0013] Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0014] The data addresses in the temporary data structure are uniformly sent to the corresponding controller, or the data in the temporary data structure are uniformly sent to the corresponding controller.

[0015] On the other hand, storing data information corresponding to the data into a temporary data structure includes:

[0016] Determine, based on the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the temporary data structure;

[0017] The data information corresponding to the data is stored in the temporary data structure according to the pool identifier, the volume identifier, the page identifier, the sector identifier and the controller identifier.

[0018] On the other hand, if it is confirmed that the data does not belong to the other controllers, the method further includes:

[0019] Determine, according to the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the current controller data structure;

[0020] The data is stored in the current controller data structure based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier.

[0021] On the other hand, the data in the corresponding backend storage area is read in sequence, including:

[0022] Reading the B+ tree node information in the backend storage area according to a preset data reading size;

[0023] Extracting metadata information according to the B+ tree node information;

[0024] The metadata information at least includes the storage address and timestamp of the root node.

[0025] On the other hand, sending the data information in the temporary data structure to the corresponding controller in a unified manner includes:

[0026] Splitting the data information in the temporary data structure according to a preset size to obtain a plurality of groups of data information blocks;

[0027] Creating a transmission thread pool; wherein the transmission thread pool contains multiple transmission threads;

[0028] establishing a plurality of transmission channels with the controller;

[0029] Allocating each of the transmission threads to each of the transmission channels respectively;

[0030] Allocating each of the data information blocks to each of the transmission threads respectively;

[0031] Transmitting the corresponding data information block to the corresponding controller through each transmission thread;

[0032] During the transmission process, each of the data information blocks is verified by a preset algorithm to determine whether the data information block is correct;

[0033] If it is confirmed that the data information block is incorrect, the corresponding transmission thread is controlled to resend the data information block, and the process returns to the step of verifying each data information block by a preset algorithm during the transmission process to determine whether the data information block is correct;

[0034] If it is confirmed that the data information block is correct, monitoring a confirmation message transmitted by the controller indicating that the data information block has been received;

[0035] When the confirmation message is received, the transmission thread that completes the data information block transmission is stopped, and the corresponding transmission channel is closed.

[0036] On the other hand, after the data information in the temporary data structure is uniformly sent to the corresponding controller, the method further includes:

[0037] Clearing the data information in the temporary data structure;

[0038] A transmission log including transmission process information of the data information is generated.

[0039] In order to solve the above technical problems, the present invention further provides a data processing device, comprising:

[0040] A reading module, used to read the data in the corresponding backend storage area in sequence;

[0041] A judgment module, used for judging whether the data belongs to other controllers according to the owner attribute of the data; if it is confirmed that the data belongs to other controllers, triggering the storage module;

[0042] The storage module is used to store the data information corresponding to the data into a temporary data structure;

[0043] The sending module is used to send the data information in the temporary data structure to the corresponding controller after all the data in the back-end storage area has been read.

[0044] In order to solve the above technical problems, the present invention further provides a data processing device, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the above-mentioned data processing method when executing the computer program.

[0047] In order to solve the above technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above data processing method are implemented.

[0048] The data processing method provided by the present invention sequentially reads the data in the corresponding back-end storage area; determines whether the data belongs to other controllers based on the owner attributes of the data; if it is confirmed that the data belongs to other controllers, stores the data information corresponding to the data in a temporary data structure; after all the data in the back-end storage area is read, the data information in the temporary data structure is uniformly sent to the corresponding controller.

[0049] The beneficial effect of the present invention is that, in a multi-controller storage system, when the current controller reads the data in the corresponding backend storage area and determines that the data belongs to other controllers according to the owner attribute of the data, the address of the data is not directly sent to the corresponding controller, but is stored in the temporary data structure of the current controller, and the data information corresponding to the other controllers is stored through the temporary data structure; after all the data in the backend storage area is read, the data information in the temporary data structure is uniformly sent to the corresponding controller. Through this one-time data transmission method, the consumption of message communication resources between controllers is effectively reduced, and the data processing efficiency and system performance are improved.

[0050] On the other hand, the present invention specifically stores the data address of the data in a temporary data structure, and uniformly sends the data address in the temporary data structure to the corresponding controller, which can improve the data transmission speed and reduce the consumption of system message resources; storing the data in a temporary data structure and uniformly sending the data in the temporary data structure to the corresponding controller can reduce the number of disk reads, increase the disk life, and improve the input and output path performance. The pool identifier, volume identifier, page identifier, sector identifier and controller identifier corresponding to the temporary data structure are determined according to the data; according to the pool identifier, volume identifier, page identifier, sector identifier and controller identifier, the data information corresponding to the data is stored in the temporary data structure, thereby realizing the storage of the data information.

[0051] In addition, the present invention also provides a data processing device, equipment and medium, with the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a data processing device provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The core of the present invention is to provide a data processing method, device, equipment and medium to solve the problem that in the current back-end storage data access of a multi-controller storage system, each non-owner node data needs to be sent to the corresponding owner node for processing through message resources, which consumes huge message resources and seriously affects system performance.

[0058] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Currently, when accessing backend storage data, multi-controller storage systems usually divide the backend storage data area to be accessed according to the number of controllers, and at the same time ensure data access load balancing, multi-controller multi-thread concurrent access, etc. Taking a four-controller storage system as an example, the specific implementation is as follows:

[0060] (1) Each controller is assigned the backend storage area address according to its own identifier (ID). Assuming that the backend storage area address space ranges from 0 to 0x3FFFFFFF, controller 1 is responsible for the data in the area from 0 to 0x0FFFFFFF, controller 2 is responsible for the data in the area from 0x10000000 to 0x1FFFFFFF, controller 3 is responsible for the data in the area from 0x20000000 to 0x2FFFFFFF, and controller 4 is responsible for the data in the area from 0x30000000 to 0x3FFFFFFF.

[0061] (2) Each controller creates multiple tasks based on the number of available threads in the system to access the backend storage data. Since the backend data being accessed has an owner attribute, it is used to determine whether the data belongs to the current controller; if it belongs to the current controller, the data is owner node data; if it does not belong to the current controller, the data is non-owner node data. Non-owner node data needs to send the data address to the corresponding owner node for processing through message resources, and is sent to the corresponding controller for processing. Therefore, when each task accesses data, it will determine whether the data belongs to this controller; if so, the data is stored in the data structure of this controller, otherwise it needs to be sent to the corresponding owner node for processing through message resources.

[0062] Take four controllers as an example. When each controller accesses data, the data for non-owner nodes needs to be sent to the other three controllers through message resources, and the corresponding controllers process the data. Therefore, when the four controllers read all the data within the scope of the controller area, a huge amount of message resources will be consumed to process non-owner data.

[0063] (3) For the owner node data read, each controller will take effect of checking the validity of the data and update it to the latest data structure, and then continue to read the next data until the data processing in all areas of the controller is completed.

[0064] (4) For data sent from other controllers, the current controller needs to re-initiate the disk read operation according to the address until all data from other controller nodes are processed.

[0065] However, for each non-owner data, the controller must send it to other controllers via messages, which results in a very large message resource overhead. At the same time, most of the data in the backend storage area corresponding to the controller is non-owner data, and all of them need to be sent to other controllers in turn for processing. The current processing method is very inefficient. Therefore, in order to solve the above problems, the present invention provides a data processing method.

[0066] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S10: Read the data in the corresponding backend storage area in sequence;

[0068] In a specific implementation, when accessing back-end storage data, a multi-controller storage system evenly divides the back-end storage data area to be accessed according to the number of controllers. Taking a four-controller storage system as an example, each controller evenly divides the back-end storage area address according to its own ID number. Assuming that the back-end storage area address space ranges from 0 to 0x3FFFFFFF, controller 1 is responsible for data in the area from 0 to 0x0FFFFFFF, controller 2 is responsible for data in the area from 0x10000000 to 0x1FFFFFFF, controller 3 is responsible for data in the area from 0x20000000 to 0x2FFFFFFF, and controller 4 is responsible for data in the area from 0x30000000 to 0x3FFFFFFF. Therefore, for controller 1, it is necessary to read the data in the corresponding back-end storage area from 0 to 0x0FFFFFFF in sequence.

[0069] It should be noted that, in this embodiment, there is no restriction on the specific process of the controller reading the data in the corresponding backend storage area, which depends on the specific implementation situation.

[0070] S11: Determine whether the data belongs to other controllers according to the owner attribute of the data; if it is confirmed that the data belongs to other controllers, proceed to step S12; if it is confirmed that the data does not belong to other controllers, proceed to step S14.

[0071] S12: storing data information corresponding to the data into a temporary data structure;

[0072] S13: After all data in the backend storage area are read, the data information in the temporary data structure is uniformly sent to the corresponding controller.

[0073] S14: Store the data into the current controller data structure.

[0074] Furthermore, since the data in the backend storage area has an owner attribute, it can indicate whether the data from the storage medium belongs to the current controller. Therefore, in order to hand over the data to the corresponding controller for processing, it is necessary to determine whether the data belongs to other controllers based on the owner attribute of the data. If it is confirmed that the data does not belong to other controllers, it is considered that the data belongs to the current controller, and the current controller directly performs a validity inventory on the data and stores it in the data structure of the current controller.

[0075] If it is confirmed that the data does not belong to the other controllers, the data needs to be sent to the corresponding controller for processing. Specifically, the data information corresponding to the data is stored in a temporary data structure. It should be noted that the temporary data structure is a storage space in the controller for temporarily storing data information corresponding to other controllers. In a multi-controller storage system, each controller has a corresponding temporary data structure. In this embodiment, the data information corresponding to the data is stored in the temporary data structure of the current controller. It should be noted that in this embodiment, there is no restriction on the specific content of the data information. For example, it can be the address of the data or the data itself, depending on the specific implementation situation.

[0076] Finally, after the current controller has finished reading all the data in the corresponding backend storage area, it will send the data information in the temporary data structure to the corresponding controller. The current controller does not need to process the non-owner data from other controllers. It is understandable that after the other controllers receive the corresponding data information, they will think that the data information belongs to this controller, and directly count the validity of the data by this controller and store it in the data structure of this controller. This one-time data transmission method effectively reduces the consumption of message communication resources between controllers.

[0077] It should be noted that in this embodiment, there is no restriction on the specific process of storing the data information into the temporary data structure, and there is no restriction on the specific process of uniformly sending the data information to the corresponding controller, which depends on the specific implementation situation.

[0078] In this embodiment, in a multi-controller storage system, when the current controller reads the data in the corresponding backend storage area and determines that the data belongs to other controllers based on the data owner attribute, the address of the data is not directly sent to the corresponding controller, but is stored in the temporary data structure of the current controller, and the data information corresponding to the other controllers is stored through the temporary data structure; after all the data in the backend storage area is read, the data information in the temporary data structure is uniformly sent to the corresponding controller. Through this one-time data transmission method, the consumption of message communication resources between controllers is effectively reduced, and the data processing efficiency and system performance are improved.

[0079] Based on the above embodiments, in some embodiments, storing data information corresponding to the data in a temporary data structure includes:

[0080] S120: storing the data address of the data into a temporary data structure, or storing the data into a temporary data structure;

[0081] Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0082] S130: Sending the data addresses in the temporary data structure to the corresponding controller in a unified manner, or sending the data in the temporary data structure to the corresponding controller in a unified manner.

[0083] In a specific implementation, in order to store the data information corresponding to the data in the temporary data structure, one way is to store the data address of the data in the temporary data structure. Correspondingly, when the data information in the temporary data structure is uniformly sent to the corresponding controller, the data address in the temporary data structure is specifically uniformly sent to the corresponding controller. Since the size of the data address is generally smaller than the size of the data itself, sending the data address can reduce the amount of data transmission, increase the data transmission speed, and reduce the consumption of system message resources.

[0084] However, in the above method, the controller receives the address of the corresponding data. Although the data transmission speed can be improved, the controller needs to read the disk again through the data address to obtain the corresponding data, that is, the disk is read multiple times for the same storage area, which is easy to reduce the life of the disk and reduce the input and output path performance. Therefore, in another method, in order to store the data information corresponding to the data in a temporary data structure, the data is specifically stored in the temporary data structure. Correspondingly, when the data information in the temporary data structure is uniformly sent to the corresponding controller, the data in the temporary data structure is specifically uniformly sent to the corresponding controller. This method can reduce the multiple disk reads of the data in the same storage area by directly sending the data to the corresponding controller, reducing the number of disk reads, increasing the life of the disk, and improving the input and output path performance.

[0085] In this embodiment, the data address of the data is stored in a temporary data structure, and the data addresses in the temporary data structure are uniformly sent to the corresponding controller, which can improve the data transmission speed and reduce the consumption of system message resources; storing the data in a temporary data structure and sending the data in the temporary data structure to the corresponding controller can reduce the number of disk reads, increase the life of the disk, and improve the input and output path performance.

[0086] Based on the above embodiments, in some embodiments, storing data information corresponding to the data in a temporary data structure includes:

[0087] S121: Determine, according to the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the temporary data structure;

[0088] S122: According to the pool identifier, the volume identifier, the page identifier, the sector identifier and the controller identifier, the data information corresponding to the data is stored in a temporary data structure.

[0089] In a multi-controller storage system, the data in the storage medium contains the location information of all data structures in the system. Therefore, in order to store the data information in a temporary data structure, in this embodiment, the pool ID, volume ID, page ID, sector ID and controller ID corresponding to the temporary data structure are determined according to the data, and the data information corresponding to the data is further stored in the temporary data structure according to the pool ID, volume ID, page ID, sector ID and controller ID. In this way, the storage of data information is realized.

[0090] Based on the above embodiment, in some embodiments, if it is confirmed that the data does not belong to other controllers, the method further includes:

[0091] S141: Determine, according to the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the current controller data structure;

[0092] S142: Store the data into the current controller data structure according to the pool identifier, volume identifier, page identifier, sector identifier and controller identifier.

[0093] In a multi-controller storage system, the data in the storage medium contains the location information of all data structures in the system. Therefore, after confirming that the data belongs to the current controller, in order to store the data in the current controller data structure, the pool ID, volume ID, page ID, sector ID and controller ID corresponding to the current controller data structure are determined based on the data; further, based on the pool ID, volume ID, page ID, sector ID and controller ID, the data information corresponding to the data is stored in the current controller data structure. In this way, the data corresponding to the current controller is stored.

[0094] Based on the above embodiments, in some embodiments, sequentially reading data in corresponding backend storage areas includes:

[0095] S100: Reading B+ tree node information in the backend storage area according to a preset data read size;

[0096] S101: extracting metadata information according to B+ tree node information;

[0097] The metadata information at least includes the storage address and timestamp of the root node.

[0098] In a specific implementation, in order to read the data in the corresponding backend storage area, the controller specifically reads the B+ tree node information in the backend storage area according to the preset data read size. In this embodiment, there is no restriction on the preset data read size. It should be noted that in a multi-controller storage system, the size of each B+ tree node stored on the disk is fixed to 512 bytes, so each task can be set to read 1MB of data from the backend storage area in turn each time, that is, 2048 B+ tree node information can be read each time, which takes into account resource consumption while having higher reading efficiency.

[0099] Further, metadata information is extracted based on the B+ tree node information, wherein the metadata information at least includes the storage address and timestamp of the root node, and when the data is subsequently stored in the data structure, the metadata information is specifically stored in the data structure.

[0100] In this way, the reading of data in the back-end storage area is realized, so that the controller can further inventory and store the validity of the data.

[0101] In order to improve the efficiency of transmitting a large amount of data information from the current controller to other controllers, based on the above embodiments, in some embodiments, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0102] S131: splitting the data information in the temporary data structure according to a preset size to obtain multiple groups of data information blocks;

[0103] S132: Create a transmission thread pool; wherein the transmission thread pool includes multiple transmission threads;

[0104] S133: Establish multiple transmission channels with the controller;

[0105] S134: Allocate each transmission thread to each transmission channel respectively;

[0106] S135: Allocate each data information block to each transmission thread respectively;

[0107] S136: Transmitting corresponding data information blocks to corresponding controllers through each transmission thread;

[0108] S137: During the transmission process, each data information block is verified by a preset algorithm to determine whether the data information block is correct; if the data information block is confirmed to be incorrect, the process proceeds to step S138; if the data information block is confirmed to be correct, the process proceeds to step S139;

[0109] S138: Control the corresponding transmission thread to resend the data information block, and return to step S137;

[0110] S139: monitoring the confirmation message of the completion of receiving the characterizing data information block transmitted by the controller;

[0111] S140: When the confirmation message is received, the transmission thread for completing the data information block transmission is stopped, and the corresponding transmission channel is closed.

[0112] Specifically, the data information in the temporary data structure is first split according to the preset size to obtain multiple groups of data information blocks to adapt to multi-threaded concurrent transmission. The block size should be optimized according to system performance and network bandwidth to achieve optimal transmission efficiency. In this embodiment, there is no limit on the preset size.

[0113] A transmission thread pool is further created in the current controller. The transmission thread pool includes multiple transmission threads, each of which is responsible for transmitting a data information block. At the same time, multiple transmission channels are established between the current controller and the controller corresponding to the non-owner data to increase the parallelism of data transmission. The transmission channel can be a physical channel (such as a network interface) or a logical channel (such as a virtual network interface), and there is no restriction on its specific type in this embodiment. Each transmission thread is assigned to each transmission channel to achieve uniform load and prevent a single channel from becoming a communication bottleneck.

[0114] Then, each data information block is assigned to each transmission thread, and each transmission thread transmits the corresponding data information block to the corresponding controller. During the transmission process, each data information block is checked by a preset algorithm to determine whether the data information block is correct to ensure the integrity and correctness of the data. In this embodiment, there is no restriction on the preset algorithm, for example, it can be a checksum algorithm, a hash algorithm or a cyclic redundancy check algorithm, depending on the specific implementation situation.

[0115] If the data information block is confirmed to be incorrect, the retransmission mechanism is triggered to control the corresponding transmission thread to resend the data information block, and return to the step of verifying each data information block through a preset algorithm during the transmission process to determine whether the data information block is correct, so as to verify again. If the data information block is confirmed to be correct, the confirmation message indicating that the data information block has been received is transmitted by the monitoring controller.

[0116] When a confirmation message is received, the data information block is confirmed to be received, the corresponding transmission thread is stopped, and the corresponding transmission channel is closed, thereby releasing the system's transmission thread resources and transmission channel resources to prepare for the next transmission task.

[0117] In this embodiment, when the data information in the temporary data structure is uniformly sent to the corresponding controller, the efficiency of transmitting a large amount of non-owner data from the current controller to other controllers can be significantly improved by using multi-threaded high concurrency and expanding the transmission channel, reducing transmission time and resource consumption. At the same time, the reliability and integrity of data transmission can be ensured through data verification, confirmation and error handling mechanisms.

[0118] Based on the above embodiment, in some embodiments, after the data information in the temporary data structure is uniformly sent to the corresponding controller, the method further includes:

[0119] S15: Clear the data information in the temporary data structure;

[0120] S16: Generate a transmission log including transmission process information of the data information.

[0121] In order to further release system resources, in the specific implementation, after the data information in the temporary data structure is uniformly sent to the corresponding controller, the data information in the temporary data structure can be further cleared to facilitate the storage of non-owner data during the next data processing. At the same time, in order to help users better understand the data processing process of the system this time, it is very important to generate a transmission log containing detailed data information.

[0122] Specifically, the transmission log should include log header information (such as log identifier, timestamp, system information and transmission type), data source and target information (such as source controller ID, target controller ID, data block ID, data block size, data block content summary), transmission process information (such as transmission start time, transmission end time, transmission channel ID, transmission thread ID, transmission status), storage status (such as source storage location, target storage location, storage time), error and retransmission information (such as error type, number of retransmissions, retransmission time), confirmation information (such as confirmation time, confirmation status) and statistical information (such as total transmission data volume, average transmission rate, total transmission time, number of successfully transmitted data blocks, number of failed transmitted data blocks). With detailed transmission logs, users can troubleshoot and optimize performance. In terms of troubleshooting, information such as error type, number of retransmissions and time in the log can help locate the root cause of the problem, such as network failure, data verification error or storage device failure, so that corresponding measures can be taken to repair it. In terms of performance optimization, data such as the transmission rate, total transmission time, and channel / thread usage in the log can reveal performance bottlenecks, such as insufficient network bandwidth, congested transmission channels, or unreasonable thread scheduling, and then optimize the network configuration, increase the transmission channel, or adjust the thread pool size to improve the system transmission efficiency and response speed. In addition, statistical information such as the number of data blocks successfully and unsuccessfully transmitted can evaluate the system stability and reliability and guide further optimization and improvement. In summary, through the log content, users can fully understand the storage and transmission of data during processing, which is convenient for troubleshooting, performance optimization, and system monitoring.

[0123] In the above embodiments, the data processing method is described in detail. The present invention also provides corresponding embodiments of the data processing device.

[0124] Figure 2 Schematic diagram of a data processing device provided by an embodiment of the present invention. Figure 2 As shown, the device comprises:

[0125] The reading module 10 is used to read the data in the corresponding backend storage area in sequence.

[0126] The judging module 11 is used to judge whether the data belongs to other controllers according to the owner attribute of the data; if it is confirmed that the data belongs to other controllers, the storage module 12 is triggered;

[0127] The storage module 12 is used to store the data information corresponding to the data into a temporary data structure;

[0128] The sending module 13 is used to send the data information in the temporary data structure to the corresponding controller after all the data in the backend storage area is read.

[0129] In some embodiments, the storage module 12 is specifically used to store the data address of the data into the temporary data structure, or store the data into the temporary data structure;

[0130] Correspondingly, the sending module 13 is specifically used to uniformly send the data addresses in the temporary data structure to the corresponding controller, or uniformly send the data in the temporary data structure to the corresponding controller.

[0131] In some embodiments, the storage module 12 includes:

[0132] A first determination submodule, configured to determine a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the temporary data structure according to the data;

[0133] The first storage submodule is used to store data information corresponding to the data into a temporary data structure according to the pool identifier, the volume identifier, the page identifier, the sector identifier and the controller identifier.

[0134] In some embodiments, it also includes:

[0135] A second determination submodule is used to determine a pool identifier, a volume identifier, a page identifier, a sector identifier and a controller identifier corresponding to the current controller data structure according to the data;

[0136] The second storage submodule is used to store the data into the current controller data structure according to the pool identifier, the volume identifier, the page identifier, the sector identifier and the controller identifier.

[0137] In some embodiments, the reading module 10 includes:

[0138] The reading submodule is used to read the B+ tree node information in the backend storage area according to the preset data reading size;

[0139] The extraction submodule is used to extract metadata information based on the B+ tree node information;

[0140] The metadata information at least includes the storage address and timestamp of the root node.

[0141] In some embodiments, the sending module 13 includes:

[0142] A splitting submodule is used to split the data information in the temporary data structure according to a preset size to obtain multiple groups of data information blocks;

[0143] Create a submodule for creating a transmission thread pool; wherein the transmission thread pool includes multiple transmission threads;

[0144] A channel establishment submodule is used to establish multiple transmission channels with the controller;

[0145] A first allocation submodule, used to allocate each transmission thread to each transmission channel respectively;

[0146] A second allocation submodule, used to allocate each data information block to each transmission thread respectively;

[0147] A transmission submodule, used for transmitting corresponding data information blocks to corresponding controllers through each transmission thread;

[0148] The check submodule is used to check each data information block through a preset algorithm during the transmission process to determine whether the data information block is correct; if the data information block is confirmed to be incorrect, the retransmission submodule is triggered; if the data information block is confirmed to be correct, the monitoring submodule is triggered;

[0149] The retransmission submodule is used to control the corresponding transmission thread to resend the data information block and trigger the check submodule;

[0150] A monitoring submodule, used for monitoring a confirmation message transmitted by the controller indicating that a data information block has been received;

[0151] The termination submodule is used to stop the transmission thread that completes the data information block transmission and close the corresponding transmission channel when receiving the confirmation message.

[0152] In some embodiments, it also includes:

[0153] Clear submodule, used to clear data information in temporary data structure;

[0154] The log generation submodule is used to generate a transmission log containing the transmission process information of the data information.

[0155] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0156] In addition, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above data processing method when executed by a processor.

[0157] Figure 3 A schematic diagram of a data processing device provided by an embodiment of the present invention. Figure 3 As shown, the data processing device includes:

[0158] A memory 20, used for storing computer programs;

[0159] The processor 21 is used to implement the steps of the data processing method mentioned in the above embodiment when executing the computer program.

[0160] The data processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0161] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0162] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the data processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include but is not limited to data involved in the data processing method.

[0163] In some embodiments, the data processing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0164] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation of the data processing device, and may include more or fewer components than shown in the figure.

[0165] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0166] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0167] The above is a detailed introduction to a data processing method, device, equipment and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0168] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A data processing method, characterized in that: include: Read the data in the corresponding backend storage area in sequence; According to the owner attribute of the data, determining whether the data belongs to other controllers; If it is confirmed that the data belongs to the other controllers, the data information corresponding to the data is stored in a temporary data structure; After all the data in the backend storage area are read, the data information in the temporary data structure is uniformly sent to the corresponding controller.

2. The data processing method according to claim 1, characterized in that: Storing data information corresponding to the data in a temporary data structure includes: Storing the data address of the data in the temporary data structure, or storing the data in the temporary data structure; Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including: The data addresses in the temporary data structure are uniformly sent to the corresponding controller, or the data in the temporary data structure are uniformly sent to the corresponding controller.

3. The data processing method according to claim 1, characterized in that: Storing data information corresponding to the data in a temporary data structure includes: Determine, based on the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the temporary data structure; The data information corresponding to the data is stored in the temporary data structure according to the pool identifier, the volume identifier, the page identifier, the sector identifier and the controller identifier.

4. The data processing method according to claim 1, characterized in that: If it is confirmed that the data does not belong to the other controllers, the method further includes: Determine, according to the data, a pool identifier, a volume identifier, a page identifier, a sector identifier, and a controller identifier corresponding to the current controller data structure; The data is stored in the current controller data structure based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier.

5. The data processing method according to claim 1, characterized in that: Read the data in the corresponding backend storage area in sequence, including: Reading the B+ tree node information in the backend storage area according to a preset data reading size; Extracting metadata information according to the B+ tree node information; The metadata information at least includes the storage address and timestamp of the root node.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The step of uniformly sending the data information in the temporary data structure to the corresponding controller includes: Splitting the data information in the temporary data structure according to a preset size to obtain a plurality of groups of data information blocks; Creating a transmission thread pool; wherein the transmission thread pool contains multiple transmission threads; establishing a plurality of transmission channels with the controller; Allocating each of the transmission threads to each of the transmission channels respectively; Allocating each of the data information blocks to each of the transmission threads respectively; Transmitting the corresponding data information block to the corresponding controller through each transmission thread; During the transmission process, each of the data information blocks is verified by a preset algorithm to determine whether the data information block is correct; If it is confirmed that the data information block is incorrect, the corresponding transmission thread is controlled to resend the data information block, and the process returns to the step of verifying each data information block by a preset algorithm during the transmission process to determine whether the data information block is correct; If it is confirmed that the data information block is correct, monitoring a confirmation message transmitted by the controller indicating that the data information block has been received; When the confirmation message is received, the transmission thread that completes the data information block transmission is stopped, and the corresponding transmission channel is closed.

7. The data processing method according to claim 6, characterized in that: After the data information in the temporary data structure is uniformly sent to the corresponding controller, the method further includes: Clearing the data information in the temporary data structure; A transmission log including transmission process information of the data information is generated.

8. A data processing device, characterized in that: include: A reading module, used to read the data in the corresponding backend storage area in sequence; A judgment module, used for judging whether the data belongs to other controllers according to the owner attribute of the data; if it is confirmed that the data belongs to other controllers, triggering the storage module; The storage module is used to store the data information corresponding to the data into a temporary data structure; The sending module is used to send the data information in the temporary data structure to the corresponding controller after all the data in the back-end storage area has been read.

9. A data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images