A data processing method, apparatus, device, and medium

Abstract

The application discloses a data processing method and device, equipment and medium, and relates to the technical field of multi-controller storage. In a multi-controller storage system, when a current controller reads data in a corresponding back-end storage area and determines that the data belongs to other controllers according to an owner attribute of the data, the address of the data is not directly sent to the corresponding controller, but is stored in a temporary data structure of the current controller, and the data information corresponding to the other controllers is stored through the temporary data structure; after all the data in the back-end storage area is read, the data information in the temporary data structure is uniformly sent to the corresponding controller. Through the one-time data transmission mode, the consumption of message communication resources between the controllers is effectively reduced, and the data processing efficiency and system performance are improved.

Description

Technical Field

[0001] This invention relates to the field of multi-controller storage technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] To achieve high reliability, mainstream storage systems generally employ a multi-controller architecture for data backup and management. Through technologies such as redundancy design, load balancing, fault isolation, and high availability, they achieve efficient, reliable, and scalable data management, making them widely used in data centers, cloud storage, and large enterprises.

[0003] Currently, multi-controller storage systems typically divide the backend storage data area equally according to the number of controllers when accessing backend storage data. Each controller then accesses the backend storage data using multiple threads within its allocated area. Since the accessed backend data has an owner attribute, each non-owner node's data needs to be processed by sending its data address to the corresponding owner node (i.e., the corresponding controller) via message resources. This results in each controller consuming a massive amount of message resources to process non-owner data when reading all data within its own area, leading to inefficiency and severely impacting system performance.

[0004] Given the above problems, how to solve the current issue in multi-controller storage systems where data from each non-owner node needs to be sent to the corresponding owner node for processing via message resources, resulting in huge message resource consumption and severely impacting system performance, is a problem that urgently needs to be solved by technical personnel in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a data processing method, apparatus, device, and medium to solve the problem that in current multi-controller storage systems, data from each non-owner node needs to be sent to the corresponding owner node for processing via message resources, resulting in huge message resource consumption and severely impacting system performance.

[0006] To solve the above-mentioned technical problems, the present invention provides a data processing method, comprising:

[0007] Read the data from the corresponding backend storage areas sequentially;

[0008] Based on the ownership attribute of the data, determine whether the data belongs to other controllers;

[0009] If it is confirmed that the data belongs to one of the other controllers, the data information corresponding to the data is stored in a temporary data structure;

[0010] After all the data in the backend storage area has been read, the data information in the temporary data structure is sent to the corresponding controller.

[0011] On the one hand, the data information corresponding to the data is stored in a temporary data structure, including:

[0012] The data address of the data is stored in the temporary data structure, or the data is stored in the temporary data structure;

[0013] Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0014] The data addresses in the temporary data structure are sent to the corresponding controller, or the data in the temporary data structure is sent to the corresponding controller.

[0015] On the other hand, storing the data information corresponding to the data in a temporary data structure includes:

[0016] Based on the data, determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the temporary data structure;

[0017] Based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier, the data information corresponding to the data is stored in the temporary data structure.

[0018] On the other hand, if it is confirmed that the data does not belong to the other controllers, the method further includes:

[0019] Based on the data, determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the current controller data structure;

[0020] The data is stored in the current controller data structure based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier.

[0021] On the other hand, data is read sequentially from the corresponding backend storage areas, including:

[0022] Read the B+ tree node information in the backend storage area according to the preset data read size;

[0023] Extract metadata information based on the B+ tree node information;

[0024] The metadata information includes at least the storage address and timestamp of the root node.

[0025] On the other hand, sending the data information in the temporary data structure to the corresponding controller in a unified manner includes:

[0026] The data information in the temporary data structure is split into multiple groups of data information blocks according to a preset size.

[0027] Create a transport thread pool; wherein the transport thread pool contains multiple transport threads;

[0028] Establish multiple transmission channels with the controller;

[0029] Each of the aforementioned transmission threads is assigned to each of the aforementioned transmission channels;

[0030] Each of the data information blocks is assigned to each of the transmission threads;

[0031] The corresponding data information block is transmitted to the corresponding controller through each of the transmission threads;

[0032] During transmission, each data information block is verified using a preset algorithm to determine whether the data information block is correct;

[0033] If the data information block is confirmed to be incorrect, the corresponding transmission thread is controlled to resend the data information block, returning to the step of verifying each data information block through a preset algorithm during transmission to determine whether the data information block is correct;

[0034] If the data information block is confirmed to be correct, then monitor the controller to transmit an acknowledgment message indicating that the data information block has been received.

[0035] Upon receiving the confirmation message, the transmission thread that has completed the transmission of the data information block is stopped, and the corresponding transmission channel is closed.

[0036] On the other hand, after uniformly sending the data information in the temporary data structure to the corresponding controller, the method further includes:

[0037] Clear the data information from the temporary data structure;

[0038] Generate a transmission log containing the transmission process information of the data.

[0039] To address the aforementioned technical problems, the present invention also provides a data processing apparatus, comprising:

[0040] The read module is used to read data sequentially from the corresponding backend storage areas;

[0041] The judgment module is used to determine whether the data belongs to other controllers based on the owner attribute of the data; if it is confirmed that the data belongs to other controllers, the storage module is triggered.

[0042] The storage module is used to store the data information corresponding to the data into a temporary data structure;

[0043] The sending module is used to send the data information in the temporary data structure to the corresponding controller after all the data in the backend storage area has been read.

[0044] To address the aforementioned technical problems, the present invention also provides a data processing device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor is used to implement the steps of the above-described data processing method when executing the computer program.

[0047] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned data processing method.

[0048] The data processing method provided by this invention involves sequentially reading data from the corresponding backend storage areas; determining whether the data belongs to other controllers based on the data's owner attribute; if the data belongs to other controllers, storing the corresponding data information in a temporary data structure; and after all data in the backend storage areas has been read, uniformly sending the data information in the temporary data structure to the corresponding controllers.

[0049] The beneficial effect of this invention lies in the fact that, in a multi-controller storage system, when the current controller reads data from the corresponding backend storage area and determines that the data belongs to other controllers based on the data's owner attribute, it does not directly send the data's address to the corresponding controller. Instead, it stores the data in a temporary data structure of the current controller, using this temporary data structure to store the corresponding data information of the other controllers. After all data in the backend storage area has been read, the data information in the temporary data structure is then sent to the corresponding controllers. This one-time data transmission method effectively reduces the consumption of message communication resources between controllers, improving data processing efficiency and system performance.

[0050] On the other hand, this invention specifically stores the data address in a temporary data structure and uniformly sends the data addresses in the temporary data structure to the corresponding controller, which can improve data transmission speed and reduce the consumption of system message resources. Storing data in a temporary data structure and uniformly sending the data in the temporary data structure to the corresponding controller can reduce the number of disk reads, increase disk lifespan, and improve the performance of input / output paths. Based on the data, the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the temporary data structure are determined; based on the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier, the data information corresponding to the data is stored in the temporary data structure, thus realizing the storage of data information.

[0051] In addition, the present invention also provides a data processing device, equipment and medium, with the same effect as above. Attached Figure Description

[0052] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a data processing device provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0057] The core of this invention is to provide a data processing method, apparatus, device, and medium to solve the problem that in current multi-controller storage systems, data from each non-owner node needs to be sent to the corresponding owner node for processing via message resources, resulting in huge message resource consumption and severely impacting system performance.

[0058] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Currently, multi-controller storage systems typically distribute the data to be accessed in the backend storage area equally according to the number of controllers, while ensuring load balancing and concurrent access by multiple controllers and threads. Taking a four-controller storage system as an example, the specific implementation is as follows:

[0060] (1) Each controller is allocated backend storage area address according to its own Identity Document (ID) number. Assuming the backend storage area address space range is 0 to 0x3FFFFFFF, then controller 1 is responsible for the data in the area from 0 to 0x0FFFFFFF, controller 2 is responsible for the data in the area from 0x10000000 to 0x1FFFFFFF, controller 3 is responsible for the data in the area from 0x20000000 to 0x2FFFFFFF, and controller 4 is responsible for the data in the area from 0x30000000 to 0x3FFFFFFF.

[0061] (2) Each controller creates multiple tasks based on the number of available threads in the system to access backend stored data. Since the data accessed in the backend has an owner attribute, it is used to determine whether the data belongs to the current controller. If it belongs to the current controller, the data is owner node data; if it does not belong to the current controller, the data is non-owner node data. Non-owner node data needs to be sent to the corresponding owner node for processing via message resources, which means it is sent to the corresponding controller for processing. Therefore, when each task accesses data, it will determine whether the data belongs to this controller; if so, the data is stored in the data structure of this controller; otherwise, it needs to be sent to the corresponding owner node for processing via message resources.

[0062] Taking four controllers as an example, when each controller accesses data, data for non-owner nodes needs to be sent to the other three controllers via message resources, and then processed by the corresponding controllers. Therefore, when the four controllers read all data within their own controller's region, they will consume a huge amount of message resources to process non-owner data.

[0063] (3) For the owner node data read, each controller will perform validity checks on these data in turn and update them to the latest data structure, and then continue to read the next data until the data processing of all areas of the controller is completed.

[0064] (4) For data sent from other controllers, the current controller needs to re-initiate the disk read operation based on the address until all data from other controller nodes has been processed.

[0065] However, for each piece of non-owner data, the controller must send a message to other controllers, resulting in significant message resource overhead. Simultaneously, most of the data in the backend storage area corresponding to the controller is non-owner data, all of which needs to be sent sequentially to other controllers for processing, making the current processing method extremely inefficient. Therefore, to solve the above problems, this invention provides a data processing method.

[0066] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S10: Read the data in the corresponding backend storage areas sequentially;

[0068] In practical implementation, when accessing backend storage data, multi-controller storage systems divide the backend storage data area to be accessed equally according to the number of controllers. Taking a four-controller storage system as an example, each controller equally distributes the backend storage area address according to its own ID number. Assuming the backend storage area address space range is 0 to 0x3FFFFFFF, then controller 1 is responsible for the data in the area from 0 to 0x0FFFFFFFF, controller 2 is responsible for the data in the area from 0x10000000 to 0x1FFFFFFF, controller 3 is responsible for the data in the area from 0x20000000 to 0x2FFFFFFF, and controller 4 is responsible for the data in the area from 0x30000000 to 0x3FFFFFFF. Therefore, for controller 1, it needs to read the data in the corresponding backend storage area from 0 to 0x0FFFFFFF sequentially.

[0069] It should be noted that this embodiment does not restrict the specific process of the controller reading data in the corresponding backend storage area, and it depends on the specific implementation.

[0070] S11: Determine whether the data belongs to other controllers based on the data owner attribute; if the data belongs to other controllers, proceed to step S12; if the data does not belong to other controllers, proceed to step S14.

[0071] S12: Store the data information corresponding to the data into a temporary data structure;

[0072] S13: After all data in the backend storage area has been read, the data information in the temporary data structure is sent to the corresponding controller.

[0073] S14: Store the data into the current controller data structure.

[0074] Furthermore, since data in the backend storage area has an owner attribute, which can characterize whether the data in the storage medium belongs to the current controller, in order to hand the data over to the corresponding controller for processing, it is necessary to determine whether the data belongs to other controllers based on the data's owner attribute. If it is confirmed that the data does not belong to other controllers, then the data is considered to belong to the current controller, and the current controller directly performs a validity check on the data and stores it in the current controller's data structure.

[0075] If it is confirmed that the data does not belong to any other controller, the data needs to be sent to the corresponding controller for processing. Specifically, the data information corresponding to the data is stored in a temporary data structure. It should be noted that a temporary data structure is a storage space within the controller used to temporarily store data information corresponding to other controllers. In a multi-controller storage system, each controller has a corresponding temporary data structure. In this embodiment, the data information corresponding to the data is stored in the temporary data structure of the current controller. It should be noted that this embodiment does not limit the specific content of the data information; for example, it can be the address of the data or the data itself, depending on the specific implementation.

[0076] Finally, after the current controller has read all the data from the corresponding backend storage area, it sends the data information in the temporary data structure to the corresponding controller. The current controller does not need to process non-owner data from other controllers. Understandably, after receiving the corresponding data information, other controllers will consider the data to belong to their own controller, directly perform a validity check on the data, and store it in their own data structure. This one-time data transmission method effectively reduces the consumption of message communication resources between controllers.

[0077] It should be noted that this embodiment does not restrict the specific process of storing data information into a temporary data structure, nor does it restrict the specific process of uniformly sending data information to the corresponding controller; it depends on the specific implementation situation.

[0078] In this embodiment, in a multi-controller storage system, when the current controller reads data from the corresponding backend storage area and determines that the data belongs to another controller based on the data's owner attribute, it does not directly send the data's address to the corresponding controller. Instead, it stores the data in a temporary data structure of the current controller, using this temporary data structure to store the corresponding data information for the other controllers. After all data in the backend storage area has been read, the data information in the temporary data structure is then sent to the corresponding controller. This one-time data transmission method effectively reduces the consumption of message communication resources between controllers, improving data processing efficiency and system performance.

[0079] Based on the above embodiments, in some embodiments, the data information corresponding to the data is stored in a temporary data structure, including:

[0080] S120: Store the data address of the data in a temporary data structure, or store the data in a temporary data structure;

[0081] Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0082] S130: Send the data addresses in the temporary data structure to the corresponding controller, or send the data in the temporary data structure to the corresponding controller.

[0083] In practical implementation, one way to store the data information corresponding to the data in a temporary data structure is to store the data address in the temporary data structure. Correspondingly, when sending the data information from the temporary data structure to the corresponding controller, the data address from the temporary data structure is specifically sent to the corresponding controller. Since the size of the data address is generally smaller than the size of the data itself, sending the data address can reduce the amount of data transmitted, increase the data transmission speed, and reduce the consumption of system message resources.

[0084] However, in the above method, the controller receives the address of the corresponding data. While this improves data transfer speed, the controller needs to read the disk again using the data address to obtain the corresponding data. This means multiple disk reads from the same storage area, which can reduce disk lifespan and degrade I / O path performance. Therefore, in another method, to store the corresponding data information in a temporary data structure, the data is specifically stored in the temporary data structure. Correspondingly, when sending the data information in the temporary data structure to the corresponding controller, the data in the temporary data structure is sent directly to the corresponding controller. This method, by directly sending data to the corresponding controller, reduces multiple disk reads from the same storage area, reduces the number of disk reads, increases disk lifespan, and improves I / O path performance.

[0085] In this embodiment, storing the data address in a temporary data structure and sending the data address in the temporary data structure to the corresponding controller can improve data transmission speed and reduce the consumption of system message resources. Storing the data in a temporary data structure and sending the data in the temporary data structure to the corresponding controller can reduce the number of disk reads, increase disk lifespan, and improve the performance of input / output paths.

[0086] Based on the above embodiments, in some embodiments, the data information corresponding to the data is stored in a temporary data structure, including:

[0087] S121: Determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the temporary data structure based on the data;

[0088] S122: Store the data information corresponding to the data into a temporary data structure based on the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier.

[0089] In a multi-controller storage system, the data in the storage medium contains the location information of all data structures in the system. Therefore, in order to store data information into a temporary data structure, this embodiment specifically determines the pool ID, volume ID, page ID, sector ID, and controller ID corresponding to the temporary data structure based on the data. Furthermore, based on the pool ID, volume ID, page ID, sector ID, and controller ID, the corresponding data information is stored into the temporary data structure. This achieves the storage of data information.

[0090] Based on the above embodiments, in some embodiments, if it is confirmed that the data does not belong to other controllers, the method further includes:

[0091] S141: Determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the current controller data structure based on the data;

[0092] S142: Store the data into the current controller data structure based on the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier.

[0093] In a multi-controller storage system, the data in the storage medium contains location information for all data structures in the system. Therefore, after confirming that the data belongs to the current controller, in order to store the data in the current controller's data structure, the pool ID, volume ID, page ID, sector ID, and controller ID corresponding to the current controller's data structure are determined based on the data. Furthermore, based on the pool ID, volume ID, page ID, sector ID, and controller ID, the corresponding data information is stored in the current controller's data structure. This achieves the storage of data corresponding to the current controller.

[0094] Based on the above embodiments, in some embodiments, data in the corresponding backend storage areas is read sequentially, including:

[0095] S100: Reads B+ tree node information in the backend storage area according to the preset data read size;

[0096] S101: Extract metadata information based on B+ tree node information;

[0097] The metadata information includes at least the storage address and timestamp of the root node.

[0098] In practical implementation, to read data from the corresponding backend storage area, the controller reads the B+ tree node information from the backend storage area according to a preset data read size. This embodiment does not limit the preset data read size. It should be noted that in a multi-controller storage system, the size of each disk's B+ tree node is fixed at 512 bytes. Therefore, each task can be set to read 1MB of data sequentially from the backend storage area each time, meaning that 2048 B+ tree node information can be read each time, achieving higher read efficiency while considering resource consumption.

[0099] Further extract metadata information based on the B+ tree node information. The metadata information includes at least the storage address and timestamp of the root node. When storing the data into the data structure later, the metadata information will be specifically stored in the data structure.

[0100] This enables the reading of data from the backend storage area, allowing the controller to further verify and store the data.

[0101] To improve the efficiency of the current controller transmitting large amounts of data to other controllers, based on the above embodiments, in some embodiments, the data information in the temporary data structure is uniformly sent to the corresponding controller, including:

[0102] S131: Split the data information in the temporary data structure according to a preset size to obtain multiple sets of data information blocks;

[0103] S132: Create a transport thread pool; the transport thread pool contains multiple transport threads;

[0104] S133: Establish multiple transmission channels with the controller;

[0105] S134: Assign each transmission thread to each transmission channel;

[0106] S135: Allocate each data block to each transmission thread;

[0107] S136: Transmit the corresponding data information block to the corresponding controller through each transmission thread;

[0108] S137: During transmission, each data information block is verified using a preset algorithm to determine whether the data information block is correct; if the data information block is confirmed to be incorrect, proceed to step S138; if the data information block is confirmed to be correct, proceed to step S139.

[0109] S138: Control the corresponding transmission thread to resend the data information block, and return to step S137;

[0110] S139: A confirmation message indicating completion of receiving the characterization data information block transmitted by the monitoring controller;

[0111] S140: When an acknowledgment message is received, stop the transmission thread that has completed the transmission of the data information block and close the corresponding transmission channel.

[0112] Specifically, the data in the temporary data structure is first split into multiple data blocks according to a preset size to accommodate concurrent multi-threaded transmission. The block size should be optimized based on system performance and network bandwidth to achieve the best transmission efficiency; however, this embodiment does not impose a limit on the preset size.

[0113] Further, a transmission thread pool is created within the current controller. This pool contains multiple transmission threads, each responsible for transmitting one data block. Simultaneously, multiple transmission channels are established between the current controller and the controller corresponding to the non-owner data to increase the parallelism of data transmission. Transmission channels can be physical channels (such as network interfaces) or logical channels (such as virtual network interfaces); in this embodiment, their specific types are not limited. Each transmission thread is assigned to a different transmission channel to achieve load evenness and prevent a single channel from becoming a communication bottleneck.

[0114] Subsequently, each data block is assigned to a separate transmission thread, and each transmission thread transmits the corresponding data block to its corresponding controller. During transmission, each data block is verified using a preset algorithm to determine its correctness, ensuring data integrity and accuracy. This embodiment does not limit the preset algorithm; it can be a checksum algorithm, hash algorithm, or cyclic redundancy check algorithm, depending on the specific implementation.

[0115] If the data block is confirmed to be incorrect, a retransmission mechanism is triggered, controlling the corresponding transmission thread to resend the data block. This returns to the step where, during transmission, each data block is verified using a preset algorithm to determine its correctness, and the verification is performed again. If the data block is confirmed to be correct, an acknowledgment message indicating successful reception of the data block is transmitted by the monitoring controller.

[0116] When an acknowledgment message is received, the receipt of the data block is confirmed, the corresponding transmission thread stops running, and the corresponding transmission channel is closed, thereby releasing the system's transmission thread resources and transmission channel resources to prepare for the next transmission task.

[0117] In this embodiment, when sending data information from the temporary data structure to the corresponding controller, the use of multi-threaded high concurrency and expanded transmission channels can significantly improve the efficiency of the current controller in transmitting large amounts of non-owner data to other controllers, reducing transmission time and resource consumption. Simultaneously, data verification, confirmation, and error handling mechanisms ensure the reliability and integrity of data transmission.

[0118] Based on the above embodiments, in some embodiments, after uniformly sending the data information in the temporary data structure to the corresponding controller, the method further includes:

[0119] S15: Clear the data information in the temporary data structure;

[0120] S16: Generate a transmission log containing transmission process information of the data.

[0121] To further free up system resources, in practice, after sending the data information in the temporary data structure to the corresponding controller, the data information in the temporary data structure can be further cleared to facilitate the storage of non-owner data in the next data processing. Meanwhile, to help users better understand the system's current data processing, generating a transmission log containing detailed data transmission information is crucial.

[0122] Specifically, the transmission log should include log header information (such as log identifier, timestamp, system information, and transmission type), data source and destination information (such as source controller ID, destination controller ID, data block ID, data block size, and data block content summary), transmission process information (such as transmission start time, transmission end time, transmission channel ID, transmission thread ID, and transmission status), storage information (such as source storage location, destination storage location, and storage time), error and retransmission information (such as error type, number of retransmissions, and retransmission time), acknowledgment information (such as acknowledgment time and acknowledgment status), and statistical information (such as total transmitted data volume, average transmission rate, total transmission time, number of successfully transmitted data blocks, and number of failed transmission data blocks). Detailed transmission logs allow users to troubleshoot and optimize performance. In terms of troubleshooting, information such as error type, number of retransmissions, and time in the logs can help pinpoint the root cause of problems, such as network failures, data verification errors, or storage device failures, allowing for appropriate corrective measures. In terms of performance optimization, data such as transmission rate, total transmission time, and channel / thread usage in the logs can reveal performance bottlenecks, such as insufficient network bandwidth, transmission channel congestion, or unreasonable thread scheduling. This allows for optimization of network configuration, increasing transmission channels, or adjusting the thread pool size, thereby improving system transmission efficiency and response speed. Furthermore, statistical information such as the number of successfully and failed data blocks can assess system stability and reliability, guiding further optimization and improvement. In summary, log content provides users with a comprehensive understanding of the storage and transmission status during data processing, facilitating troubleshooting, performance optimization, and system monitoring.

[0123] In the above embodiments, the data processing method has been described in detail. The present invention also provides embodiments corresponding to the data processing apparatus.

[0124] Figure 2 This is a schematic diagram of a data processing apparatus provided in an embodiment of the present invention. Figure 2 As shown, the device includes:

[0125] The reading module 10 is used to read data from the corresponding backend storage areas sequentially.

[0126] The judgment module 11 is used to determine whether the data belongs to other controllers based on the data's owner attribute; if it is confirmed that the data belongs to other controllers, the storage module 12 is triggered.

[0127] Storage module 12 is used to store the data information corresponding to the data into a temporary data structure;

[0128] The sending module 13 is used to send the data information in the temporary data structure to the corresponding controller after all data in the backend storage area has been read.

[0129] In some embodiments, the storage module 12 is specifically used to store the data address of the data into a temporary data structure, or to store the data into a temporary data structure;

[0130] Correspondingly, the sending module 13 is specifically used to uniformly send the data addresses in the temporary data structure to the corresponding controller, or uniformly send the data in the temporary data structure to the corresponding controller.

[0131] In some embodiments, the storage module 12 includes:

[0132] The first determination submodule is used to determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the temporary data structure based on the data.

[0133] The first storage submodule is used to store the data information corresponding to the data into a temporary data structure based on the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier.

[0134] In some embodiments, it also includes:

[0135] The second determination submodule is used to determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the current controller data structure based on the data.

[0136] The second storage submodule is used to store data into the current controller data structure based on the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier.

[0137] In some embodiments, the reading module 10 includes:

[0138] The read submodule is used to read B+ tree node information from the backend storage area according to a preset data read size.

[0139] The extraction submodule is used to extract metadata information based on B+ tree node information;

[0140] The metadata information includes at least the storage address and timestamp of the root node.

[0141] In some embodiments, the sending module 13 includes:

[0142] The splitting submodule is used to split the data information in the temporary data structure according to a preset size to obtain multiple sets of data information blocks;

[0143] Create a submodule to create a transport thread pool; the transport thread pool contains multiple transport threads.

[0144] The channel establishment submodule is used to establish multiple transmission channels with the controller;

[0145] The first allocation submodule is used to allocate each transmission thread to each transmission channel respectively;

[0146] The second allocation submodule is used to allocate each data information block to each transmission thread respectively;

[0147] The transmission submodule is used to transmit the corresponding data information blocks to the corresponding controllers through each transmission thread.

[0148] The verification submodule is used to verify each data information block during transmission using a preset algorithm to determine whether the data information block is correct. If the data information block is confirmed to be incorrect, the retransmission submodule is triggered; if the data information block is confirmed to be correct, the monitoring submodule is triggered.

[0149] The retransmission submodule is used to control the corresponding transmission thread to resend the data information block and trigger the verification submodule;

[0150] The monitoring submodule is used to monitor the confirmation message indicating that the characteristic data information block transmitted by the controller has been received.

[0151] The termination submodule is used to stop the transmission thread that has completed the transmission of data information blocks and close the corresponding transmission channel when an acknowledgment message is received.

[0152] In some embodiments, it also includes:

[0153] The clear submodule is used to clear data information in temporary data structures;

[0154] The log generation submodule is used to generate transmission logs that contain information about the data transmission process.

[0155] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0156] In addition, the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data processing method.

[0157] Figure 3 This is a schematic diagram of a data processing device provided in an embodiment of the present invention. Figure 3 As shown, the data processing device includes:

[0158] Memory 20 is used to store computer programs;

[0159] The processor 21 is configured to implement the steps of the data processing method as described in the above embodiments when executing a computer program.

[0160] The data processing device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0161] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0162] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the data processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the data processing method.

[0163] In some embodiments, the data processing device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0164] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the data processing device and may include more or fewer components than illustrated.

[0165] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0166] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] The foregoing has provided a detailed description of a data processing method, apparatus, device, and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0168] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data processing method, characterized in that, include: Read the data from the corresponding backend storage areas sequentially; Based on the ownership attribute of the data, determine whether the data belongs to other controllers; If it is confirmed that the data belongs to one of the other controllers, the data information corresponding to the data is stored in a temporary data structure; After all the data in the backend storage area has been read, the data information in the temporary data structure is sent to the corresponding controller. Sending the data information in the temporary data structure to the corresponding controller in a unified manner includes: The data information in the temporary data structure is split into multiple groups of data information blocks according to a preset size. Create a transport thread pool; wherein the transport thread pool contains multiple transport threads; Establish multiple transmission channels with the controller; Each of the aforementioned transmission threads is assigned to each of the aforementioned transmission channels; Each of the data information blocks is assigned to each of the transmission threads; The corresponding data information block is transmitted to the corresponding controller through each of the transmission threads; During transmission, each data information block is verified using a preset algorithm to determine whether the data information block is correct; If the data information block is confirmed to be incorrect, the corresponding transmission thread is controlled to resend the data information block, returning to the step of verifying each data information block through a preset algorithm during transmission to determine whether the data information block is correct; If the data information block is confirmed to be correct, then monitor the controller to transmit an acknowledgment message indicating that the data information block has been received. Upon receiving the confirmation message, the transmission thread that has completed the transmission of the data information block is stopped, and the corresponding transmission channel is closed. After uniformly sending the data information in the temporary data structure to the corresponding controller, the method further includes: Clear the data information from the temporary data structure; A transmission log containing the transmission process information of the data is generated. The transmission log includes at least log header information, data source and destination information, transmission process information, storage status, error and retransmission information, acknowledgment information, and statistical information, so that users can locate the root cause of the fault based on the error and retransmission information, and identify performance bottlenecks and evaluate system stability and reliability based on the statistical information.

2. The data processing method according to claim 1, characterized in that, The data information corresponding to the data is stored in a temporary data structure, including: The data address of the data is stored in the temporary data structure, or the data is stored in the temporary data structure; Correspondingly, the data information in the temporary data structure is uniformly sent to the corresponding controller, including: The data addresses in the temporary data structure are sent to the corresponding controller, or the data in the temporary data structure is sent to the corresponding controller.

3. The data processing method according to claim 1, characterized in that, The data information corresponding to the data is stored in a temporary data structure, including: Based on the data, determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the temporary data structure; Based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier, the data information corresponding to the data is stored in the temporary data structure.

4. The data processing method according to claim 1, characterized in that, If it is confirmed that the data does not belong to the other controllers, the method further includes: Based on the data, determine the pool identifier, volume identifier, page identifier, sector identifier, and controller identifier corresponding to the current controller data structure; The data is stored in the current controller data structure based on the pool identifier, the volume identifier, the page identifier, the sector identifier, and the controller identifier.

5. The data processing method according to claim 1, characterized in that, Read the data from the corresponding backend storage areas sequentially, including: Read the B+ tree node information in the backend storage area according to the preset data read size; Extract metadata information based on the B+ tree node information; The metadata information includes at least the storage address and timestamp of the root node.

6. A data processing apparatus, characterized in that, include: The read module is used to read data sequentially from the corresponding backend storage areas; The judgment module is used to determine whether the data belongs to other controllers based on the owner attribute of the data; if it is confirmed that the data belongs to other controllers, the storage module is triggered. The storage module is used to store the data information corresponding to the data into a temporary data structure; The sending module is used to send the data information in the temporary data structure to the corresponding controller after all the data in the backend storage area has been read; The sending module includes: The splitting submodule is used to split the data information in the temporary data structure according to a preset size to obtain multiple sets of data information blocks; Create a submodule for creating a transport thread pool; wherein the transport thread pool contains multiple transport threads; The channel establishment submodule is used to establish multiple transmission channels with the controller; The first allocation submodule is used to allocate each of the transmission threads to each of the transmission channels respectively; The second allocation submodule is used to allocate each of the data information blocks to each of the transmission threads respectively; A transmission submodule is used to transmit the corresponding data information block to the corresponding controller through each of the transmission threads; The verification submodule is used to verify each data information block during transmission using a preset algorithm to determine whether the data information block is correct; if the data information block is confirmed to be incorrect, the retransmission submodule is triggered; if the data information block is confirmed to be correct, the monitoring submodule is triggered. The retransmission submodule is used to control the corresponding transmission thread to resend the data information block and trigger the verification submodule; The monitoring submodule is used to monitor the confirmation message transmitted by the controller, which indicates that the data information block has been received. The termination submodule is used to stop the transmission thread that has completed the transmission of the data information block when the confirmation message is received, and to close the corresponding transmission channel. Also includes: The clearing submodule is used to clear the data information in the temporary data structure; The log generation submodule is used to generate a transmission log containing transmission process information of the data information. The transmission log includes at least log header information, data source and destination information, transmission process information, storage status, error and retransmission information, acknowledgment information, and statistical information, so that users can locate the root cause of the fault based on the error and retransmission information, and identify performance bottlenecks and evaluate system stability and reliability based on the statistical information.

7. A data processing device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 5.

Images