Data reading and writing method and device, electronic equipment and storage medium

Abstract

The invention provides a data reading and writing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that a data processing request of a server side is monitored, the data processing request is used for requesting to store configuration data of the server side, in response to the data processing request, the configuration data generated by the server side is summarized, and the configuration data is stored in the server side; and writing the configuration data of the server side into the distributed storage cluster through a pre-packaged data processing interface of the distributed storage cluster, and in response to a configuration data reading request of the server side corresponding to the back-end storage, synchronizing the configuration data to the back-end storage through the data processing interface. Through the distributed storage cluster, reliable storage of the global configuration data of the server side is realized, a single-point fault is prevented, consistency of the global configuration data is ensured, and the data synchronization effect of the iSCSI server side is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data reading and writing method, device, electronic equipment and storage medium. Background Art

[0002] ISCSI (Internet Small Computer System Interface) combines the existing ISCSI interface with Ethernet technology, and connects the iSCSI server (Target) and client (Initiator) based on the TCP / IP protocol, so that the encapsulated iSCSI data packets can be transmitted on the general Internet, and finally the iSCSI server is mapped as a storage space (disk) to the connected and authenticated client. Since iSCSI can send the SCSI protocol originally used only for the local machine through the TCP / IP network, the connection distance can be extended to an unlimited geographical area, and the number of connected servers is unlimited, realizing online expansion and even dynamic deployment.

[0003] At present, the configuration data in the iSCSI server (Target) is stored in a distributed manner, and the volume mapping information in the configuration data is saved in the local configuration file. Therefore, if the configuration data needs to be synchronized, it is obtained by reading and writing the local configuration file, and each synchronization node needs to be operated once, resulting in a high time consumption for command execution. If there is a node failure, it may cause inconsistency with the configuration data of other nodes after recovery, affecting the data synchronization effect of the iSCSI server. Summary of the invention

[0004] In view of this, the present invention aims to propose a data reading and writing method, device, electronic device and storage medium to solve the problem of low efficiency in obtaining configuration data by reading and writing local configuration files, and to achieve efficient and reliable data reading and writing.

[0005] According to a first aspect of the present invention, a data reading and writing method is provided, which is applied to a distributed storage cluster, and the method comprises:

[0006] Monitoring a data processing request from a server; the data processing request is used to request configuration data of a storage server;

[0007] In response to the data processing request, aggregating the configuration data generated by the server;

[0008] Writing the configuration data of the server into the distributed storage cluster through a data processing interface prepackaged by the distributed storage cluster;

[0009] In response to a request from the server to read configuration data corresponding to the backend storage, the configuration data is synchronized to the backend storage through the data processing interface.

[0010] Optionally, the monitoring server-side data processing request; the data processing request is used to request to store the configuration data of the server-side, including:

[0011] Using a preset monitoring thread to monitor the data processing request sent by the server;

[0012] The data processing request is parsed to determine that the server requests the distributed storage cluster to store the configuration data of the server.

[0013] Optionally, the aggregating the configuration data generated by the server in response to the data processing request includes:

[0014] In response to the data processing request, traverse the global configuration of the server to determine volume mapping configuration information, path number information, and authentication information generated by the server;

[0015] The volume mapping configuration information, path number information and authentication information are queued and cached to obtain aggregated configuration data.

[0016] Optionally, writing the configuration data of the server into the distributed storage cluster through a data processing interface prepackaged by the distributed storage cluster includes:

[0017] Pre-encapsulating the preset interfaces in the distributed storage cluster into multiple types of data processing interfaces; wherein the data processing interfaces include data writing interfaces and data modification interfaces;

[0018] In response to the configuration data write request from the server, calling a data write interface corresponding to the configuration data write request in the distributed storage cluster;

[0019] Writing the configuration data of the server into a preset storage area of ​​the distributed storage cluster through the data writing interface;

[0020] In response to the request for modifying configuration data from the server, determining a modification operation on configuration data of a preset storage area;

[0021] Calling the data modification interface corresponding to the configuration data modification request in the distributed storage cluster;

[0022] The modification operation is performed through the data modification interface to update the configuration information of the preset storage area.

[0023] Optionally, the data processing interface includes a data reading interface, and in response to a request for reading configuration data from a server corresponding to a backend storage, synchronizing the configuration data to the backend storage through the data processing interface includes:

[0024] In response to a request for reading configuration data from the server corresponding to the backend storage, calling a data reading interface corresponding to the request for reading configuration data in the distributed storage cluster;

[0025] The configuration data of the preset storage area of ​​the distributed storage cluster is synchronized to the backend storage through the data reading interface.

[0026] Optionally, before the monitoring server-side data processing request, the method further includes:

[0027] A distributed storage cluster is pre-deployed on the server side, and a connection between the distributed storage cluster and the server side is established; wherein the distributed cluster includes a primary node and secondary nodes for reading and writing data.

[0028] Optionally, after synchronizing the configuration data to the backend storage through the data processing interface in response to the request for reading configuration data from the server corresponding to the backend storage, the method further includes:

[0029] Monitoring the master node status of the distributed storage cluster; wherein the distributed storage cluster includes a cluster alarm interface;

[0030] If the master node is detected to be abnormal, a target master node is selected from the slave nodes, and the configuration data is backed up to the target master node;

[0031] The cluster alarm interface is used to send abnormal prompt information to the server side.

[0032] According to a second aspect of the present invention, a data reading and writing device is provided, which is applied to a distributed storage cluster, and the device comprises:

[0033] A monitoring request module, used for monitoring a data processing request from a server; the data processing request is used for requesting to store configuration data from the server;

[0034] A data aggregation module, used to aggregate the configuration data generated by the server in response to the data processing request;

[0035] A data writing module, used to write the configuration data of the server side into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;

[0036] The data reading module is used to respond to a request from the server to read configuration data corresponding to the backend storage, and synchronize the configuration data to the backend storage through the data processing interface.

[0037] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0038] processor;

[0039] a memory for storing instructions executable by the processor;

[0040] The processor is configured to execute the instructions to implement the data reading and writing method as described above.

[0041] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data reading and writing method described above are implemented.

[0042] The data reading and writing method provided by the embodiment of the present invention monitors the data processing request of the server side, responds to the data processing request, summarizes the configuration data generated by the server side, writes the configuration data of the server side to the distributed storage cluster through the data processing interface pre-encapsulated by the distributed storage cluster, and responds to the read configuration data request of the corresponding back-end storage of the server side, and synchronizes the configuration data to the back-end storage through the data processing interface. The embodiment of the present invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single point failure, and the distributed storage cluster can adapt to the growth of the data volume of the server side. By responding to the read and write requests in time, it ensures that the back-end storage and the server side can quickly synchronize the configuration data, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data, and further improves the data synchronization effect of the iSCSI server.

[0043] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 is a flowchart of a data reading and writing method provided by an embodiment of the present invention;

[0046] Figure 2 yes Figure 1 Flow chart of step 101 in the data reading and writing method provided by an embodiment of the present invention;

[0047] Figure 3 yes Figure 1 Flow chart of step 102 in the data reading and writing method provided by an embodiment of the present invention;

[0048] Figure 4 yes Figure 1 Flow chart of step 103 in the data reading and writing method provided by the embodiment of the present invention;

[0049] Figure 5 yes Figure 1 Flow chart of step 104 in the data reading and writing method provided by the embodiment of the present invention;

[0050] Figure 6 is a flowchart of another data reading and writing method provided by an embodiment of the present invention;

[0051] Figure 7 is a scenario schematic diagram of a data reading and writing method provided by an embodiment of the present invention;

[0052] Figure 8 It is a structural schematic diagram of a data reading and writing device provided by an embodiment of the present invention;

[0053] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0055] Reference Figure 1 , shows a flowchart of the steps of a data reading and writing method provided by an embodiment of the present invention, which is applied to a distributed storage cluster. The method may include:

[0056] Step 101, monitoring a data processing request on a server side; the data processing request is used to request to store configuration data on the server side.

[0057] In the embodiment of the present invention, the iSCSI protocol mainly includes two parts: iSCSI initiator and iSCSI target. The initiator is deployed on the client, and the target is deployed on the storage server. The iSCSI protocol is the process of mapping the remote procedure call of iSCSI to the IP protocol. Figure 7 , shows a scenario diagram of a data reading and writing method provided by an embodiment of the present invention. In this embodiment, a distributed storage cluster is built to store the global configuration information of the ISCSI target in the distributed storage cluster in a key / value manner. When the ISCSI server is working, the reading and writing of application data is directly written to the back-end storage, and the configuration data such as volume mapping information, path information, and chap information are written to the distributed storage cluster. If the back-end storage needs to obtain the configuration data of the ISCSI target, it can be read from the distributed storage cluster. The global configuration information of the ISCSI target is stored through the distributed storage cluster, which avoids the low efficiency of the traditional method of reading and writing data from local files, can support a larger amount of configuration data, and enables the back-end storage to share the global configuration information with the ISCSI server.

[0058] It should be noted that the distributed storage cluster in this embodiment adopts etcd (etc Distribute), which is a distributed consistent KV storage system for shared configuration and service discovery. It is a highly available distributed key-value pair storage database. etcd internally uses the Raft protocol as a consistency algorithm. The Raft protocol algorithm is a distributed consistency algorithm used to ensure data consistency in distributed systems.

[0059] Specifically, when the server is started, the distributed storage cluster initializes the data processing request monitoring mechanism. The monitoring mechanism can adopt polling, event-driven and other methods. The monitoring mechanism continuously listens to the data processing requests from the server. Once a request is received, it is processed immediately. The received data processing request can be parsed to extract the type, content, target storage location and other information of the request, and the request can be classified into different processing tasks, such as configuration data storage request, configuration data reading request, etc.

[0060] Step 102: In response to the data processing request, summarize the configuration data generated by the server.

[0061] In the embodiment of the present invention, when the monitoring mechanism captures the data processing request, it immediately responds to the request, starts the configuration data aggregation process, collects the generated configuration data from various modules or services on the server side, and aggregates the collected configuration data to ensure the integrity and consistency of the data. The aggregation process may include operations such as data merging, deduplication, and formatting, and the aggregated configuration data is stored in a temporary storage area, such as a memory cache or a local file system, for subsequent processing.

[0062] It should be noted that, specifically, the global configuration information related to iscsi target includes the following: volume mapping configuration information: Key: pool_name (storage pool name) and target_name (iscsi server target name), Value: version (initialized to 0), lun_id (volume name), target_ip_list (IP list of bound hosts), target_initor_list (bound initiator list); path number information: Key: path_num (number of paths), value: max_path_num (maximum number of paths), min_path_num (minimum number of paths); authentication information: Key: chap_user (CHAP user name), Value: including incoming / outcoming (authentication direction) version (initialized to 0), password (encrypted password), initor_list (bound initiator list).

[0063] Step 103: write the configuration data of the server into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster.

[0064] In an embodiment of the present invention, the configuration data of the server side is written into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster, wherein the preset interface in the distributed storage cluster is pre-packaged as multiple types of data processing interfaces, and the data processing interface includes a data writing interface and a data modification interface. The distributed storage cluster responds to the write configuration data request of the server side, calls the data writing interface corresponding to the write configuration data request in the distributed storage cluster, and writes the configuration data of the server side into the preset storage area of ​​the distributed storage cluster through the data writing interface; responds to the modification configuration data request of the server side, determines the modification operation of the configuration data of the preset storage area, calls the data modification interface corresponding to the modification configuration data request in the distributed storage cluster, performs the modification operation through the data modification interface, and updates the configuration information of the preset storage area.

[0065] Step 104, in response to the request for reading configuration data from the server corresponding to the backend storage, the configuration data is synchronized to the backend storage through the data processing interface.

[0066] In an embodiment of the present invention, when the iSCSI server is working, application data is directly written to the backend storage. The backend storage corresponding to the server includes physical storage devices such as disk arrays (RAID) or virtual storage devices such as virtual disks (Virtual Disks), etc., which are not specifically limited here. The application data of the iSCSI server can be directly stored in the backend device. When the backend storage needs global configuration data when running the application data, it requests the distributed storage cluster to read the configuration data. The distributed storage cluster responds to the read configuration data request of the backend storage corresponding to the server, and synchronizes the configuration data to the backend storage through the data processing interface.

[0067] The data reading and writing method provided by the embodiment of the present invention monitors the data processing request of the server side, responds to the data processing request, summarizes the configuration data generated by the server side, writes the configuration data of the server side to the distributed storage cluster through the data processing interface pre-encapsulated by the distributed storage cluster, and responds to the read configuration data request of the corresponding back-end storage of the server side, and synchronizes the configuration data to the back-end storage through the data processing interface. The embodiment of the present invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single point failure, and the distributed storage cluster can adapt to the growth of the data volume of the server side. By responding to the read and write requests in time, it ensures that the back-end storage and the server side can quickly synchronize the configuration data, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data, and further improves the data synchronization effect of the iSCSI server.

[0068] Further, see Figure 2 , showing Figure 1 A flowchart of step 101 in a data reading and writing method is provided. The method is substantially the same as the data reading and writing method provided in the first embodiment of the present invention. Step 101 may include:

[0069] Step 201, using a preset monitoring thread to monitor a data processing request sent by a server;

[0070] Step 202: parse the data processing request to determine that the server requests the distributed storage cluster to store the configuration data of the server.

[0071] It should be noted that, in an embodiment of the present invention, when the server side is started, the distributed storage cluster initializes one or more monitoring threads, and the monitoring threads are responsible for listening to the data processing requests from the server side. Specifically, a specific listening port can be assigned to the monitoring thread to ensure that the data processing requests from the server side can be sent through the port. The monitoring thread continuously listens to the designated port, and once a data processing request is received, it is immediately cached into the memory or temporary storage for subsequent processing.

[0072] In this embodiment, the cached data processing requests can be preliminarily filtered and classified to identify requests that need to be processed by the distributed storage cluster, extract data processing requests from the cache, use a predefined parser to parse the request content, extract key information such as request type, data type, storage requirements, etc., and verify the parsed requests to ensure the legitimacy and integrity of the requests. Based on the parsed request information, determine the type and storage location of the configuration data that needs to be stored on the server side, so that the configuration data can be stored in the distributed storage cluster.

[0073] The embodiment of the present invention can capture data processing requests from the server in real time by presetting a monitoring thread, ensuring that the requests can be processed in a timely manner, and storing configuration data in a distributed storage cluster, thereby improving data reliability and redundancy and preventing single point failures.

[0074] Further, see Figure 3 , showing Figure 1 A flowchart of step 102 in a data reading and writing method is provided. The method is substantially the same as the data reading and writing method provided in the first embodiment of the present invention. Step 102 may include:

[0075] Step 301, in response to a data processing request, traverse the global configuration of the server side to determine the volume mapping configuration information, path number information and authentication information generated by the server side;

[0076] Step 302: queue and cache the volume mapping configuration information, the number of paths information, and the authentication information to obtain aggregated configuration data.

[0077] It should be noted that, in an embodiment of the present invention, the distributed storage cluster responds to the data processing request of the server side, traverses the global configuration of the server side, and determines the volume mapping configuration information, path number information and authentication information generated by the server side. Specifically, when the monitoring thread captures the data processing request, it immediately responds to the request, starts the configuration traversal process, traverses the global configuration of the server side, and obtains all relevant configuration information. In this embodiment, the traversal can be performed recursively or iteratively to ensure that no configuration items are missed.

[0078] Specifically, during the traversal process, the configuration information related to volume mapping is identified and extracted, and the configuration information includes the name of the volume, mapping path, access rights, etc.; the traversal is continued to extract the configuration information related to the number of paths, and the path number information includes the number of paths, the hierarchical structure of the paths, the access frequency of the paths, etc.; the authentication information is extracted; the configuration information related to authentication is extracted, and the authentication information includes user identity authentication, permission control, access tokens, etc. The volume mapping configuration information, path number information and authentication information are queued and cached to obtain summarized configuration data.

[0079] In this embodiment, one or more queue caches are initialized on the server side for temporarily storing the extracted configuration information. The queue cache can manage data in a first-in-first-out (FIFO) manner, and the extracted volume mapping configuration information, path number information, and authentication information are queued in sequence. Each configuration information item is used as an independent queue element to manage the queue to ensure that the capacity and performance of the queue meet the requirements. When all configuration information is successfully queued, the data in the queue is summarized. The summary process may include data merging, deduplication, formatting, and other operations to ensure that the final configuration data is complete and consistent, so that the summarized configuration data can be stored in the storage location specified by the distributed storage cluster.

[0080] The embodiment of the present invention ensures that all relevant configuration information is extracted through global configuration traversal, can respond to data processing requests quickly and accurately, and generate summarized configuration data. It ensures that configuration information is not lost during the processing process through queue caching, thereby improving the efficiency of extracting configuration information and reducing processing time.

[0081] Further, see Figure 4 , showing Figure 1 A flowchart of step 103 in a data reading and writing method is provided. The method is substantially the same as the data reading and writing method provided in the first embodiment of the present invention. Step 103 may include:

[0082] Step 401, pre-encapsulate the preset interfaces in the distributed storage cluster into multiple types of data processing interfaces; wherein the data processing interfaces include data writing interfaces and data modification interfaces;

[0083] Step 402, in response to the configuration data write request from the server, calling a data write interface corresponding to the configuration data write request in the distributed storage cluster;

[0084] Step 403, writing the configuration data of the server into a preset storage area of ​​the distributed storage cluster through a data writing interface;

[0085] Step 404, in response to the request for modifying configuration data from the server, determining a modification operation on the configuration data of the preset storage area;

[0086] Step 405, calling the data modification interface corresponding to the configuration data modification request in the distributed storage cluster;

[0087] Step 406: Perform a modification operation through the data modification interface to update the configuration information of the preset storage area.

[0088] In an embodiment of the present invention, the preset interfaces in the distributed storage cluster are pre-encapsulated into multiple types of data processing interfaces, and the data processing interfaces include data writing interfaces and data modification interfaces. Specifically, in the distributed storage cluster, multiple types of data processing interfaces are pre-defined, and the data processing interfaces can be divided into multiple types according to storage requirements, including data writing interfaces, data modification interfaces, and data reading interfaces. By way of example, in this embodiment, the encapsulated data processing interfaces are comprehensively tested to ensure their stability and performance in different scenarios. The interfaces that encapsulate storage configuration data include data writing interfaces, data reading interfaces, and data modification interfaces, wherein the data writing interface is wirite_iscsi_conf_data(key, value), the data reading interface is read_iscsi_conf_data(key, value); and the data modification interface is modify_iscsi_conf_data(key, value).

[0089] Specifically, in response to a write configuration data request from the server, a data write interface corresponding to the write configuration data request in the distributed storage cluster is called. After receiving the write configuration data request, the distributed storage cluster immediately responds to the request and starts the data write process. According to the request content, the pre-packaged data write interface in the distributed storage cluster is called, and the data write interface is executed to write the configuration data to the distributed storage cluster. The configuration data on the server side is written to a preset storage area of ​​the distributed storage cluster through the data write interface. According to the type of configuration data and access requirements, a suitable preset storage area is selected. The storage area may be a node in the distributed storage cluster or a specific storage partition.

[0090] Specifically, in this embodiment, after receiving the request to modify the configuration data, the distributed storage cluster immediately responds to the request, starts the data modification process, and determines the modification operation to be performed according to the request content. The modification operation includes data update, data deletion, data append, etc. The configuration data to be modified is located in the preset storage area to ensure the accuracy of the modification operation. According to the modification operation type, the pre-packaged data modification interface in the distributed storage cluster is called, and the data modification interface is executed to modify the configuration data in the preset storage area. Through the data modification interface, the determined modification operation is executed to update the configuration information in the preset storage area. After the modification operation is completed, the modified configuration data is verified to ensure the successful execution of the modification operation.

[0091] The embodiments of the present invention pre-package data processing interfaces, respond to write and modify requests, and execute data write and modify operations, thereby realizing efficient, reliable, and flexible processing of server-side configuration data. Through the preset storage areas of the distributed storage cluster, the reliability and redundancy of data are improved to prevent single point failures. The distributed storage cluster can dynamically expand storage capacity and processing capabilities according to demand to adapt to the growth of server-side data volume. Through reasonable modification operation execution and data verification strategies, it is ensured that configuration data can be updated quickly and efficiently.

[0092] Further, see Figure 5 , showing Figure 1 A flowchart of step 104 in a data reading and writing method is provided, which is substantially the same as the data reading and writing method provided in the first embodiment of the present invention. The data processing interface includes a data reading interface, and step 104 may include:

[0093] Step 501, in response to a request for reading configuration data corresponding to backend storage from a server, calling a data reading interface corresponding to the request for reading configuration data in a distributed storage cluster;

[0094] Step 502: Synchronize the configuration data of the preset storage area of ​​the distributed storage cluster to the backend storage through the data reading interface.

[0095] It should be noted that, in an embodiment of the present invention, the distributed storage cluster responds to the read configuration data request of the server-side corresponding backend storage. After receiving the read configuration data request, it immediately responds to the request, starts the data reading process, calls the data reading interface corresponding to the read configuration data request in the distributed storage cluster, executes the data reading interface, and reads the configuration data from the distributed storage cluster. Specifically, through the data reading interface, the configuration data is read from the preset storage area of ​​the distributed storage cluster, and the read configuration data is synchronized to the backend storage of the server-side. The synchronization process can adopt full synchronization or incremental synchronization. The appropriate synchronization strategy is selected according to actual needs. After the synchronization operation is completed, the synchronized configuration data is verified to ensure the successful execution of the synchronization operation.

[0096] In the embodiment of the present invention, the back-end storage of the iSCSI server refers to the physical or virtual storage device that actually stores the data. These devices communicate with the client through the iSCSI protocol. The back-end storage of the iSCSI server includes physical storage devices such as disk arrays (RAID) or virtual storage devices such as virtual disks (Virtual Disks), etc., which are not specifically limited here. The application data of the iSCSI server can be directly stored in the back-end device.

[0097] The embodiments of the present invention improve the reliability and redundancy of data and prevent single point failures through the preset storage area of ​​the distributed storage cluster. The distributed storage cluster can dynamically expand the storage capacity and processing power according to demand to adapt to the growth of the server-side data volume. Through a reasonable synchronization strategy, it is ensured that the configuration data can be quickly and efficiently synchronized to the back-end storage, so that the back-end storage and the ISCSI server target can share global configuration information.

[0098] Reference Figure 6 , shows a flow chart of steps of another data reading and writing method provided by an embodiment of the present invention, which method is basically the same as the data reading and writing method provided by the first embodiment of the present invention, except that the method may further include:

[0099] Step 105, pre-deploy a distributed storage cluster on the server side, and create a connection between the distributed storage cluster and the server side; wherein the distributed cluster includes a primary node and secondary nodes for reading and writing data.

[0100] In an embodiment of the present invention, a distributed storage cluster is deployed on the server side in advance, a multi-node etcd cluster is built, and a connection between the distributed storage cluster and the server side is created. The distributed cluster includes a master node and a slave node for reading and writing data to ensure the fault tolerance of the data. In this embodiment, the brain split phenomenon is avoided and an odd number of nodes are deployed. For example, there are 4 nodes. If 2 nodes have different networks from the other 2 nodes, it is equivalent to the cluster being divided into two network areas, which will cause the entire etcd cluster to be unusable, or there are two master ends, that is, the master nodes of the two areas can be used, and the system cannot determine which area to select for configuring data storage; if it is an odd number of nodes, etcd executes the majority principle internally, that is, as long as the number of normally available nodes is greater than half of the total number of nodes in the original cluster, the cluster can provide services normally.

[0101] Step 101, monitoring a data processing request of a server; the data processing request is used to request to store configuration data of the server;

[0102] Step 102, in response to the data processing request, summarizing the configuration data generated by the server;

[0103] Step 103, writing the configuration data of the server into the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster;

[0104] Step 104, in response to the request for reading configuration data from the server corresponding to the backend storage, the configuration data is synchronized to the backend storage through the data processing interface.

[0105] On the basis of achieving the beneficial effects of the first embodiment, the embodiment of the present invention pre-deploys a distributed storage cluster on the server side and creates a connection between the distributed storage cluster and the server side. Through the distributed storage cluster, a high-availability storage solution is provided to prevent single point failures. The distributed storage cluster supports dynamic expansion to meet the growing storage needs of server-side configuration data.

[0106] Specifically, after synchronizing the configuration data to the backend storage through the data processing interface in response to the request for reading configuration data from the server corresponding to the backend storage, the method further includes:

[0107] First, monitoring the master node status of the distributed storage cluster; wherein the distributed storage cluster includes a cluster alarm interface;

[0108] Secondly, if the master node is detected to be abnormal, a target master node is selected from the secondary nodes, and the configuration data is backed up to the target master node;

[0109] Secondly, the cluster alarm interface is used to send abnormal prompt information to the server side.

[0110] It should be noted that, in the embodiment of the present invention, when the distributed storage cluster is started, the master node status monitoring mechanism is initialized. The monitoring mechanism can adopt heartbeat detection, status polling and other methods. The master node regularly sends heartbeat signals to the monitoring mechanism to indicate that it is operating normally. The monitoring mechanism continuously receives heartbeat signals and records the status of the master node. The monitoring mechanism regularly polls the status information of the master node, including memory usage, I / O, etc., to ensure the health status of the master node. The cluster alarm interface is initialized to send abnormal prompt information to the server side when the master node status is abnormal. If the master node status is abnormal, the target master node is selected from the secondary nodes, and the configuration data is backed up to the target master node. The target master node is selected from the secondary nodes of the distributed storage cluster. The selection process can be evaluated based on factors such as the load situation, health status, and historical performance of the secondary nodes. The configuration data in the master node is backed up to the target master node. The backup process should ensure the integrity and consistency of the data. After completing the configuration data backup, the target secondary node is switched to the master node to ensure the normal operation of the distributed storage cluster.

[0111] Specifically, the cluster alarm interface is used to send abnormal prompt information to the server side. After the abnormal status of the master node is detected and the secondary node election and configuration data backup are completed, the abnormal prompt information is generated. The alarm information should include details such as the abnormal type, abnormal time, and processing measures. The cluster alarm interface is called to send the generated abnormal prompt information to the server side. The server side can record the sending log of the alarm information, including the sending time, recipient, alarm content and other information, to facilitate subsequent troubleshooting.

[0112] The embodiment of the present invention ensures the high availability of the distributed storage cluster through cluster secondary node election and master node switching, prevents single point failure, configures the data backup process, ensures data consistency, avoids data loss or damage, and can send abnormal prompt information to the server in time through the cluster alarm interface to ensure that problems can be responded to and handled quickly, thereby achieving high availability and data consistency of the distributed storage cluster.

[0113] Reference Figure 8 , shows a schematic diagram of the structure of a data reading and writing device provided by an embodiment of the present invention, which is applied to a distributed storage cluster, and the device includes:

[0114] A monitoring request module 601 is used to monitor a data processing request from a server; the data processing request is used to request to store configuration data from the server;

[0115] A data aggregation module 602, configured to aggregate the configuration data generated by the server in response to the data processing request;

[0116] A data writing module 603 is used to write the configuration data of the server into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster;

[0117] The data reading module 604 is used to respond to a request from the server to read configuration data corresponding to the backend storage, and synchronize the configuration data to the backend storage through the data processing interface.

[0118] Furthermore, the monitoring request module 601 includes:

[0119] A monitoring submodule, used to monitor the data processing request sent by the server using a preset monitoring thread;

[0120] The parsing submodule is used to parse the data processing request and determine that the server requests the distributed storage cluster to store the configuration data of the server.

[0121] Furthermore, the data aggregation module 602 includes:

[0122] A traversal submodule, for traversing the global configuration of the server side in response to the data processing request, and determining the volume mapping configuration information, path number information and authentication information generated by the server side;

[0123] The cache submodule is used to queue cache the volume mapping configuration information, path number information and authentication information to obtain summarized configuration data.

[0124] Furthermore, the data writing module 603 includes:

[0125] The encapsulation submodule is used to encapsulate the preset interfaces in the distributed storage cluster into multiple types of data processing interfaces in advance; wherein the data processing interface includes a data writing interface and a data modification interface;

[0126] A first calling submodule, configured to call a data writing interface corresponding to the write configuration data request in the distributed storage cluster in response to the write configuration data request of the server;

[0127] A writing submodule, used to write the configuration data of the server into a preset storage area of ​​the distributed storage cluster through the data writing interface;

[0128] A determination submodule, configured to determine a modification operation on the configuration data of a preset storage area in response to a request for modifying configuration data from the server;

[0129] The second calling submodule is used to call the data modification interface corresponding to the configuration data modification request in the distributed storage cluster;

[0130] The modification submodule is used to perform the modification operation through the data modification interface to update the configuration information of the preset storage area.

[0131] Further, the data processing interface includes a data reading interface, and the data reading module 604 includes:

[0132] A third calling submodule is used to respond to the configuration data reading request corresponding to the backend storage of the server side, and call the data reading interface corresponding to the configuration data reading request in the distributed storage cluster;

[0133] The synchronization submodule is used to synchronize the configuration data of the preset storage area of ​​the distributed storage cluster to the backend storage through the data reading interface.

[0134] Furthermore, the device also includes:

[0135] The cluster creation module is used to pre-deploy a distributed storage cluster on the server side and to create a connection between the distributed storage cluster and the server side; wherein the distributed cluster includes a primary node and a secondary node for reading and writing data.

[0136] Furthermore, the device also includes:

[0137] A monitoring node module, used to monitor the master node status of the distributed storage cluster; wherein the distributed storage cluster includes a cluster alarm interface;

[0138] A data backup module is used to select a target master node from the secondary nodes and back up the configuration data to the target master node if the state of the master node is detected to be abnormal;

[0139] The abnormal prompt module is used to send abnormal prompt information to the server side using the cluster alarm interface.

[0140] The data reading and writing device provided by the embodiment of the present invention monitors the data processing request of the server side, responds to the data processing request, summarizes the configuration data generated by the server side, writes the configuration data of the server side to the distributed storage cluster through the data processing interface pre-packaged by the distributed storage cluster, and responds to the request for reading configuration data of the corresponding back-end storage of the server side, and synchronizes the configuration data to the back-end storage through the data processing interface. The embodiment of the present invention realizes the reliable storage of the global configuration data of the server side through the distributed storage cluster, prevents single point failure, and the distributed storage cluster can adapt to the growth of the data volume of the server side. By responding to the read and write requests in time, it ensures that the back-end storage and the server side can quickly synchronize the configuration data, realizes the efficient, reliable and flexible processing of the configuration data of the server side, ensures the consistency of the global configuration data, and further improves the data synchronization effect of the iSCSI server.

[0141] Reference Fig. 9 , an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0142] Memory 703, used for storing computer programs;

[0143] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0144] Monitoring a data processing request from a server; the data processing request is used to request configuration data of a storage server;

[0145] In response to the data processing request, aggregating the configuration data generated by the server;

[0146] Writing the configuration data of the server into the distributed storage cluster through a data processing interface prepackaged by the distributed storage cluster;

[0147] In response to a request from the server to read configuration data corresponding to the backend storage, the configuration data is synchronized to the backend storage through the data processing interface.

[0148] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0149] The communication interface is used for communication between the above terminal and other devices.

[0150] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0151] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the data reading and writing method described in any one of the above embodiments.

[0153] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data reading and writing device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0154] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0155] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data reading and writing method, characterized in that: Applied to a distributed storage cluster, the method includes: Monitoring a data processing request from a server; the data processing request is used to request configuration data of a storage server; In response to the data processing request, aggregating the configuration data generated by the server; Writing the configuration data of the server into the distributed storage cluster through a data processing interface prepackaged by the distributed storage cluster; In response to a request from the server to read configuration data corresponding to the backend storage, the configuration data is synchronized to the backend storage through the data processing interface.

2. The method according to claim 1, characterized in that The monitoring server-side data processing request; The data processing request is used to request the configuration data of the storage server, including: Using a preset monitoring thread to monitor the data processing request sent by the server; The data processing request is parsed to determine that the server requests the distributed storage cluster to store the configuration data of the server.

3. The method according to claim 1, characterized in that The step of aggregating the configuration data generated by the server in response to the data processing request includes: In response to the data processing request, traverse the global configuration of the server to determine volume mapping configuration information, path number information, and authentication information generated by the server; The volume mapping configuration information, path number information and authentication information are queued and cached to obtain aggregated configuration data.

4. The method according to claim 1, characterized in that: Writing the configuration data of the server side into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster includes: Pre-encapsulating the preset interfaces in the distributed storage cluster into multiple types of data processing interfaces; wherein the data processing interfaces include data writing interfaces and data modification interfaces; In response to the configuration data write request from the server, calling a data write interface corresponding to the configuration data write request in the distributed storage cluster; Writing the configuration data of the server into a preset storage area of ​​the distributed storage cluster through the data writing interface; In response to the request for modifying configuration data from the server, determining a modification operation on configuration data of a preset storage area; Calling the data modification interface corresponding to the configuration data modification request in the distributed storage cluster; The modification operation is performed through the data modification interface to update the configuration information of the preset storage area.

5. The method according to claim 1, characterized in that The data processing interface includes a data reading interface, and in response to a request for reading configuration data from a server corresponding to a backend storage, synchronizing the configuration data to the backend storage through the data processing interface includes: In response to a request for reading configuration data from the server corresponding to the backend storage, calling a data reading interface corresponding to the request for reading configuration data in the distributed storage cluster; The configuration data of the preset storage area of ​​the distributed storage cluster is synchronized to the backend storage through the data reading interface.

6. The method according to claim 1, characterized in that Before the monitoring server-side data processing request, the method further includes: A distributed storage cluster is pre-deployed on the server side, and a connection between the distributed storage cluster and the server side is established; wherein the distributed cluster includes a primary node and secondary nodes for reading and writing data.

7. The method according to claim 6, characterized in that After responding to the request for reading configuration data from the server corresponding to the backend storage, synchronizing the configuration data to the backend storage through the data processing interface, the method further includes: Monitoring the master node status of the distributed storage cluster; wherein the distributed storage cluster includes a cluster alarm interface; If the master node is detected to be abnormal, a target master node is selected from the slave nodes, and the configuration data is backed up to the target master node; The cluster alarm interface is used to send abnormal prompt information to the server side.

8. A data reading and writing device, characterized in that: Applied to a distributed storage cluster, the device comprises: A monitoring request module, used for monitoring a data processing request from a server; the data processing request is used for requesting to store configuration data from the server; A data aggregation module, used to aggregate the configuration data generated by the server in response to the data processing request; A data writing module, used to write the configuration data of the server side into the distributed storage cluster through a data processing interface pre-packaged by the distributed storage cluster; The data reading module is used to respond to a request from the server to read configuration data corresponding to the backend storage, and synchronize the configuration data to the backend storage through the data processing interface.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the data reading and writing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the data reading and writing method according to any one of claims 1 to 7 is implemented.

Images