Local disk data compensation method, device, equipment and storage medium

Abstract

An embodiment of the present application provides a local disk data compensation method, apparatus, device and storage medium. The method includes, after the target microservice is started, obtaining the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice, and then determining the number of combined data and the number of characteristic data. If the number of combined data is less than the number of characteristic data, determining the differential data offset position based on the number of combined data and the preset characteristic data length, and determining the differential data identifier, differential characteristic data and differential structured data, appending the differential structured data and differential characteristic data to the local disk can simply and conveniently determine the differential data offset position, and then obtain differential structured data. By appending the differential structured data and differential characteristic data to the local disk to compensate for the data of the local disk, the data retrieval speed is improved and the service recovery time is reduced during the synchronization process.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a local disk data compensation method, apparatus, device, and storage medium. Background Art

[0002] Currently, in order to enhance the system's disaster recovery capabilities, distributed systems are becoming increasingly accepted by the public, and the shared storage brought by distribution is also increasingly used by the public. In the security field, the optimal way is to store feature files directly on the hard disk because of the large size of individual files, large storage database capacity, and slow query speed. However, due to the characteristics of distributed shared storage, redundant backup of stored data and data sharing, the data reading speed becomes relatively slow.

[0003] Distributed shared storage is a stable distributed shared storage. If a node in a cluster experiences downtime, data synchronization in related technologies requires reading all files from the distributed shared storage and copying them to the local disks of other cluster nodes. When the data volume is large, synchronization can take a very long time due to the limited read speed of the distributed shared storage. During the synchronization process, data retrieval services cannot be quickly provided, resulting in slow data retrieval and long service recovery times. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a local disk data compensation method, device, equipment and storage medium for solving the technical problems of slow data retrieval speed and long service recovery time during synchronization.

[0005] In response to the above problems, the present invention provides a local disk data compensation method, which includes: when the target microservice is started, obtaining the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice, the local disk stores the characteristic data and structured data of the target microservice, and the shared storage stores the characteristic data of the target microservice; determining the number of combined data according to the combined data capacity and the preset combined data length, and determining the number of characteristic data according to the characteristic data capacity and the preset characteristic data length; if the number of combined data is less than the number of characteristic data, determining the difference data offset position based on the combined data number and the preset characteristic data length, and determining the difference data identifier and the difference characteristic data; determining the difference structured data based on the difference data identifier, and appending the difference structured data and the difference characteristic data to the local disk to perform data compensation on the local disk.

[0006] In one embodiment of the present invention, the difference data offset position is determined based on the number of combined data and the preset feature data length, and the difference data identifier and the difference feature data are determined, including: determining the combined feature length according to the number of combined data and the preset feature data length; determining the difference data offset position of the difference data in the shared storage based on the combined feature length; starting from the feature data at the difference data offset position in the shared storage, consuming the obtained feature data and the initial data identifier as the difference feature data and the difference data identifier, and the initial data identifier of each feature data is also stored in the shared storage.

[0007] In one embodiment of the present invention, determining the differential structured data based on the differential data identifier includes: querying from a preset database based on the differential data identifier to obtain the differential structured data, wherein the preset database stores the structured data of the target microservice and the initial data identifier of the feature data corresponding to the structured data.

[0008] In one embodiment of the present invention, appending the differential structured data and the differential feature data to the local disk includes: combining the differential structured data and the differential feature data to obtain a file to be stored, the data length of the file to be stored being the preset combined data length; and appending the file to be stored to the local disk.

[0009] In one embodiment of the present invention, before obtaining the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice, the method includes: during the startup of the target microservice, obtaining the backup information existence status and the current startup node of the target microservice; if the backup information existence status is existence, determining the pointing node and pointing disk of the target microservice based on the backup information of the target microservice; if the current startup node is different from the pointing node, restarting and drifting the target microservice by calling the service restart interface of the cluster until the current startup node after the restart drift is the same as the pointing node.

[0010] In one embodiment of the present invention, if the current startup node is different from the pointing node, the target microservice is restarted and drifted by calling the service restart interface of the cluster until the current startup node after the restart drift is the same as the pointing node. The method also includes: if the target microservice is started, obtaining an idle disk of an idle node in the environment, and the idle node is different from the current startup node; and determining the idle disk as a backup disk of the target microservice.

[0011] In one embodiment of the present invention, if the backup information existence status is non-existent, the method further includes: obtaining the current number of startup services and the number of local disks of the cluster; determining the minimum number of disks based on the current number of startup services, and the minimum number of disks is greater than or equal to twice the current number of startup services; if the number of local disks is greater than the minimum number of disks, backing up the feature data and structured data of the target microservice to a backup disk, and the backup disk is different from the local disk mounted by the target microservice.

[0012] In one embodiment of the present invention, the method further includes: obtaining the service status and disk status of the target microservice; if the service status is normal and the disk status is normal, determining the service backup status as backed up, and synchronizing the file data of the local disk and the backup disk by starting a synchronization thread; if the service status is abnormal, or the disk status is abnormal, determining the service backup status as not backed up, and not performing data backup for the time being.

[0013] In one embodiment of the present invention, if the current startup node of the target microservice crashes, the backup node of the target microservice is restored to the new current startup node; new backup information is generated based on the new current startup node, new local disk information, new backup node, and new backup disk information, and the historical backup information of the target microservice is replaced with the new backup information.

[0014] In one embodiment of the present invention, after restoring the backup node of the target microservice as the new current startup node, the method further includes: obtaining an idle node and an idle disk; if there is no idle node or no idle disk, not performing data backup; polling the idle node and the idle disk at a preset interval until the idle node and the idle disk exist; determining the idle node and the idle disk as a backup node and a backup disk, and performing data backup.

[0015] An embodiment of the present invention also provides a local disk data compensation device, which includes: a capacity acquisition module, which is used to obtain the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice after the target microservice is started, the local disk stores the characteristic data and structured data of the target microservice, and the shared storage stores the characteristic data of the target microservice; a data quantity determination module, which is used to determine the combined data quantity according to the combined data capacity and the preset combined data length, and determine the characteristic data quantity according to the characteristic data capacity and the preset characteristic data length; a difference determination module, which is used to determine the difference data offset position based on the combined data quantity and the preset characteristic data length if the combined data quantity is less than the characteristic data quantity, and determine the difference data identifier and the difference characteristic data; a data compensation module, which is used to determine the difference structured data based on the difference data identifier, and append the difference structured data and the difference characteristic data to the local disk to perform data compensation on the local disk.

[0016] An embodiment of the present invention also provides an electronic device, comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement a method as described in any one of the above embodiments.

[0017] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to enable the computer to execute the method as described in any one of the above embodiments.

[0018] As described above, the local disk data compensation method, device, equipment, and storage medium provided by the present invention have the following beneficial effects:

[0019] The method comprises the following steps: after a target microservice is started, obtaining the combined data capacity of a local disk mounted by the target microservice and the feature data capacity stored in a shared storage of the target microservice, determining the combined data quantity based on the combined data capacity and a preset combined data length, and determining the feature data quantity based on the feature data capacity and the preset feature data length; if the combined data quantity is less than the feature data quantity, determining the difference data offset position based on the combined data quantity and the preset feature data length, determining the difference data identifier and the difference feature data, determining the difference structured data based on the difference data identifier, appending the difference structured data and the difference feature data to the local disk to compensate for data on the local disk; storing the full amount of feature data in the full amount of data in the shared storage, and storing the feature data and structured data on the local disk; comparing the combined data quantity on the local disk with the feature data quantity in the shared storage to determine whether data compensation is required on the local disk; and simply and conveniently determining the difference data offset position to obtain the difference structured data; appending the difference structured data and the difference feature data to the local disk to compensate for data on the local disk, thereby improving data retrieval speed and reducing service recovery time during the synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a local disk data compensation method shown in an exemplary embodiment of the present application.

[0021] Figure 2 This is a flowchart of a specific example of a local disk data compensation method shown in an exemplary embodiment of the present application.

[0022] Figure 3 This is a flowchart of another specific example of a local disk data compensation method shown in an exemplary embodiment of the present application.

[0023] Figure 4 It is a block diagram of a local disk data compensation device shown in an exemplary embodiment of the present application.

[0024] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0026] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0027] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a local disk data compensation method shown in an exemplary embodiment of the present application. Figure 1 As shown, the method includes at least steps S101 to S104, which are described in detail as follows:

[0028] Step S101: After the target microservice is started, the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice are obtained.

[0029] The target microservice is the microservice currently being started in the cluster. There may be many microservices in the cluster, some of which may be in the startup state, some may be starting, and some may not yet be started. In this embodiment, the target microservice includes the microservice that has started.

[0030] The target microservice's service data is typically stored using a primary disk and a backup disk. The current primary disk of the target microservice is the local disk mounted on the target microservice in this embodiment of the application. When the target microservice's primary disk changes, the local disk (local disk) mounted on the target microservice in this embodiment is changed to the new primary disk (which may be the original backup disk).

[0031] In one embodiment, when a service is started, it is first determined whether the started service has backup information, and then the service status and disk status of the entire cluster environment are determined, and the backup status (service backup status) is confirmed based on the service status and disk status.

[0032] See Figure 2 , Figure 2 This is a flowchart of a specific example of a local disk data compensation method shown in an exemplary embodiment of the present application. Figure 2 As shown, in one embodiment, before obtaining the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice, the method further includes:

[0033] Step S201, during the startup process of the target microservice, obtain the backup information existence status and the current startup node of the target microservice. At this time, the target microservice has not yet been successfully started. The backup information existence status can be determined by searching the database that stores the backup information of multiple microservices to determine the backup information existence status corresponding to the target microservice. If the backup information corresponding to the target microservice exists in the database, the backup information existence status is "exist". Otherwise, if the backup information corresponding to the target microservice does not exist in the database, the backup status existence status is "non-existent". An exemplary method is to search in the database based on the service identifier of the target microservice. The backup information in the database is pre-stored corresponding to the service identifier corresponding to the backup information. The method for determining the current startup node (i.e., the master node) can be implemented in a manner known to those skilled in the art and is not limited here.

[0034] Step S202: If the backup information exists, determine the target node and disk of the target microservice based on the backup information of the target microservice. The backup information includes the target node and disk, which can be pre-set by a person skilled in the art. Generally, the target node and disk are the primary node and primary disk of the target microservice (if a primary-backup node switch occurs, the primary node and primary disk after the switch).

[0035] In step S203, if the current startup node is different from the target node, the target microservice is restarted and drifted by calling the cluster's service restart interface until the current startup node after the restart drift is the same as the target node. In one example, if the current startup node is inconsistent with the target node, the target microservice can be restarted by calling the service restart node of the cluster where the target microservice resides. Each restart will change the current startup node until the changed current startup node is the same as the target node. Since the nodes can be pre-set in a certain order, when the target microservice is restarted, the current startup node can be drifted in this order.

[0036] In this embodiment, after the current startup node is adjusted to be the same as the pointing node, the directory can be mounted on the backup disk recorded in the database, and the data on the local disk is used as the data source.

[0037] In this embodiment, if the current startup node is different from the target node, the target microservice is restarted and drifted by calling the service restart interface of the cluster until the current startup node after the restart drift is the same as the target node. The method further includes:

[0038] If the target microservice is started, obtain the free disk of the idle node in the environment. The idle node is different from the currently started node.

[0039] Identify the idle disk as the backup disk of the target microservice and determine the service backup status as backupable.

[0040] For example, when a service is started, it is first determined whether there is backup information in the database (the database and the preset database can be the same database or different databases, which is not limited here) for the service being started (target microservice). If it exists, it is directed to start according to the node pointed to in the database (pointing to node) and the disk pointed to (pointing to disk). Among them, an exemplary directional method is: determine whether the currently started node (currently started node) is consistent with the node in the database (pointing to node). If not, call the service restart interface of the cluster to restart the service until the node started by the service is consistent with the node in the database, and mount the directory to the backup disk recorded in the database (each disk has a disk name, which is determined by the name), and use the data of the local disk as the data source. After the startup is completed, the node information and disk information of the corresponding service in the database are deleted and the new node information and disk information are updated to the database. If there are idle disks in other nodes in the environment, the idle local disks of other nodes are marked as backup disks and recorded in the database. Then, the judgment of the consistency of distributed shared storage and local disk data is started, and data synchronization and data backup are started. The data backup method can be an incremental backup in file form by an incremental method known to those skilled in the art.

[0041] Continue to see Figure 2 In this embodiment, if the backup information existence status is not present, the method further includes:

[0042] Step S204, obtaining the number of currently started services and local disks of the cluster;

[0043] Step S205, determining the minimum number of disks according to the number of currently started services, wherein the minimum number of disks is greater than or equal to twice the number of currently started services;

[0044] Step S206: If the number of local disks is greater than the minimum number of disks, the feature data and structured data of the target microservice are backed up to a backup disk, which is different from the local disk mounted by the target microservice.

[0045] For example, if the backup information corresponding to the service does not exist in the database. Then determine the number of service startups (the number of currently started services) N and the number of local disks in the local disk (the number of local disks, the total number of disks that can be used for local storage). If the number of local disks in the cluster is greater than or equal to 2N, then perform data backup of the local disk. The node where the backup disk is located is not the same node as the running node. Theoretically, add 1 to the running node. If it does not exist after adding 1, then backup is performed for node 1 (the initial node or other nodes set by those skilled in the art). The backup method can directly use the incremental backup file method known to those skilled in the art to back up data. The service startup node data and disk data and the backup node data and disk data are stored in the database. In an optional embodiment, if the number of disks is less than twice the number of services started, each service does not perform data backup, which can better ensure the reliability of the entire system.

[0046] In one embodiment, the method further comprises:

[0047] Get the service status and disk status of the target microservice;

[0048] If the service status is normal and the disk status is normal, the service backup status is determined to be ready for backup, and the file data of the local disk and the backup disk are synchronized by starting the synchronization thread;

[0049] If the service status is abnormal or the disk status is abnormal, the service backup status is determined to be not backed up, and data backup is temporarily suspended. In this case, you can poll the disk status at regular intervals, such as thirty minutes, until the disk status returns to normal, and then perform data backup. The service status can indicate whether the target microservice is started and whether it is running normally after startup. If the target microservice starts or runs normally, the service status is normal; otherwise, the service status is abnormal. The disk status indicates the status of the primary and backup disks of the current target microservice. If both the primary and backup disks are normal, backup between the primary and backup disks can be performed.

[0050] Through the above method, it is possible to pre-determine whether the target microservice needs to be started to compensate for local disk data, and provide a method for starting the target microservice and selecting nodes and disks.

[0051] The local disk stores the target microservice's feature data and structured data, while the shared storage stores the target microservice's feature data. It should be noted that the feature data and structured data on the local disk can be stored by combining one feature data item with one structured data item. Alternatively, the feature data and structured data can be stored in an associated manner, but it must be ensured that one feature data item corresponds to one structured data item. If a feature data item and a structured data item are considered a data pair, any missing data item in the data pair is stored as a placeholder. This ensures that the storage length of each data pair is consistent.

[0052] In this embodiment, the service data of the target microservice includes but is not limited to structured data and feature data (that is, pure feature data and long features mentioned later), among which, the structured data is stored in the database (preset database) in full, the pure feature data is stored in the distributed shared storage (that is, the shared storage in this embodiment), and the combined data of feature data and structured data is stored in the local disk.

[0053] The combined data capacity refers to the capacity of the combined data currently stored on the local disk, and the feature data capacity refers to the total feature data capacity of the target microservice in the distributed shared storage. The specific acquisition method can be implemented using methods known to those skilled in the art. Each combined data item has the same size.

[0054] In one embodiment, if the service backup status is not backed up, such as when an extreme situation occurs at a node, the backup node is restored and the backup node is used as the primary node to re-determine the backup node and backup disk information. If there is no additional backup node and backup disk information, data backup is not performed.

[0055] Step S102: determining the amount of combined data according to the combined data capacity and the preset combined data length, and determining the amount of characteristic data according to the characteristic data capacity and the preset characteristic data length.

[0056] An exemplary method for determining the amount of feature data is as follows:

[0057] N1=C1 / L1 formula (1),

[0058] Among them, N1 is the number of feature data, C1 is the total capacity of shared storage (feature data capacity), and L1 is the length of a single feature (preset feature data length).

[0059] An exemplary method for determining the amount of combined data is as follows:

[0060] N2=C2 / L2 Formula (2),

[0061] Where N2 is the number of combined data, C2 is the size of the file on the local disk (combined data capacity), and L2 is the sum of the length of a single feature and the length of a single structured data, i.e., the preset combined data length. It can be understood that L2 = L1 + L0, where L2 is the preset combined data length, L1 is the preset feature data length, and L0 is the length of a single structured data.

[0062] It should be noted that each piece of structured data in the combined data (hereinafter referred to as the combined file) stored on disk is also added with a fixed length. If some structured data is missing, 0 is used as a placeholder to ensure that the size of the combined file is consistent.

[0063] Step S103: If the number of combined data is less than the number of characteristic data, the offset position of the difference data is determined based on the number of combined data and the preset characteristic data length, and the difference data identifier and the difference characteristic data are determined.

[0064] In one embodiment, determining the offset position of the difference data based on the number of combined data and the preset characteristic data length, and determining the difference data identifier and the difference characteristic data includes:

[0065] Determine the combined feature length according to the number of combined data and the preset feature data length. One way to determine the combined feature length is Lz=N2*Ly, where Lz is the combined feature length, N2 is the number of combined data, and Ly is the preset feature data length.

[0066] Determine the difference data offset position of the difference data in the shared storage based on the combined feature length. Since the data is stored in a certain order during data storage, if the number of combined data is less than the number of feature data, the missing difference data is the feature data after the start of the difference data offset position. The existing feature data can be determined by the combined feature length, and the end position of the existing feature data is the difference data offset position.

[0067] Consumption starts from the feature data at the offset position of the difference data in the shared storage, and the obtained feature data and initial data identifier are determined as the difference feature data and the difference data identifier. The shared storage also stores the initial data identifier of each feature data.

[0068] In other words, starting from the offset position of the difference data, the feature data stored thereafter is the difference data. In the shared storage, not only the feature data but also the initial data identifier of each feature data is stored. At least a part of the feature data corresponds to structured data, and the structured data is stored in the database. The database also stores an initial data identifier corresponding to the structured data. In this way, based on the initial data identifier, the structured data corresponding to a certain feature data can be queried.

[0069] In this embodiment, determining the difference structured data based on the difference data identifier includes:

[0070] Based on the difference data identifier, the difference structured data is queried from the preset database to obtain the difference structured data. The preset database stores the structured data of the target microservice and the initial data identifier of the feature data corresponding to the structured data.

[0071] Through the above method, it is possible to determine which feature data (difference feature data) is missing from the local disk, as well as the initial data identifier (difference data identifier) ​​corresponding to each feature data. The structured data corresponding to the missing feature data can be queried using the initial data identifier. Of course, it is also possible that the structured data corresponding to a certain difference data identifier is not found in the preset database. In this case, the preset placeholder rules can be used to reserve the data in the subsequent files to be stored.

[0072] Step S104 : determining the differential structured data based on the differential data identifier, and appending the differential structured data and the differential feature data to the local disk to perform data compensation on the local disk.

[0073] In one embodiment, appending the differential structured data and the differential feature data to the local disk includes:

[0074] Combining the difference structured data and the difference feature data into files to obtain a file to be stored, where the data length of the file to be stored is the preset combined data length;

[0075] Append the files to be stored to the local disk.

[0076] Because the data in the distributed shared storage is designed to be full data, the data volume (number of feature data) must be greater than or equal to the number in the local disk (number of combined data). If the data in the local disk is less (that is, the number of feature data described by the number of combined data), then the offset position of the difference file is obtained based on the number of feature data * the feature file size (preset feature data length), and consumption starts from the data at the offset position. After consuming the feature data, the corresponding structured data is queried from the database based on the recordID of the feature data, and the files are combined according to a fixed format and fixed length. In this way, the purpose of file consistency between the local disk and the distributed shared storage is achieved. After each piece of data is processed, it is appended to the local disk using the file append method.

[0077] After the service has been started, determine whether the number of corresponding feature files on the disk mounted by the service (local disk) is consistent with the number of feature files on the distributed shared storage. If they are consistent, no data synchronization operation is performed; if the data amount is inconsistent, data synchronization compensation is performed.

[0078] In one embodiment, if the current startup node of the target microservice goes down, the backup node of the target microservice is restored as the new current startup node;

[0079] Generate new backup information based on the new current startup node, new local disk information, new standby node, and new standby disk information, and replace the historical backup information of the target microservice with the new backup information.

[0080] The backup information can be stored in the database. By updating the historical backup information in the database, it can be ensured that the backup information of the target microservice stored in the database is the latest data.

[0081] In one embodiment, after restoring the standby node of the target microservice as the new current startup node, the method further includes:

[0082] Get free nodes and free disks;

[0083] If there are no idle nodes or no idle disks, data backup will not be performed;

[0084] Poll idle nodes and idle disks at preset intervals until idle nodes and idle disks are found.

[0085] Identify idle nodes and idle disks as backup nodes and backup disks, and perform data backup.

[0086] For example, when an extreme situation occurs at a node, the node where the service is located goes down, and after the standby node is restored to a service node, the node and disk information of the original service originally recorded in the database are cleared, and the standby node is used as the main node, and the standby node and standby disk information are recorded in the database. If there is no additional standby node and standby disk information, the data of the node will not be backed up, and a poll will be conducted every 30 minutes (or other time lengths set by those skilled in the art) to inquire whether there are enough nodes and free disk information. If so, the data of the unbacked up disk will be backed up. It should be noted that the preset time length can be a fixed value or a random value, which is not limited here.

[0087] See also Figure 3 , Figure 3 This is a specific flow chart of a local disk data compensation method shown in an exemplary embodiment of the present application. Figure 3As shown, it is pre-defined that all structured data is stored in the database, pure feature data (feature data) is stored in distributed shared storage, and the local disk stores the combination of feature data and structured data. When the service is started, first determine whether the started service has backup information, then determine the service status and disk status in the entire cluster environment, confirm whether the backup is in place based on the service status and disk status, and finally determine whether the data in the local disk is consistent with the number of corresponding feature files in the distributed shared storage. When an extreme situation occurs at a node, the standby node is restored, the node and disk information of the original service is cleared, and the standby node is used as the main node to re-determine the standby node and standby disk information. If there is no additional standby node and standby disk information, no backup is performed, and a 30-minute poll is performed to check whether there is enough node and free disk information. If so, the data on the unbacked-up disk is backed up. Continue to see Figure 2 When the original master node goes down, a master-slave node switch is performed. When starting the target microservice, determine whether there is a backup node. If not, determine whether the number of local disks in the cluster's current number of started services N and the number of local disks is greater than the minimum number of disks 2N. If the number of local disks is greater than or equal to the minimum number of disks 2N, start the service but do not back up the data. If the number of local disks is less than the minimum number of disks 2N, start the service randomly and back up the data on the selected node. An optional way to select a node is to start the service on the node + 1 (node ​​+ 1 node as a backup node) and mount the disk on node + 1 as a backup disk for local disk data (backup disk or backup disk). The above node + 1 node and node + 1 node disk information are recorded in the database. If node is the maximum value among the nodes, the disk of node 1 can be used as a backup disk for local disk data. If a standby node (also called a backup node) exists, the standby node information is dispatched, the standby data disk is mounted, and the long code file (feature data) corresponding to each partition on the standby disk is determined to be consistent with the long code file (feature data) on Ceph (a distributed storage system, shared storage). If not, the Ceph file location is calculated based on the data volume (determining the offset of the differential data), and the data is incrementally added to the local disk based on the file location. The data on Ceph is the full data volume, which must be greater than or equal to the amount on the local disk. If the local disk has less data, the offset of the differential file is calculated by multiplying the local disk capacity (feature data quantity) by the feature file size, and data consumption begins at this offset. To determine whether the data is consistent, the Ceph disk only contains long features (feature data), while the local disk file data contains long features plus structured files (structured data). Therefore, the calculation is based on the total Ceph capacity divided by the length of a single feature. The calculation method for local disk data is: the file size on the local disk divided by (feature length + structured data).

[0088] The local disk data compensation method provided in the above embodiment is exemplarily described below through another specific embodiment. An example is given in which a business system has 15 services started, 50 local disks, 5 cluster nodes, and no backup data in the database.

[0089] When the service is started, there is no directed startup because there is no backup data in the database. Since the number of services started is 15 and the number of local disks is 50, which is greater than 15 times 2, data backup is required. At startup, the startup node and backup node of each service must be on different servers. Therefore, if service 0 is on Node0, the backup disk of service 0 must be on Node1 or other nodes. If Node1 has no free disk, add 1 to Node1, and so on. Because the service is started with a brand new disk, it is necessary to determine whether the data on the distributed shared storage is consistent with the data volume on the local disk. If there is data on the distributed shared storage that needs to be synchronized to the local disk, synchronization is performed, and structured data is found through recordID (initial data identifier, which is also a differential data identifier). After the combination is completed, it is saved on the local disk, and the synchronization thread is started to synchronize the file data of the local disk and the backup disk.

[0090] The local disk data compensation method provided by this embodiment obtains the combined data capacity of the local disk mounted by the target microservice and the characteristic data capacity stored in the shared storage of the target microservice after the target microservice is started, determines the number of combined data according to the combined data capacity and the preset combined data length, determines the number of characteristic data according to the characteristic data capacity and the preset characteristic data length, if the number of combined data is less than the number of characteristic data, determines the offset position of the difference data based on the number of combined data and the preset characteristic data length, determines the difference data identifier and the difference characteristic data, determines the difference structured data based on the difference data identifier, and converts the difference structured data and the characteristic data into a differential data. The differential feature data is appended to the local disk to compensate for the data on the local disk. By storing the full amount of feature data in the full data in the shared storage, and storing the feature data and structured data in the local disk, the combined data volume in the local disk is compared with the feature data volume in the shared storage to determine whether data compensation is needed in the local disk. The offset position of the differential data can be determined simply and conveniently to obtain differential structured data. By appending the differential structured data and differential feature data to the local disk to compensate for the data on the local disk, the data retrieval speed is improved and the service recovery time is reduced during the synchronization process.

[0091] The method of the above embodiment greatly improves the processing capacity when the original node unexpectedly crashes by incrementally backing up the local disk data of the service on other nodes. By calculating the data volume (the amount of combined data and the amount of feature data), it is determined whether the local disk data is missing, and the data of the local disk and the distributed shared storage disk are incrementally synchronized to shorten the service recovery time. By backing up the combined data of the local disk during the process, the local disk data is continuously backed up to the backup disk in the backup node. When an extreme situation occurs, the service can be quickly pulled up, and the data consistency of the distributed shared storage and local disk can be restored, providing high-speed data retrieval capabilities. By allocating a single local disk based on a stable distributed shared storage system, the data of each disk is not shared and logical redundancy is not performed, thereby accelerating the reading of features. When there is sufficient disk, each microservice uses physical methods to perform separate data backups on different cluster nodes. After a server crashes, it is directly dispatched to the backup cluster node to quickly take over the backup data, thereby quickly restoring the high-speed query effect.

[0092] See also Figure 4 , Figure 4 FIG. 1 is a block diagram of a local disk data compensation device shown in an exemplary embodiment of the present application. Figure 4 As shown, this embodiment provides a local disk data compensation device 400, which includes:

[0093] Capacity acquisition module 401 is used to obtain the combined data capacity of the local disk mounted by the target microservice and the feature data capacity stored in the shared storage of the target microservice after the target microservice is started. The local disk stores the feature data and structured data of the target microservice, and the shared storage stores the feature data of the target microservice.

[0094] The data quantity determination module 402 is used to determine the quantity of combined data according to the combined data capacity and the preset combined data length, and to determine the quantity of characteristic data according to the characteristic data capacity and the preset characteristic data length;

[0095] A difference determination module 403 is configured to determine a difference data offset position based on the number of combined data and a preset feature data length, and determine a difference data identifier and difference feature data if the number of combined data is less than the number of feature data;

[0096] The data compensation module 404 is configured to determine the differential structured data based on the differential data identifier, and append the differential structured data and the differential feature data to the local disk to perform data compensation on the local disk.

[0097] In one embodiment, the difference determination module is configured to: determine the combined feature length based on the number of combined data and the preset feature data length; determine the difference data offset position of the difference data in the shared storage based on the combined feature length; start consuming the feature data at the difference data offset position in the shared storage, and determine the obtained feature data and initial data identifier as the difference feature data and difference data identifier. The initial data identifier of each feature data is also stored in the shared storage.

[0098] In one embodiment, the data compensation module is configured to: query from a preset database based on the difference data identifier to obtain the difference structured data, where the preset database stores the structured data of the target microservice and the initial data identifier of the feature data corresponding to the structured data.

[0099] In one embodiment, the data compensation module is further configured to: combine the differential structured data and the differential feature data to obtain a file to be stored, where the data length of the file to be stored is a preset combined data length; and append the file to be stored to the local disk.

[0100] In one embodiment, the device also includes a startup module, which is used to obtain the backup information existence status and current startup node of the target microservice during the startup process of the target microservice; if the backup information existence status is existence, determine the pointing node and pointing disk of the target microservice based on the backup information of the target microservice; if the current startup node is different from the pointing node, restart the target microservice by calling the service restart interface of the cluster until the current startup node after the restart drift is the same as the pointing node.

[0101] In one embodiment, the startup module is further configured to: if the current startup node is different from the pointing node, restart the target microservice by calling the service restart interface of the cluster until the current startup node after the restart drift is the same as the pointing node. The method also includes: if the target microservice is started, obtain the idle disk of the idle node in the environment, and the idle node is different from the current startup node; determine the idle disk as the backup disk of the target microservice.

[0102] In one embodiment, the startup module is further configured to: if the backup information existence status is non-existent, obtain the current number of startup services and the number of local disks of the cluster; determine the minimum number of disks based on the current number of startup services, and the minimum number of disks is greater than or equal to twice the current number of startup services; if the number of local disks is greater than the minimum number of disks, back up the feature data and structured data of the target microservice to the backup disk, and the backup disk is different from the local disk mounted by the target microservice.

[0103] In one embodiment, the device also includes a backup control module for obtaining the target microservice and disk status; if the service status is normal and the disk status is normal, the service backup status is determined to be backed up, and the file data of the local disk and the backup disk are synchronized by starting a synchronization thread; if the service status is abnormal, or the disk status is abnormal, the service backup status is determined to be not backed up, and data backup is not performed for the time being.

[0104] In one embodiment, the device also includes a switching control module, which is used to restore the backup node of the target microservice to a new current startup node if the current startup node of the target microservice crashes; generate new backup information based on the new current startup node, new local disk information, new backup node and new backup disk information, and replace the historical backup information of the target microservice with the new backup information.

[0105] In one embodiment, the switching control module is configured to: after restoring the backup node of the target microservice to the new current startup node, obtain the idle nodes and idle disks; if there are no idle nodes or idle disks, do not perform data backup; poll the idle nodes and idle disks at preset intervals until there are idle nodes and idle disks; determine the idle nodes and idle disks as backup nodes and backup disks, and perform data backup.

[0106] In this embodiment, the device is essentially provided with multiple modules for executing the method in any of the above embodiments. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0107] See also Figure 5 , an embodiment of the present invention further provides an electronic device 500, including a processor 501, a memory 502 and a communication bus 503;

[0108] The communication bus 503 is used to connect the processor 501 and the memory 502;

[0109] The processor 501 is configured to execute the computer program stored in the memory 502 to implement one or more methods in the above embodiments.

[0110] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon.

[0111] The computer program is used to enable a computer to execute any one of the methods in the first embodiment.

[0112] An embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute the instructions (instructions) of the steps included in embodiment 1 of the embodiment of the present application.

[0113] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0114] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0115] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A local disk data compensation method, characterized in that: The method comprises: After the target microservice is started, the combined data capacity of the local disk mounted by the target microservice and the feature data capacity stored in the shared storage of the target microservice are obtained, where the local disk stores the feature data and structured data of the target microservice, and the shared storage stores the feature data of the target microservice; Determine the amount of combined data according to the combined data capacity and the preset combined data length, and determine the amount of characteristic data according to the characteristic data capacity and the preset characteristic data length; If the number of the combined data is less than the number of the characteristic data, determining the offset position of the difference data based on the number of the combined data and the preset characteristic data length, and determining the difference data identifier and the difference characteristic data; Determining differential structured data based on the differential data identifier, and appending the differential structured data and the differential feature data to the local disk to perform data compensation on the local disk; Determining the difference data offset position based on the number of combined data and the preset characteristic data length, and determining the difference data identifier and the difference characteristic data includes: Determining a combined feature length according to the combined data quantity and the preset feature data length; determining the difference data offset position of the difference data in the shared storage based on the combined characteristic length; Consumption starts from the feature data at the offset position of the difference data in the shared storage, and the obtained feature data and initial data identifier are determined as the difference feature data and the difference data identifier. The shared storage also stores the initial data identifier of each feature data.

2. The local disk data compensation method according to claim 1, wherein: Determining the difference structured data based on the difference data identifier includes: The differential structured data is obtained by querying a preset database based on the differential data identifier, wherein the preset database stores the structured data of the target microservice and the initial data identifier of the feature data corresponding to the structured data.

3. The local disk data compensation method according to claim 1, wherein: Appending the difference structured data and the difference feature data to the local disk includes: Combining the difference structured data and the difference characteristic data to obtain a file to be stored, wherein the data length of the file to be stored is the preset combined data length; Append the file to be stored to the local disk.

4. The local disk data compensation method according to claim 1, wherein: Before obtaining the combined data capacity of the local disk mounted by the target microservice and the feature data capacity stored in the shared storage of the target microservice, the method includes: During the startup of the target microservice, obtaining the backup information existence status and the current startup node of the target microservice; If the backup information exists, determine the target node and the target disk of the target microservice based on the backup information of the target microservice; If the current startup node is different from the target node, the target microservice is restarted and drifted by calling the service restart interface of the cluster until the current startup node after the restart and drift is the same as the target node.

5. The local disk data compensation method according to claim 4, wherein: If the current startup node is different from the target node, restarting and drifting the target microservice by calling a service restart interface of the cluster until the current startup node after the restart and drift is the same as the target node, the method further includes: If the target microservice is started, obtain the idle disk of the idle node in the environment, where the idle node is different from the currently started node; The idle disk is determined as a backup disk for the target microservice.

6. The local disk data compensation method according to claim 4, wherein: If the backup information existence status is non-existent, the method further includes: Get the current number of started services and local disks in the cluster; Determine a minimum number of disks according to the number of currently started services, where the minimum number of disks is greater than or equal to twice the number of currently started services; If the number of local disks is greater than the minimum number of disks, the feature data and structured data of the target microservice are backed up to a backup disk, which is different from the local disk mounted by the target microservice.

7. The local disk data compensation method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtain the service status and disk status of the target microservice; If the service status is normal and the disk status is normal, the service backup status is determined to be backupable, and the file data of the local disk and the backup disk are synchronized by starting a synchronization thread; If the service status is abnormal, or the disk status is abnormal, the service backup status is determined to be not backed up, and data backup is not performed temporarily.

8. The local disk data compensation method according to any one of claims 1 to 6, characterized in that: If the current startup node of the target microservice goes down, restore the backup node of the target microservice to be the new current startup node; New backup information is generated based on the new current startup node, the new local disk information, the new standby node, and the new standby disk information, and the historical backup information of the target microservice is replaced with the new backup information.

9. The local disk data compensation method according to claim 8, wherein: After restoring the standby node of the target microservice as the new current startup node, the method further includes: Get free nodes and free disks; If there is no idle node or no idle disk, no data backup is performed; Performing polling to query the idle nodes and the idle disks at preset intervals until the idle nodes and the idle disks exist; The idle node and the idle disk are determined as backup nodes and backup disks, and data backup is performed.

10. A local disk data compensation device, characterized in that: The device comprises: A capacity acquisition module is configured to, after a target microservice is started, acquire the combined data capacity of the local disk mounted on the target microservice and the feature data capacity stored in the shared storage of the target microservice, wherein the local disk stores the feature data and structured data of the target microservice, and the shared storage stores the feature data of the target microservice; a data quantity determination module, configured to determine the quantity of combined data according to the combined data capacity and the preset combined data length, and to determine the quantity of characteristic data according to the characteristic data capacity and the preset characteristic data length; a difference determination module, configured to determine, if the number of the combined data is less than the number of the characteristic data, an offset position of the difference data based on the number of the combined data and the preset characteristic data length, and determine a difference data identifier and difference characteristic data; a data compensation module, configured to determine differential structured data based on the differential data identifier, and append the differential structured data and the differential feature data to the local disk to perform data compensation on the local disk; The difference determination module is configured to: determine the combined feature length based on the number of combined data and the preset feature data length; determine the difference data offset position of the difference data in the shared storage based on the combined feature length; start consuming the feature data at the difference data offset position in the shared storage, and determine the obtained feature data and initial data identifier as the difference feature data and difference data identifier. The initial data identifier of each feature data is also stored in the shared storage.

11. An electronic device, characterized in that: Includes processor, memory and communication bus; The communication bus is used to connect the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that A computer program is stored thereon, The computer program is used to cause the computer to execute the method according to any one of claims 1 to 9.

Images