Data restoration method and device, processor and readable storage medium

Abstract

The invention discloses a data restoration method and device, a processor and a readable storage medium. The method comprises the steps that in response to a received restoration instruction, difference data between initial data and target data is determined, the restoration instruction is used for indicating restoration of the target data, and the difference data is used for indicating the difference between the initial data and the target data; based on the difference data, segmenting the target data to obtain a plurality of fragment data; in the multiple pieces of fragment data, target fragment data needing to be repaired is determined; and repairing the target data based on the storage position of the target fragment data. According to the method and the device, the technical problem of low data restoration efficiency in the file transmission process is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of file migration, and in particular to a data repair method, device, processor and readable storage medium. Background Art

[0002] At present, with the rapid development of technologies such as cloud computing and big data, as well as the growth of data volume and storage demand, the widespread application of file migration technology promotes digital transformation and information technology innovation, and becomes one of the keys to promote innovation and development in various industries. Enterprises may need to migrate a large amount of file data from local storage to cloud platforms or new data centers, which makes the demand for file migration increasingly urgent. File migration is the process of transferring files from one location to another. In this process, factors such as data integrity, migration speed, and resource usage must also be considered, which makes "file data verification" and "file data repair" important steps to ensure data quality. File data verification is used to verify the accuracy, completeness, and consistency of data to avoid errors, loss, or inconsistencies. File data repair is a key measure to prevent data loss and damage, and helps maintain data integrity and reliability. In an actual production environment, users can use file data verification to promptly discover and solve potential problems and ensure the quality of data after migration. Then, file data repair is used to ensure the accuracy and completeness of data during the migration process and reduce the risk of data loss and damage.

[0003] In the related art, the traditional file copying method can only simply transfer data, and cannot verify and repair data in real time, resulting in a time-consuming and laborious migration process. Therefore, modern file migration tools are usually used. Although some migration tools support verification and repair functions, there are still some problems that need to be solved. First, the retrieval efficiency is too low: Although some tools provide verification and repair functions, they still have room for improvement in data retrieval efficiency. Due to the large amount of data or high complexity, traditional tools often cannot achieve efficient data verification and rapid error location, resulting in less than ideal execution efficiency. Secondly, collaborative computing is difficult: Existing file migration tools have limitations in utilizing host computing power. Usually, traditional tools can only use host computing power in one direction, and it is difficult to perform collaborative computing through the source data machine and the destination data machine, resulting in low resource utilization when the tool is verifying and repairing, limiting the efficiency and speed of the file migration process. In addition, too many resources are occupied: the file verification and repair process often occupies a large amount of resources, including computing resources, storage resources, etc., resulting in large resource consumption, affecting the stability and efficiency of the overall file migration process. Therefore, there is a technical problem of low efficiency in repairing data during file transfer.

[0004] Currently, no effective solution has been proposed to the above-mentioned technical problem of low efficiency in repairing data during file transmission. Summary of the invention

[0005] Embodiments of the present invention provide a data repair method, device, processor, and readable storage medium to at least solve the technical problem of low efficiency in repairing data during file transmission.

[0006] According to one aspect of an embodiment of the present invention, a method for repairing data is provided. The method may include: in response to receiving a repair instruction, determining difference data between initial data and target data, wherein the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data; based on the difference data, segmenting the target data to obtain multiple fragment data; determining the target fragment data that needs to be repaired among the multiple fragment data; and repairing the target data based on the storage location of the target fragment data.

[0007] Optionally, in response to receiving a repair instruction, determining difference data between the initial data and the target data includes: in response to receiving the repair instruction, generating a difference list, wherein the difference list is used to indicate target data that differs from the initial data; and determining difference data between the initial data and the target data based on the difference list.

[0008] Optionally, based on the difference data, the target data is segmented to obtain a plurality of fragment data, including: based on the difference data, determining a slice length, wherein the slice length is used to indicate the number of bytes or bits of the fragment data; comparing the slice length with the length of the target data, and in response to the length of the target data being greater than or equal to the slice length, segmenting the target data to obtain a plurality of fragment data; in response to the length of the target data being less than the slice length, deleting the target data.

[0009] Optionally, among multiple fragment data, target fragment data that needs to be repaired is determined, including: matching the multiple fragment data with initial data to obtain a matching result, wherein the matching result is used to indicate whether the fragment data successfully matches or fails to match the initial data; in response to the matching result being a matching failure, determining the fragment data corresponding to the matching result as the target fragment data that needs to be repaired.

[0010] Optionally, based on the storage position of the target fragment data, the target data is repaired, including: based on the storage position of the target fragment data, determining the storage position in the initial data; and replacing the fragment data after the storage position of the target fragment data with the initial data after the storage position of the initial data.

[0011] Optionally, the data repair method also includes: in response to completion of target data repair, determining a resource occupation value during the repair process, wherein the resource occupation value is used to indicate a ratio of resources occupied by the repair process; in response to the resource occupation value being greater than a resource occupation threshold, determining a target resource occupation value, wherein the target resource value is used to indicate a ratio of resources of the file migration system that the repair process is expected to occupy; and based on the target resource occupation value, updating the slice length of the fragment data.

[0012] According to another aspect of an embodiment of the present invention, a data repair device is also provided. The device may include: a first determination unit, configured to determine difference data between initial data and target data in response to receiving a repair instruction, wherein the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data; a segmentation unit, configured to segment the target data based on the difference data to obtain a plurality of fragment data; a second determination unit, configured to determine the target fragment data to be repaired from the plurality of fragment data; and a repair unit, configured to repair the target data based on the storage location of the target fragment data.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program processor runs, the device where the storage medium is located is controlled to execute the data repair method in the embodiment of the present invention.

[0014] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein the data repair method in the embodiment of the present invention is executed when the program is run.

[0015] According to another aspect of an embodiment of the present invention, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the data repair method in the embodiment of the present invention is implemented.

[0016] In an embodiment of the present invention, in response to receiving a repair instruction, the difference data between the initial data and the target data is determined; based on the difference data, the target data is segmented to obtain a plurality of fragment data; among the plurality of fragment data, the target fragment data that needs to be repaired is determined; based on the storage location of the target fragment data, the target data is repaired. In other words, the present invention can determine the difference data between the initial data and the target data according to the received repair instruction during the file transmission process, thereby repairing the target data according to the difference data, solving the technical problem of low efficiency of data repair during the file transmission process, and achieving the technical effect of improving the efficiency of data repair during the file transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a data repair method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a platform architecture of a file checking and repairing method based on an agent migration mechanism according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of virtualization nesting according to an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a MySQL semi-synchronous replication architecture according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a Redis cluster architecture according to an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a Prometheus deployment architecture according to an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of a source-destination host migration control relationship according to an embodiment of the present invention;

[0025] Figure 8 is a flow chart of a file migration method according to an embodiment of the present invention;

[0026] Fig. 9 is a flow chart of a real-time difference checking and repairing method according to an embodiment of the present invention;

[0027] Fig.10 is a flow chart of a fast file comparison method according to an embodiment of the present invention;

[0028] Fig.11 is a flow chart of an optimal step calculation method according to an embodiment of the present invention;

[0029] Fig.12 is a flow chart of an extraction hash calculation method according to an embodiment of the present invention;

[0030] Fig.13 is a flow chart of a data sample extraction method according to an embodiment of the present invention;

[0031] Fig.14 is a schematic diagram of a merging logic according to an embodiment of the present invention;

[0032] Fig.15 is a flow chart of an efficient difference repair method according to an embodiment of the present invention;

[0033] Fig.16 is a flow chart of a slice repair operation method according to an embodiment of the present invention;

[0034] Fig.17 is a flow chart of a dynamic shredded file reorganization mechanism according to an embodiment of the present invention;

[0035] Fig.18 is a flow chart of a file repair compression transmission mechanism according to an embodiment of the present invention;

[0036] Fig.19 is a flow chart of host interaction and concurrent task scheduling according to an embodiment of the present invention;

[0037] Fig. 20 is a schematic diagram of host-to-host interaction and concurrent task scheduling according to an embodiment of the present invention;

[0038] Fig.21 It is a schematic diagram of a data repair device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, functional component or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, functional components or devices.

[0041] According to an embodiment of the present invention, an embodiment of a method for repairing data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Figure 1 is a flow chart of a data repair method according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0043] Step S101, in response to receiving a repair instruction, determining difference data between initial data and target data.

[0044] In the technical solution provided in step S101 of the present invention, the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data. The initial data may also be referred to as an initial file, source data, source file, source data file or metadata information, and the target data may also be referred to as a target file, target data or target data file.

[0045] In this embodiment, during the file transmission process, when a repair instruction is received, the difference data between the initial data and the target data is determined.

[0046] Optionally, the initial data and the target data are compared according to the received repair instruction to generate a difference list, and then the difference data is determined according to the difference list.

[0047] For example, when a comparison instruction is issued through the client, the source data machine and the target data machine will work together to perform a rapid file comparison and generate a detailed difference list through the file comparison.

[0048] Step S102: segment the target data based on the difference data to obtain a plurality of fragment data.

[0049] In the technical solution provided in the above step S102 of the present invention, the fragment data may also be referred to as slice data or slice data set.

[0050] In this embodiment, after the difference data between the initial data and the target data is determined in step S101, the target data is segmented according to the difference data to obtain a plurality of fragment data.

[0051] Optionally, the slice length is determined based on the difference data, where the slice length is used to indicate the number of bytes or bits of the fragment data; the slice length is compared with the length of the target data. When the length of the target data is greater than or equal to the slice length, it means that the target data can be segmented by the slice length. Based on this, the target data can be segmented to obtain multiple fragment data.

[0052] Optionally, when the length of the target data is less than the slice length, it indicates that the target data cannot meet the requirement of the slice length for segmentation. Based on this, the target data can be deleted and the initial file can be retransmitted from the source host to the target host.

[0053] Step S103: determining target segment data that needs to be repaired among the multiple segment data.

[0054] In the technical solution provided in the above step S103 of the present invention, the target segment data may also be referred to as target slice data.

[0055] In this embodiment, after a plurality of segment data are obtained in step S102, target segment data that needs to be repaired is determined among the plurality of segment data.

[0056] Optionally, the plurality of segment data are matched with the initial data to obtain a matching result, wherein the matching result is used to indicate whether the segment data matches the initial data successfully or fails to match.

[0057] For example, the slice data is hashed and compared with the data on the source host to obtain a matching result.

[0058] Optionally, when the matching result is a mismatch, it indicates that some segment data in the target data is different from the initial data. Based on this, the segment data corresponding to the matching result can be determined as the target segment data that needs to be repaired.

[0059] Step S104: repair the target data based on the storage location of the target segment data.

[0060] In the technical solution provided in the above step S104 of the present invention, after the target segment data to be repaired is determined in step S103, the target data is repaired according to the storage location of the target segment data.

[0061] In this embodiment, the storage position in the initial data is determined according to the storage position of the target segment data; that is, the storage position of the corresponding target segment data in the initial data is determined.

[0062] Optionally, after determining the storage position in the initial data, the fragment data after the storage position of the target fragment data is replaced with the initial data after the storage position of the initial data.

[0063] For example, the initial data after the initial data storage location is uploaded again, and the uploaded initial data replaces the target data after the storage location of the target segment data in the target data. That is, the initial data after the initial data storage location is uploaded again.

[0064] It should be noted that the above embodiments can be executed by a data repair device.

[0065] In the above steps S101 to S104 of the present invention, in response to receiving a repair instruction, the difference data between the initial data and the target data is determined; based on the difference data, the target data is segmented to obtain multiple fragment data; among the multiple fragment data, the target fragment data that needs to be repaired is determined; based on the storage location of the target fragment data, the target data is repaired. In other words, the present invention can determine the difference data between the initial data and the target data according to the received repair instruction during the file transmission process, and thus repair the target data according to the difference data, thereby solving the technical problem of low efficiency of data repair during the file transmission process and achieving the technical effect of improving the efficiency of data repair during the file transmission process.

[0066] The above method of this embodiment is further introduced below.

[0067] As an optional implementation method, in response to receiving a repair instruction, determining the difference data between the initial data and the target data includes: in response to receiving the repair instruction, generating a difference list, wherein the difference list is used to indicate the target data that is different from the initial data; based on the difference list, determining the difference data between the initial data and the target data.

[0068] In this embodiment, when a repair instruction is received, a difference list is generated, and the difference data between the initial data and the target data is determined according to the difference list.

[0069] For example, when a fast file comparison operation is started, the system first generates a comparison order list based on the preset sorting rules (such as file name, size, modification time, etc.). This list determines which files need to be compared first. Then, the system sends this list to the agent on the target machine, and the agent processes the comparison task based on the received list and generates a difference list.

[0070] As an optional implementation method, based on the difference data, the target data is segmented to obtain multiple fragment data, including: based on the difference data, determining the slice length, wherein the slice length is used to indicate the number of bytes or bits of the fragment data; comparing the slice length with the length of the target data, and in response to the length of the target data being greater than or equal to the slice length, segmenting the target data to obtain multiple fragment data; in response to the length of the target data being less than the slice length, deleting the target data.

[0071] In this embodiment, the slice length is determined based on the difference data; the slice length is compared with the length of the target data, and when the length of the target data is not less than the slice length, the target data is divided to obtain multiple fragment data; when the length of the target data is less than the slice length, the target data is deleted.

[0072] Optionally, slice-based file differential repair is a method of repairing damaged or missing files, where the file is split into multiple small pieces or slices, each slice containing a portion of the file's data. With slice-based file differential repair, when a file is damaged or lost, these errors can be automatically detected and repaired without re-downloading the entire file, improving the efficiency of repairing file data.

[0073] As an optional implementation method, among multiple fragment data, target fragment data that needs to be repaired is determined, including: matching the multiple fragment data with initial data to obtain matching results, wherein the matching results are used to indicate whether the fragment data successfully matches or fails to match the initial data; in response to the matching result being a matching failure, determining the fragment data corresponding to the matching result as the target fragment data that needs to be repaired.

[0074] In this embodiment, a plurality of segment data are matched with the initial data to obtain a matching result; when the matching result is a matching failure, the segment data corresponding to the matching result is determined as the target segment data that needs to be repaired.

[0075] For example, multiple fragments of data are matched with the initial data through hash comparison. Extractive hash calculation refers to extracting part of the content (usually a string or number of fixed length) from a piece of text or data, and then performing hash calculation on the extracted content to generate a unique hash value. This method can be used to quickly compare the similarity or uniqueness of a large amount of data, and is often used in application scenarios such as data deduplication, similarity matching, and data indexing.

[0076] As an optional implementation method, the target data is repaired based on the storage position of the target fragment data, including: determining the storage position in the initial data based on the storage position of the target fragment data; replacing the fragment data after the storage position of the target fragment data with the initial data after the storage position of the initial data.

[0077] In this embodiment, the storage position in the initial data is determined according to the storage position of the target segment data; and the segment data after the storage position of the target segment data is replaced with the initial data after the storage position of the initial data.

[0078] Optionally, the fragment data after the storage position of the target fragment data is replaced with the initial data after the storage position of the initial data, and the fragmented file optimization mechanism is used to reorganize the storage position of the file and organize the file fragments into continuous blocks to improve the reading and writing speed of the file.

[0079] As an optional implementation manner, the data repair method also includes: in response to completion of target data repair, determining a resource occupancy value during the repair process, wherein the resource occupancy value is used to indicate the ratio of resources occupied by the repair process; in response to the resource occupancy value being greater than a resource occupancy threshold, determining a target resource occupancy value, wherein the target resource value is used to indicate the ratio of resources of the file migration system that the repair process is expected to occupy; and based on the target resource occupancy value, updating the slice length of the fragment data.

[0080] In this embodiment, when the target data is repaired, the resource occupancy value in the repair process is determined; when the resource occupancy value is greater than the resource occupancy threshold, the target resource occupancy value is determined; and according to the target resource occupancy value, the slice length is updated.

[0081] For example, the resource occupancy status of the repair process is calculated to determine whether the resource occupancy is less than a set value; if so, the allocated slice size is recalculated and the slice size value is updated.

[0082] It should be noted that the above embodiments can be executed by a data repair device.

[0083] In this embodiment, in response to receiving a repair instruction, the difference data between the initial data and the target data is determined; based on the difference data, the target data is segmented to obtain a plurality of fragment data; among the plurality of fragment data, the target fragment data that needs to be repaired is determined; based on the storage location of the target fragment data, the target data is repaired. In other words, the present invention can determine the difference data between the initial data and the target data according to the received repair instruction during the file transmission process, and thus repair the target data according to the difference data, thereby solving the technical problem of low efficiency of data repair during the file transmission process and achieving the technical effect of improving the efficiency of data repair during the file transmission process.

[0084] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0085] At present, with the rapid development of technologies such as cloud computing and big data, as well as the growth of data volume and storage demand, the widespread application of file migration technology promotes digital transformation and information technology innovation, and becomes one of the keys to promote innovation and development in various industries. Enterprises may need to migrate a large amount of file data from local storage to cloud platforms or new data centers, which makes the demand for file migration increasingly urgent. File migration is the process of transferring files from one location to another. In this process, factors such as data integrity, migration speed, and resource usage must also be considered, which makes "file data verification" and "file data repair" important steps to ensure data quality. File data verification is used to verify the accuracy, completeness, and consistency of data to avoid errors, loss, or inconsistencies. File data repair is a key measure to prevent data loss and damage, and helps maintain data integrity and reliability. In an actual production environment, users can use file data verification to promptly discover and solve potential problems and ensure the quality of data after migration. Then, file data repair is used to ensure the accuracy and completeness of data during the migration process and reduce the risk of data loss and damage.

[0086] In the related art, the traditional file copying method can only simply transfer data, and cannot verify and repair data in real time, resulting in a time-consuming and laborious migration process. Therefore, modern file migration tools are usually used. Although some migration tools support verification and repair functions, there are still some problems that need to be solved. First, the retrieval efficiency is too low: Although some tools provide verification and repair functions, they still have room for improvement in data retrieval efficiency. Due to the large amount of data or high complexity, traditional tools often cannot achieve efficient data verification and rapid error location, resulting in less than ideal execution efficiency. Secondly, collaborative computing is difficult: Existing file migration tools have limitations in utilizing host computing power. Usually, traditional tools can only use host computing power in one direction, and it is difficult to perform collaborative computing through the source data machine and the destination data machine, resulting in low resource utilization when the tool is verifying and repairing, limiting the efficiency and speed of the file migration process. In addition, too many resources are occupied: the file verification and repair process often occupies a large amount of resources, including computing resources, storage resources, etc., resulting in large resource consumption, affecting the stability and efficiency of the overall file migration process. Therefore, there is a technical problem of low efficiency in repairing data during file transfer. Currently, no effective solution has been proposed to the above-mentioned technical problem of low efficiency in repairing data during file transmission.

[0087] However, the embodiment of the present invention proposes a file verification and repair method based on the proxy migration mechanism, which generates a detailed difference list by comparing the files at the sending end and the receiving end, determines whether there is differential information based on the difference list, determines the slice size based on the differential information, and cuts the file according to the slice size to obtain a number of slice data; sequentially hashes the single slice data with the data on the source host to determine whether there is a difference; if so, re-upload all slice data subsequent to the current slice data, and calculate the resource occupancy status of the repair process after merging them into a repair file, and determine whether the resource occupancy is less than the set value; if so, recalculate the allocated slice size and update the slice size value. The technical problem of low efficiency in repairing data during file transmission is solved, and the technical effect of improving the efficiency of meter information storage is achieved.

[0088] The embodiments of the present invention are further described below.

[0089] Figure 2 It is a schematic diagram of the platform architecture of a file verification and repair method based on a proxy migration mechanism according to an embodiment of the present invention. The platform of the file verification and repair method based on the proxy migration mechanism includes: an administrator, a cloud operation user, a migration client, a monitoring system, an EIP, a K8S control node, multiple front ends, a back end, a Redis forwarder, and a primary database and a backup database.

[0090] In this embodiment, the implementation environment is deployed in a Kubernetes containerized management manner, and the platform achieves high-concurrency processing of platform instructions through clustered deployment of Mysql and Redis.

[0091] Optionally, Figure 3 It is a schematic diagram of a virtualization nesting according to an embodiment of the present invention, and the virtualization nesting includes containerization technology, a virtualization platform and a physical machine. The method implementation environment of the present invention is deployed in a Kubernetes containerized management manner, wherein the Kubernetes part will adopt a virtualization nesting manner, using containerization technology based on the virtualization platform.

[0092] Optionally, the platform uses VPC resources configured with a capacity greater than 16c|64g, builds a Kubernetes environment on 9 hosts as the foundation for the migration control platform, and uses automatic orchestration of application containerization to achieve high availability of the system.

[0093] Optionally, Figure 4 is a schematic diagram of a MySQL semi-synchronous replication architecture according to an embodiment of the present invention, such as Figure 4As shown, the platform of the method of the present invention realizes high concurrent processing of platform instructions through clustered deployment of Mysql and Redis. Master library (1 host): Install MySQL database and configure it as the master library, responsible for data writing, updating and deleting operations. Slave library (1 host): Install MySQL database and configure it as the slave library, responsible for data reading and backup.

[0094] Optionally, master-slave replication architecture: Use the master-slave replication function that comes with MySQL to achieve high availability and fault tolerance of the database. One host acts as the master database, responsible for data writing, updating, and deleting operations; the other host acts as the slave database, responsible for data backup and reading operations. When the master database fails, the slave database can be switched to the new master database to ensure service continuity and data integrity. Master-slave replication can be implemented using the built-in function of MySQL, such as Figure 4 shown.

[0095] Optionally, MySQL's master-slave replication uses a semi-synchronous replication mode. In traditional master-slave replication, the replication of the slave database is asynchronous. After the master database executes a transaction, it immediately returns a response to the client without waiting for the slave database to complete the transaction. This asynchronous replication may cause data delays between the master and slave databases, and cause the slave database to be unable to update in time. After adopting semi-synchronous replication, when completing a transaction, it waits for at least one slave database to successfully replicate the transaction before returning a response to the client. Although the performance of data operations may be affected to a certain extent, it can effectively solve the problem of master-slave data consistency.

[0096] Optionally, Figure 5 is a schematic diagram of a Redis cluster architecture according to an embodiment of the present invention, such as Figure 5 As shown in the figure, the cluster function of Redis is used to achieve high availability and fault tolerance of the Redis database.

[0097] Optionally, in this Redis cluster, 6 nodes need to be created, 3 of which are master nodes and the other 3 are slave nodes. Each node stores a portion of the data and can process the client's read and write requests. When the master node fails, the cluster will force the slave node to upgrade itself to the new master node to ensure service continuity and data integrity.

[0098] Optionally, each node has a corresponding master-slave relationship, and the specific examples are as follows: the port number of node 1 (master) is 18000. The port number of node 2 (master) is 18000. The port number of node 3 (master) is 18000. The port number of node 4 (slave) is 18001. The port number of node 5 (slave) is 18001. The port number of node 6 (slave) is 18001.

[0099] Optionally, Figure 6 is a schematic diagram of a Prometheus deployment architecture according to an embodiment of the present invention, such as Figure 6 As shown in the figure, the platform uses Prometheus combined with Grafana to promptly detect and handle fault problems.

[0100] Optionally, use Grafana to visualize the data. Point the data source configured in Grafana to Prometheus to ensure that Grafana can receive all collected data. Create a dashboard to visualize the data, such as displaying the CPU utilization percentage and host memory usage.

[0101] Optionally, Prometheus uses a node collection client to monitor the operating status of three main functional nodes, and uses a Redis collection client and a MySQL collection client to monitor the status of two types of database clusters, and reports the above monitoring information to the monitoring platform for processing.

[0102] Figure 7 FIG. 1 is a schematic diagram of a source-destination host migration control relationship according to an embodiment of the present invention. Figure 7 As shown in the figure, a dedicated migration agent is designed for the source and target hosts to be migrated, and it is ensured that the agent only needs a 1-core processor and 1GB of memory to run smoothly. During the migration process, the migration agent will make full use of the resources of the source and target machines for reasonable scheduling.

[0103] Figure 8 is a flow chart of a file migration method according to an embodiment of the present invention. Figure 8 As shown, the file migration method includes the following steps:

[0104] Step S801, traversing all file data.

[0105] In this embodiment, when the system starts migration, it first needs to traverse all files in full to ensure that each file is scanned and identified.

[0106] Optionally, during the traversal process, the system will record metadata information of the file, including file size and path, etc.

[0107] Step S802: Full file transfer.

[0108] In this embodiment, in the full file transfer stage, the file is transferred from the source location to the target location.

[0109] Step S803, incremental file recording.

[0110] In this embodiment, during the transmission process, the system will record the incremental information of the file in real time.

[0111] Step S804: incremental file transmission.

[0112] In this embodiment, after the full transmission is completed, the system will continue to synchronize the incremental data recorded in real time.

[0113] Step S805, stop incremental transmission.

[0114] In this embodiment, during the file transfer process, the user can choose to stop the incremental transfer at any time. The host that stops the incremental transfer can enter the "waiting for migration and cutover" state.

[0115] Step S806, start checking and repairing.

[0116] In this embodiment, if the user chooses to start checking and repairing, step S807 is executed; if the user does not choose to start checking and repairing, the transmission is terminated.

[0117] Optionally, the user can choose to perform a file transfer review and repair. This step ensures that the files at the destination are consistent with the files at the source, including confirming that all files were successfully transferred and taking necessary corrective actions, such as retransmitting missing or damaged files.

[0118] Step S807: Real-time difference checking and repairing.

[0119] In this embodiment, the difference check and repair of the transmitted files are performed in real time.

[0120] Optionally, real-time difference checking and repair refers to the process of real-time monitoring and comparison of system data during system operation, discovering differences between data, and immediately repairing and adjusting. This process can effectively detect data anomalies and errors, ensure data accuracy and integrity, and improve system stability and reliability.

[0121] Step 808, recheck and repair.

[0122] In this embodiment, the re-check and repair process needs to be manually checked and ends after the check is correct. If an error occurs during the check, step S806 is executed.

[0123] Fig. 9 is a flow chart of a real-time difference checking and repairing method according to an embodiment of the present invention. Fig. 9 As shown, the real-time difference verification and repair includes the following steps:

[0124] Step S901, start checking and repairing.

[0125] In this embodiment, after the host stops incrementing, it enters the "pending migration and cutover" state. At this time, the user can selectively perform file transfer verification and repair operations. If the user attempts to use verification and repair, the host will enter the "real-time difference verification and repair" process.

[0126] Step S902: Issue a comparison instruction.

[0127] In this embodiment, after the host starts checking and repairing, a comparison instruction is issued through the agent client.

[0128] Step S903: fast file comparison.

[0129] In this embodiment, the source data machine and the target data machine receive the data and the source data machine and the target data machine will cooperate to perform a fast file comparison.

[0130] Optionally, fast file comparison is a technology for detecting differences between two or more files, which is often used in scenarios such as backup, version control, and system migration.

[0131] Step S904: generating a difference list.

[0132] In this embodiment, a detailed difference list is generated by comparing the files of the source data machine with the files of the target data machine.

[0133] Step S905: whether a difference is found.

[0134] In this embodiment, based on the generated difference list, it is determined whether there are differences between the files of the source data machine and the files of the target data machine. If there are differences, step S906 is executed; if there are no differences, step S907 is executed.

[0135] Step S906: efficient difference repair.

[0136] In this embodiment, efficient difference repair is performed on the files of the target data machine.

[0137] Optionally, efficient difference repair is based on a slice-based difference repair method. Slice-based difference repair refers to a method of dividing a piece of data into multiple different segments (or slices), and then performing difference repair on these segments. This method is suitable for relatively large data or files. By slicing the data, differences in the data can be found more efficiently and repaired.

[0138] Step S907, recheck and repair.

[0139] In this embodiment, the re-check and repair process needs to be manually checked and ends after the check is correct. If an error occurs during the check, step S901 is executed.

[0140] Fig.10 is a flowchart of a fast file comparison method according to an embodiment of the present invention. Fig.10 As shown, the fast file comparison method includes the following steps:

[0141] Step S1001, generating a comparison order list.

[0142] In this embodiment, when the fast file comparison operation is started, the system first generates a comparison order list according to preset sorting rules (such as file name, size, modification time, etc.). This list determines which files need to be compared first.

[0143] Step S1002, sending a command to the target machine to perform synchronous operation.

[0144] In this embodiment, the system sends this list to an agent on the target machine to process the comparison tasks in parallel.

[0145] Step S1003, optimal data extraction step selection.

[0146] In this embodiment, before starting the actual file content comparison, the system performs a step to determine the most appropriate data extraction step value.

[0147] Optionally, the optimal data extraction step refers to the process of selecting data samples through certain step rules during the data sample extraction process. The optimal data extraction step can help select data samples more scientifically and effectively during the data sampling process to ensure that the sampling results are representative and reliable for the entire data set.

[0148] Optionally, this step value is calculated based on the performance of the host in order to balance the speed and accuracy of the comparison. The smaller the step value, the denser the information extracted during the comparison process, and the corresponding comparison speed may be reduced; the larger the step value, the faster the comparison, but some subtle differences may be missed.

[0149] Step S1004, scanning the next file data.

[0150] In this embodiment, the next file data is scanned sequentially.

[0151] Optionally, the system enters a specific file comparison phase, in which the system will traverse the files to be compared one by one in the order in the comparison order list.

[0152] Step S1005, extract hash calculation.

[0153] In this embodiment, a hash operation is performed on the scanned file content.

[0154] Optionally, the hash operation can convert the file content into a unique hash value of fixed length, which can be used to quickly compare whether the file contents are identical.

[0155] Optionally, extractive hash calculation refers to extracting part of the content (usually a fixed-length string or number) from a piece of text or data, and then performing a hash calculation on the extracted content to generate a unique hash value. This method can be used to quickly compare the similarity or uniqueness of a large amount of data, and is often used in application scenarios such as data deduplication, similarity matching, and data indexing.

[0156] Step S1006, outputting hash value information.

[0157] In this embodiment, after the hash operation is completed, the information of the hash value is output.

[0158] Step S1007, sending a session message to confirm the operation status of the target machine.

[0159] In this embodiment, the session message is sent to confirm the operation status of the target machine, that is, to confirm the operation status of the target agent.

[0160] Step S1008: the target machine completes the operation.

[0161] In this embodiment, it is determined whether the target machine has completed the operation. If the operation is completed, step S1011 is executed; if the operation is not completed, step S1009 is executed.

[0162] Step S1009, delaying program execution.

[0163] In this embodiment, if the target machine has not completed the operation, it is necessary to delay the program execution and wait for the target machine to complete the operation before continuing to the next step.

[0164] Step S1010, confirm again.

[0165] In this embodiment, it is confirmed again whether the target machine has completed the operation.

[0166] Step S1011, comparing target machine information.

[0167] In this embodiment, information of the target machine is compared.

[0168] Step S1012: whether it is the last file.

[0169] In this embodiment, it is determined whether it is the last file. If it is the last file, step S1013 is executed; if it is not the last file, step S1004 is executed.

[0170] Step S1013: output differentiation information.

[0171] In this embodiment, differential information of the file is output.

[0172] Fig.11 is a flow chart of an optimal step calculation method according to an embodiment of the present invention. Fig.11 As shown, the optimal step calculation method includes the following steps:

[0173] Step S1101, initializing the step reference value.

[0174] In this embodiment, the step reference value is initialized to be the total amount of data divided by 1024. The reference value is initially "total amount of data / 1024". For example, the step reference value of 1 GB is 1024 KB.

[0175] Step S1102, setting a step value reference value.

[0176] In this embodiment, the current step value is set to 2 / 3 of the reference value.

[0177] Step S1103, simulate and generate a hash value.

[0178] In this embodiment, the hash value generation is simulated, and the host CPU and memory usage are calculated.

[0179] Step S1104, determining the estimated allocated resources.

[0180] In this embodiment, the estimated allocated resources are determined. If both the CPU and memory usage are less than or equal to 5% of the estimated allocated amount, step S1102 is executed. If both the CPU and memory usage are greater than or equal to 5% of the estimated allocated amount, step S1105 is executed.

[0181] Step S1105, setting the step value again.

[0182] In this embodiment, the step value is set to 3 / 2 of the current value.

[0183] Step S1106, outputting the current step value as the optimal data selection step value.

[0184] In this embodiment, an optimal step value is output.

[0185] Fig.12 is a flowchart of a method for extracting hash calculation according to an embodiment of the present invention. Fig.12 As shown, the extraction hash calculation method includes the following steps:

[0186] Step S1201, construct an all-0 sequence.

[0187] In this embodiment, an all-zero sequence is constructed, ie, all data are set to zero value.

[0188] Step S1202, selecting a step value according to the best data to perform data extraction.

[0189] In this embodiment, a step value is selected according to the best data to extract data from the source file or database.

[0190] Step S1203: data sample extraction algorithm.

[0191] In this embodiment, data sample extraction is a process of randomly or selectively selecting a portion of data records or data rows from a computer file to represent the entire file, which can help analysts understand the characteristics of the data set more quickly. Through data sample extraction, large-scale data can be effectively processed, data processing efficiency can be improved, and the accuracy and reliability of analysis results can be ensured.

[0192] Step S1204, performing all-0 sequence interpolation processing on the extraction result of the extraction algorithm.

[0193] In this embodiment, the result of the decimation algorithm is interpolated with the all-zero sequence.

[0194] Step S1205, hash value calculation.

[0195] In this embodiment, a hash calculation is performed on the interpolated extracted values.

[0196] Step S1206, generating a random sampling hash value.

[0197] In this embodiment, a random sampling hash value is generated for data verification or comparison to ensure the integrity and accuracy of the data.

[0198] Fig.13 is a flow chart of a data sample extraction method according to an embodiment of the present invention. Fig.13 As shown, the data sample extraction method includes the following steps:

[0199] Step S1301, setting the extraction character spacing.

[0200] In this embodiment, the extracted character spacing is set to an optimal step.

[0201] Step S1302, traverse the data samples and extract 1 byte for each character spacing.

[0202] In this embodiment, all data samples are traversed and 1 byte is extracted for each character spacing.

[0203] Step S1303, traverse to the end of the file.

[0204] In this embodiment, it is determined whether the traversal reaches the end of the file. If the traversal reaches the end of the file, step S1304 is executed; if the traversal does not reach the end of the file, step S1302 is executed.

[0205] Step S1304, merging data values.

[0206] In this embodiment, the data sample extraction needs to achieve the purpose of simplifying the calculation by merging the data.

[0207] Optionally, merging data values ​​is the process of combining identical data items from two or more data sets into one data set, which can simplify the data analysis process, reduce duplicate data, and obtain a more comprehensive and accurate data set.

[0208] Fig.14 is a schematic diagram of a merging logic according to an embodiment of the present invention, such as Fig.14 As shown, common methods of merging data values ​​include merging rows, columns, or using association keys to merge data from different data sources. The merging logic merges data 1, data 2, and data 3, etc., which are separated by character spacing, through SumFunction.

[0209] Fig.15 is a flow chart of an efficient difference repair method according to an embodiment of the present invention. Fig.15 As shown, the efficient difference repair method includes the following steps:

[0210] Step S1501, input differentiation information.

[0211] In this embodiment, differential information obtained by comparison is input.

[0212] Step S1502: Slice-based data repair.

[0213] In this embodiment, sliced ​​file differential repair is a method of repairing a damaged or lost file, wherein the file is divided into multiple small pieces or slices, each slice containing a portion of the file's data.

[0214] Optionally, when files are damaged or lost, slice-based file difference repair can automatically detect and repair these errors without re-downloading the entire file. The present method adopts an automatic slice size adjustment method, using an intelligent algorithm to dynamically adjust the slice size to meet the performance requirements of different hardware devices and files of different sizes.

[0215] Step S1503: whether it is the last file.

[0216] In this embodiment, it is determined whether it is the last file. If it is the last file, step S1504 is executed; if it is not the last file, step S1501 is executed.

[0217] Step S1504, output the verification and repair results.

[0218] In this embodiment, the result of checking and repairing is output.

[0219] Fig.16 is a flow chart of a slice repair operation method according to an embodiment of the present invention. Fig.16 As shown, the slice repair operation method includes the following steps:

[0220] Step S1601, input differentiation information.

[0221] In this embodiment, differential information obtained by comparison is input.

[0222] Step S1602: whether a depth value has been allocated.

[0223] In this embodiment, whether a depth value, that is, a Z value, has been allocated, if the depth value has been allocated, step S1604 is executed, and if the depth value has not been allocated, step S1603 is executed.

[0224] Step S1603, initialize the slice size.

[0225] In this embodiment, the slice size is initialized, for example, Z=50.

[0226] Step S1604, start data repair.

[0227] In this embodiment, data repair is started according to the slice size.

[0228] Step S1605, determining whether the file size is greater than or equal to the depth value.

[0229] In this embodiment, it is determined whether the file size is greater than or equal to the depth value, that is, it is determined whether the file size is greater than or equal to the depth value; if the file size is greater than or equal to the depth value, step S1606 is executed; if the file size is not greater than or equal to the depth value, step S1613 is executed.

[0230] Step S1606, cutting the file using the slice size as the depth value.

[0231] In this embodiment, the file is cut with the slice size as the depth value, that is, with the slice size as Z MB

[0232] Crop the file.

[0233] Step S1607, performing hash comparison based on the slice data.

[0234] In this embodiment, hash comparison is performed based on the cut slice data.

[0235] Step S1608, compare the next slice data with the hash of the source host.

[0236] In this embodiment, a hash comparison is performed on the next slice data with the source host.

[0237] Step S1609, whether there is any error.

[0238] In this embodiment, it is determined whether the comparison is incorrect. If so, step S1610 is executed; if not, step S1608 is executed.

[0239] Step S1610, retransmit all slice data after the current slice.

[0240] In this embodiment, all slice data after the current slice are retransmitted.

[0241] Step S1611, merge slice files.

[0242] In this embodiment, all slice files are merged.

[0243] Step S1612, calculating resource occupancy.

[0244] In this embodiment, resource usage during the repair process is calculated.

[0245] Step S1613, delete the file.

[0246] In this embodiment, the current file is deleted.

[0247] Step S1614, retransmit the file.

[0248] In this embodiment, the file is re-uploaded.

[0249] Step S1615, retransmission is completed.

[0250] In this embodiment, the retransmission is completed.

[0251] Step S1616: Occupancy < 80% of the expected allocated resources.

[0252] In this embodiment, it is determined whether the occupied resources are less than 80% of the expected allocated resources. If the occupied resources are less than 80% of the expected allocated resources, step S1618 is executed; if the occupied resources are not less than 80% of the expected allocated resources, step S1617 is executed.

[0253] Step S1617, output the verification and repair results.

[0254] In this embodiment, the result of checking and repairing is output.

[0255] Step S1618, reset the slice size.

[0256] In this embodiment, the slice size is reset, for example, the slice size Z=Z / 1.15.

[0257] Fig.17 is a flow chart of a dynamic fragmented file reorganization mechanism according to an embodiment of the present invention. Fig.17 As shown, the dynamic fragmented file reorganization mechanism includes the following steps:

[0258] Step S1701, resource evaluation.

[0259] In this embodiment, the idle resources of the current system are analyzed, including CPU usage, memory usage, and disk I / O capacity.

[0260] Step S1702: output resource evaluation results.

[0261] In this embodiment, the resource evaluation result of step S1701 is output.

[0262] Step S1703: whether the system is idle.

[0263] In this embodiment, it is determined whether the system is idle. If the system is idle, step S1704 is executed; if the system is not idle, step S1705 is executed.

[0264] Step S1704, fragment reassembly.

[0265] In this embodiment, before transmission, if possible, an attempt is made to locally reassemble the file segments into a complete file. This can be done by creating a temporary file and writing each file segment to the file in sequence. This process can be performed in parallel if system resources permit.

[0266] Optionally, the fragmented file optimization mechanism is a technology used to deal with fragmentation problems in computer file systems. Fragmentation refers to the fact that files are scattered in discontinuous fragments on the disk when they are stored, which will cause the file reading and writing speed to slow down and reduce system performance. The fragmented file optimization mechanism organizes the file fragments into continuous blocks by reorganizing the storage location of the file to improve the file reading and writing speed.

[0267] Step S1705, clean up the system.

[0268] In this embodiment, temporary storage space is cleared to release system resources.

[0269] Fig.18FIG. 1 is a flowchart of a file repair compression transmission mechanism according to an embodiment of the present invention. Fig.18 As shown, the file repair compression transmission mechanism includes the following steps:

[0270] Step S1801: Check whether the file compression is completed.

[0271] In this embodiment, it is determined whether the file has been compressed. If the file has been compressed, step S1805 is executed; if the file has not been compressed, step S1802 is executed.

[0272] Step S1802, reading a character and the number of repetitions of the character from a compressed file.

[0273] In this embodiment, a character is read from a compressed file and the number of repetitions of the character is determined.

[0274] Step S1803: Check whether the consecutive numbers have been written.

[0275] In this embodiment, it is determined whether the continuous number has been written. If the continuous number has been written, step S1804 is executed; if the continuous number has not been written, step S1802 is executed.

[0276] Step S1804, writing the characters read from the compressed file into the recovery file.

[0277] In this embodiment, characters are written from the restored file and read from the compressed file.

[0278] Step S1805, close the compressed file.

[0279] In this embodiment, the compressed file is closed.

[0280] Optionally, the compression transmission mechanism is a technology that reduces the amount of data transmitted over the network by compressing the data. During the data transmission process, the data becomes more compact after being compressed, thereby reducing the bandwidth and transmission time required for transmission. Through the compression transmission mechanism, network resources can be saved, transmission speed can be increased, and the possibility of network congestion can be reduced during the data transmission process.

[0281] Fig.19 is a flow chart of host interaction and concurrent task scheduling according to an embodiment of the present invention, such as Fig.19 As shown, the host interaction and concurrent task scheduling includes the following steps:

[0282] Step S1901, the source machine sends a command to the target machine to start the interactive task.

[0283] In this embodiment, when the interactive and concurrent tasks start, the source machine first sends an instruction to the target host to start the interactive task.

[0284] Step S1902: The source creates a thread pool.

[0285] In this embodiment, both the source host and the target host will create a thread pool.

[0286] Step S1903, receiving a task request.

[0287] In this embodiment, once a task request related to verification or repair is received, the source host will send the task request to the target host, including a list of files to be migrated and related parameters.

[0288] Step S1904, task allocation.

[0289] In this embodiment, after receiving the task request, the target host will assign the task to a thread in the thread pool for processing based on factors such as the size and type of the file.

[0290] Step S1905, task execution.

[0291] In this embodiment, the thread executes the received task.

[0292] Step S1906, close the thread pool to record the migration log.

[0293] In this embodiment, after the task execution is completed, the thread pool will automatically close and record the migration log related to the verification and repair.

[0294] Optionally, host interaction and concurrency refer to the way in which different hosts communicate and interact with each other in a computer network. Concurrency refers to the ability to handle multiple tasks or connections at the same time, while host interaction refers to the process of communication and data transfer between hosts.

[0295] Optionally, in the file verification and repair method, the host uses an interactive and concurrent method to perform comparison and repair operations for each task. Specifically, when performing file verification, the host compares data with the user in an interactive manner to ensure the integrity and correctness of the file. In the repair operation, the host uses concurrent technology to process multiple tasks at the same time, improving operational efficiency and speed. In this way, the host can perform file verification and repair more effectively, improving the security and stability of the system. At the same time, this interactive and concurrent method also reflects the intelligence and efficiency of the host in processing tasks, providing users with more convenient and reliable services.

[0296] Fig. 20 is a schematic diagram of host-to-host interaction and concurrent task scheduling according to an embodiment of the present invention, such as Fig. 20As shown, a to g in the host-to-host interaction and concurrent task scheduling represent different tasks, among which step 1 is the completion of task a on the source machine; step 2 is the message to the target machine to check task e; step 3 is the target pool feedback that task e is not completed; step 4 is the completion of task b and stacking; step 5 is to check task f; step 6 is the completion of task f and the comparison task; step 7 is the popping of task b; step 8 is to repeat step 2.

[0297] In this embodiment, by comparing the files at the sending end and the receiving end, a detailed difference list is generated, and based on the difference list, it is determined whether there is differential information, the slice size is determined according to the differential information, and the file is cut according to the slice size to obtain a number of slice data; a single slice data is hashed and compared with the data on the source host in sequence to determine whether there is a difference; if so, all slice data subsequent to the current slice data are re-uploaded, and after merging into a repair file, the resource occupancy status of the repair process is calculated to determine whether the resource occupancy is less than the set value; if so, the allocated slice size is recalculated and the slice size value is updated. The technical problem of low efficiency in repairing data during file transmission is solved, and the technical effect of improving the efficiency of meter information storage is achieved.

[0298] According to an embodiment of the present invention, a data repair device is also provided. It should be noted that the data repair device can be used to execute the data repair method in embodiment 1.

[0299] Fig.21 FIG. 1 is a schematic diagram of a data repair device according to an embodiment of the present invention. Fig.21 As shown, the data repairing device 2100 may include: a first determining unit 2101, a segmenting unit 2102, a second determining unit 2103 and a repairing unit 2104.

[0300] The first determining unit 2101 is used to determine difference data between the initial data and the target data in response to receiving a repair instruction, wherein the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data.

[0301] The segmentation unit 2102 is used to segment the target data based on the difference data to obtain a plurality of segment data.

[0302] The second determining unit 2103 is used to determine the target segment data that needs to be repaired among the multiple segment data.

[0303] The repair unit 2104 is used to repair the target data based on the storage location of the target segment data.

[0304] Optionally, the first determination unit 2101 may include: a generation module, used to generate a difference list in response to receiving a repair instruction, wherein the difference list is used to indicate target data that differs from the initial data; and a first determination module, used to determine the difference data between the initial data and the target data based on the difference list.

[0305] Optionally, the segmentation unit 2102 may include: a second determination module, used to determine the slice length based on the difference data, wherein the slice length is used to indicate the number of bytes or bits of the fragment data; a segmentation module, used to compare the slice length with the length of the target data, and in response to the length of the target data being greater than or equal to the slice length, segmenting the target data to obtain multiple fragment data; and a deletion module, used to delete the target data in response to the length of the target data being less than the slice length.

[0306] Optionally, the second determination unit 2103 may include: a matching module, used to match multiple fragment data with the initial data to obtain a matching result, wherein the matching result is used to indicate whether the fragment data successfully matches or fails to match the initial data; a third determination module, used to determine the fragment data corresponding to the matching result as the target fragment data that needs to be repaired in response to the matching result being a matching failure.

[0307] Optionally, the repair unit 2104 may include: a fourth determination unit, used to determine the storage position in the initial data based on the storage position of the target fragment data; and a replacement unit, used to replace the fragment data after the storage position of the target fragment data with the initial data after the storage position of the initial data.

[0308] Optionally, the data repair device 2100 may further include: a third determination unit, for determining a resource occupancy value during the repair process in response to completion of the target data repair, wherein the resource occupancy value is used to indicate the ratio of resources of the file migration system occupied by the repair process; a fourth determination unit, for determining a target resource occupancy value in response to the resource occupancy value being greater than a resource occupancy threshold, wherein the target resource value is used to indicate the ratio of resources expected to be occupied by the repair process; and an update unit, for updating a slice length of the fragment data based on the target resource occupancy value, wherein the slice length is used to indicate the number of bytes or bits of the fragment data.

[0309] In this embodiment, in response to receiving a repair instruction, the difference data between the initial data and the target data is determined; based on the difference data, the target data is segmented to obtain a plurality of fragment data; among the plurality of fragment data, the target fragment data that needs to be repaired is determined; based on the storage location of the target fragment data, the target data is repaired. In other words, the present invention can determine the difference data between the initial data and the target data according to the received repair instruction during the file transmission process, and thus repair the target data according to the difference data, thereby solving the technical problem of low efficiency of data repair during the file transmission process and achieving the technical effect of improving the efficiency of data repair during the file transmission process.

[0310] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The storage medium includes a stored program, wherein the program executes the data repair method in Embodiment 1.

[0311] According to an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the data repair method in embodiment 1 is executed when the program is run.

[0312] According to an embodiment of the present invention, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the data repair method in Embodiment 1 is implemented.

[0313] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0314] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0315] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0316] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0317] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0318] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent functional component, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software functional component, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program codes.

[0319] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for repairing data, It is characterized in that include: In response to receiving a repair instruction, determining difference data between the initial data and the target data, wherein the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data; Based on the difference data, the target data is segmented to obtain a plurality of fragment data; Determining target segment data that needs to be repaired among the plurality of segment data; The target data is repaired based on the storage location of the target segment data.

2. The method according to claim 1, It is characterized in that In response to receiving the repair instruction, determining difference data between the initial data and the target data includes: In response to receiving the repair instruction, generating a difference list, wherein the difference list is used to indicate the target data that is different from the initial data; Based on the difference list, the difference data between the initial data and the target data is determined.

3. The method according to claim 1, It is characterized in that Based on the difference data, the target data is segmented to obtain a plurality of fragment data, including: Determine a slice length based on the difference data, wherein the slice length is used to indicate the number of bytes or bits of the segment data; In response to the length of the target data being greater than or equal to the slice length, segmenting the target data to obtain a plurality of the fragment data; In response to the length of the target data being smaller than the slice length, the target data is deleted.

4. The method according to claim 1, It is characterized in that Determining target segment data that needs to be repaired among the plurality of segment data includes: Matching the plurality of the fragment data with the initial data to obtain a matching result, wherein the matching result is used to indicate whether the fragment data successfully matches the initial data or fails to match; In response to the matching result being the matching failure, the segment data corresponding to the matching result is determined as the target segment data that needs to be repaired.

5. The method according to claim 1, It is characterized in that Repairing the target data based on the storage location of the target fragment data includes: determining the storage location in the initial data based on the storage location of the target segment data; The fragment data after the storage position of the target fragment data is replaced with the initial data after the storage position of the initial data.

6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: In response to the target data being repaired, determining a resource occupation value during the repair process, wherein the resource occupation value is used to indicate a ratio of resources of the file migration system occupied by the repair process; In response to the resource occupancy value being greater than a resource occupancy threshold, determining the target resource occupancy value, wherein the target resource value is used to indicate the resource ratio that the repair process is expected to occupy; Based on the target resource occupancy value, the slice length of the segment data is updated.

7. A data repair device, It is characterized in that include: A first determining unit, configured to determine difference data between the initial data and the target data in response to receiving a repair instruction, wherein the repair instruction is used to instruct to repair the target data, and the difference data is used to indicate the difference between the initial data and the target data; a segmentation unit, configured to segment the target data based on the difference data to obtain a plurality of segment data; A second determining unit, configured to determine a target segment data to be repaired among the plurality of segment data; A repair unit is used to repair the target data based on the storage location of the target fragment data.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A processor, It is characterized in that The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer program product, It is characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images