Data processing method, device and equipment between heterogeneous storage media

Abstract

An embodiment of the present invention provides a method, device, and equipment for processing data between heterogeneous storage media. The data processing method includes: obtaining at least one file to be processed in a source storage medium; performing calculations on the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed; obtaining second verification information corresponding to each of the at least one file to be processed stored in a target storage medium, where the source storage medium and the target storage medium are heterogeneous storage media; and judging whether the at least one file to be processed is a difference file between the source storage medium and the target storage medium based on the first verification information and the second verification information. The technical solution provided by this embodiment effectively implements a difference detection operation on data between heterogeneous storage media, which is beneficial to ensuring the accuracy and reliability of data migration or data copying in application scenarios where data migration or data copying is performed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device and equipment for processing data between heterogeneous storage media. Background Art

[0002] With the rapid development of science and technology, the application areas of cloud storage services that provide massive, secure, low-cost, and highly reliable data are becoming increasingly widespread. In big data migration scenarios, especially when migrating data between heterogeneous storage media, it is often necessary to compare data differences between the source and target storage media. For example, when migrating data from a user-built distributed Hadoop cluster to the cloud, it is necessary to compare data differences between the Hadoop cluster and the cloud storage media.

[0003] For heterogeneous storage media, existing data processing methods often skip data integrity verification, resulting in difficulty in timely detection of problems when abnormalities occur in data migration operations. Summary of the Invention

[0004] The embodiments of the present invention provide a method, apparatus and device for processing data between heterogeneous storage media, which can solve the problem in the prior art that heterogeneous storage media cannot be directly used because their default data verification methods are often different.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing data between heterogeneous storage media, including:

[0006] Obtaining at least one file to be processed from a source storage medium;

[0007] Performing calculations on the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed;

[0008] Obtaining second verification information corresponding to each of the at least one to-be-processed files stored in a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media;

[0009] Based on the first verification information and the second verification information, it is determined whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

[0010] In a second aspect, an embodiment of the present invention provides a data processing device between heterogeneous storage media, including:

[0011] A first acquisition module is used to acquire at least one file to be processed in a source storage medium;

[0012] A first calculation module is configured to calculate the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed;

[0013] The first acquisition module is further configured to acquire second verification information corresponding to each of the at least one to-be-processed files stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media;

[0014] The first processing module is configured to determine, based on the first verification information and the second verification information, whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the data processing method between heterogeneous storage media in the above-mentioned first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer storage medium for storing a computer program, wherein the computer program enables a computer to implement the data processing method between heterogeneous storage media in the first aspect when executed.

[0017] In a fifth aspect, an embodiment of the present invention provides a method for processing data between heterogeneous storage media, including:

[0018] Obtaining at least one file to be processed stored in a source storage medium;

[0019] Performing calculations on the at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed;

[0020] The at least one to-be-processed file and the corresponding verification information are transmitted to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0021] In a sixth aspect, an embodiment of the present invention provides a data processing device between heterogeneous storage media, including:

[0022] A second acquisition module is used to acquire at least one to-be-processed file stored in a source storage medium;

[0023] A second calculation module is used to calculate the at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed;

[0024] The second processing module is configured to transmit the at least one to-be-processed file and the corresponding verification information to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0025] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data processing method between heterogeneous storage media in the above-mentioned fifth aspect.

[0026] In an eighth aspect, an embodiment of the present invention provides a computer storage medium for storing a computer program, wherein the computer program enables a computer to implement the data processing method between heterogeneous storage media in the fifth aspect when executed.

[0027] The data processing method, device and equipment between heterogeneous storage media provided in this embodiment obtain at least one file to be processed in the source storage medium, calculate the at least one file to be processed, obtain the first verification information corresponding to each of the at least one file to be processed, and then obtain the second verification information corresponding to each of the at least one file to be processed stored in the target storage medium, and based on the first verification information and the second verification information, determine whether the at least one file to be processed is a difference file between the source storage medium and the target storage medium, thereby effectively realizing the difference detection operation on the data between heterogeneous storage media, so that in the application scenario of data migration or incremental data copying, the difference problem can be discovered in time; when it is necessary to perform a migration operation on the difference file, it is also possible to check whether the migration operation of the difference file is normal. Specifically, the migration process verification information and the post-migration process verification information corresponding to the difference file can be obtained, and based on the migration process verification information and the post-migration process verification information, it is detected whether the migration operation of the difference file is normal, thereby effectively ensuring the accuracy and reliability of data migration or incremental data copying. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A schematic diagram illustrating the principle of a data processing method between heterogeneous storage media provided by an embodiment of the present invention;

[0030] Figure 2A schematic diagram of a flow chart of a method for processing data between heterogeneous storage media provided by an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a process for calculating the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed provided in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of a process for transmitting the difference file to the target storage medium provided in an embodiment of the present invention;

[0033] Figure 5 A schematic flow chart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention;

[0034] Figure 6 A schematic flow chart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention;

[0035] Figure 7 A schematic flow chart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention;

[0036] Figure 8 A schematic diagram of a source storage medium and a target storage medium provided in an application embodiment of the present invention;

[0037] Figure 9 A flowchart of a method for processing data between heterogeneous storage media provided by an application embodiment of the present invention;

[0038] Figure 10 A schematic diagram of the structure of a data processing device between heterogeneous storage media provided by an application embodiment of the present invention;

[0039] Figure 11 For Figure 10 A schematic structural diagram of an electronic device corresponding to a data processing device between heterogeneous storage media provided in the illustrated embodiment;

[0040] Figure 12 A schematic structural diagram of another data processing device between heterogeneous storage media provided by an embodiment of the present invention;

[0041] Figure 13 For Figure 12 A schematic structural diagram of an electronic device corresponding to a data processing device between heterogeneous storage media provided in the illustrated embodiment. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one.

[0044] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0045] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0046] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0047] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0048] To facilitate understanding of the specific implementation process and implementation effect of the data processing method in this embodiment, the following briefly describes the relevant technologies:

[0049] In big data migration scenarios, especially when migrating data between heterogeneous storage media, it's often necessary to compare data differences between the source and target storage media. For example, when migrating data from a user-built distributed Hadoop cluster to the cloud, it's necessary to compare data differences between the Hadoop cluster and the cloud storage media. Once the difference data is obtained, migration operations can be performed on the difference data, effectively verifying the integrity of the migrated data and further ensuring its correctness and integrity.

[0050] However, for heterogeneous storage media, since the default data verification methods of heterogeneous storage media are often different, they cannot be used directly. For example, for Hadoop-type storage media, the default data verification method may be a 32-bit Cyclic Redundancy Check (CRC32) algorithm; for Simple Storage Service (S3)-type storage media, the default data verification method may be a Message Digest Algorithm (MD5) algorithm; for cloud object storage service type (for example, OSS) storage media, the default data verification method may be a 64-bit Cyclic Redundancy Check (CRC64) algorithm. Therefore, in the application scenario of big data migration, the existing data processing method can often only skip the data integrity check. In this way, when an abnormality occurs in the data migration operation, it is easy to fail to discover the problem in time.

[0051] In order to avoid the above-mentioned defect of not being able to discover problems in a timely manner, in some application scenarios where incremental copying of data is required, a difference comparison can be performed on the data stored between heterogeneous storage media. Specifically, the source data and target data that need to be compared are read out through the source storage medium and the target storage medium respectively, and the file name and file size of the source data, and the file name and file size of the target data are obtained; then, based on the file name and file size of the source data, and the file name and file size of the target data, it is identified whether the source data and the target data are consistent.

[0052] However, this implementation is not only prone to file recognition errors, for example, when the file names of two files are exactly the same, but only a single character is changed between the source and target data, the difference between the source and target data will not be recognized. In addition, when performing difference comparisons between homogeneous storage media, the above implementation often requires rereading the data on the target storage medium and calculating a checksum. The checksum refers to the verification information used for error detection and integrity verification in the fields of data processing and data communication. This greatly increases the utilization of data processing resources and reduces data processing efficiency.

[0053] In order to solve the above technical problems, this embodiment provides a data processing method, device and equipment between heterogeneous storage media, wherein the data processing method obtains at least one file to be processed in the source storage medium, calculates at least one file to be processed, obtains first verification information corresponding to each of the at least one file to be processed, and then extracts second verification information corresponding to each of the at least one file to be processed stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media, and based on the first verification information and the second verification information, it is determined whether the at least one file to be processed is a difference file between the source storage medium and the target storage medium, thereby effectively realizing the difference identification operation of the files between the source storage medium and the target storage medium, which is beneficial to ensure the accuracy and reliability of data migration or incremental data copying in application scenarios.

[0054] The following embodiments of the present invention are described in detail with reference to the accompanying drawings. In the absence of conflicts between the embodiments, the following embodiments and features therein may be combined with each other.

[0055] Figure 1 A schematic diagram illustrating the principle of a data processing method between heterogeneous storage media provided by an embodiment of the present invention; Figure 2 A flowchart of a data processing method between heterogeneous storage media provided by an embodiment of the present invention; Figure 1-Figure 2 As shown, this embodiment provides a method for processing data between heterogeneous storage media. The execution subject of this method is a data processing device between heterogeneous storage media. It is understood that the data processing device between heterogeneous storage media can be implemented as software, or a combination of software and hardware. The data processing device between heterogeneous storage media can be independent of the heterogeneous storage media, or the data processing device between heterogeneous storage media can be integrated into the heterogeneous storage media. Specifically, the data processing method between heterogeneous storage media may include:

[0056] Step S201: Acquire at least one file to be processed in a source storage medium.

[0057] Step S202: Calculate at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed.

[0058] Step S203: obtaining second verification information corresponding to at least one to-be-processed file stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0059] Step S204: Based on the first verification information and the second verification information, determine whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

[0060] The following is a detailed description of the specific implementation process and results of each of the above steps:

[0061] Step S201: Acquire at least one file to be processed in a source storage medium.

[0062] Among them, the source storage medium may refer to a storage medium that can store at least one file to be processed (which may correspond to at least one file type), and the source storage medium may be a private storage medium or a cloud storage medium, and the target storage medium refers to a storage medium that can realize file migration operations or file processing operations. The above-mentioned target storage medium and the source storage medium are heterogeneous storage media. Specifically, heterogeneous storage media refer to storage media with different structures, for example: the source storage medium is a tape drive, and the target storage medium is a mechanical hard disk; or, the source storage medium is a read-only memory, and the target storage medium is an optical disk, etc. The above-mentioned source storage medium and target storage medium are heterogeneous storage media.

[0063] Specifically, when a data processing device capable of implementing the data processing method is independently provided on a source storage medium, the data processing device can be communicatively connected to the source storage medium and then obtain at least one file to be processed stored in the source storage medium via the established communication link. When the data processing device is provided on the source storage medium, the data processing device can directly access the source storage medium and thereby obtain at least one file to be processed. When the data processing device is provided on a target storage medium, the data processing device can directly establish a communication link between the source storage medium and the target storage medium and obtain at least one file to be processed via the established communication link.

[0064] Of course, those skilled in the art may also adopt other methods to obtain at least one file to be processed in the source storage medium, as long as the stability and reliability of obtaining the at least one file to be processed can be guaranteed, which will not be described in detail here.

[0065] Step S202: Calculate at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed.

[0066] After obtaining at least one file to be processed, the at least one file to be processed can be analyzed and calculated, so as to obtain the first verification information corresponding to each of the at least one file to be processed. Specifically, the source storage medium stores a verification algorithm for calculating the at least one file to be processed. It can be understood that different types of source storage media can correspond to different verification algorithms. For example: for Hadoop type storage media, its default verification algorithm can be a 32-bit Cyclic Redundancy Check (CRC32) algorithm; for Simple Storage Service (S3) type storage media, its default verification algorithm can be a Message Digest Algorithm (MD5) algorithm; for cloud object storage service type (for example: OSS) storage media, its default verification algorithm can be a 64-bit Cyclic Redundancy Check (CRC64) algorithm. The verification algorithm can be obtained through the source storage medium, and then the verification algorithm is used to analyze and calculate at least one file to be processed, so as to obtain the first verification information corresponding to at least one file to be processed. In specific implementation, the first verification information may include any of the following categories: data checksum, check code, XOR verification information, etc. Of course, those skilled in the art can determine other types of first verification information based on the corresponding verification algorithm, as long as the accuracy and reliability of the determination of the first verification information can be guaranteed, and no further details will be given here.

[0067] Step S203: obtaining second verification information corresponding to at least one to-be-processed file stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0068] The target storage medium stores second verification information corresponding to each of the at least one to-be-processed files. The second verification information may be generated by sending the at least one to-be-processed file and its corresponding verification information to the target storage medium after the source storage medium obtains the verification information corresponding to each of the at least one to-be-processed files. In some instances, before obtaining the at least one to-be-processed file from the source storage medium, the method in this embodiment may further include: obtaining the at least one to-be-processed file stored in the source storage medium at a historical moment; calculating historical verification information corresponding to each of the at least one to-be-processed files; and transmitting the historical verification information as the second verification information to the target storage medium along with the at least one to-be-processed file.

[0069] Specifically, at the historical moment t0, at least one file to be processed (W1, W2, W3 and W4) is stored in the source storage medium. As time goes by, the at least one file to be processed in the source storage medium may change or be updated. For example, at the current moment t1, at least one file to be processed (W1, W2, W3, W4 and W5) is stored in the source storage medium. At this time, at least one file to be processed has been updated. Therefore, at the current moment t1, the at least one file to be processed stored in the source storage medium may be the same as or different from the at least one file to be processed stored in the source storage medium at the historical moment t0.

[0070] After obtaining at least one to-be-processed file stored in the source storage medium at a historical time, a verification calculation can be performed on the at least one to-be-processed file to calculate historical verification information corresponding to each of the at least one to-be-processed files. Specifically, the verification algorithm used to perform the verification calculation on the at least one to-be-processed file can be a default verification algorithm stored in the source storage medium. After obtaining the historical verification information, the historical verification information can be transmitted as second verification information to the target storage medium along with the at least one to-be-processed file.

[0071] In some instances, using historical verification information as second verification information and transmitting it to the target storage medium together with at least one file to be processed may include: using the second verification information as the header information of the corresponding file to be processed and transmitting it to the target storage medium together with the file to be processed, thereby effectively storing the second verification information in the target storage medium, thereby facilitating the acquisition of the second verification information corresponding to each of the at least one file to be processed stored in the target storage medium.

[0072] It should be noted that, in this embodiment, the execution order between step S203 and step S202 is not limited to the order defined by the above description. Those skilled in the art may adjust the execution order between step S202 and step S203 according to specific application scenarios or application requirements. For example, step S203 may be executed before step S202, or step S203 and step S202 may be executed simultaneously, which will not be repeated here.

[0073] Step S204: Based on the first verification information and the second verification information, determine whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

[0074] Among them, after obtaining the first verification information and the second verification information, the first verification information and the second verification information can be analyzed and processed to determine whether at least one to-be-processed file is a difference file between the source storage medium and the target storage medium based on the analysis and processing results, that is, to determine whether there is a difference between the to-be-processed file in the source storage medium and the to-be-processed file in the target storage medium based on the analysis and processing results; when the to-be-processed file in the source storage medium is the same as the file stored in the target storage medium, it can be determined that the to-be-processed file is not a difference file between the source storage medium and the target storage medium; when the to-be-processed file in the source storage medium is different from the file stored in the target storage medium, it can be determined that the to-be-processed file is a difference file between the source storage medium and the target storage medium.

[0075] Specifically, based on the first verification information and the second verification information, determining whether at least one file to be processed is a difference file between the source storage medium and the target storage medium may include: when the first verification information and the second verification information are the same, determining that the file to be processed corresponding to the first verification information is not a difference file; when the first verification information and the second verification information are different, determining that the file to be processed corresponding to the first verification information is a difference file.

[0076] For example, a source storage medium stores at least one file to be processed, which may include W1, W2, W3, W4, and W5. The first verification information corresponding to each of the files to be processed may be S11, S12, S13, S14, and S15. The second verification information corresponding to each of the files to be processed obtained in the target storage medium may be S21, S22, S23, S24, and S25. After obtaining the first and second verification information, the first and second verification information may be compared, i.e., comparing whether S11 is identical to S21, whether S12 is identical to S22, whether S13 is identical to S23, whether S14 is identical to S24, and whether S15 is identical to S25. If S11 is identical to S21, S13 is identical to S23, and S14 is identical to S24, it may be determined that W1, W3, and W4 corresponding to S11, S13, and S14 are not difference files between the source storage medium and the target storage medium. When S12 is different from S22 and S15 is different from S25, it can be determined that W2 and W5 corresponding to the above S12 and S15 are difference files between the source storage medium and the target storage medium, thereby effectively realizing the difference detection operation of data between heterogeneous storage media.

[0077] After determining that the file to be processed corresponding to the first verification information is a difference file, the method in this embodiment further includes: displaying the first verification information, the second verification information, and the difference file.

[0078] Among them, after determining that the file to be processed corresponding to the first verification information is a difference file, the first verification information, the second verification information and the difference file can be displayed through the preset interface. Specifically, the first verification information, the second verification information and the difference file can be displayed at different positions of the preset interface, so that the user can obtain relevant information of the difference file more directly and quickly.

[0079] It can be understood that when the first verification information, the second verification information and the difference file are displayed through the preset interface, in order to improve the quality and efficiency of information display, the first verification information, the second verification information and the difference file can be displayed in different forms. For example, a floating window, an information prompt box, or a pop-up window can be used to display the first verification information, the second verification information and the difference file. In order to improve the quality and effect of information display, different fonts and different colors can be used to display the first verification information, the second verification information and the difference file.

[0080] In addition, when the first verification information, the second verification information and the difference file are displayed through a preset interface, the preset interface may include a control for closing the display interface. The user can input an execution operation on the control through the preset interface. The above-mentioned execution operation can be any one of a click operation and a sliding operation. After obtaining the execution operation input by the user on the control, the window for displaying the first verification information, the second verification information and the difference file can be closed based on the execution operation.

[0081] In addition, when the first verification information, the second verification information and the difference file are displayed through a preset interface, the preset interface may include a control for displaying detailed information. The user can input an execution operation on the control through the preset interface. The above-mentioned execution operation can be any one of a click operation and a sliding operation. After obtaining the execution operation input by the user on the control, the detailed information corresponding to the first verification information, the second verification information and the difference file can be displayed based on the execution operation, so that the user can quickly and directly understand the specific information of the difference detection.

[0082] The data processing method between heterogeneous storage media provided in this embodiment obtains at least one file to be processed in the source storage medium, calculates the at least one file to be processed, obtains first verification information corresponding to each of the at least one file to be processed, and then obtains second verification information corresponding to each of the at least one file to be processed stored in the target storage medium. Based on the first verification information and the second verification information, it is determined whether the at least one file to be processed is a difference file between the source storage medium and the target storage medium, thereby effectively realizing the difference detection operation of the data between the heterogeneous storage media. This is beneficial to ensure the accuracy and reliability of data migration or incremental data copying in application scenarios.

[0083] Figure 3 The present invention provides a flowchart of calculating at least one file to be processed to obtain the first verification information corresponding to each of the at least one file to be processed; based on the above embodiment, continue to refer to the attached Figure 3 As shown, this embodiment provides a specific implementation method for calculating at least one file to be processed. Specifically, in this embodiment, calculating at least one file to be processed and obtaining first verification information corresponding to each of the at least one file to be processed may include:

[0084] Step S301: Obtain a checksum algorithm for performing checksum calculation on at least one file to be processed.

[0085] Step S302: Calculate at least one file to be processed using a checksum algorithm to obtain a first data checksum corresponding to each of the at least one file to be processed.

[0086] Among them, for the source storage medium, the source storage medium stores a checksum algorithm related to the storage medium type. Therefore, after the storage medium type of the source storage medium is determined, the checksum algorithm used in the source storage medium to perform checksum calculations on at least one to-be-processed file can be determined. For example, for a Hadoop type storage medium, its default checksum algorithm may be a 32-bit Cyclic Redundancy Check (CRC32) algorithm; for a Simple Storage Service (S3) type storage medium, its default checksum algorithm may be a Message Digest Algorithm (MD5) algorithm; for a cloud object storage service type (e.g., OSS) storage medium, its default checksum algorithm may be a 64-bit Cyclic Redundancy Check (CRC64) algorithm.

[0087] In order to accurately obtain the first data checksum corresponding to at least one file to be processed, a checksum algorithm for performing checksum calculation on at least one file to be processed can be obtained through a source storage medium, and then the checksum algorithm is used to calculate the at least one file to be processed to obtain the first data checksum corresponding to at least one file to be processed, thereby effectively ensuring the accuracy and reliability of obtaining the first verification information.

[0088] In this embodiment, by obtaining a checksum algorithm for performing verification calculations on at least one file to be processed, and then using the checksum algorithm to calculate at least one file to be processed, a first data checksum corresponding to each of the at least one file to be processed is obtained, thereby effectively ensuring the accuracy and reliability of obtaining the first verification information, and thereby improving the accuracy of difference judgment based on the first verification information. In addition, based on the data checksum stored in the target storage medium, the integrity verification of the migrated data and the incremental copy function can be efficiently completed when migrating data between heterogeneous storage media.

[0089] In some examples, after determining that the file to be processed corresponding to the first verification information is a difference file, the method in this embodiment may further include: transmitting the difference file to a target storage medium.

[0090] If the file to be processed is a differential file, it indicates that the differential file exists between the source storage medium and the target storage medium. In this way, in an application scenario where a file migration operation or an incremental file copy is required, the differential file can be transferred to the target storage medium.

[0091] For example: a file to be processed A is stored in a source storage medium, and then after the file to be processed A is transferred to a target storage medium, after the user adds, deletes or modifies the content of the file to be processed A in the source storage medium, the file to be processed A` in the source storage medium can be obtained. After obtaining the file to be processed A`, a difference detection operation can be performed on the file to be processed A` stored in the source storage medium and the file to be processed A stored in the target storage medium, and the first verification information corresponding to the file to be processed A` in the source storage medium and the second verification information corresponding to the file to be processed in the target storage medium are analyzed and compared. When the first verification information is different from the second verification information, it can be determined that the file to be processed A` is a difference file between the source storage medium and the target storage medium.

[0092] In some instances, when a differential file is transferred to a target storage medium, the data transfer progress corresponding to the differential file can be obtained, and the data transfer progress can be displayed through a preset interface. For example, the data transfer progress can be displayed in the preset interface in the form of a progress bar, so that the user can directly and quickly understand the process of data migration of the differential file.

[0093] For details, please refer to the attached Figure 4 As shown, in this embodiment, transferring the difference file to the target storage medium may include:

[0094] Step S401: Generate a distributed transmission task corresponding to the difference file.

[0095] Step S402: Determine the distributed nodes corresponding to the distributed transmission task.

[0096] Step S403: Utilize the distributed nodes to transmit the difference file to the target storage medium.

[0097] In the process of transferring the difference file to the target storage medium, in order to ensure the stability and reliability of the file transfer operation, a distributed transmission task (MapReduce task) corresponding to the difference file can be generated. The above-mentioned distributed transmission task can correspond to one or more distributed nodes. After the distributed transmission task is generated, the distributed node corresponding to the distributed transmission task can be determined, and then the distributed node can be used to transfer the difference file to the target storage medium.

[0098] For example, when the number of difference files is three, a distributed transmission task can be generated based on the above three difference files, and then the distributed nodes corresponding to the above distributed transmission task can be determined. The number of distributed nodes can be three. After determining the above three distributed nodes, the distributed nodes can be used to transmit the difference files to the target storage medium, that is, the three difference files can be transmitted to the target storage medium using three distributed nodes respectively, thereby effectively ensuring the stability and reliability of transmitting the difference files to the target storage medium.

[0099] In one implementation, the target storage medium can be used as a data lake, which can provide a single storage for all data. For example, when the target storage medium is an OSS object storage system (OSS), OSS can be used as a data lake. In this case, the number of source storage media for data transmission with OSS can be one or more. If there are multiple source storage media, data migration operations can be performed between the multiple source storage media and OSS. Furthermore, the types of data migrated between different source storage media and OSS can be the same or different. The migrated data types can include at least one of the following: structured data (e.g., rows and columns), semi-structured data (e.g., logs, XML data), unstructured data (e.g., documents, emails, PDF files, etc.), and binary data (e.g., images, audio, video). When different source storage media migrate stored data to OSS, OSS aggregates the data from the different source storage media, thereby achieving unified storage operations for the data from the different source storage media. This effectively facilitates the management and viewing of all data.

[0100] Figure 5 A flow chart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention; based on the above embodiment, refer to the attached Figure 5 As shown, when the difference file is transferred to the target storage medium, the method in this embodiment may further include:

[0101] Step S501: Calculate the in-transmission verification information corresponding to the difference file.

[0102] In the process of transmitting the difference file to the target storage medium, in order to detect whether the transmission process of the difference file is normal, the in-transmission verification information corresponding to the difference file can be calculated. In some instances, calculating the in-transmission verification information corresponding to the difference file may include: obtaining a verification algorithm stored in the target storage medium for verifying data; using the verification algorithm to calculate the difference file to obtain the in-transmission data checksum corresponding to the difference file.

[0103] Specifically, the target storage medium stores a verification algorithm for verifying data. In order to detect whether the transmission operation of the difference file is normal, the data processing device used to implement the data processing method can obtain the verification algorithm for verifying data stored in the target storage medium. After obtaining the above-mentioned verification algorithm, the verification algorithm can be used to calculate the difference file, so as to obtain the data checksum in transmission corresponding to the difference file.

[0104] Step S502: Obtain target verification information obtained by calculating the received difference file by the target storage medium.

[0105] In the process of transferring the difference file from the source storage medium to the target storage medium, the target verification information obtained by the target storage medium by calculating the received difference file can be obtained. It can be understood that the target verification information can be obtained by the target storage medium using the verification algorithm stored in the target storage medium for verifying data to analyze and calculate the difference file.

[0106] Step S503: Based on the verification information during transmission and the target verification information, it is detected whether the transmission of the difference file is normal.

[0107] After obtaining the in-transmission checksum information and the target checksum information, the in-transmission checksum information and the target checksum information can be analyzed and processed to detect whether the transmission operation of the difference file is normal. In some examples, in this embodiment, detecting whether the transmission of the difference file is normal based on the in-transmission checksum information and the target checksum information can include: if the in-transmission checksum information and the target checksum information are the same, then determining that the transmission of the difference file is normal, that is, indicating that the data migration operation corresponding to the difference file is normal; if the in-transmission checksum information and the target checksum information are different, then determining that the transmission of the difference file is abnormal, that is, indicating that the data migration operation corresponding to the difference file is abnormal.

[0108] Specifically, when the verification information during transmission is the same as the target verification information, it means that the difference file stored in the source storage medium is the same as the difference file transmitted to the target storage medium, and thus it can be determined that the transmission operation of the difference file is normal; when the verification information during transmission is different from the target verification information, it means that the difference file stored in the source storage medium is different from the difference file transmitted to the target storage medium, for example: the difference file transmission operation is incomplete, data is lost in the difference file during transmission, etc., and thus it can be determined that the transmission operation of the difference file is abnormal, thereby effectively realizing that whether the transmission operation of the difference file is normal can be accurately detected based on the verification information during transmission and the target verification information, further improving the accuracy and reliability of the use of the data processing method.

[0109] For example, there is a difference file A between the source storage medium and the target storage medium. After migrating the difference file A to the target storage medium, the target storage medium can obtain the migrated file A` corresponding to the difference file A, and then obtain the verification algorithm in the target storage medium for analyzing and processing the migrated file A`, calculate the difference file A using the verification algorithm to obtain verification information a, and then obtain the target storage medium to calculate the migrated file A` to obtain verification information a`, and then compare the verification information a and the verification information a`. When the verification information a is the same as the verification information a`, it means that the migrated file A` is the same as the difference file A, that is, the migration operation of the difference file A is normal; when the migrated file A` is different from the difference file A, it means that the migration operation of the difference file A is abnormal.

[0110] In some instances, the method in this embodiment may further include: displaying a first prompt message in a preset interface when it is determined that the differential file transmission is normal; and displaying a second prompt message in the preset interface when it is determined that the differential file transmission is abnormal.

[0111] Among them, after obtaining the detection result of whether the transmission of the difference file is normal, different prompt information can be generated based on different detection results. Specifically, when it is determined that the transmission of the difference file is normal, a first prompt information can be displayed in the preset interface, and the first prompt information is used to identify that the transmission operation of the difference file is in a normal state; when it is determined that the transmission of the difference file is abnormal, a second prompt information can be displayed in the preset interface, and the second prompt information is used to identify that the transmission operation of the difference file is in an abnormal state. This effectively enables the user to quickly and directly understand the detection result of the transmission of the difference file through the displayed prompt information.

[0112] It can be understood that when the first prompt information / second prompt information is displayed through the preset interface, in order to improve the quality and efficiency of information display, the first prompt information / second prompt information can be displayed in different forms, for example: the first prompt information / second prompt information can be displayed in a floating window, an information prompt box, or a pop-up window. In order to improve the quality and effect of information display, the first prompt information / second prompt information can be displayed in different fonts and different colors. For example, the first prompt information can be displayed in green and the second prompt information can be displayed in red.

[0113] In addition, when the first prompt information / second prompt information is displayed through a preset interface, the preset interface may include a control for closing the display interface. The user can input an execution operation on the control through the preset interface. The above-mentioned execution operation can be any one of a click operation and a sliding operation. After obtaining the execution operation input by the user on the control, the window for displaying the first prompt information / second prompt information can be closed based on the execution operation.

[0114] In addition, when the first prompt information / second prompt information is displayed through a preset interface, the preset interface may include a control for displaying detailed information. The user can input an execution operation on the control through the preset interface. The above-mentioned execution operation can be any one of a click operation and a sliding operation. After obtaining the execution operation input by the user on the control, detailed information corresponding to the first prompt information / second prompt information can be displayed based on the execution operation, so that the user can quickly and directly understand the specific information of the difference file transfer process.

[0115] Figure 6 A flowchart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention; based on any of the above embodiments, continue to refer to the attached Figure 6 As shown, before calculating at least one file to be processed, the method in this embodiment may further include:

[0116] Step S601: Acquire a stored file corresponding to at least one file to be processed stored in a target storage medium;

[0117] Step S602: screening at least one to-be-processed file based on the stored files to obtain a target processing file included in the at least one to-be-processed file, wherein the target processing file is identical to the corresponding stored file.

[0118] Among them, before calculating at least one file to be processed, in order to reduce the computational amount of data processing, a stored file corresponding to the at least one file to be processed stored in the target storage medium can be obtained, and then the at least one file to be processed can be filtered based on the stored file. Specifically, the filtering operation on the at least one file to be processed can be implemented based on the name information, file size, file format, editing timestamp corresponding to the file, keywords included in the file and other information of the stored file, so that the target processing file included in the at least one file to be processed can be obtained. It can be understood that the target processing file is at least a part of the at least one file to be processed, and the target processing file is the same as the corresponding stored file.

[0119] In some instances, filtering at least one file to be processed based on stored files to obtain a target processing file included in the at least one file to be processed may include: obtaining a first file name and a first file size corresponding to each of the at least one file to be processed, and a second file name and a second file size corresponding to the stored file; filtering at least one file to be processed based on the first file name, the first file size, the second file name, and the second file size to obtain a target processing file included in the at least one file to be processed.

[0120] Specifically, the source storage medium may store a first file name and a first file size corresponding to each of the at least one to-be-processed files, and the target storage medium may store a second file name and a second file size corresponding to the stored files. After obtaining the first file name, the first file size, the second file name, and the second file size, the first file name, the first file size, the second file name, and the second file size may be analyzed and processed to implement a screening operation on the at least one to-be-processed file, thereby obtaining a target processing file included in the at least one to-be-processed file.

[0121] In some instances, screening at least one file to be processed based on the first file name, the first file size, the second file name, and the second file size, and obtaining the target processing file included in the at least one file to be processed may include: when the first file name and the second file name are the same, and the first file size is different from the second file size, determining the file to be processed corresponding to the first file name as a differential file; when the first file name and the second file name are the same, and the first file size is the same as the second file size, determining the file to be processed corresponding to the first file name as the target processing file.

[0122] For example, the first file name corresponding to at least one file to be processed stored in the source storage medium may include: W1, W2, W3, W4, W5, W6, and the first file size corresponding to the at least one file to be processed may include: S11, S12, S13, S14, S15, S16. The second file name of the stored file corresponding to the at least one file to be processed stored in the target storage medium may include: W1, W2, W4, W5, W6, and the second file size corresponding to the at least one file to be processed may include: S21, S22, S24, S25, S26.

[0123] When the above-mentioned first file name is the same as the second file name, and the first file size is the same as the second file size, for example: for the above-mentioned W1, W2, and W5 files, when S11=S21, S12=S22, and S15=S25, it can be determined that the stored file in the target storage medium is the same as at least one to-be-processed file in the source storage medium. At this time, the to-be-processed file in the above-mentioned source storage medium that is the same as the stored file can be determined as the target processing file, so as to further perform file difference verification based on the verification information of the file in the source storage medium and the verification information obtained in the target storage medium.

[0124] When the first file name and the second file name are the same, and the first file size and the second file size are different, for example: for the above-mentioned W4 and W6 files, when S14≠S24 and S16≠S26, it can be determined that the stored file in the target storage medium is different from at least one to-be-processed file in the source storage medium, that is, it can be determined that there is a difference between at least one to-be-processed file in the source storage medium and the stored file in the target storage medium. Therefore, there is no need to perform a difference detection operation on the above-mentioned stored files, which further improves the quality and efficiency of the data processing method.

[0125] In this embodiment, by obtaining a stored file corresponding to at least one file to be processed stored in a target storage medium, and then screening at least one file to be processed based on the stored file, a target processing file included in at least one file to be processed is obtained. This effectively enables a preliminary screening of at least one file to be processed before a difference detection operation is performed on at least one file to be processed, thereby obtaining the target processing file included in at least one file to be processed, and then a difference detection operation can be performed on the target processing file. This effectively narrows the data processing scope for the difference detection operation and further improves the quality and efficiency of the data processing method.

[0126] Figure 7 A flow chart of another method for processing data between heterogeneous storage media provided by an embodiment of the present invention; Figure 7 As shown, this embodiment provides another method for processing data between heterogeneous storage media. The execution subject of this method is a data processing device between heterogeneous storage media. It is understood that the data processing device between heterogeneous storage media can be implemented as software, or a combination of software and hardware. The data processing device between heterogeneous storage media can be independent of the heterogeneous storage media, or the data processing device between heterogeneous storage media can be integrated into the heterogeneous storage media. Specifically, the data processing method between heterogeneous storage media may include:

[0127] Step S701: Acquire at least one to-be-processed file stored in a source storage medium.

[0128] Step S702: Calculate at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed.

[0129] Among them, the specific implementation method and implementation effect of the above steps S701-step S702 in this embodiment are similar to the specific implementation method and implementation effect of steps S201-step S202 in the above embodiment. Please refer to the above statements for details and will not be repeated here.

[0130] Step S703: transmitting at least one to-be-processed file and corresponding verification information to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0131] Among them, after obtaining at least one file to be processed and the corresponding verification information, the at least one file to be processed and the corresponding verification information can be transmitted to the target storage medium. Specifically, transmitting at least one file to be processed and the corresponding verification information to the target storage medium can include: using the verification information as the header information of the corresponding file to be processed, and transmitting it to the target storage medium together with the file to be processed.

[0132] The data processing method between heterogeneous storage media provided in this embodiment obtains at least one file to be processed stored in a source storage medium, calculates the at least one file to be processed, obtains verification information corresponding to each of the at least one file to be processed, and then transmits the at least one file to be processed and the corresponding verification information to a target storage medium, thereby effectively realizing a data migration operation, and making the target storage medium store verification information corresponding to the migrated data, thereby facilitating the acquisition of verification information corresponding to the migrated data stored in the target storage medium, so as to realize a file difference detection operation based on the verification information.

[0133] For specific applications, refer to the attached Figure 8-Figure 9 As shown, taking the privatized Hadoop cluster as the source storage medium and the cloud object storage OSS as the target storage medium as an example, the source storage medium at this time stores the CRC32 checksum algorithm for obtaining the data checksum, and the target storage medium stores the CRC64 checksum algorithm for obtaining the data checksum.

[0134] This application embodiment provides a method for processing data between heterogeneous storage media. By effectively utilizing the native data checksums in the source and target storage media, this method can efficiently complete the integrity check operation and incremental copy operation of the migrated data. Specifically, when performing the data migration operation for the first time, only the data migration operation needs to be performed; in subsequent data migration processes, if an incremental copy operation is required, a difference detection operation needs to be performed first, and then the migration and copy operation is performed. Specifically, in the application scenario of incremental copy, the data processing method may include the following steps:

[0135] Step 1: Obtain a list of files on the source storage medium and the target storage medium by recursively querying the source directory and the target directory.

[0136] The source storage medium stores a source directory, through which a file list on the source storage medium can be obtained; the target storage medium stores a target directory, through which a file list on the target storage medium can be obtained.

[0137] Step 2: Obtain the file name and file size of the source file based on the file list in the source storage medium; obtain the file name and file size of the target file based on the file list in the target storage medium.

[0138] Step 3: Compare the file names and file sizes of the source file and the target file, and perform a preliminary detection of the differences between the source file and the target file.

[0139] For a source file and a target file with the same file name, if the file size of the source file is different from the file size of the target file, the source file and the target file can be recorded as differential files; if the file size of the source file is the same as the file size of the target file, the source file and the target file can be further analyzed and processed to further determine whether the source file and the target file are differential files.

[0140] Step 4: For source files and target files with the same file name and file size, the data checksum corresponding to the source file can be calculated; for the target file, there is no need to read the file and recalculate the data checksum. The data checksum corresponding to the target file stored in the target storage medium can be directly obtained, and then a difference detection operation can be performed based on the data checksum corresponding to the source file and the data checksum corresponding to the target file.

[0141] Among them, after the source storage medium obtains the data checksum corresponding to the source file, the source file and the data checksum corresponding to the source file can be transmitted to the target storage medium. Specifically, the obtained data checksum can be a customized data checksum, and then the customized data checksum is stored in the metadata of the source file. The metadata is used to store some identification information of the source file and customized information, etc. At this time, the customized data checksum can be used as header information in the metadata, and then the header information is sent to the target storage medium together with the source file, so that the target storage medium can store the data checksum corresponding to the target file.

[0142] Specifically, when the data checksum corresponding to the source file is the same as the data checksum corresponding to the target file, it means that the content of the source file is the same as the content of the target file, and the source file and the target file can be determined to be the same file; when the data checksum corresponding to the source file is different from the data checksum corresponding to the target file, it means that the source file has been modified, and it can be determined that there is a difference between the source file and the target file, that is, the source file can be determined as a difference file.

[0143] Step 5: After the difference files are determined, all the difference files may be output and a difference file list Manifest corresponding to the difference files may be generated.

[0144] Step 6: Read the difference file list Manifest, and generate a copy file list based on the difference file list Manifest.

[0145] Step 7: Generate a distributed copy task for the copied file, and perform a copy operation on the difference file based on the distributed copy task.

[0146] Among them, after determining the difference file, in order to keep the data between the target storage medium and the source storage medium consistent, the difference file can be migrated. It should be noted that only the difference file with differences in the source storage medium needs to be migrated to the target storage medium, and there is no need to migrate other files, thereby effectively realizing the incremental copy operation of the quantity.

[0147] Step 8: At the end of the file copy operation, the data checksum during the copy process is calculated using the verification algorithm on the target storage medium, and then the target storage medium is determined to calculate the copied file using the stored verification algorithm to obtain the post-copy data checksum corresponding to the copied file in the target storage medium; based on the data checksum during the process and the post-copy data checksum, detect whether the file copy operation is normal.

[0148] Specifically, if the data checksum during the process is the same as the checksum after the copy, the file copy operation is determined to be normal; if the data checksum during the process is different from the checksum after the copy, the file copy operation is determined to be abnormal. In some examples, after determining that the file operation is abnormal, the difference file stored in the source file can be copied until all source files are completely copied.

[0149] It should be noted that the method in this embodiment can be applied to various application scenarios involving data difference comparison or data migration, such as: data difference detection between terminals, data migration between terminals, application scenarios of calling microservices, etc.

[0150] When detecting data differences between terminals, it is possible to compare whether there are differences in the data between the two terminals. At this time, one terminal includes a source storage medium for storing data, and the other terminal includes a target storage medium for storing data; then the above implementation method can be used to perform a difference detection operation on the data between the two terminals.

[0151] In the application scenario of calling microservices, relevant data for implementing a service is configured in the cloud. When a user needs to call the above service, a data call request can be sent to the cloud. After obtaining the data call request, the cloud can determine the data processing resources for implementing the above service and send the data processing resources to the user end. In the process of sending the data processing resources from the cloud to the user end, the above method can be used to implement data migration operations, which can ensure the integrity and reliability of data migration; when the user end obtains the data processing resources, the functional operations corresponding to the service can be implemented based on the data processing resources.

[0152] The data processing method between heterogeneous storage media provided in this embodiment can achieve the following effects:

[0153] During the file copy process, the data checksum during the file copy process is calculated, and then the data checksum of the received file on the target storage medium is calculated. Then, based on the data checksum during the transmission process and the data checksum corresponding to the target storage medium, the integrity check operation of the migrated file is performed. This effectively avoids the problem of being unable to compare data checksums between heterogeneous storage media and ensures the integrity of the migrated data.

[0154] In addition, by obtaining the data checksum of the source file before each data copy and directly saving it to the metadata of the target file, the data checksum stored in the target storage medium can be directly called, effectively avoiding repeated calculation operations of the data checksum and further reducing data calculation costs;

[0155] In addition, by calculating the original data checksum of the file in the source storage medium and obtaining the target data checksum stored in the target storage medium, and then based on the original data checksum and the target data checksum, the file difference detection operation can be effectively implemented, and then the existing difference files can be migrated based on the difference detection operation, thereby effectively realizing high-performance difference comparison operations.

[0156] Figure 10 A schematic diagram of a data processing device between heterogeneous storage media provided by an application embodiment of the present invention; Figure 10 As shown, this embodiment provides a data processing device between heterogeneous storage media, which is used to perform the above Figure 2 The data processing method between heterogeneous storage media shown in the figure, specifically, the data processing device may include:

[0157] A first acquisition module 11 is configured to acquire at least one file to be processed from a source storage medium;

[0158] A first calculation module 12 is configured to calculate at least one file to be processed and obtain first verification information corresponding to each of the at least one file to be processed;

[0159] A first acquisition module 11 is configured to acquire second verification information corresponding to at least one to-be-processed file stored in a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media;

[0160] The first processing module 13 is configured to determine whether at least one to-be-processed file is a difference file between a source storage medium and a target storage medium based on the first verification information and the second verification information.

[0161] In some instances, when the first calculation module 12 calculates at least one file to be processed and obtains first verification information corresponding to each of the at least one file to be processed, the first calculation module 12 is used to execute: obtaining a checksum algorithm for performing verification calculations on the at least one file to be processed; and calculating the at least one file to be processed using the checksum algorithm to obtain a first data checksum corresponding to each of the at least one file to be processed.

[0162] In some instances, before obtaining at least one to-be-processed file from the source storage medium, the first obtaining module 11, the first calculating module 12, and the first processing module 13 in this embodiment may be configured to perform the following steps:

[0163] A first acquisition module 11 is configured to acquire at least one to-be-processed file stored in a source storage medium at a historical moment;

[0164] A first calculation module 12 is configured to calculate historical verification information corresponding to at least one file to be processed;

[0165] The first processing module 13 is configured to transmit the historical verification information as the second verification information and at least one file to be processed to a target storage medium.

[0166] In some instances, when the first processing module 13 uses historical verification information as second verification information and transfers it together with at least one file to be processed to the target storage medium, the first processing module 13 is used to execute: using the second verification information as the header information of the corresponding file to be processed and transferring it together with the file to be processed to the target storage medium.

[0167] In some instances, when the first processing module 13 determines whether at least one file to be processed is a difference file between the source storage medium and the target storage medium based on the first verification information and the second verification information, the first processing module 13 is used to execute: when the first verification information and the second verification information are the same, it is determined that the file to be processed corresponding to the first verification information is not a difference file; when the first verification information and the second verification information are different, it is determined that the file to be processed corresponding to the first verification information is a difference file.

[0168] In some examples, after determining that the to-be-processed file corresponding to the first verification information is a difference file, the method in this embodiment may further include: displaying the first verification information, the second verification information, and the difference file.

[0169] In some examples, after determining that the to-be-processed file corresponding to the first verification information is a difference file, the first processing module 13 in this embodiment is configured to perform the following steps: transmitting the difference file to a target storage medium.

[0170] In some instances, when the first processing module 13 transfers the difference file to the target storage medium, the first processing module 13 is used to perform: generating a distributed transmission task corresponding to the difference file; determining a distributed node corresponding to the distributed transmission task; and using the distributed node to transfer the difference file to the target storage medium.

[0171] In some instances, when the first processing module 13 transfers the difference file to the target storage medium, the first processing module 13 in this embodiment is used to perform: calculating the in-transmission verification information corresponding to the difference file; obtaining the target verification information obtained by the target storage medium by calculating the received difference file; and detecting whether the transmission of the difference file is normal based on the in-transmission verification information and the target verification information.

[0172] In some instances, when the first processing module 13 calculates the in-transmission verification information corresponding to the difference file, the first processing module 13 can be used to execute: obtaining a verification algorithm for verifying data stored in the target storage medium; using the verification algorithm to calculate the difference file to obtain the in-transmission data checksum corresponding to the difference file.

[0173] In some instances, when the first processing module 13 detects whether the transmission of the difference file is normal based on the verification information during transmission and the target verification information, the first processing module 13 can be used to execute: when the verification information during transmission is the same as the target verification information, it is determined that the transmission of the difference file is normal; when the verification information during transmission is different from the target verification information, it is determined that the transmission of the difference file is abnormal.

[0174] In some instances, the method in this embodiment may further include: displaying a first prompt message in a preset interface when it is determined that the differential file transmission is normal; and displaying a second prompt message in the preset interface when it is determined that the differential file transmission is abnormal.

[0175] In some examples, before performing calculations on at least one file to be processed, the first acquisition module 11 and the first processing module 13 in this embodiment may be configured to perform the following steps:

[0176] A first acquisition module 11 is configured to acquire a stored file corresponding to at least one file to be processed and stored in a target storage medium;

[0177] The first processing module 13 is configured to screen at least one to-be-processed file based on the stored files to obtain a target processing file included in the at least one to-be-processed file, wherein the target processing file is identical to the corresponding stored file.

[0178] In some instances, when the first processing module 13 filters at least one file to be processed based on the stored files to obtain the target processing file included in the at least one file to be processed, the first processing module 13 is used to execute: obtaining the first file name and first file size corresponding to each of the at least one file to be processed, and the second file name and second file size corresponding to the stored file; filtering at least one file to be processed based on the first file name, first file size, second file name and second file size to obtain the target processing file included in the at least one file to be processed.

[0179] In some instances, when the first processing module 13 screens at least one file to be processed based on the first file name, the first file size, the second file name, and the second file size to obtain a target processing file included in the at least one file to be processed, the first processing module 13 is used to execute: when the first file name and the second file name are the same, and the first file size is different from the second file size, the file to be processed corresponding to the first file name is determined as a differential file; when the first file name and the second file name are the same, and the first file size is the same as the second file size, the file to be processed corresponding to the first file name is determined as the target processing file.

[0180] Figure 10 The device shown can perform Figures 1-6 、 Figure 8-Figure 9 For the method of the embodiment shown in FIG. 1 , reference may be made to the description of the part not described in detail in the embodiment. Figures 1-6 、 Figure 8-Figure 9 The implementation process and technical effects of this technical solution can be found in Figures 1-6 、 Figure 8-Figure 9 The description in the illustrated embodiment will not be repeated here.

[0181] In one possible design, Figure 10 The structure of the data processing device between the heterogeneous storage media shown can be implemented as an electronic device, which can be a mobile phone, tablet computer, server and other devices. Figure 11 As shown, the electronic device may include: a first processor 21 and a first memory 22. The first memory 22 is used to store the corresponding electronic device to perform the above Figures 1-6 、 Figure 8-Figure 9 In the embodiment shown, the first processor 21 is configured to execute the program stored in the first memory 22 .

[0182] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the first processor 21, the following steps can be implemented:

[0183] Obtaining at least one file to be processed from a source storage medium;

[0184] Calculating at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed;

[0185] Obtaining second verification information corresponding to at least one to-be-processed file stored in a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media;

[0186] Based on the first verification information and the second verification information, it is determined whether at least one to-be-processed file is a difference file between a source storage medium and a target storage medium.

[0187] Furthermore, the first processor 21 is also used to execute the aforementioned Figures 1-6 、 Figure 8-Figure 9 All or part of the steps in the illustrated embodiments.

[0188] The structure of the electronic device may further include a first communication interface 23 for the electronic device to communicate with other devices or a communication network.

[0189] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by electronic devices, which includes instructions for executing the above Figures 1-6 、 Figure 8-Figure 9 The program involved in the data processing method between heterogeneous storage media in the method embodiment shown.

[0190] Figure 12 This is a structural diagram of another data processing device between heterogeneous storage media provided by an embodiment of the present invention; This embodiment provides a data processing device between heterogeneous storage media, which is used to perform the above Figure 2 The data processing method between heterogeneous storage media shown in the figure, specifically, the data processing device may include:

[0191] A second acquisition module 31 is configured to acquire at least one to-be-processed file stored in a source storage medium;

[0192] A second calculation module 32 is configured to perform calculations on at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed;

[0193] The second processing module 33 is configured to transmit at least one to-be-processed file and corresponding verification information to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0194] In some instances, when the second processing module 33 transmits at least one file to be processed and the corresponding verification information to the target storage medium, the second processing module 33 is used to execute: using the verification information as the header information of the corresponding file to be processed and transmitting it to the target storage medium together with the file to be processed.

[0195] Figure 12 The device shown can perform Figure 7-Figure 9 For the method of the embodiment shown in FIG. 1 , reference may be made to the description of the part not described in detail in the embodiment. Figure 7-Figure 9 The implementation process and technical effects of this technical solution can be found in Figure 7-Figure 9 The description in the illustrated embodiment will not be repeated here.

[0196] In one possible design, Figure 12 The structure of the data processing device between the heterogeneous storage media shown can be implemented as an electronic device, which can be a mobile phone, tablet computer, server and other devices. Figure 13 As shown, the electronic device may include: a second processor 41 and a second memory 42. The second memory 42 is used to store the corresponding electronic device to perform the above Figure 7-Figure 9 In the embodiment shown, the second processor 41 is configured to execute the program stored in the second memory 42 .

[0197] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the second processor 41, the following steps can be implemented:

[0198] Obtaining at least one file to be processed stored in a source storage medium;

[0199] Calculating at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed;

[0200] At least one to-be-processed file and corresponding verification information are transmitted to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media.

[0201] Furthermore, the second processor 41 is also used to execute the aforementioned Figure 7-Figure 9 All or part of the steps in the illustrated embodiments.

[0202] The structure of the electronic device may further include a second communication interface 43 for the electronic device to communicate with other devices or a communication network.

[0203] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by electronic devices, which includes instructions for executing the above Figure 7-Figure 9 The program involved in the data processing method between heterogeneous storage media in the method embodiment shown.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0205] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0209] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0210] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0211] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0212] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for processing data between heterogeneous storage media, characterized in that: include: Obtain at least one to-be-processed file stored in a source storage medium at a historical moment; Calculating historical verification information corresponding to each of the at least one file to be processed; transmitting the historical verification information as second verification information and at least one to-be-processed file to a target storage medium; Acquire at least one file to be processed in the source storage medium; Performing calculations on the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed; Obtaining second verification information corresponding to each of the at least one to-be-processed files stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media, and a verification algorithm stored in the source storage medium is different from a verification algorithm stored in the target storage medium; Based on the first verification information and the second verification information, it is determined whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

2. The method according to claim 1, characterized in that Calculating the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed includes: Obtaining a checksum algorithm for performing checksum calculation on the at least one file to be processed; The checksum algorithm is used to calculate the at least one file to be processed to obtain a first data checksum corresponding to each of the at least one file to be processed.

3. The method according to claim 1, characterized in that Determining, based on the first verification information and the second verification information, whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium includes: When the first verification information is the same as the second verification information, determining that the to-be-processed file corresponding to the first verification information is not a difference file; When the first verification information is different from the second verification information, it is determined that the to-be-processed file corresponding to the first verification information is a difference file.

4. The method according to claim 3, characterized in that After determining that the to-be-processed file corresponding to the first verification information is a difference file, the method further includes: The difference file is transferred to the target storage medium.

5. The method according to claim 4, characterized in that Transmitting the difference file to the target storage medium includes: generating a distributed transmission task corresponding to the difference file; Determining a distributed node corresponding to the distributed transmission task; The difference file is transmitted to the target storage medium using the distributed nodes.

6. The method according to claim 4, characterized in that When transmitting the difference file to the target storage medium, the method further includes: Calculating in-transit checksum information corresponding to the difference file; Obtaining target verification information obtained by the target storage medium calculating the received difference file; Based on the verification information during transmission and the target verification information, it is detected whether the transmission of the difference file is normal.

7. The method according to claim 6, characterized in that Calculating in-transit checksum information corresponding to the difference file includes: Obtaining a verification algorithm for verifying data stored in the target storage medium; The difference file is calculated using the checksum algorithm to obtain a checksum of the in-transmission data corresponding to the difference file.

8. The method according to claim 6, characterized in that Detecting whether the transmission of the difference file is normal based on the verification information during transmission and the target verification information includes: When the verification information during the transmission is identical to the target verification information, it is determined that the difference file transmission is normal; When the verification information during the transmission is different from the target verification information, it is determined that the difference file transmission is abnormal.

9. The method according to any one of claims 1 to 8, characterized in that Before calculating the at least one file to be processed, the method further includes: Acquire a stored file corresponding to the at least one file to be processed stored in the target storage medium; The at least one to-be-processed file is screened based on the stored file to obtain a target processing file included in the at least one to-be-processed file, wherein the target processing file is the same as the corresponding stored file.

10. The method according to claim 9, characterized in that Screening the at least one to-be-processed file based on the stored file to obtain a target processing file included in the at least one to-be-processed file includes: Obtaining a first file name and a first file size corresponding to each of the at least one to-be-processed file, and a second file name and a second file size corresponding to the stored file; The at least one file to be processed is screened based on the first file name, the first file size, the second file name, and the second file size to obtain a target processing file included in the at least one file to be processed.

11. A method for processing data between heterogeneous storage media, characterized in that: include: Obtaining at least one file to be processed stored in a source storage medium; Performing calculations on the at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed; The at least one file to be processed and the corresponding verification information are transmitted to a target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media, and the verification algorithm stored in the source storage medium is different from the verification algorithm stored in the target storage medium.

12. A data processing device between heterogeneous storage media, characterized in that: include: A first acquisition module is used to acquire at least one to-be-processed file stored in a source storage medium at a historical moment; A first calculation module, configured to calculate historical verification information corresponding to each of the at least one file to be processed; A first processing module is configured to transmit the historical verification information as second verification information and at least one file to be processed to a target storage medium; The first acquisition module is further configured to acquire at least one file to be processed from the source storage medium; The first calculation module is further configured to calculate the at least one file to be processed to obtain first verification information corresponding to each of the at least one file to be processed; The first acquisition module is further configured to acquire second verification information corresponding to each of the at least one to-be-processed files stored in the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media, and a verification algorithm stored in the source storage medium is different from a verification algorithm stored in the target storage medium; The first processing module is further configured to determine, based on the first verification information and the second verification information, whether the at least one to-be-processed file is a difference file between the source storage medium and the target storage medium.

13. An electronic device, characterized in that: include: A memory, a processor; wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the data processing method between heterogeneous storage media as described in any one of claims 1 to 10 is implemented.

14. A data processing device between heterogeneous storage media, characterized in that: include: A second acquisition module is used to acquire at least one to-be-processed file stored in a source storage medium; A second calculation module is used to calculate the at least one file to be processed to obtain verification information corresponding to each of the at least one file to be processed; The second processing module is used to transmit the at least one file to be processed and the corresponding verification information to the target storage medium, wherein the source storage medium and the target storage medium are heterogeneous storage media, and the verification algorithm stored in the source storage medium is different from the verification algorithm stored in the target storage medium.

15. An electronic device, characterized in that: include: A memory, a processor; wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the data processing method between heterogeneous storage media as claimed in claim 11 is implemented.

Images