Data consistency analysis method, device and equipment and readable storage medium

Abstract

The invention discloses a data consistency analysis method, device and equipment and a readable storage medium, and is applied to the technical field of data storage, and the method comprises the following steps: determining a prefix, a suffix and an offset position corresponding to error report data according to the error report data of a data consistency tool and a configuration file; wherein the error report data comprises data related to the error data, and the offset position is offset in the storage object; determining a target object name based on the prefix, the suffix and the offset position, and querying data position information of the error report data based on the target object name; obtaining read-out data based on the data position information, and comparing the read-out data with the error report data to obtain a comparison result; and performing analysis based on the comparison result and the log information corresponding to the error report data, and determining an error report analysis report.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data consistency analysis method, device, equipment and readable storage medium. Background Art

[0002] Data consistency is one of the performance capabilities of storage and is the red line of all storage. During the R&D process, data consistency problems are prone to occur due to the update and optimization of business IO processes, and the analysis of consistency problems is cumbersome and complicated. Currently, when analyzing data consistency problems, it is usually based on manual analysis, which makes the accuracy and efficiency of data consistency analysis low.

[0003] It can be seen that how to improve the accuracy of data consistency analysis is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a data consistency analysis method, device, equipment and readable storage medium, which solve the technical problem of low accuracy of data consistency analysis in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a data consistency analysis method, comprising:

[0006] Determine the prefix, suffix and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is an offset within the storage object;

[0007] Determine a target object name based on the prefix, the suffix and the offset position, and query data location information of the error data based on the target object name;

[0008] Acquire read data based on the data location information, and compare the read data with the error data to obtain a comparison result;

[0009] An analysis is performed based on the comparison result and the log information corresponding to the error data to determine an error analysis report.

[0010] On the one hand, the analysis based on the comparison result and the log information corresponding to the error data to determine the error analysis report includes:

[0011] Determining whether the read data and the written data in the error reporting data are consistent based on the comparison result;

[0012] When it is determined that the read data and the written data are inconsistent, performing analysis based on the written data and the error data to determine a timeline for fault injection recovery;

[0013] The error analysis report is generated based on the timeline of the fault injection recovery.

[0014] On the one hand, before analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report, it also includes:

[0015] Query the corresponding object storage service list according to the target object name;

[0016] Collect process record logs and object storage service logs according to the object storage service list; wherein the log information includes the process record logs and the object storage service logs.

[0017] On the one hand, determining the prefix, suffix and offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool includes:

[0018] Determine the prefix, the suffix and the offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool by using a data consistency analysis tool, wherein the data consistency analysis tool is a tool running on a storage node, and includes:

[0019] Acquire the error data corresponding to the data consistency tool and the configuration file corresponding to the data consistency tool;

[0020] Determine the client and the drive letter corresponding to the error data according to the error data and the configuration file;

[0021] Determine the volume name corresponding to the drive letter according to the client and the drive letter corresponding to the error data;

[0022] Determining, in the storage cluster, a prefix of the target query object corresponding to the target object name according to the volume name;

[0023] The offset position of the error data on the disk is determined according to the error reporting data, and the suffix of the target query object and the offset position are determined according to the offset position on the disk, where the offset position is the offset inside the target query object.

[0024] On the one hand, after analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report, it also includes:

[0025] Based on the error analysis report, the causes of data inconsistency are analyzed from multiple aspects to determine.

[0026] On the one hand, the cause of the data inconsistency includes at least one of data redundancy, concurrency control error, hardware failure, software failure or operation error.

[0027] On the one hand, the acquiring of the read data based on the data location information, and comparing the read data with the error data to obtain a comparison result include:

[0028] Determine an object storage service list according to the target object name;

[0029] Acquire data operation information and data location information of the object corresponding to the target object name according to the object storage service list;

[0030] The written data is read from the disk according to the data operation information and the data position information to obtain the read data.

[0031] The embodiment of the present invention further provides a data consistency analysis device, which is applied to a storage node and includes:

[0032] A first information determination module, configured to determine a prefix, a suffix and an offset position corresponding to the error data according to the error data and a configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is an offset within the storage object;

[0033] A second information determination module, configured to determine a target object name based on the prefix, the suffix and the offset position, and query data location information of the error data based on the target object name;

[0034] A comparison result determination module, used for acquiring read data based on the data location information, comparing the read data with the error data, and obtaining a comparison result;

[0035] The analysis module is used to analyze the log information corresponding to the comparison result and the error data to determine the error analysis report.

[0036] The embodiment of the present invention further provides a data consistency analysis device, including:

[0037] Memory for storing computer programs;

[0038] A processor is used to execute the computer program to implement the steps of the above-mentioned data consistency analysis method.

[0039] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data consistency analysis method are implemented.

[0040] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned data consistency analysis method when executed by a processor.

[0041] To solve the above technical problems, an embodiment of the present invention provides a data consistency analysis method, which may include: determining the prefix, suffix and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is the offset within the storage object; determining the target object name based on the prefix, suffix and offset position, and querying the data location information of the error data based on the target object name; obtaining the read data based on the data location information, comparing the read data with the error data, and obtaining a comparison result; analyzing based on the comparison result and the log information corresponding to the error data, and determining an error analysis report.

[0042] It can be seen from the above technical scheme that the beneficial effects of the present invention are: compared with the current need for manual data inconsistency analysis, the present invention realizes automatic collection of relevant data by integrating the data consistency problem analysis process, and generates analysis reports by parsing error data and log information corresponding to the error data, thereby realizing automated analysis, reducing human intervention, and avoiding time waste due to rework due to errors after the manual operation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a data consistency analysis method provided by an embodiment of the present invention;

[0045] Figure 2 An example diagram of a data consistency analysis process provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of a data consistency analysis device provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of a data consistency analysis device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0050] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] Next, a data consistency analysis method provided by an embodiment of the present invention is introduced in detail. Figure 1 A data consistency analysis method provided by an embodiment of the present invention may include:

[0052] S101, determining a prefix, a suffix and an offset position corresponding to the error data according to the error data and a configuration file of a data consistency tool; wherein the error data includes data related to the error data, and the offset position is an offset within a storage object.

[0053] The execution subject of this embodiment is an electronic device. The electronic device in this embodiment can be a computer, a mobile phone, etc. The data consistency tool in this embodiment is vdbench, which is an open source tool that provides a data verification function. The tool itself records the source data and feature data written, and then reads the written data for comparison. The error reporting data of the data consistency tool in this embodiment refers to the data that reports an error when reading. The configuration file in this embodiment includes which clients and volumes vdbench runs on, which disks the client writes, and the way to write to the disk. The use of prefixes in this embodiment is mainly to avoid naming conflicts, improve readability, simplify maintenance work, and avoid naming conflicts. For example, when migrating or merging data between multiple databases, the duplication of table names or column names can be avoided by using different prefixes. In addition, the prefix can also help developers quickly understand the purpose and module to which the table or column belongs. In practical applications, the selection of prefixes should follow certain naming specifications, such as using lowercase letters, numbers and underscores, and avoiding the use of special characters, spaces or illegal characters. At the same time, the length of the prefix should be moderate and should not be too long to facilitate program coding and maintenance. This embodiment does not limit the specific prefix. For example, the prefix in this embodiment can be dbs_data.xxxxxxxx, usermgmt_data.xxxxxxxx. The suffix in this embodiment generally refers to a part of the database table name or field name, which is used to indicate the type, purpose or other specific information of the data. For example, the suffix in this embodiment can be .txt, .doc / .docx, .xls / .xlsx, etc. The offset position in this embodiment refers to the byte distance between the starting address of the data on the storage medium (such as a hard disk, a solid-state drive or an optical disk) and a reference point (usually the starting position of the file or the starting position of the partition). This offset is used to locate and access a specific data block or file. On a hard disk or solid-state drive, data is divided into a series of sectors, each of which usually contains 512 bytes or 4096 bytes (for modern hard disks). When you request to read a specific file, the file system calculates the starting sector of the file and the offset within the sector to be read, and then sends this information to the hard disk controller to correctly read the data. This embodiment does not limit the specific offset position. For example, the offset position in this embodiment may be 000000; or the offset position in this embodiment may also be 000001; or the offset position in this embodiment may also be 01000. The offset in the storage object in this embodiment means that if you have a text file, the file content is "Hello, world!". If you want to read the 7th character (that is, "w") in the file, you need to specify an offset of 6 (because counting usually starts from 0). In this way, the file system will skip the first 6 characters from the beginning of the file and then start reading from the 7th character.

[0054] It should be further explained that the above-mentioned determination of the prefix, suffix and offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool may include:

[0055] The data consistency analysis tool is used to determine the prefix, suffix, and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool. The data consistency analysis tool is a tool running on the storage node, including:

[0056] S1011, obtaining error data corresponding to the data consistency tool and a configuration file corresponding to the data consistency tool;

[0057] S1012, determining the client and the drive letter corresponding to the error data according to the error data and the configuration file;

[0058] S1013, determining the volume name corresponding to the drive letter according to the client and drive letter corresponding to the error data;

[0059] S1014, determining, in the storage cluster according to the volume name, a prefix of the target query object corresponding to the target object name;

[0060] S1015, determining the offset position of the error data on the disk according to the error data, and determining the suffix of the target query object and the offset position according to the offset position on the disk, wherein the offset position is the offset inside the target query object.

[0061] This embodiment provides a specific method for determining the prefix, suffix, and offset position, and how to determine data location information based on the prefix, suffix, and offset position, thereby improving the accuracy of determining the prefix, suffix, and offset position. The offset within the target query object in this embodiment corresponds to the offset position described above.

[0062] S102, determining a target object name based on the prefix, the suffix and the offset position, and querying data location information of the error data based on the target object name.

[0063] The target object name in this embodiment is determined by a prefix, an offset position, and a suffix. This embodiment does not limit the specific target object name. For example, the target object name in this embodiment may be dbs_data.xxxxxxxx.000000a; or the target object name in this embodiment may also be dbs_data.xxxxxxxx.010000a. In this embodiment, the specific data location information of the error data may be queried based on the target object name. For example, in this embodiment, the data location information of the object may be queried as an OSS (Object Storage Service) service list. Or in this embodiment, the data location information may be queried as a local file system: this is the most traditional and most common data storage method, suitable for devices such as personal computers and servers. Data is stored in the form of files on a hard disk or other storage medium, and can be managed and accessed through an operating system. Or the data location information in this embodiment may also be a network attached storage (NAS): NAS is a device specifically used for file storage, which provides file-level data storage services through a network connection. NAS devices usually have their own file systems and support multiple users to access and share files at the same time. Or the data location information in this embodiment may also be a storage area network (SAN): SAN is a high-speed network used to connect servers and storage devices. It provides block-level data storage services, allowing servers to directly access the physical disks of storage devices. SAN is commonly used in enterprise environments that require high performance and high availability. Or the data location information in this embodiment can also be a database management system (DBMS): For structured data, such as table data in a relational database, data location information is usually stored in the database. The database management system provides powerful data query, update and transaction processing capabilities. Or the data location information in this embodiment can also be a distributed file system: such as HDFS (Hadoop Distributed File System), which is a distributed file system for big data processing. The data is divided into multiple blocks and distributed on multiple nodes to achieve high availability and scalability. Or the data location information in this embodiment can also be object storage: In addition to OSS, there are other object storage services, such as Amazon S3, Google Cloud Storage, etc. Object storage is suitable for unstructured data, such as pictures, videos, backup files, etc.

[0064] S103, acquiring read data based on the data location information, and comparing the read data with the error data to obtain a comparison result.

[0065] The reason why this embodiment performs a comparison again is to ensure the accuracy of the error information and determine the specific cause of the error. This embodiment does not limit the specific comparison results. For example, the comparison result in this embodiment can be that the read data and the actual written data are consistent; or the comparison result in this embodiment can also be that the read data and the actual written data are inconsistent. The comparison result in this embodiment can determine the specific inconsistency information between the actual written data and the read data. The data consistency tool writes 100 on the client, and there is no problem in the io path. The io path is modified and changed to 102, which is considered inconsistent. The client writes it. For example, when saving money, take 100 to store, and after processing it becomes 99. The error data includes the actual 100 you want to store, and the read data is 102. When you actually read it, you find that the read data is 99, indicating that it was written incorrectly.

[0066] It should be further explained that the above-mentioned acquisition of read data based on data location information, comparison of the read data with the error data, and obtaining the comparison result may include: determining the object storage service list according to the target object name; acquiring the data operation information and data location information of the object corresponding to the target object name according to the object storage service list; reading the written data from the disk according to the data operation information and the data location information to obtain the read data. The data operation information in this embodiment refers to the modification of the data or the modification of the data path during the whole process, so that the read data can be accurately obtained.

[0067] S104: Analyze the log information corresponding to the comparison result and the error data to determine an error analysis report.

[0068] This embodiment does not limit specific log information. For example, the log information in this embodiment may be an OSS log, which records the IO process of the object, the time of reading and writing, and the process experienced; or the log information in this embodiment may also include a CCS log, which refers to the debug log in Code Composer Studio (CCS). This embodiment analyzes the comparison results and the error data and its corresponding log information to determine the specific cause of the data inconsistency; or this embodiment may also determine the specific cause of the error data.

[0069] It should be further explained that in order to improve the comprehensiveness of the log-based analysis, the above analysis based on the comparison results and the log information corresponding to the error data, before determining the error analysis report, can also include: querying the corresponding object storage service list according to the target object name; collecting the process record log and the object storage service log according to the object storage service list; wherein the log information includes the process record log and the object storage service log. The process record log in this embodiment is the CCS log, and the object storage service log in this embodiment is the OSS log. This embodiment performs analysis based on both logs at the same time, which can improve the comprehensiveness of the content of the error analysis report generated based on the log.

[0070] It should be further explained that, in order to improve the accuracy of the error analysis report, the above-mentioned analysis based on the comparison result and the log information corresponding to the error data to determine the error analysis report may include: determining whether the read data and the written data in the error data are consistent based on the comparison result; when it is determined that the read data and the written data are inconsistent, parsing based on the written data and the error data to determine the timeline of fault injection recovery; generating an error analysis report based on the timeline of fault injection recovery. The timeline of fault injection recovery in this embodiment can also be just the fault timeline. In this embodiment, when the read data and the written data are inconsistent, or the timeline of fault injection recovery is determined, so as to determine the specific error analysis report according to the timeline of fault injection recovery. It can be understood that the timeline of fault injection recovery is a detailed process that describes the time required for the system to recover to a normal state after a fault (or error) is deliberately introduced into the system. By analyzing the timeline of fault injection recovery, the cause of the inconsistency between the read data and the written data can be accurately determined, and other data in the time period can be discovered in time so as to be corrected in time.

[0071] It should be further explained that after analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report, the following may be further included: analyzing and determining the cause of the data inconsistency from multiple aspects based on the error analysis report. The multiple aspects in this embodiment may include the operating system environment, the number of nodes used in the entire process, the bandwidth of the network card, the number of network cards, the central processing unit information, the storage pool information, the erasure code strategy, and what operations were performed on the object of the error, thereby improving the comprehensiveness of the error analysis.

[0072] It should be further explained that the above-mentioned reasons for data inconsistency include at least one of data redundancy, concurrency control error, hardware failure, software failure or operation error. The specific reasons for the inconsistency between read data and written data are not limited in this embodiment. For example, the error reporting reason in this embodiment may be the organization form of the storage pool; or it may be that the data organization is wrong; or it may be a timing problem, resulting in the old data replacing the new data.

[0073] A data consistency analysis method provided by an embodiment of the present invention may include: S101, determining the prefix, suffix and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is the offset within the storage object; S102, determining the target object name based on the prefix, suffix and offset position, and querying the data location information of the error data based on the target object name; S103, obtaining the read data based on the data location information, comparing the read data with the error data, and obtaining the comparison result; S104, analyzing based on the comparison result and the log information corresponding to the error data, and determining the error analysis report. Compared with the current need for manual data inconsistency analysis, the present invention realizes automatic collection of relevant data by integrating the data consistency problem analysis process, and generates an analysis report by parsing the error data and the log information corresponding to the error data, thereby realizing automated analysis, reducing human intervention, avoiding time waste caused by rework due to errors during manual operation, and improving the accuracy and efficiency of data consistency analysis.

[0074] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 An example flow chart of a data consistency analysis provided by an embodiment of the present invention may specifically include:

[0075] 1. Configure the data consistency analysis tool on the storage node.

[0076] 2. Input the error data and configuration files into the data consistency analysis tool.

[0077] 3. Perform analysis based on data consistency analysis tools.

[0078] The process of data consistency analysis may include:

[0079] S201, determining the client and the drive letter corresponding to the error data according to the error data and the configuration file.

[0080] S202: Determine the volume name corresponding to the drive letter according to the client and the drive letter.

[0081] S203: Determine a prefix of the query object name according to the volume name in the storage cluster.

[0082] The prefix in this embodiment is equivalent to the name.

[0083] S204, determining the suffix of the query object name and the offset within the query object name according to the offset position of the error data on the disk, and determining the complete object name based on the prefix, suffix and the offset within the query object name.

[0084] In this embodiment, the offset position on the disk can be directly determined based on the error data. For example, the 9-megabyte position corresponding to the error data, the offset position of an object is the offset of the object is 4, 9 / 4 is rounded to 2, 2 is the offset of the object, the suffix of the object name, and 1 is the offset.

[0085] S205: Query the corresponding OSS service list according to the object name.

[0086] S206, obtaining metadata information and data location information of the object according to the OSS service list, and reading the written data from the disk according to the metadata information and data location information.

[0087] The metadata information in this embodiment refers to the size, modification time and data operation information of the object. The data location information in this embodiment refers to the location of the generated data in the corresponding OSS service. The data read and written from the disk in this embodiment refers to the original data that is converted to C when it is written to A.

[0088] S207, collecting CSS logs and OSS logs according to the OSS service list.

[0089] S208: Analyze and compare the read data and error data collected from the disk to analyze whether the written data and the read data are consistent.

[0090] The error data in this embodiment may include the actual written data and the read data when the error is reported. The first step is to compare the read data collected from the disk with the read data when the error is reported. If they are the same, it means there is a problem with the data consistency. If they are different, it is necessary to compare with the actual written data in the error data. If it is consistent with the actual written data, it means that there is no problem with the data itself. If it is inconsistent with the actual written data, it means that there is a problem with the data.

[0091] S208, when there is inconsistency, parse and output the timeline of reading, writing and fault injection recovery based on the CSS log and the OSS log.

[0092] This embodiment can also collect system logs and related logs of the object where the error data is located. This embodiment finds out whether the data read on the analysis disk is consistent with the vdbench error data. If not, the cause of the data error is analyzed: version error, data generation process error, data LBA (Logical Block Addressing) error or data is not verification data.

[0093] S210, based on step S208 and step S209, and the environmental information analysis error cause, generate a data consistency analysis report.

[0094] The data consistency analysis report in this embodiment corresponds to the error analysis report above. This embodiment can automatically parse the test environment and store cluster environment information throughout the process. The data consistency analysis report may include environment information, data information, timeline information, etc.

[0095] The present invention is a design method based on a distributed block storage data analysis tool. It realizes automatic collection of relevant data by integrating the data consistency problem analysis process (collecting the offset position of error data on the volume, finding the corresponding volume according to the drive letter mounted by the client, obtaining the name of the object where the error data is located according to the existing data, and obtaining the object information, object data distribution, and object data on each copy according to the object name). By parsing and collating error reporting data and collected data, analyzing logs to comb the timeline of reading, writing, and fault recovery, and generating analysis reports, automatic analysis is realized, human intervention is reduced, and time waste caused by rework due to mistakes after manual operation is avoided. Improve R&D efficiency, enhance product competitiveness, improve user experience, and increase product competitiveness. The key point of the present invention is: based on the design method of the distributed block storage data analysis tool, the integration of the data consistency problem analysis process realizes automatic collection of relevant data, parsing and collating error reporting data and collected data, parsing and collating storage logs to comb the timeline of reading, writing, and fault recovery, and finally generating an analysis report to realize automatic analysis.

[0096] The following is an introduction to a data consistency analysis device provided in an embodiment of the present invention. The data consistency analysis device described below and the data consistency analysis method described above can be referenced to each other.

[0097] Figure 3 A schematic diagram of a data consistency analysis device provided by an embodiment of the present invention may include:

[0098] The first information determination module 100 is used to determine the prefix, suffix and offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is the offset inside the storage object;

[0099] A second information determination module 200, configured to determine a target object name based on the prefix, the suffix and the offset position, and query data location information of error reporting data based on the target object name;

[0100] A comparison result determination module 300, configured to obtain read data based on the data location information, and compare the read data with the error reporting data to obtain a comparison result;

[0101] The analysis module 400 is used to analyze the log information corresponding to the comparison result and the error data to determine an error analysis report.

[0102] Further, based on the above embodiment, the analysis module 400 may include:

[0103] A judgment unit, configured to determine whether the read data and the written data in the error reporting data are consistent based on the comparison result;

[0104] A timeline determination unit, configured to determine a timeline for fault injection recovery by performing analysis based on the written data and the error data when it is determined that the read data and the written data are inconsistent;

[0105] The error analysis report determining unit is used to generate the error analysis report based on the timeline of the fault injection recovery.

[0106] Further, based on any of the above embodiments, the data consistency analysis device may further include:

[0107] An object storage service list determination module is used to query a corresponding object storage service list according to the target object name;

[0108] A log information determination module is used to collect process record logs and object storage service logs according to the object storage service list; wherein the log information includes the process record logs and the object storage service logs.

[0109] Further, based on any of the above embodiments, the first information determining module 100 may include:

[0110] The first information determination unit is used to determine the prefix, the suffix and the offset position corresponding to the error data according to the error data and the configuration file of the data consistency analysis tool by using a data consistency analysis tool, wherein the data consistency analysis tool is a tool running on a storage node, and includes:

[0111] A configuration file determining subunit, used for acquiring the error data corresponding to the data consistency tool and the configuration file corresponding to the data consistency tool;

[0112] a drive letter determination subunit, configured to determine a client and a drive letter corresponding to the error data according to the error data and the configuration file;

[0113] A volume name determination subunit, configured to determine a volume name corresponding to the drive letter according to the client and the drive letter corresponding to the error data;

[0114] A prefix determination subunit, used to determine, in the storage cluster, a prefix of a target query object corresponding to the target object name according to the volume name;

[0115] The offset position determination sub-tern is used to determine the offset position of the error data on the disk according to the error data, determine the suffix of the target query object and the offset position according to the offset position on the disk, and the offset position is the offset inside the target query object.

[0116] Further, based on any of the above embodiments, the data consistency analysis device may further include:

[0117] The inconsistency cause determination module is used to analyze and determine the causes of data inconsistency from multiple aspects based on the error analysis report.

[0118] Further, based on any of the above embodiments, the cause of the data inconsistency includes at least one of data redundancy, concurrency control error, hardware failure, software failure or operation error.

[0119] Further, based on any of the above embodiments, the comparison result determination module 300 may include:

[0120] A list determining unit, configured to determine an object storage service list according to the target object name;

[0121] An information determining unit, configured to obtain data operation information and data location information of an object corresponding to the target object name according to the object storage service list;

[0122] The read data writing unit is used to read the written data from the disk according to the data operation information and the data position information to obtain the read data.

[0123] It should be noted that the order of the modules and units in the above data consistency analysis device can be changed without affecting the logic.

[0124] Figure 3 The description of the features in the corresponding embodiments can be found in Figure 3 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0125] The data consistency analysis device provided by the embodiment of the present invention may include: a first information determination module 100, which is used to determine the prefix, suffix and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is the offset within the storage object; a second information determination module 200, which is used to determine the target object name based on the prefix, the suffix and the offset position, and query the data location information of the error data based on the target object name; a comparison result determination module 300, which is used to obtain the read data based on the data location information, compare the read data with the error data, and obtain the comparison result; an analysis module 400, which is used to analyze based on the comparison result and the log information corresponding to the error data, and determine the error analysis report. Compared with the current need for manual data inconsistency analysis, the present invention realizes automatic collection of relevant data by integrating the data consistency problem analysis process, and generates an analysis report by parsing the error data and the log information corresponding to the error data, thereby realizing automatic analysis, reducing human intervention, avoiding time waste caused by rework due to mistakes during manual operation, and improving the accuracy of data consistency analysis.

[0126] A data consistency analysis device provided by an embodiment of the present invention is introduced below. The data consistency analysis device described below and the data consistency analysis method described above can refer to each other.

[0127] Figure 4 A schematic diagram of the structure of a data consistency analysis device provided by an embodiment of the present invention, such as Figure 4 As shown, the data consistency analysis device includes: a memory 60 for storing a computer program;

[0128] The processor 61 is used to implement the steps of the data consistency analysis method in the above embodiment when executing a computer program.

[0129] The data consistency analysis device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0130] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0131] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the data consistency analysis method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data in the data consistency analysis process, etc.

[0132] In some embodiments, the data consistency analysis device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0133] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the data consistency analysis device, and may include more or fewer components than those shown in the figure.

[0134] It is understandable that if the data consistency analysis method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program code.

[0135] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data consistency analysis method are implemented.

[0136] Based on this, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data consistency analysis method when executed by a processor.

[0137] The above is a detailed introduction to a data consistency analysis method, device, equipment and readable storage medium provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0138] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0139] The above is a detailed introduction to a data consistency analysis method, device, equipment and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A data consistency analysis method, characterized in that: Applicable to storage nodes, including: Determine the prefix, suffix and offset position corresponding to the error data according to the error data and configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is an offset within the storage object; Determine a target object name based on the prefix, the suffix and the offset position, and query data location information of the error data based on the target object name; Acquire read data based on the data location information, and compare the read data with the error data to obtain a comparison result; An analysis is performed based on the comparison result and the log information corresponding to the error data to determine an error analysis report.

2. The data consistency analysis method according to claim 1, characterized in that: The analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report includes: Determining whether the read data and the written data in the error reporting data are consistent based on the comparison result; When it is determined that the read data and the written data are inconsistent, performing analysis based on the written data and the error data to determine a timeline for fault injection recovery; The error analysis report is generated based on the timeline of the fault injection recovery.

3. The data consistency analysis method according to claim 1, characterized in that: Before analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report, the method further includes: Query the corresponding object storage service list according to the target object name; Collect process record logs and object storage service logs according to the object storage service list; wherein the log information includes the process record logs and the object storage service logs.

4. The data consistency analysis method according to claim 1, characterized in that: The determining of the prefix, suffix and offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool includes: Determine the prefix, the suffix and the offset position corresponding to the error data according to the error data and the configuration file of the data consistency tool by using a data consistency analysis tool, wherein the data consistency analysis tool is a tool running on a storage node, and includes: Acquire the error data corresponding to the data consistency tool and the configuration file corresponding to the data consistency tool; Determine the client and the drive letter corresponding to the error data according to the error data and the configuration file; Determine the volume name corresponding to the drive letter according to the client and the drive letter corresponding to the error data; Determining, in the storage cluster, a prefix of the target query object corresponding to the target object name according to the volume name; The offset position of the error data on the disk is determined according to the error reporting data, and the suffix of the target query object and the offset position are determined according to the offset position on the disk, where the offset position is the offset inside the target query object.

5. The data consistency analysis method according to claim 1, characterized in that: After analyzing the log information corresponding to the comparison result and the error data to determine the error analysis report, the method further includes: Based on the error analysis report, the causes of data inconsistency are analyzed from multiple aspects to determine.

6. The data consistency analysis method according to claim 5, characterized in that: The cause of the data inconsistency includes at least one of data redundancy, concurrency control error, hardware failure, software failure or operation error.

7. The data consistency analysis method according to any one of claims 1 to 6, characterized in that: The acquiring the read data based on the data location information, and comparing the read data with the error data to obtain a comparison result, includes: Determine an object storage service list according to the target object name; Acquire data operation information and data location information of the object corresponding to the target object name according to the object storage service list; The written data is read from the disk according to the data operation information and the data position information to obtain the read data.

8. A data consistency analysis device, characterized in that: Applicable to storage nodes, including: A first information determination module, configured to determine a prefix, a suffix and an offset position corresponding to the error data according to the error data and a configuration file of the data consistency tool; wherein the error data includes data related to the error data, and the offset position is an offset within the storage object; A second information determination module, configured to determine a target object name based on the prefix, the suffix and the offset position, and query data location information of the error data based on the target object name; A comparison result determination module, used for acquiring read data based on the data location information, comparing the read data with the error data, and obtaining a comparison result; The analysis module is used to analyze the log information corresponding to the comparison result and the error data to determine the error analysis report.

9. A data consistency analysis device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data consistency analysis method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data consistency analysis method according to any one of claims 1 to 7 are implemented.

Images