A data checking method, device and equipment of a distributed system and a medium

Abstract

The application provides a data checking method and device of a distributed system, equipment and a medium, and relates to the technical field of data processing. The method sets the same data structure for each record file generated under the same business job, determines the first field value of the checking field in the target file and the second field value of the checking field in each to-be-processed file according to the data structure information of the target file after determining the to-be-checked file according to the file identifier of the target file in the data checking stage. Then, the checking record meeting the checking condition is generated according to the comparison result of the first field value and each second field value. The above checking process does not require manual participation, has the advantages of high checking efficiency and avoidance of human errors.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data verification method, apparatus, device, and medium for a distributed system. Background Technology

[0002] A distributed system is a loosely coupled system consisting of multiple processors interconnected through communication lines. Compared with a single-machine system, it has many advantages such as resource sharing, high reliability, and fast communication and is therefore widely used.

[0003] In a distributed system, different jobs and job steps may be executed on different node servers. The job log files generated during the job process are split into multiple parts and stored in different storage spaces. Business processing often requires checking and analyzing these job log files. For example, when counting the problem records generated in a certain job process, it is necessary to search all job log files generated by that business process and check whether these job log files contain the target field that represents the problem record.

[0004] Currently, data verification for distributed systems is primarily performed manually. Specifically, verifiers need to locate the corresponding verification file based on an index indicating the verification scope and then manually verify that file. This verification method suffers from problems such as long verification cycles and significant errors. Summary of the Invention

[0005] This application provides a data verification method, apparatus, device, and medium for a distributed system, which addresses the problems of long verification cycles and large errors in current verification methods.

[0006] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a data verification method for a distributed system, the method comprising:

[0008] In response to a check instruction, file information and check information of a target file are received; wherein, the target file is a record file generated when any business operation is executed in batch processing; the file information includes a file identifier and data structure information representing the data structure of the target file, and the check information includes a check field and check conditions associated with the check field, wherein the check field is at least one field in the data structure corresponding to the target file;

[0009] Based on the file identifier, the storage space to be checked is determined, and each file to be checked corresponding to the target file is determined from the storage space; wherein, the file to be checked and the target file are record files generated under the same business operation, and the data structure of each record file generated under the same business operation is the same;

[0010] The first field value of the inspection field in the target file and the second field value of the inspection field in each of the files to be inspected are determined based on the data structure information; an inspection record that meets the inspection conditions is generated based on the comparison results between the first field value and each of the second field values.

[0011] In some possible embodiments, the file identifier is a storage index of the target file in the storage space, and the file identifier includes a first identifier representing the target file, a second identifier representing the business job corresponding to the target file, and a third identifier representing the batch processing number of the business job.

[0012] In some possible embodiments, the distributed system includes multiple storage partitions, each corresponding to a unique data platform; the data platform is used to store the generated record files after batch processing and executing business jobs in the corresponding storage partitions;

[0013] The file information also includes the scope of inspection; determining the storage space to be inspected based on the file identifier includes:

[0014] The storage index of the target file is determined based on the file identifier;

[0015] The storage space to be checked is determined based on the check scope and the storage index.

[0016] In some possible embodiments, the inspection scope includes a first inspection scope characterized by inspections performed on a data platform basis, and a second inspection scope characterized by inspections performed on a storage partition basis; determining the storage space to be inspected based on the inspection scope and the storage index includes:

[0017] If the scope of inspection is the first scope of inspection, then the data platform that generates the target file is determined according to the storage index, and all storage partitions under the data platform are used as the storage space;

[0018] If the scope of inspection is the second scope of inspection, then the target storage partition for storing the target file is determined according to the storage index, and the target storage partition is used as the storage space.

[0019] In some possible embodiments, the verification conditions include duplicate checks and range checks; generating a verification record that satisfies the verification conditions based on the comparison results of the first field value and each of the second field values ​​includes:

[0020] If the verification condition is a duplicate check, then the first target field value is determined from each of the second field values ​​based on the comparison results; wherein the first target field value is the same as the first field value;

[0021] If the check condition is a range check, then the second target field value is determined from each of the second field values ​​based on the comparison result; wherein the second target field value is within a specified numerical range, and the specified numerical range is determined based on the first field value;

[0022] The inspection record is generated based on the number of target field values; wherein the target field value is either the first target field value or the second target field value.

[0023] In some possible embodiments, the data structure information is used to record the fields contained in the data structure corresponding to the target file, as well as the offset and length of each field.

[0024] Secondly, embodiments of this application provide a data verification device for a distributed system, the device comprising:

[0025] The information acquisition module is configured to respond to a check instruction by receiving file information and check information of a target file; wherein, the target file is a record file generated when any business operation is executed in batch processing; the file information includes a file identifier and data structure information representing the data structure of the target file, and the check information includes a check field and check conditions associated with the check field, wherein the check field is at least one field in the data structure corresponding to the target file;

[0026] The file acquisition module is configured to determine the storage space to be checked based on the file identifier, and to determine each file to be checked corresponding to the target file from the storage space; wherein the file to be checked and the target file are record files generated under the same business operation, and the data structure of each record file generated under the same business operation is the same;

[0027] The inspection record module is configured to perform the following actions: determine the first field value of the inspection field in the target file and the second field value of the inspection field in each of the files to be inspected based on the data structure information; and generate an inspection record that meets the inspection conditions based on the comparison results between the first field value and each of the second field values.

[0028] In some possible embodiments, the file identifier is a storage index of the target file in the storage space, and the file identifier includes a first identifier representing the target file, a second identifier representing the business job corresponding to the target file, and a third identifier representing the batch processing number of the business job.

[0029] In some possible embodiments, the distributed system includes multiple storage partitions, each corresponding to a unique data platform; the data platform is used to store the generated record files after batch processing and executing business jobs in the corresponding storage partitions;

[0030] The file information also includes the scope of inspection; the file acquisition module is configured to determine the storage space to be inspected based on the file identifier.

[0031] The storage index of the target file is determined based on the file identifier;

[0032] The storage space to be checked is determined based on the check scope and the storage index.

[0033] In some possible embodiments, the inspection scope includes a first inspection scope characterized by inspections performed on a data platform basis, and a second inspection scope characterized by inspections performed on a storage partition basis; the file acquisition module is configured to perform the step of determining the storage space to be inspected based on the inspection scope and the storage index.

[0034] If the scope of inspection is the first scope of inspection, then the data platform that generates the target file is determined according to the storage index, and all storage partitions under the data platform are used as the storage space;

[0035] If the scope of inspection is the second scope of inspection, then the target storage partition for storing the target file is determined according to the storage index, and the target storage partition is used as the storage space.

[0036] In some possible embodiments, the verification conditions include duplicate checks and range checks; the verification record module is configured to generate a verification record that satisfies the verification conditions based on the comparison results between the first field value and each of the second field values.

[0037] If the verification condition is a duplicate check, then the first target field value is determined from each of the second field values ​​based on the comparison results; wherein the first target field value is the same as the first field value;

[0038] If the check condition is a range check, then the second target field value is determined from each of the second field values ​​based on the comparison result; wherein the second target field value is within a specified numerical range, and the specified numerical range is determined based on the first field value;

[0039] The inspection record is generated based on the number of target field values; wherein the target field value is either the first target field value or the second target field value.

[0040] In some possible embodiments, the data structure information is used to record the fields contained in the data structure corresponding to the target file, as well as the offset and length of each field.

[0041] Thirdly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, causes the processor to implement any of the methods in the first aspect.

[0042] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods in the first aspect.

[0043] Fifthly, according to an embodiment of this application, a computer program product includes computer instructions stored in a computer-readable storage medium; when a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to implement any of the methods in the first aspect.

[0044] In this embodiment, by setting the same data structure for all record files generated under the same business operation, after determining the file to be checked based on the file identifier of the target file during the data verification stage, the first field value of the verification field in the target file and the second field value of the verification field in each file to be processed are determined based on the data structure information of the target file. Then, verification records that meet the verification conditions are generated based on the comparison results of the first field value and each second field value. The above verification process requires no manual intervention and has the advantages of high verification efficiency and avoidance of human error.

[0045] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing this disclosure. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the manual inspection process provided in the embodiments of this application;

[0047] Figure 2 A flowchart illustrating the overall process of a data verification method for a distributed system provided in this application embodiment;

[0048] Figure 3 This is a schematic diagram of the storage structure in a distributed system provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the formatted storage index path provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the duplicate check and range check provided in the embodiments of this application;

[0051] Figure 6 This is a structural diagram of the data verification 600 of the distributed system provided in the embodiments of this application;

[0052] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0054] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0055] As mentioned earlier, data verification for distributed systems is currently mainly performed manually. Verifiers need to locate the corresponding verification file based on the index indicating the verification scope and then manually verify that file. Specifically, as follows... Figure 1 As shown, inspectors need to log in to the database beforehand and find the file index based on the work batch number and record file name to obtain the storage path of the inspection files. Then, they retrieve the corresponding inspection files from the script directory according to the storage path. Finally, they manually check the content of each inspection file. It is evident that the above process, from file querying and retrieval to inspection, requires manual intervention, resulting in a long inspection cycle and the potential for human error.

[0056] To address the aforementioned problems, the inventive concept of this application is as follows: By setting the same data structure for all record files generated under the same business operation, after determining the file to be checked based on the file identifier of the target file during the data verification stage, the first field value of the verification field in the target file and the second field value of the verification field in each file to be processed are determined based on the data structure information of the target file. Then, verification records that meet the verification conditions are generated based on the comparison results of the first field value and each second field value. The above verification process requires no manual intervention and has the advantages of high verification efficiency and avoidance of human error.

[0057] Next, as follows Figure 2 As shown, Figure 2 This application illustrates the overall flow of a data verification method for a distributed system provided in an embodiment of the present application, specifically including:

[0058] Step 201: In response to the verification instruction, receive file information and verification information of the target file; wherein, the target file is a record file generated when any business operation is executed in batch processing; the file information includes a file identifier and data structure information representing the data structure of the target file, and the verification information includes a verification field and verification conditions associated with the verification field, wherein the verification field is at least one field in the data structure corresponding to the target file;

[0059] To facilitate understanding of the technical solution of this application, a banking scenario is used as an example below. It should be understood that the banking scenario example is only for the purpose of introducing the solution and is not a limitation on the application scenario. The technical solution of this application is applicable to any distributed system application scenario.

[0060] In banking scenarios, batch processing of business operations is often required. For example, when executing an "interest settlement" operation, the list of businesses to be settled needs to be batch-processed for "interest settlement". When any business operation is executed in batch processing, a log file is generated to record the operation process. In a distributed system, the log files generated by batch processing may be stored in different storage spaces.

[0061] In this embodiment, the same data structure is set for the record files generated by the same business operation. That is, the number, position, and meaning of fields in each record file generated by any business operation are the same. Suppose the business requirement is to check the number of errors in the interest settlement business of a certain branch. Then, any record file in the interest settlement business of that branch can be selected as the target file, and then the file to be checked can be determined based on the file information and the check information of the target file.

[0062] Specifically, the file information in this embodiment includes a file identifier and data structure information representing the target file's data structure. The file identifier is the storage index of the target file in the storage space; the storage space where the target file is stored in the distributed system can be found based on the file information. This file identifier may include a first identifier representing the target file, a second identifier representing the business operation corresponding to the target file, and a third identifier representing the batch processing count of the business operation.

[0063] The first identifier is the filename of the target file, which is unique in all record files generated by any business operation. The second identifier is the business operation that generated the target file; taking the interest calculation operation as an example, the second identifier here represents the interest calculation operation. The third identifier represents the batch processing number of the business operation. The first to third identifiers indicate the storage index of the target file.

[0064] In this embodiment, the data structure information is used to record the fields contained in the corresponding data structure of the target file, as well as the offset and length of each field. That is, the position of each field in the target file can be determined through the data structure information, thereby determining the field value of each field in the target file.

[0065] Step 202: Determine the storage space to be checked based on the file identifier, and determine each file to be checked corresponding to the target file from the storage space; wherein, the file to be checked and the target file are record files generated under the same business operation, and the data structure of each record file generated under the same business operation is the same;

[0066] The distributed system in this embodiment includes multiple storage partitions, each corresponding to a unique data platform. Each data platform stores the generated record files in its corresponding storage partition after batch processing business operations. Taking the aforementioned banking scenario as an example, the distributed system is equipped with servers or server clusters for each branch. Each branch's server or server cluster corresponds to multiple storage partitions, meaning each storage partition corresponds to a unique branch.

[0067] The document information in this application embodiment also includes a verification scope, which includes a first verification scope characterized by verification on a data platform basis and a second verification scope characterized by verification on a storage partition basis; the storage space to be verified is determined based on the target storage partition and the verification scope.

[0068] When performing step 202 above, the storage index of the target file needs to be determined based on the file identifier, and then the storage space to be checked is determined according to the check range and the storage index.

[0069] As mentioned earlier, in a distributed system, the log files generated by the same business operation are divided into multiple copies and stored in different storage partitions. The storage distribution in the above-mentioned banking scenario in a distributed system can be as follows: Figure 3 As shown, any branch (corresponding to a unique server or server cluster) contains multiple storage partitions, and each storage partition contains numerous record files. In implementation, the storage path for each record file can be set in a database form to determine its storage index based on the record file's identifier information.

[0070] In this embodiment, the first verification scope represents verification based on the region where the bank branch is located. In a distributed system, this corresponds to verification based on the server or server cluster corresponding to the branch. The second verification scope represents verification based on storage partitions. Assuming the Qinghai branch performs error statistics for interest settlement business A, any record file containing error records is selected from the numerous batch processing records of interest settlement business A and designated as the target file. Then, the storage index of the target file is determined based on its file identifier. Assuming the target file is a record file generated during the interest settlement operation of the Qinghai branch, its filename is FILE. The batch number of the operation that generated the target file is 001, then the identifier information of the target file is obtained as the first identifier FILE, the second identifier A, and the third identifier 001. Based on these file identifiers, the storage index of the target file based on the interest settlement operation is now known as Qh / Qh1 / C_001 / FILE_A_001. In Qh / Qh1 / C_001 / FILE_A_001, the first "Qh" represents the Qinghai branch, the second "Qh1" represents the corresponding server, the third "C_001" represents the target storage partition where the target file is stored, and the last "FILE_A_001" represents the file identifier of the target file.

[0071] Next, we will continue with the example of the Qinghai branch's error reporting statistics for interest settlement business A. Assume that the Qinghai provincial branch has a server cluster in the distributed system, containing servers Qh1 to Qh2. NIf the scope of the target file to be checked is the first check scope mentioned above, then the storage space to be checked is all storage partitions under server Qh1. Correspondingly, if the scope of the check is the second check scope mentioned above, then the storage space to be checked is servers Qh1 to Qh... N The target storage partition contains the target file in each of the following storage partitions.

[0072] Specifically, such as Figure 4 As shown, in the example above, the first check scope indicates that all storage partitions under the corresponding server of the Qinghai branch will be checked. Therefore, after determining the storage index Qh / Qh1 / C_001 / FILE_A_001 of the target file based on the target file's identification information, the storage index is formatted, and the processed storage index is Qh / Qh* / C* / FILE*. Here, "Qh*" represents all servers of the Qinghai branch, "C*" represents all storage partitions under server Qh*, and "FILE*" represents all record files stored in storage partition C*. At this time, all storage partitions C* within the server cluster Qh* of the Qinghai branch are retrieved from the storage space, and the record files stored in all storage partitions C* are used as the files to be checked.

[0073] Accordingly, if the second verification scope is used, after determining the storage index Qh / Qh1 / C_001 / FILE_A_001 of the target file based on its identification information, the storage index is formatted, resulting in a storage index of Qh / Qh1 / C_001 / FILE*. This means that all record files stored in the target storage partition C_001, where the target file resides, are used as files to be verified.

[0074] Step 203: Determine the first field value of the inspection field in the target file and the second field value of the inspection field in each of the files to be inspected based on the data structure information; generate an inspection record that meets the inspection conditions based on the comparison results of the first field value and each of the second field values.

[0075] As mentioned earlier, the data structure information in this embodiment indicates the fields contained in the corresponding data structure of the target file, as well as the offset and length of each field. That is, the position of each field in the target file can be determined through the data structure information, and thus the field value of each field in the target file can be determined. In this embodiment, all record files generated by the same business operation have the same data structure, so the first field value of the search field in the target file and the second field value in each file to be checked can be determined based on the data structure information of target file 1.

[0076] Therefore, a verification record that meets the verification conditions is generated based on the comparison results between the first field value and each of the second field values. The verification conditions in this embodiment include deduplication verification and range verification. If the verification condition is deduplication verification, a first target field value is determined from each of the second field values ​​based on the comparison results; wherein the first target field value is the same as the first field value. If the verification condition is range verification, a second target field value is determined from each of the second field values ​​based on the comparison results; wherein the second target field value is within a specified numerical range, and the specified numerical range is determined based on the first field value.

[0077] Assume the data structure of the parsing business contains fields A and B, where field A represents errors and its value can be 1 or 0 (i.e., 1 indicates an error, 0 indicates no error). Field B represents the specific reason for the error. Therefore, when performing error statistics for the interest settlement business, any record file containing errors can be used as the target file. Describing and checking for duplicates in field A or performing a range check in field B can determine the number of errors for the interest settlement business within the check range.

[0078] Specifically, such as Figure 5 As shown, the plagiarism detection process first obtains the first field value (1) of field A in the target file, then obtains the second field value of field A in each target file. The number of second field values ​​of 1 is then counted. Range checking determines the detection range of a field value based on the first field value in the target file. Figure 5 In the example of error count statistics shown, since the target file contains errors, field B, which represents the cause of the errors, must be a non-empty field. Therefore, field B can be used as the search field to check whether field B in each file to be checked is a non-empty field, i.e., whether the second field value of field B is null. If the second field value is non-empty, it indicates that the file to be checked contains errors. The error count of the current record can then be accumulated to complete the error count statistics for interest settlement business.

[0079] In this embodiment, the search field is any field or a combination of fields in the data structure corresponding to the target file. During the verification stage, only manual selection of the search field from the record data structure form and input of the target file's file identifier are required to automatically obtain the target file's data structure information and storage index. Then, the file to be verified is determined according to the pre-set verification range, and verified according to the pre-set verification conditions. The above verification process has the advantages of convenient operation, high verification efficiency, and avoidance of human error.

[0080] Based on the same inventive concept, this application provides a data verification device 600 for a distributed system, specifically as follows: Figure 6 As shown, it includes:

[0081] The information acquisition module 601 is configured to respond to a check instruction by receiving file information and check information of a target file; wherein, the target file is a record file generated when any business operation is executed in batch processing; the file information includes a file identifier and data structure information representing the data structure of the target file, and the check information includes a check field and check conditions associated with the check field, wherein the check field is at least one field in the data structure corresponding to the target file;

[0082] The file acquisition module 602 is configured to determine the storage space to be checked based on the file identifier, and determine each file to be checked corresponding to the target file from the storage space; wherein the file to be checked and the target file are record files generated under the same business operation, and the data structure of each record file generated under the same business operation is the same;

[0083] The inspection record module 603 is configured to perform the following actions: determine the first field value of the inspection field in the target file and the second field value of the inspection field in each of the files to be inspected based on the data structure information; and generate an inspection record that meets the inspection conditions based on the comparison results between the first field value and each of the second field values.

[0084] In some possible embodiments, the file identifier is a storage index of the target file in the storage space, and the file identifier includes a first identifier representing the target file, a second identifier representing the business job corresponding to the target file, and a third identifier representing the batch processing number of the business job.

[0085] In some possible embodiments, the distributed system includes multiple storage partitions, each corresponding to a unique data platform; the data platform is used to store the generated record files after batch processing and executing business jobs in the corresponding storage partitions;

[0086] The file information also includes the scope of inspection; the file acquisition module 602 is configured to: execute the process of determining the storage space to be inspected based on the file identifier.

[0087] The storage index of the target file is determined based on the file identifier;

[0088] The storage space to be checked is determined based on the check scope and the storage index.

[0089] In some possible embodiments, the inspection scope includes a first inspection scope characterized by inspections performed on a data platform basis, and a second inspection scope characterized by inspections performed on a storage partition basis; the file acquisition module 602 is configured to: perform the step of determining the storage space to be inspected based on the inspection scope and the storage index.

[0090] If the scope of inspection is the first scope of inspection, then the data platform that generates the target file is determined according to the storage index, and all storage partitions under the data platform are used as the storage space;

[0091] If the scope of inspection is the second scope of inspection, then the target storage partition for storing the target file is determined according to the storage index, and the target storage partition is used as the storage space.

[0092] In some possible embodiments, the verification conditions include duplicate checks and range checks; the verification record module 603 is configured to generate verification records that meet the verification conditions based on the comparison results between the first field value and each of the second field values, and the verification record module 603 is configured to:

[0093] If the verification condition is a duplicate check, then the first target field value is determined from each of the second field values ​​based on the comparison results; wherein the first target field value is the same as the first field value;

[0094] If the check condition is a range check, then the second target field value is determined from each of the second field values ​​based on the comparison result; wherein the second target field value is within a specified numerical range, and the specified numerical range is determined based on the first field value;

[0095] The inspection record is generated based on the number of target field values; wherein the target field value is either the first target field value or the second target field value.

[0096] In some possible embodiments, the data structure information is used to record the fields contained in the data structure corresponding to the target file, as well as the offset and length of each field.

[0097] The following reference Figure 7 To describe an electronic device 130 according to this embodiment of the present application. Figure 7 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0098] like Figure 7 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0099] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0100] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0101] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0102] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0103] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, which can be executed by the processor 131 of the aforementioned device to perform the aforementioned method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0104] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor 131, implement any of the methods in the data verification method for a distributed system provided in this application.

[0105] In an exemplary embodiment, various aspects of the data verification method for a distributed system provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the data verification method for a distributed system according to various exemplary embodiments of this application described above.

[0106] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] The program product for data verification in a distributed system according to embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0109] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0110] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as Java or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0111] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0112] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0118] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of data auditing of a distributed system, characterized in that, The method comprises: in response to the check instruction, receiving file information and check information of a target file; wherein the target file is a record file generated when a batch processing executes any business job; the file information comprises a file identifier and data structure information representing a data structure of the target file; the check information comprises a check field and a check condition associated with the check field, and the check field is at least one field in the data structure corresponding to the target file; determining a storage space to be checked based on the file identifier, and determining each file to be checked corresponding to the target file from the storage space; wherein the file to be checked and the target file are record files generated under the same business job, and the data structures of each record file generated under the same business job are the same; determining a first field value of the check field in the target file according to the data structure information, and a second field value of the check field in each of the files to be checked; and generating a check record satisfying the check condition according to the comparison result of the first field value and each of the second field values.

2. The method of claim 1, wherein, The file identifier is a storage index of the target file in the storage space, and the file identifier comprises a first identifier representing the target file, a second identifier representing the business job corresponding to the target file, and a third identifier representing the batch processing times of the business job.

3. The method of claim 2, wherein, The distributed system comprises a plurality of storage partitions, and each storage partition corresponds to a unique data platform; the data platform is used to store the record file generated after the batch processing executes the business job in each storage partition corresponding to the data platform; The file information further comprises a check range; the determination of the storage space to be checked based on the file identifier comprises: determining the storage index of the target file based on the file identifier; determining the storage space to be checked according to the check range and the storage index.

4. The method of claim 3, wherein, The check range comprises a first check range representing the check in units of data platforms, and a second check range representing the check in units of storage partitions; The determination of the storage space to be checked according to the check range and the storage index comprises: if the check range is the first check range, determining the data platform generating the target file according to the storage index, and taking all storage partitions under the data platform as the storage space; if the check range is the second check range, determining the target storage partition storing the target file according to the storage index, and taking the target storage partition as the storage space.

5. The method of claim 1, wherein, The check condition comprises duplicate check and range check; the generation of the check record satisfying the check condition according to the comparison result of the first field value and each of the second field values comprises: if the check condition is duplicate check, determining a first target field value from each of the second field values according to the comparison result; wherein the first target field value is the same as the first field value. If the check condition is a range check, a second target field value is determined from each of the second field values according to the comparison result; wherein the second target field value is within a specified numerical range, and the specified numerical range is determined according to the first field value; The check record is generated based on the number of target field values; wherein the target field value is the first target field value or the second target field value.

6. The method according to any one of claims 1 to 5, characterized in that, The data structure information is used to record the fields contained in the data structure corresponding to the target file, and the offset and field length of each field.

7. A data verification device for a distributed system, characterized in that, The apparatus comprises: An information obtaining module configured to receive file information and check information of a target file in response to a check instruction; wherein the target file is a record file generated when any business job is executed in batch processing; the file information comprises a file identifier and data structure information representing a data structure of the target file; the check information comprises a check field and a check condition associated with the check field, and the check field is at least one field in the data structure corresponding to the target file; A file obtaining module configured to determine a storage space to be checked based on the file identifier, and determine each file to be checked corresponding to the target file from the storage space; wherein the file to be checked and the target file are record files generated under the same business job, and each record file generated under the same business job has the same data structure; A check record module configured to determine a first field value of the check field in the target file and a second field value of the check field in each of the files to be checked according to the data structure information; and generate a check record satisfying the check condition according to the comparison result of the first field value and each of the second field values.

8. An electronic device, comprising: Comprise: A memory for storing program instructions; A processor for invoking the program instructions stored in the memory, and performing the steps included in the method according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program comprises program instructions, which, when executed by a computer, cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterised in that, The computer program product comprises computer program code, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1-6. The computer program product comprises computer program code, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1-6.

Images