Data checking method and device under distributed system, electronic equipment and storage medium

Abstract

The invention provides a data checking method and device under a distributed system, electronic equipment and a storage medium. The method comprises the following steps: determining target file identification information corresponding to at least one target distributed file to be subjected to data checking; according to the target file identification information, obtaining a target index path of each target distributed file in the at least one target distributed file; according to the target index path of each target distributed file, determining all partition files of each target distributed file; performing statistics to obtain the file line number of each partition file; and determining a data checking result of each target distributed file according to the file line number of each partition file and all partition files of each target distributed file. By means of the data checking method and device, the technical problem that in the prior art, data checking in a distributed system is tedious in operation and large in processing amount, and then the processing efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed systems, and in particular to a data verification method and device, an electronic device and a storage medium in a distributed system. Background Art

[0002] Currently, many users (especially enterprise-level systems) use a dual-core system with a mainframe and distributed parallel processing.

[0003] Compared with the host system, the distributed files in the distributed system will be divided into multiple partition files; at the same time, a partition (in order to improve the scalability and availability of the system, the data is divided into several parts and stored on different server nodes) will be divided into many pieces, and the data will also increase exponentially, which will bring great trouble to the verification work. The difference from the host system is that the host system runs under the same container, while the distributed system has a cluster. Different jobs and job steps may run on different servers, and the generated files will also be divided into multiple copies. The verification personnel need to view the results in the following way to verify: log in to the database server-enter the ISB library-find the file index table-find the desired index data according to the batch number and job or batch number and file name, copy the index file name, log in to the HDFS (i.e., distributed file system) server-enter the HDFS script directory-enter the statistical command and the specific file path-execute the command to view the results.

[0004] Furthermore, when performing data verification in a distributed system, only one file can be counted each time, but often a job will generate a large number of files, which brings a huge amount of processing volume to the verification personnel.

[0005] It can be seen that in the related technology, data verification in a distributed system has technical problems such as cumbersome operations and huge processing volume, which leads to low processing efficiency. Summary of the invention

[0006] The present application provides a data verification method and device, electronic device and storage medium in a distributed system, so as to at least solve the technical problems existing in the related art that data verification in a distributed system is complicated in operation, has a large amount of processing, and thus leads to low processing efficiency.

[0007] According to one aspect of an embodiment of the present application, a data verification method in a distributed system is provided, comprising:

[0008] Determine target file identification information corresponding to at least one target distributed file to be data-verified;

[0009] According to the target file identification information, obtaining a target index path of each target distributed file in the at least one target distributed file;

[0010] According to the target index path of each target distributed file, all partition files of each target distributed file are determined;

[0011] Count the number of file lines in each partition file;

[0012] According to the number of file lines of each partition file and all partition files of each target distributed file, a data verification result of each target distributed file is determined.

[0013] Optionally, as in the aforementioned method, acquiring the target index path of each target distributed file in the at least one target distributed file according to the target file identification information includes:

[0014] Searching the index table through the target file identification information to obtain all candidate index paths corresponding to the target file identification information, wherein the index table records a one-to-one correspondence between distributed files and index paths;

[0015] According to all the candidate index paths, a target index path of each target distributed file is determined.

[0016] Optionally, as in the aforementioned method, determining all partition files of each target distributed file according to the target index path of each target distributed file includes:

[0017] Based on the target index path, determining all target clusters where each target distributed file is distributed;

[0018] The number of files of all the partition files is determined according to the number of the target clusters; and all the partition files are determined by traversing the target cluster corresponding to each target index path.

[0019] Optionally, as in the aforementioned method, the counting of the number of file lines of each partition file includes:

[0020] Determine a statistical method corresponding to the number of files;

[0021] According to the statistical method corresponding to the number of files, statistics are performed on each partition file to obtain the number of file lines of each partition file.

[0022] Optionally, as in the aforementioned method, counting each partition file according to the statistical method corresponding to the number of files to obtain the number of file lines of each partition file includes:

[0023] When the number of files is one, the number of file lines of the partition file is obtained by reading according to the target index path and using the HDFS interface method;

[0024] The step of determining the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file includes:

[0025] The number of file lines of the partition file is mapped to a target query interface to obtain the data verification result of the target distributed file, wherein the target query interface is an interface for a user to input identification information of the target file.

[0026] Optionally, as in the aforementioned method, counting each partition file according to the statistical method corresponding to the number of files to obtain the number of file lines of each partition file includes:

[0027] In the case where there are multiple files, the target index path is processed into a specified index path in a spark program format;

[0028] Put the specified index path into a spark container and obtain the spark object;

[0029] The spark object calls the spark statistical file interface to count the partition files corresponding to each specified index path to obtain the number of file lines of each partition file;

[0030] The number of file lines of each partition file is stored in a preset library table;

[0031] The step of determining the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file includes:

[0032] According to all partition files of each target distributed file, by mapping the number of file rows of each partition file in the preset library table to the target query interface, the data verification result corresponding to each target distributed file in the at least one target distributed file is obtained, wherein the target query interface is an interface for obtaining the target file identification information determined by the user.

[0033] Optionally, as described in any of the preceding methods, the determining the target file identification information corresponding to at least one target distributed file to be data-verified includes:

[0034] Determine the target file identification information corresponding to one of the target distributed files according to the acquired batch number and file name; or,

[0035] All candidate distributed files corresponding to the acquired batch number and / or job name are determined; based on a selection operation of selecting multiple target distributed files from all the candidate distributed files, target file identification information corresponding to the multiple target distributed files is determined.

[0036] According to another aspect of the embodiment of the present application, a data checking device in a distributed system is also provided, including:

[0037] A first determination module, used to determine target file identification information corresponding to at least one target distributed file to be data-verified;

[0038] An acquisition module, configured to acquire a target index path of each target distributed file in the at least one target distributed file according to the target file identification information;

[0039] A second determination module, configured to determine all partition files of each target distributed file according to a target index path of each target distributed file;

[0040] Statistics module, used to count the number of file lines in each partition file;

[0041] The result module is used to determine the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file.

[0042] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0043] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method steps in any of the above embodiments when executed.

[0044] In an embodiment of the present application, through the method in this embodiment, based on the target file identification information corresponding to at least one target distributed file to be data verified, the number of file lines of each partition file corresponding to each target distributed file in all partition files can be obtained, and then the data verification result of each target distributed file can be determined, thereby eliminating the need to manually perform data verification on each file one by one, which can effectively simplify manual operations and improve verification efficiency; and thus can overcome the technical problems existing in the related art that data verification in a distributed system is cumbersome and has a large amount of processing, thereby leading to low processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 It is a flowchart of an optional data verification method in a distributed system according to an embodiment of the present application;

[0048] Figure 2 is a flow chart of another optional data verification method in a distributed system according to an embodiment of the present application;

[0049] Figure 3 It is a flowchart of an optional data verification method in a distributed system according to an application example of the present application;

[0050] Figure 4 is a schematic diagram of an optional distributed file according to an application example of the present application;

[0051] Figure 5 is a structural block diagram of an optional data checking device in a distributed system according to an embodiment of the present application;

[0052] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] According to one aspect of an embodiment of the present application, a data verification method in a distributed system is provided. Optionally, in this embodiment, the data verification method in a distributed system can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal via a network, and can be used to provide services (such as advertising push services, application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0056] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0057] The data verification method in a distributed system of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The terminal can also execute the data verification method in a distributed system of the embodiment of the present application by a client installed thereon.

[0058] In the related art, in order to check the file data volume in a distributed system, one can only log in to the liunx server for statistics. The operation is complicated and cumbersome, the readability is poor, and only a single file can be counted at a time. In addition, the statistical information only includes the file name and the number of lines.

[0059] Taking the data checking method in the distributed system in this embodiment executed by the server as an example, Figure 1 A data verification method in a distributed system provided in an embodiment of the present application includes the following steps:

[0060] Step S101: determining target file identification information corresponding to at least one target distributed file to be data-checked.

[0061] The data verification method in a distributed system in this embodiment can be applied to scenarios where data in a distributed system needs to be verified, for example: a scenario where a distributed file in a distributed system is verified, and a scenario where multiple distributed files in a distributed system are verified simultaneously; for other types of scenarios, the above-mentioned data verification method in a distributed system is also applicable unless there is a contradiction.

[0062] In this embodiment, the target distributed file is a file that is distributedly stored in a distributed system, and the target distributed file may be stored in one or more nodes in the distributed system. For example, different job steps of a job may be processed in different nodes, and then the sub-files corresponding to each job step are stored in each node.

[0063] In this embodiment, the target file identification information can be determined by accepting the information input by the user on the interactive interface. For example, when the target distributed file is a file under an enterprise-level distributed system, the file identification information may include but is not limited to: batch number (referring to the date and time of running the batch processing, such as June 20, 2023), job name (name of the job, job: XML name, there are N jobs under the batch processing). Batch processing refers to batch processing of a certain object or thing, file name.

[0064] In addition, the target file identification information corresponding to the one or more target distributed files may also be determined through the one or more target distributed files selected by the user on the interactive interface.

[0065] For example, when it is necessary to count all file results of a certain job, the job name can be used as the target file identification information to obtain the data verification results of all target distributed files corresponding to the job name; when it is necessary to count all file results of a certain batch, the batch number of the batch can be used as the target file identification information to obtain the data verification results of all target distributed files corresponding to the batch.

[0066] Step S102: acquiring a target index path of each target distributed file in at least one target distributed file according to the target file identification information.

[0067] After obtaining the target file identification information, the index path of each distributed file may include the file identification information of the distributed file. Therefore, based on the above target file identification information, a query can be performed in all index paths to obtain the target index path of each target distributed file in at least one target distributed file.

[0068] The target index path is used to indicate the path of the target distributed file in the distributed system.

[0069] Step S103: determining all partition files of each target distributed file according to the target index path of each target distributed file.

[0070] After obtaining the target index path of the target distributed file, when a target distributed file has only one target index path, then the target distributed file has only one partition file under the target index path; when a target distributed file has multiple target index paths, then the target distributed file has a partition file under each target index path.

[0071] After the target index path of each target distributed file is determined, a scan may be performed in the cluster corresponding to the target index path to further determine all partition files of each target distributed file.

[0072] Step S104, counting the number of file lines of each partition file.

[0073] After scanning and obtaining each partition file, the number of file lines in each partition file may be read to obtain the number of file lines in each partition file.

[0074] In general, each time a job step corresponding to a partition file is executed, a file line number is added to the partition file.

[0075] Step S105 : determining a data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file.

[0076] After obtaining the number of file lines of each partition file, based on all partition files of each target distributed file mentioned above, the total number of file lines of each target distributed file can be obtained, and based on the total number of file lines of each target distributed file, the data verification result of each target distributed file can be obtained.

[0077] Through the method in this embodiment, based on the target file identification information corresponding to at least one target distributed file to be data verified, the number of file lines of each partition file corresponding to each target distributed file in all partition files can be obtained, and the data verification result of each target distributed file can be determined, thereby eliminating the need to manually perform data verification on each file one by one, which can effectively simplify manual operations and improve verification efficiency; and thus can overcome the technical problems existing in the related art that data verification in a distributed system is cumbersome and requires a large amount of processing, which leads to low processing efficiency.

[0078] As an optional embodiment, as in the above method, the step S102 obtains the target index path of each target distributed file in at least one target distributed file according to the target file identification information, including the following steps:

[0079] Step S201, querying in an index table through target file identification information to obtain all candidate index paths corresponding to the target file identification information, wherein the index table records a one-to-one correspondence between distributed files and index paths.

[0080] Step S202: determining a target index path for each target distributed file based on all candidate index paths.

[0081] One of the optional implementations could be:

[0082] After obtaining the target file identification information, a query can be performed in the index table based on the target file identification information. In each index path in the index table, there is a field corresponding to the file identification information. Therefore, the index path whose field information includes the target file identification information can be determined as a candidate index path by data matching. Among all the candidate index paths, the target index path of each target distributed file is determined based on the target file identification sub-information corresponding to each target distributed file in the target file identification information.

[0083] In addition, another optional implementation may be:

[0084] It is also possible to directly determine the target distributed file corresponding to each target file identification sub-information. For any target file identification sub-information, a query is performed in the index table through the target file identification sub-information, and then all candidate index paths corresponding to the target file identification sub-information can be obtained, and all candidate index paths corresponding to the target file identification sub-information can be used as the target index paths of the target distributed file corresponding to the target file identification sub-information.

[0085] Through the method in this embodiment, all target index paths corresponding to each target distributed file can be quickly determined, thereby facilitating the subsequent determination of all partition files corresponding to each target distributed file.

[0086] As an optional embodiment, as in the above method, the step S103 determines all partition files of each target distributed file according to the target index path of each target distributed file, including the following steps:

[0087] Step S301: based on the target index path, determine all target clusters where each target distributed file is distributed.

[0088] After obtaining the target index path, the target cluster corresponding to each target index path can be determined; further, according to the target index path corresponding to each target distributed file, all target clusters of all partition files corresponding to each target distributed file can be determined.

[0089] Step S302, determining the number of files of all partitions according to the number of target clusters;

[0090] Step S303: determine all partition files by traversing the target cluster corresponding to each target index path.

[0091] After determining all target clusters for each target distributed file, the number of all target clusters corresponding to at least one target partition file can be determined based on the total number of all target clusters; and the number of files of all partition files can be determined according to the number of all target clusters.

[0092] For each target index path, the file identification information corresponding to the corresponding target distributed file in the target index path is used to traverse the target cluster corresponding to the target index path to obtain the partition file corresponding to the target index path; according to this method, all partition files can be determined.

[0093] like Figure 2 As shown, as an optional embodiment, as in the above method, the step S104 of obtaining the number of file lines of each partition file by counting includes the following steps:

[0094] Step S401, determining a statistical method corresponding to the number of files;

[0095] Step S402: Count each partition file according to the counting method corresponding to the number of files to obtain the number of file lines of each partition file.

[0096] That is to say, the statistical methods for counting all partition files are different depending on the number of files; and after determining the statistical method corresponding to the number of files, the statistical method can be used to count each partition file to obtain the number of file lines of each partition file.

[0097] As an optional embodiment, as in the aforementioned method, step S402 performs statistics on each partition file according to the statistical method corresponding to the number of files to obtain the number of file lines of each partition file, including the following steps:

[0098] When there is only one file, the number of file lines of the partition file is obtained by reading the target index path through the HDFS interface method.

[0099] The step S105 determines the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file, including:

[0100] The number of file lines of the partition file is mapped to the target query interface to obtain the data verification result of the target distributed file, wherein the target query interface is an interface for the user to input the target file identification information.

[0101] When there is only one file, the number of file lines of the partition file can be directly read through the HDFS API program (ie, the HDFS interface method).

[0102] Optionally, you can link to an HDFS file server, create an HDFS container, and inject the above-mentioned target index path into the HDFS container; then, obtain the file operation object (i.e., the number of file lines) through the HDFS container, put the file operation object into a JAVA IO stream (a tool class for JAVA operation files) object, and map the file operation object to the target query interface through the JAVA IO stream to obtain a data verification result corresponding to the target distributed file that is returned to the target query interface.

[0103] Through the method in this embodiment, when the number of partition files corresponding to at least one target distributed file is one, the data verification result of the at least one target distributed file can be obtained.

[0104] As an optional embodiment, as in the aforementioned method, step S402 performs statistics on each partition file according to the statistical method corresponding to the number of files to obtain the number of file lines of each partition file, including the following steps:

[0105] Step S601, when there are multiple files, the target index path is processed into a specified index path in the spark program format;

[0106] Step S602: put the specified index path into a spark container and obtain a spark object.

[0107] Step S603, the spark object calls the spark statistical file interface to count the partition files corresponding to each specified index path to obtain the number of file lines of each partition file.

[0108] Step S604, storing the number of file lines of each partition file in a preset library table.

[0109] In this embodiment, when there are multiple files, the target index path is processed into a specified index path in the spark Chen Xu format. For example, when the target index path is / home / qh / qh1 / A001 / testFile.A001, the specified index path after being processed into the spark program format is: / home / qh / qh* / A001 / testFile.A*.

[0110] Then you can create a spark container by linking to a spark server (a fast and versatile computing engine designed for large-scale data processing), put the processed specified index path into the spark container, and get a spark object; the spark object counts the partition files corresponding to each specified index path by calling the spark statistical file interface to obtain the number of file lines of each partition file; then the number of file lines of each partition file is stored in the preset library table.

[0111] The step S105 determines the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file, including:

[0112] Step S605, according to all partition files of each target distributed file, by mapping the number of file lines of each partition file in the preset library table to the target query interface, obtain the data verification result corresponding to each target distributed file in at least one target distributed file, wherein the target query interface is an interface for obtaining the target file identification information determined by the user.

[0113] In this embodiment, when the number of file lines of each partition file is stored in the preset library table, all the file line numbers read can be stored according to all the partition files corresponding to each target distributed file. Furthermore, when the number of file lines of each partition file in the preset library table is mapped to the target query interface, it can also be mapped directly according to the order of storage; thereby obtaining the data verification result corresponding to each target distributed file in multiple target distributed files.

[0114] As an optional embodiment, as in any of the methods of the preceding items, the step S101 of determining the target file identification information corresponding to at least one target distributed file to be data-verified comprises the following steps:

[0115] Step S701, determining target file identification information corresponding to a target distributed file according to the acquired batch number and file name;

[0116] In this embodiment, the user can enter the batch number and the file name in the interactive interface, and then obtain a target distributed file uniquely corresponding to the batch number and the file name, and target file identification information corresponding to the target distributed file.

[0117] Step S702, determining all candidate distributed files corresponding to the acquired batch number and / or job name; based on the selection operation of selecting multiple target distributed files from all candidate distributed files, determining target file identification information corresponding to the multiple target distributed files.

[0118] In this embodiment, the user can enter the batch number and / or job name in the interactive interface, and then obtain all candidate distributed files corresponding to the batch number and / or job name; in general, there are multiple distributed files corresponding to the same batch number and / or job name.

[0119] After all candidate distributed files are determined, all candidate distributed files can be displayed in an interactive interface, and then the user can select from all candidate distributed files displayed in the interactive interface through a selection operation to obtain multiple target distributed files, and then the target file identification information corresponding to the multiple target distributed files can be determined.

[0120] Through the method in this embodiment, multiple selection modes can be provided for users to select target distributed files, which is convenient for data verification of each target distributed file at a later stage.

[0121] like Figure 3 As shown, an application example of applying any of the above embodiments to any distributed system is provided:

[0122] 1) Get page parameters:

[0123] Users can select batch numbers, file names or enter job names on the page, or select multiple distributed file lists by themselves.

[0124] 2) File statistics can include the following two statistical methods:

[0125] 2.1: Single file statistics: statistics of a single distributed file at a time;

[0126] 2.2: Multi-file statistics: statistics of multiple distributed files at a time.

[0127] 3) Get input parameters:

[0128] 3.1: Based on the batch number and file name: Based on these two parameters, a specific index path can be obtained later;

[0129] 3.2: Multiple files selected by the user based on batch number and / or job name: Multiple index paths can be obtained later based on these three parameters.

[0130] 4) ISB index library table:

[0131] The index library table records the specific index path of each file.

[0132] 5) Get the index path:

[0133] 5.1: Get a single file index path: Get the specific index path of the distributed file that meets the requirements of [3.1];

[0134] 5.2: Get the index path of all files under the job or the index path of the file selected by the user: Get the specific index path of all distributed files that meet the requirements of [3.2].

[0135] 6) Is it a regional file or a provincial (e.g., provincial branches of an enterprise) / national (e.g., including the national head office and all provincial branches) file?

[0136] Because of the distributed system, a distributed file will be divided into many parts, such as Figure 4 shown.

[0137] 7)Statistical type:

[0138] 7.1: Partition file statistics: Statistics of a single file in a partition (such as Figure 4 (as indicated by 11 in the figure);

[0139] 7.2: Provincial / national file statistics: Statistics of all provincial / national files. Provincial / national files include multiple partition files (such as Figure 4 (as indicated by 12 in the figure).

[0140] 8) Get or process the file path:

[0141] 8.1: Get the specific path of the file. There is no need to process the index path for a partition file. The HDFS API program can process it directly.

[0142] 8.2: Process the index path: Convert the index path into spark program format for post-processing.

[0143] Thus, through the above method, some personnel without professional technical operation can also perform data verification. Further, the above method can generate a document data verification result by interacting with the corresponding interactive interface (for example, clicking a button, etc.), and the verification result can also include information such as the job name, job step, file line number, time, etc.

[0144] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0146] According to another aspect of the embodiments of the present application, a data checking device in a distributed system for implementing the above-mentioned data checking method in a distributed system is also provided. Figure 5 is a structural block diagram of an optional data checking device in a distributed system according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0147] A first determination module 1, used to determine target file identification information corresponding to at least one target distributed file to be data verified;

[0148] An acquisition module 2 is used to acquire a target index path of each target distributed file in at least one target distributed file according to the target file identification information;

[0149] A second determination module 3, configured to determine all partition files of each target distributed file according to a target index path of each target distributed file;

[0150] Statistics module 4, used to count the number of file lines in each partition file;

[0151] The result module 5 is used to determine the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file.

[0152] It should be noted that the first determination module 1 in this embodiment can be used to execute the above step S101, the acquisition module 2 in this embodiment can be used to execute the above step S102, the second determination module 3 in this embodiment can be used to execute the above step S103, the statistical module 4 in this embodiment can be used to execute the above step S104, and the result module 5 in this embodiment can be used to execute the above step S105.

[0153] The device in this embodiment, in addition to the above modules, may also include a module for executing any method in any of the above-mentioned embodiments of the data verification method in a distributed system.

[0154] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 1 The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0155] According to another aspect of an embodiment of the present application, there is also provided an electronic device for implementing the above-mentioned data verification method in a distributed system. The electronic device may be a server, a terminal, or a combination thereof.

[0156] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0157] Memory 1503, used for storing computer programs;

[0158] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0159] Step S101: determining target file identification information corresponding to at least one target distributed file to be data-checked.

[0160] Step S102: acquiring a target index path of each target distributed file in at least one target distributed file according to the target file identification information.

[0161] Step S103: determining all partition files of each target distributed file according to the target index path of each target distributed file.

[0162] Step S104, counting the number of file lines of each partition file.

[0163] Step S105 : determining a data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file.

[0164] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0165] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0166] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0167] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0168] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0169] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0170] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0171] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data verification method in a distributed system, characterized in that: include: Determine target file identification information corresponding to at least one target distributed file to be data-verified; According to the target file identification information, obtaining a target index path of each target distributed file in the at least one target distributed file; According to the target index path of each target distributed file, all partition files of each target distributed file are determined; Count the number of file lines in each partition file; According to the number of file lines of each partition file and all partition files of each target distributed file, a data verification result of each target distributed file is determined.

2. The method according to claim 1, characterized in that The acquiring, according to the target file identification information, a target index path of each target distributed file in the at least one target distributed file comprises: Searching the index table through the target file identification information to obtain all candidate index paths corresponding to the target file identification information, wherein the index table records a one-to-one correspondence between distributed files and index paths; According to all the candidate index paths, a target index path of each target distributed file is determined.

3. The method according to claim 1, characterized in that Determining all partition files of each target distributed file according to the target index path of each target distributed file includes: Based on the target index path, determining all target clusters where each target distributed file is distributed; The number of files of all the partition files is determined according to the number of the target clusters; and all the partition files are determined by traversing the target cluster corresponding to each target index path.

4. The method according to claim 3, characterized in that: The statistics of the number of file lines of each partition file include: Determine a statistical method corresponding to the number of files; According to the statistical method corresponding to the number of files, statistics are performed on each partition file to obtain the number of file lines of each partition file.

5. The method according to claim 4, characterized in that The counting of each partition file according to the counting method corresponding to the number of files to obtain the number of file lines of each partition file includes: When the number of files is one, the number of file lines of the partition file is obtained by reading according to the target index path and using the HDFS interface method; The step of determining the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file includes: The number of file lines of the partition file is mapped to a target query interface to obtain the data verification result of the target distributed file, wherein the target query interface is an interface for a user to input identification information of the target file.

6. The method according to claim 4, characterized in that The counting of each partition file according to the counting method corresponding to the number of files to obtain the number of file lines of each partition file includes: In the case where there are multiple files, the target index path is processed into a specified index path in a spark program format; Put the specified index path into a spark container and obtain the spark object; The spark object calls the spark statistical file interface to count the partition files corresponding to each specified index path to obtain the number of file lines of each partition file; The number of file lines of each partition file is stored in a preset library table; The step of determining the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file includes: According to all partition files of each target distributed file, by mapping the number of file rows of each partition file in the preset library table to the target query interface, the data verification result corresponding to each target distributed file in the at least one target distributed file is obtained, wherein the target query interface is an interface for obtaining the target file identification information determined by the user.

7. The method according to any one of claims 1 to 6, characterized in that The determining of target file identification information corresponding to at least one target distributed file to be data-verified includes: Determine the target file identification information corresponding to one of the target distributed files according to the acquired batch number and file name; or, All candidate distributed files corresponding to the acquired batch number and / or job name are determined; based on a selection operation of selecting multiple target distributed files from all the candidate distributed files, target file identification information corresponding to the multiple target distributed files is determined.

8. A data checking device in a distributed system, characterized in that: include: A first determination module, used to determine target file identification information corresponding to at least one target distributed file to be data-verified; An acquisition module, configured to acquire a target index path of each target distributed file in the at least one target distributed file according to the target file identification information; A second determination module, configured to determine all partition files of each target distributed file according to a target index path of each target distributed file; Statistics module, used to count the number of file lines in each partition file; The result module is used to determine the data verification result of each target distributed file according to the number of file lines of each partition file and all partition files of each target distributed file.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method steps of any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method steps described in any one of claims 1 to 7 when run.

Images