A data quality inspection method and system based on a master multi-node

Abstract

The application provides a data quality inspection method and system based on a master and multiple sub-nodes, which comprises the following steps: dividing the data to be inspected into multiple groups, and sending each group of data to be inspected to a corresponding sub-node; judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, and if so, performing task scheduling and task execution; if not, waiting or regulating resources, and then performing task scheduling and task execution, and gathering the inspection result data to the master node; when map display is needed, extracting the corresponding inspection result data, and rendering and displaying according to the corresponding rendering type. The application groups the data to be inspected, and inspects each group of data by using multiple sub-nodes, so that the data quality inspection method based on a master and multiple sub-nodes improves the data quality inspection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data quality inspection, and more particularly, to a data quality inspection method and system based on a master and multiple sub-nodes. BACKGROUND

[0002] In the fields of artificial intelligence, big data, cloud computing, etc., the development of digital economy is actively promoted, and the accuracy and reliability of data in these fields are required to be extremely high. Therefore, effective data quality inspection means are needed to ensure the quality of data to support the healthy development of digital economy.

[0003] In the construction of new urbanization, the level of urban management and service is improved through big data, Internet of Things, etc. In the construction of smart city, data quality inspection is the premise to ensure the accuracy and security of urban data.

[0004] In the current data quality inspection, there are differences in rule understanding and different implementation of quality inspection work, which will lead to quality problems of data results and make it more difficult to ensure the authenticity of data.

[0005] The current BS-end data quality inspection service is affected by network environment, bandwidth, and server CPU performance. In the case of large data results package, real-time data quality inspection will inevitably have the problems of slow response speed and poor running efficiency. SUMMARY

[0006] The present application provides a data quality inspection method and system based on a master and multiple sub-nodes to improve the efficiency of data quality inspection.

[0007] According to the first aspect of the present application, a data quality inspection method based on a master and multiple sub-nodes is provided, comprising:

[0008] Step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, and sending each group of data to be inspected to the corresponding sub-node;

[0009] Step S2, judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, if sufficient, executing task scheduling and task execution; if not sufficient, waiting or regulating resources, and then executing task scheduling and task execution;

[0010] Step S3, when the task quality inspection is successful, storing the quality inspection result data in the sub-node result database, and gathering the quality inspection result data in all sub-node result databases into the master node result database;

[0011] Step S4, when map display is needed, extracting the quality inspection result data from the master node result database, and rendering and displaying according to the corresponding rendering type.

[0012] According to a second aspect of the present application, a data quality inspection system based on a master multi-sub-node is provided, comprising:

[0013] a division module for grouping and dividing the data to be inspected, obtaining a plurality of groups of data to be inspected, and sending each group of data to be inspected to a corresponding sub-node;

[0014] a judgment module for judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, and if so, performing task scheduling and task execution; if not, waiting or regulating resources, and then performing task scheduling and task execution;

[0015] a storage module for storing the inspection result data into the sub-node result database after the task inspection is successful, and gathering the inspection result data in all sub-node result databases into the master node result database;

[0016] a rendering module for extracting the inspection result data from the master node result database when map display is needed, and rendering and displaying according to the corresponding rendering type.

[0017] The data quality inspection method and system based on a master multi-sub-node provided by the present application divides the data to be inspected into a plurality of groups, sends each group of data to be inspected to a corresponding sub-node, judges whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, and if so, performs task scheduling and task execution; if not, waits or regulates resources, and then performs task scheduling and task execution, and gathers the inspection result data into the master node; when map display is needed, extracts the corresponding inspection result data, and renders and displays according to the corresponding rendering type. The data to be inspected is divided into groups, and the inspection work is performed in parallel on a plurality of sub-nodes, the data quality inspection method based on a master multi-sub-node improves the data quality inspection efficiency, and the data that passes the inspection is used for rendering to realize map display, so that the displayed map is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A data quality inspection method flowchart based on a master multi-sub-node provided by an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a data quality inspection request and a judgment node resource;

[0020] Figure 3 A schematic diagram of inspection task scheduling and multi-channel message communication;

[0021] Figure 4 A schematic diagram of creating an inspection task and inspecting the task;

[0022] Figure 5 a schematic diagram for rendering the quality inspection result data;

[0023] Figure 6 a structural block diagram of a data quality inspection system based on a master and multiple sub-nodes provided by another embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the protection scope of the present application. In addition, the technical features in each of the embodiments or in a single embodiment provided by the present application can be combined with each other at will to form a feasible technical solution, and this combination is not restricted by the sequence of steps and / or the structure mode, but should be based on the fact that it can be realized by those of ordinary skill in the art. When the combination of technical solutions appears contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope of the present application.

[0025] Referring to Figure 1 , a data quality inspection method based on a master and multiple sub-nodes of one embodiment of the present application is provided, mainly including the following steps:

[0026] Step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, and sending each group of data to be inspected to a corresponding sub-node.

[0027] In one embodiment of the present application, the step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, includes:

[0028] According to the division, organization and spatial position of the uploading user of the data to be inspected, clustering and grouping all the data to be inspected uploaded by the user to obtain multiple groups of data to be inspected;

[0029] The step of sending each group of data to be inspected to a corresponding sub-node includes:

[0030] According to the physical position of each group of data to be inspected, obtaining multiple sub-nodes within the adjacent range of each group of data to be inspected;

[0031] Assigning a serial number w to each group of data to be inspected, and generating a corresponding hash value according to the serial number w;

[0032] According to the hash value corresponding to each group of data to be inspected, each group of data to be inspected is allocated to a corresponding sub-node, wherein a correspondence between the hash value of the data to be inspected and the serial number of the sub-node is established in advance.

[0033] It can be understood that when the master node receives each user uploaded data to be inspected, since the data volume of all data to be inspected is very large, it is necessary to group all data to be inspected. In the embodiment of the application, based on the geographical location of the user uploading the data to be inspected, including the division, organization and spatial location to which the user belongs, the data to be inspected is clustered and grouped, the data to be inspected uploaded by users with close geographical locations is divided into a group, a plurality of groups of data to be inspected are obtained, and each group of data to be inspected is allocated to a sub-node for inspection.

[0034] Step S2, according to the data volume of the data to be inspected on each sub-node, it is judged whether the computing resources on the sub-node are sufficient, if sufficient, task scheduling and task inspection are performed; if not sufficient, waiting or resource regulation is performed, and then task scheduling and task execution are performed.

[0035] It can be understood that when the master node receives each user uploaded data to be inspected, since the data volume of all data to be inspected is very large, it is necessary to group all data to be inspected. In the embodiment of the application, based on the geographical location of the user uploading the data to be inspected, including the division, organization and spatial location to which the user belongs, the data to be inspected is clustered and grouped, the data to be inspected uploaded by users with close geographical locations is divided into a group, a plurality of groups of data to be inspected are obtained, and each group of data to be inspected is allocated to a sub-node for inspection. Figure 2 When the sub-node data is uploaded and the data inspection is applied, it is judged according to the data volume of the data to be inspected whether the sub-node currently has sufficient computing resources, if yes, the scheduling and execution of the inspection task can be performed. However, if the sub-node currently does not have sufficient computing resources, the data inspection needs to be performed after the sub-node resources are released or the data to be inspected is re-distributed to the sub-node resources for inspection.

[0036] It can be understood that after each group of data to be inspected is allocated to the corresponding sub-node, it is judged whether the sub-node currently has sufficient computing resources, if the sub-node currently has sufficient computing resources, the data to be inspected is inspected on the sub-node, if the sub-node currently does not have sufficient computing resources, the sub-node needs to be waited for or re-distributed.

[0037] In an embodiment of the application, the step S2, according to the data volume of the data to be inspected on each sub-node, it is judged whether the computing resources on the sub-node are sufficient, if sufficient, task scheduling and task inspection are performed; if not sufficient, waiting or resource regulation is performed, and then task scheduling and task execution are performed.

[0038] Step S21, according to the number of cores of the sub-node and the current available memory, the maximum number of processes that can be supported by the sub-node is calculated.

[0039] It can be understood that when it is judged whether the sub-node currently has sufficient computing resources, the maximum number of processes that can be supported by the sub-node is calculated first. Specifically, the maximum number of processes that can be supported by the sub-node is calculated according to the number of cores of the sub-node and the current available memory, including:

[0040] Pmax = MIN(P cpu ,P mem );

[0041] wherein P cpu is the maximum number of processes that the CPU of the sub-node can support, and P mem is the maximum number of processes that the available memory of the sub-node can support;

[0042] P cpu = ;

[0043] wherein CPU 核数 is the number of cores of the sub-node server, CPU 安全阈值 is a preset safety threshold, usually taken as 80%, and CPU per is the number of cores occupied by a single process;

[0044] P mem = ;

[0045] wherein MEM 内存 is the available memory of the sub-node, MEM 缓冲内存 is a preset buffer memory ratio, usually taken as 20%, and MEM per is the memory required by a single process.

[0046] In step S22, it is judged whether the computing resources on the sub-node are sufficient according to the number of processes required by the data to be inspected on the sub-node and the maximum number of processes that the sub-node can support at present.

[0047] It can be understood that after the maximum number of processes that the sub-node can support at present is calculated, it is judged according to the number of processes required by the data to be inspected whether the sub-node has sufficient computing resources to support the inspection work of the data to be inspected.

[0048] Specifically, when the number of processes P task required by the inspection work of the data to be inspected is less than P max , the computing resources on the sub-node are sufficient, and then the inspection task is normally started to enter the inspection process; when P task ≥ P max , the computing resources on the sub-node are insufficient, and then the inspection task is put into a waiting queue to wait for the release of the sub-node resources, and then enters the inspection process, or the sub-node resources are re-allocated to the inspection task, and then enters the inspection process.

[0049] When the computing resource on the sub-node is insufficient, the urgency of the quality inspection task is obtained, and whether the current quality inspection task is waiting or reassigning node resource is determined according to the urgency of the quality inspection task. Specifically, if the urgency of the current quality inspection task is less than a preset urgency, the quality inspection task is put into a waiting queue to wait for the release of the sub-node resource, and then enters the quality inspection process. If the urgency of the current quality inspection task is greater than or equal to the preset urgency, the quality inspection task is reassigned to other sub-nodes with sufficient computing resources and the smallest distance value, and enters the quality inspection process, wherein the distance value refers to the distance between the physical address of the user uploading the data to be inspected and the physical address of the sub-node.

[0050] When the computing resource on the sub-node is sufficient, the quality inspection task of the data to be inspected is started to perform the quality inspection process. Specifically, referring to Figure 2 When the resource of the sub-node is sufficient, the quality inspection task of the data to be inspected is started, the quality inspection task is put into a quality inspection queue, and communication with the quality inspection program is started to enter the quality inspection process of the data to be inspected through the quality inspection program.

[0051] In an embodiment of the present application, when the computing resource on the sub-node is sufficient, the quality inspection task is normally started to enter the quality inspection process, including:

[0052] The quality inspection task is created, and the quality inspection task includes the pre-checking scheme, the data to be inspected and the quality inspection scheme;

[0053] According to the pre-checking scheme, the pre-checking of the data to be inspected is performed;

[0054] After the pre-checking is passed, the quality inspection program is started, the data to be inspected is inspected according to the quality inspection scheme, and the quality inspection result data is obtained.

[0055] Referring to Figure 3 When the computing resource on the sub-node is sufficient, the quality inspection task is scheduled, and the quality inspection resource is reasonably allocated to ensure the smooth and orderly progress of the quality inspection task. At the same time, the module is also responsible for establishing and maintaining the service end of the multi-channel quality inspection message channel processing program, and maintaining the message intercommunication between the communication channel client of the quality inspection rendering module and the quality inspection processing program. Wherein, the reasonable allocation of resources includes the management of the computing node, the architecture design of one master and multiple sub-nodes, etc.

[0056] Then, the multi-channel quality inspection task communication program is started, the quality inspection program is started to enter the data quality inspection process.

[0057] Referring to Figure 4For a single quality inspection task, the quality inspection task includes a pre-check scheme, to-be-inspected data and a quality inspection scheme. Rule matching will be performed on the to-be-inspected data, the pre-check scheme and the data packet quality inspection scheme (that is, one special to-be-inspected data will correspond to one pre-check scheme and one quality inspection scheme). The pre-check scheme defines which pre-checks need to be performed on the to-be-inspected data. When the pre-checks of the to-be-inspected data pass, the subsequent quality inspection process will be performed.

[0058] The pre-checks include data format correctness checking, table space and attribute table connection testing, data space coordinate system checking and the like of the to-be-inspected data. The quality inspection scheme defines a quality inspection factor sequence, that is, which factors of the to-be-inspected data are to be inspected.

[0059] The quality inspection factor sequence includes but is not limited to layer attribute field type / length / decimal place checking, element code consistency, numerical range compliance, number uniqueness, field mandatory item, layer internal attribute consistency, layer attribute consistency, spatial graph topological relationship, broken line checking, broken surface checking, geometry checking, table integrity, table data structure consistency, table data code consistency, table data range compliance, field mandatory item and the like.

[0060] According to the quality inspection factor sequence defined by the quality inspection scheme, the quality inspection of multiple quality inspection factors of the to-be-inspected data is performed in parallel through multiple processes to obtain quality inspection result data.

[0061] Step S3: When the task quality inspection is successful, the quality inspection result data is stored in the sub-node result database, and the quality inspection result data in all sub-node result databases is aggregated into the master node result database.

[0062] It can be understood that after the quality inspection of each group of to-be-inspected data is completed on each sub-node, the quality inspection result data after the quality inspection is passed is stored in the result database of the sub-node, and then the quality inspection result data in the result database of each sub-node is aggregated into the result database of the master node to form a master-slave architecture.

[0063] Step S4: When map display is needed, the quality inspection result data is extracted from the master node result database and rendered and displayed according to the corresponding rendering type.

[0064] It can be understood that in the field of map display application, when these data need to be rendered into map display, the data after the quality inspection needs to be rendered. In an embodiment of the present application, step S4: when map display is needed, the quality inspection result data is extracted from the master node result database and rendered and displayed according to the corresponding rendering type, which includes:

[0065] When a map display is needed, the quality inspection result data is extracted from the master node result library, and according to the data condition of the quality inspection result data, a corresponding rendering type is determined, the rendering type including GIS service rendering, java service rendering and WEB front-end rendering.

[0066] Based on the rendering type, the quality inspection result data is rendered and displayed.

[0067] Among them, according to the data condition of the quality inspection result data, the corresponding rendering type is determined, including:

[0068] When the data amount of the quality inspection result data exceeds the preset data amount, GIS service rendering is adopted;

[0069] When the data amount of the quality inspection result data does not exceed the preset data amount, java service rendering is adopted;

[0070] For the quality inspection result data that does not need to be stored in the warehouse, WEB front-end rendering is adopted.

[0071] Referring to Figure 5 , a flowchart for different types of rendering of quality inspection result data is shown, mainly including GIS service rendering, java service rendering and WEB front-end rendering.

[0072] Specifically, when it is GIS service rendering, the rendering and display of the quality inspection result data based on the rendering type includes:

[0073] Based on the rendering request, the corresponding quality inspection result data is obtained as the GIS geographic data to be rendered;

[0074] The WEBGL tile rendering is performed on the GIS geographic data to be rendered;

[0075] When it is java service rendering, the rendering and display of the quality inspection result data based on the rendering type includes:

[0076] According to the rendering request, the corresponding quality inspection result data is obtained as the data to be rendered;

[0077] The vectorization and pooling are performed on the data to be rendered, and the tile data after pooling is extracted for WEBGL tile rendering;

[0078] When it is WEB front-end rendering, the rendering and display of the quality inspection result data based on the rendering type includes:

[0079] According to the rendering request, the corresponding quality inspection result data is obtained as the data to be rendered;

[0080] The vectorization and hierarchical gridding are performed on the data to be rendered, and the hierarchical tiles are cached;

[0081] extracting the layered tiles, and rendering the layered tiles by WEBGL vector data.

[0082] It can be understood that the rendering type depends on the data, for example: when the collective quantity of single quality inspection data is greater than 200,000 nodes, GIS service rendering is used, wherein the GIS service is a rendering mode depending on the GIS platform capability to quickly feedback the data, such as feedback in the form of an image or a binary format, and more depends on the image processing capability of the server.

[0083] When the number of nodes is less than 200,000, Java service rendering can be used, wherein the Java service is mainly used for data optimization processing, such as sparse node number of small scale rendering, and fine scale needs to be fine, and more depends on the CPU processing performance of the server.

[0084] For data that does not need to be stored in the warehouse (i.e., process data), Web front-end rendering is used, wherein the front-end Shader shader is used to draw and color each node position color, and more depends on the GPU performance of the client

[0085] According to the data quality inspection condition, the rendering type is determined, the quality inspection result data is rendered and displayed based on the rendering type, and the map data display is realized.

[0086] It can be understood that if the data needs to be displayed on the map, the corresponding quality inspection result data is extracted from the master node database, and the quality inspection result data is rendered by different types of rendering methods to realize the drawing and display of the map.

[0087] Referring to Figure 6 , another embodiment of the data quality inspection system based on a master node and multiple sub-nodes is provided, and the system comprises:

[0088] The division module 601 is configured to group and divide the to-be-inspected data to obtain multiple groups of to-be-inspected data, and send each group of to-be-inspected data to a corresponding sub-node.

[0089] The judgment module 602 is configured to judge whether the computing resources on the sub-node are sufficient according to the data quantity of the to-be-inspected data on each sub-node, and if sufficient, execute task scheduling and task execution; if not sufficient, wait or perform resource regulation, and then execute task scheduling and task execution.

[0090] The storage module 603 is configured to store the quality inspection result data in the sub-node result database when the task quality inspection is successful, and gather the quality inspection result data in all sub-node result databases to the master node result database.

[0091] The rendering module 604 is configured to extract the quality inspection result data from the master node result library when map display is required, and perform rendering display according to a corresponding rendering type.

[0092] It can be understood that the data quality inspection system based on a master and multiple sub-nodes provided by the present application corresponds to the data quality inspection method based on a master and multiple sub-nodes provided by the above-mentioned embodiments, and the related technical features of the data quality inspection system based on a master and multiple sub-nodes can refer to the related technical features of the data quality inspection method based on a master and multiple sub-nodes, which will not be described here again.

[0093] The data quality inspection method and system based on a master and multiple sub-nodes provided by the embodiments of the present application divide the data to be inspected into multiple groups, send each group of data to be inspected to a corresponding sub-node, judge whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, execute task scheduling and task execution if the computing resources are sufficient, execute task scheduling and task execution again after waiting or resource regulation if the computing resources are insufficient, and aggregate the quality inspection result data into the master node. When map display is required, the corresponding quality inspection result data is extracted, and rendering display is performed according to the corresponding rendering type. The data to be inspected is divided into groups, and the quality inspection work is performed in parallel on multiple sub-nodes. The data quality inspection method based on a master and multiple sub-nodes improves the data quality inspection efficiency, and the data that passes the quality inspection is used for rendering to realize map display, so that the displayed map is more accurate.

[0094] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0095] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0096] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0097] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0099] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to cover all such additional variations and modifications as falling within the scope of the application.

[0100] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data quality inspection method based on one master and multiple slave nodes, characterized in that, include: Step S1: Divide the data to be inspected into groups to obtain multiple groups of data to be inspected, and send each group of data to be inspected to the corresponding sub-node. Step S2: Based on the amount of data to be inspected on each sub-node, determine whether the computing resources on the sub-node are sufficient. If sufficient, execute task scheduling and task execution. If there is not enough, then wait or adjust resources before executing task scheduling and task execution. Step S3: After the task quality inspection is successful, the quality inspection result data is stored in the sub-node result database, and the quality inspection result data in all sub-node result databases is aggregated into the main node result database. Step S4: When map display is required, extract the quality inspection result data from the main node result library and render and display it according to the corresponding rendering type. Step S4, when map display is required, involves extracting quality inspection result data from the main node result library and rendering and displaying it according to the corresponding rendering type, including: When map display is required, quality inspection result data is extracted from the main node result library. Based on the data of the quality inspection result data, the corresponding rendering type is determined. The rendering type includes GIS service rendering, Java service rendering, and WEB front-end rendering. The quality inspection result data is rendered and displayed based on the rendering type; The step of determining the corresponding rendering type based on the data of the quality inspection results includes: If the amount of the quality inspection result data exceeds the preset amount, GIS service rendering will be used. If the amount of the quality inspection result data does not exceed the preset amount of data, Java service rendering will be used. For the quality inspection result data that does not need to be stored in the database, web front-end rendering is used.

2. The data quality inspection method according to claim 1, characterized in that, Step S1 involves grouping the data to be inspected into multiple groups, including: Based on the district, organization, and spatial location of the user who uploaded the data to be inspected, all user-uploaded data to be inspected are clustered and grouped to obtain multiple groups of data to be inspected. The step of sending each set of data to be inspected to the corresponding sub-node includes: Based on the physical location of each set of data to be inspected, obtain multiple sub-nodes within the vicinity of each set of data to be inspected. Assign a sequence number w to each group of data to be inspected, and generate the corresponding hash value based on the sequence number w; Based on the hash value corresponding to each group of data to be inspected, each group of data to be inspected is assigned to a corresponding sub-node. The correspondence between the hash value of the data to be inspected and the sub-node number is established in advance.

3. The data quality inspection method according to claim 2, characterized in that, Step S2, based on the amount of data to be inspected on each sub-node, determines whether the computing resources on the sub-node are sufficient, including: Calculate the maximum number of processes that a node can currently support, based on the number of cores and the currently available memory. Based on the number of processes required for the data to be inspected on the sub-node and the maximum number of processes that the sub-node can currently support, determine whether the computing resources on the sub-node are sufficient.

4. The data quality inspection method according to claim 3, characterized in that, The calculation of the maximum number of processes that a sub-node can currently support based on the number of cores and the currently available memory includes: P max = MIN(P cpu ,P mem ); Among them, P cpu P represents the maximum number of processes that a node's CPU can support. mem The maximum number of processes that the available memory of a node can support; P cpu = ； Among them, CPU 核数 For the number of cores in the distributed server, CPU 安全阈值 To preset a safety threshold, the CPU per This represents the number of CPU cores used by a single process. P mem = ； Among them, MEM 内存 MEM is the available memory for each node. 缓冲内存 To preset the buffer memory ratio, MEM per The memory required for a single process; The step of determining whether the computing resources on a branch node are sufficient based on the number of processes required for the data to be inspected on the branch node and the maximum number of processes that the branch node can currently support includes: When P task < P max If the computing resources on the sub-nodes are sufficient, the quality inspection task will start normally and enter the quality inspection process; if P task ≥ P max If the computing resources on a sub-node are insufficient, the quality inspection task will be placed in a waiting queue and allowed to enter the quality inspection process after the sub-node resources are released, or the sub-node resources will be reallocated to the quality inspection task before it enters the quality inspection process.

5. The data quality inspection method according to claim 4, characterized in that, When P task ≥ P max If the computing resources on a sub-node are insufficient, the quality inspection task will be placed in a waiting queue until the sub-node resources are released before entering the quality inspection process. Alternatively, sub-node resources can be reallocated to the quality inspection task before it enters the quality inspection process. This includes: When the computing resources on the sub-node are insufficient, the urgency of the quality inspection task is obtained. If the urgency is less than the preset urgency, the quality inspection task is placed in the waiting queue and waits for the sub-node resources to be released before entering the quality inspection process. If the urgency level is greater than or equal to the preset urgency level, the quality inspection task will be reassigned to another sub-node with sufficient computing resources and the smallest distance value, and the quality inspection process will begin. Here, the distance value refers to the distance between the physical address of the user who uploaded the data to be inspected and the physical address of the sub-node.

6. The data quality inspection method according to claim 4, characterized in that, If the computing resources on the sub-node are sufficient, the quality inspection task will be started normally and the quality inspection process will begin, including: Create a quality inspection task, which includes a preliminary inspection plan, data to be inspected, and a quality inspection plan. According to the aforementioned pre-inspection plan, a pre-inspection is performed on the data to be inspected. After the pre-emptive check is passed, the quality inspection procedure is started, and the data to be inspected is inspected according to the quality inspection plan to obtain the quality inspection result data.

7. The data quality inspection method according to claim 6, characterized in that, The preliminary inspection plan includes data format correctness checks, tablespace and attribute library connection tests, and data space coordinate system checks. The quality inspection plan defines a sequence of quality inspection factors. The process of performing quality inspection on the data to be inspected according to the quality inspection plan and obtaining quality inspection result data includes: Based on the quality inspection factor sequence defined in the quality inspection scheme, multiple processes are used to perform quality inspections on multiple quality inspection factors of the data to be inspected in parallel to obtain the quality inspection result data.

8. The data quality inspection method according to claim 1, characterized in that, When it is GIS During service rendering, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is obtained as the GIS geographic data to be rendered based on the rendering request; Perform WEBGL tile rendering on the GIS geographic data to be rendered; When rendering for a Java service, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is obtained based on the rendering request and used as the data to be rendered. The data to be rendered is vectorized and pooled, and the pooled tile data is extracted for WEBGL tile rendering. When rendering for a web front-end, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is obtained based on the rendering request and used as the data to be rendered. The data to be rendered is vectorized and layered into a mesh, and the layered tiles are cached; Extract the layered tiles and render them using WEBGL vector data.

9. A data quality inspection system based on one master and multiple slave nodes, characterized in that, include: The partitioning module is used to group the data to be inspected into multiple groups, and send each group of data to the corresponding sub-node. The judgment module is used to determine whether the computing resources on each sub-node are sufficient based on the amount of data to be inspected on each sub-node. If sufficient, task scheduling and task execution are performed. If there is not enough, then wait or adjust resources before executing task scheduling and task execution. The storage module is used to store the quality inspection results in the sub-node result database after the task quality inspection is successful, and to aggregate the quality inspection results data in all sub-node result databases into the master node result database. The rendering module is used to extract quality inspection result data from the main node result library and render and display it according to the corresponding rendering type when map display is required. The rendering module is used to extract quality inspection result data from the master node result library when map display is required, and to render and display the data according to the corresponding rendering type, including: When map display is required, quality inspection result data is extracted from the main node result library. Based on the data of the quality inspection result data, the corresponding rendering type is determined. The rendering type includes GIS service rendering, Java service rendering, and WEB front-end rendering. The quality inspection result data is rendered and displayed based on the rendering type; The step of determining the corresponding rendering type based on the data of the quality inspection results includes: If the amount of the quality inspection result data exceeds the preset amount, GIS service rendering will be used. If the amount of the quality inspection result data does not exceed the preset amount of data, Java service rendering will be used. For the quality inspection result data that does not need to be stored in the database, web front-end rendering is used.

Images