Data quality inspection method and system based on one main node and multiple sub-nodes

Abstract

The invention provides a data quality inspection method and system based on one master and multiple partial nodes, and the method comprises the steps: dividing to-be-inspected data into multiple groups, and transmitting each group of to-be-inspected data to a corresponding partial node; according to the data volume of the data to be subjected to quality inspection on each partial node, judging whether computing resources on the partial nodes are sufficient or not, and if yes, executing task scheduling and task execution; if the quality inspection result data are not enough, waiting or resource regulation and control are carried out, then task scheduling and task execution are carried out, and quality inspection result data are gathered into a main node; and when map display needs to be carried out, extracting corresponding quality inspection result data, and carrying out rendering display according to the corresponding rendering type. According to the method, the data to be subjected to quality inspection is grouped, the quality inspection is performed on each group of data by using the plurality of sub-nodes, and the data quality inspection efficiency is improved based on a data quality inspection mode of one main node and multiple sub-nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data quality inspection, and more particularly, to a data quality inspection method and system based on a master and multiple sub-nodes. BACKGROUND

[0002] In the fields of artificial intelligence, big data, cloud computing, etc., the development of digital economy is actively promoted, and the accuracy and reliability of data in these fields are required to be extremely high. Therefore, effective data quality inspection means are needed to ensure the quality of data to support the healthy development of digital economy.

[0003] In the construction of new urbanization, the level of urban management and service is improved through big data, Internet of Things, etc., and in the construction of smart city, data quality inspection is the premise to ensure the accuracy and security of urban data.

[0004] In the current data quality inspection, there are differences in rule understanding and different implementation of quality inspection work, which will lead to quality problems of data results and make it more difficult to ensure the authenticity of data.

[0005] The current BS-end data quality inspection service is affected by network environment, bandwidth, and server CPU performance. In the case of large data results package, real-time data quality inspection will inevitably have the problems of slow response speed and poor running efficiency. SUMMARY

[0006] The present application provides a data quality inspection method and system based on a master and multiple sub-nodes to solve the technical problems in the prior art and improve the efficiency of data quality inspection.

[0007] According to a first aspect of the present application, a data quality inspection method based on a master and multiple sub-nodes is provided, comprising: Step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, and sending each group of data to be inspected to the corresponding sub-node; Step S2, judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, if sufficient, executing task scheduling and task execution, if not sufficient, waiting or resource control, and then executing task scheduling and task execution; Step S3, when the task quality inspection is successful, storing the quality inspection result data in the sub-node result database, and gathering the quality inspection result data in all sub-node result databases into the master node result database; Step S4, when map display is needed, extracting the quality inspection result data from the master node result database, and rendering and displaying according to the corresponding rendering type.

[0008] According to a second aspect of the present application, a data quality inspection system based on a master and multiple sub-nodes is provided, comprising: The dividing module is used for grouping and dividing the to-be-inspected data to obtain multiple groups of to-be-inspected data, and sending each group of to-be-inspected data to a corresponding sub-node; The judging module is used for judging whether the computing resources on the sub-node are sufficient according to the data amount of the to-be-inspected data on each sub-node, and if sufficient, performing task scheduling and task execution; if not sufficient, waiting or performing resource control, and then performing task scheduling and task execution again; The storing module is used for storing the inspection result data into the sub-node result database after the task inspection is successful, and gathering the inspection result data in all sub-node result databases into the master node result database; The rendering module is used for extracting the inspection result data from the master node result database when map display is needed, and performing rendering display according to the corresponding rendering type.

[0009] The data inspection method and system based on one master and multiple sub-nodes provided by the application divide the to-be-inspected data into multiple groups, send each group of to-be-inspected data to a corresponding sub-node, judge whether the computing resources on the sub-node are sufficient according to the data amount of the to-be-inspected data on each sub-node, perform task scheduling and task execution if sufficient, perform waiting or resource control if not sufficient, and then perform task scheduling and task execution again, gather the inspection result data into the master node, extract the corresponding inspection result data when map display is needed, and perform rendering display according to the corresponding rendering type. The to-be-inspected data is divided into groups, and the inspection work is performed in parallel on multiple sub-nodes, the data inspection method based on one master and multiple sub-nodes improves the data inspection efficiency, and the data that passes the inspection is used for rendering to realize map display, so that the displayed map is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A data inspection method flow chart based on one master and multiple sub-nodes is provided for an embodiment of the application; Figure 2 A data inspection request and a judging node resource schematic diagram; Figure 3 A task inspection scheduling and multi-channel message communication schematic diagram; Figure 4 A creating inspection task and task inspection schematic diagram; Figure 5 A rendering of inspection result data schematic diagram; Figure 6 A structure block diagram of a data inspection system based on one master and multiple sub-nodes is provided for another embodiment of the application. DETAILED DESCRIPTION

[0011] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. In addition, the technical features in each of the embodiments or in a single embodiment provided by the present application can be combined with each other at will to form a feasible technical solution, and such combination is not restricted by the sequence of steps and / or the mode of structural composition. However, it should be considered that when the combination of technical solutions contradicts each other or cannot be realized, such combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0012] Referring to Figure 1 , a data quality inspection method based on one master and multiple sub-nodes is provided, and mainly includes the following steps: Step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, and sending each group of data to be inspected to a corresponding sub-node.

[0013] In an embodiment of the present application, the step S1, grouping and dividing the data to be inspected to obtain multiple groups of data to be inspected, includes: According to the division, organization and spatial position of the uploading user of the data to be inspected, clustering and grouping all the data to be inspected uploaded by the user to obtain multiple groups of data to be inspected; The step of sending each group of data to be inspected to a corresponding sub-node includes: According to the physical position of each group of data to be inspected, obtaining multiple sub-nodes in the adjacent range of each group of data to be inspected; Assigning a serial number w to each group of data to be inspected, and generating a corresponding hash value according to the serial number w; According to the hash value corresponding to each group of data to be inspected, assigning each group of data to be inspected to a corresponding sub-node, wherein a corresponding relationship between the hash value of the data to be inspected and the serial number of the sub-node is established in advance.

[0014] It can be understood that when the master node receives each user uploaded data to be inspected, since the data volume of all data to be inspected is very large, it is necessary to group all data to be inspected. In the embodiment of the application, based on the geographical location of the user uploading the data to be inspected, including the division, organization and spatial location, the data to be inspected is clustered and grouped, the data to be inspected uploaded by users with close geographical location is divided into a group, a plurality of groups of data to be inspected are obtained, and each group of data to be inspected is distributed to the nearby sub-node for inspection.

[0015] Step S2, according to the data volume of the data to be inspected on each sub-node, it is judged whether the computing resources on the sub-node are sufficient, if sufficient, task scheduling and task inspection are performed; if not sufficient, waiting or resource regulation is performed, and then task scheduling and task execution are performed.

[0016] It can be understood that when the master node receives each user uploaded data to be inspected, since the data volume of all data to be inspected is very large, it is necessary to group all data to be inspected. In the embodiment of the application, based on the geographical location of the user uploading the data to be inspected, including the division, organization and spatial location, the data to be inspected is clustered and grouped, the data to be inspected uploaded by users with close geographical location is divided into a group, a plurality of groups of data to be inspected are obtained, and each group of data to be inspected is distributed to the nearby sub-node for inspection. Figure 2 When the sub-node data is uploaded and the data inspection is applied, according to the data volume of the data to be inspected, it is judged whether the current computing resources of the sub-node are sufficient, if yes, the scheduling and execution of the inspection task can be performed. However, if the current computing resources of the sub-node are not sufficient, the data inspection needs to be performed after the release of the sub-node resources, or the data to be inspected needs to be re-distributed to the sub-node resources for the inspection work.

[0017] It can be understood that after each group of data to be inspected is distributed to the corresponding sub-node, it is judged whether the current computing resources of the sub-node are sufficient, if the current computing resources of the sub-node are sufficient, the data to be inspected is inspected on the sub-node, if the current computing resources of the sub-node are not sufficient, the sub-node needs to be waited or re-distributed.

[0018] In an embodiment of the application, the step S2, according to the data volume of the data to be inspected on each sub-node, it is judged whether the computing resources on the sub-node are sufficient, includes: Step S21, according to the number of cores and the current available memory of the sub-node, the maximum process number that can be supported by the sub-node at present is calculated.

[0019] It can be understood that when it is judged whether the current computing resources of the sub-node are sufficient, the maximum process number that can be supported by the sub-node at present is calculated. Specifically, the maximum process number that can be supported by the sub-node at present is calculated according to the number of cores and the current available memory of the sub-node, including: P max = MIN(P cpu ,P mem ); Wherein, P cpu is the maximum process number that can be supported by the CPU of the sub-node, P memThe maximum number of processes that can be supported by the available memory of the sub-node; P cpu = ; CPU 核数 is the number of sub-node server cores, CPU 安全阈值 is a preset safety threshold, usually taken as 80%, and CPU per is the CPU core number occupied by a single process; P mem = ; MEM 内存 is the available memory of the sub-node, MEM 缓冲内存 is a preset buffer memory ratio, usually taken as 20%, and MEM per is the memory required by a single process.

[0020] In step S22, it is judged whether the computing resources on the sub-node are sufficient according to the number of processes required by the data to be inspected on the sub-node and the maximum number of processes that can be supported by the sub-node at present.

[0021] It can be understood that after the maximum number of processes that can be supported by the sub-node at present is calculated, it is judged according to the number of processes required by the data to be inspected whether the sub-node has sufficient computing resources to support the inspection work of the data to be inspected.

[0022] Specifically, when the number of processes P task required by the inspection work of the data to be inspected is less than P max , the computing resources on the sub-node are sufficient, and then the inspection task is normally started to enter the inspection process; when P task ≥ P max , the computing resources on the sub-node are insufficient, and then the inspection task is put into a waiting queue to wait for the release of the sub-node resources, and then enters the inspection process, or the sub-node resources are re-allocated to the inspection task, and then enters the inspection process.

[0023] When the computing resources on the sub-node are insufficient, the urgency of the inspection task is obtained, and it is determined according to the urgency of the inspection task whether the current inspection task is to wait or to re-allocate node resources. Specifically, if the urgency of the current inspection task is less than a preset urgency, the inspection task is put into a waiting queue to wait for the release of the sub-node resources, and then enters the inspection process. If the urgency of the current inspection task is greater than or equal to the preset urgency, the inspection task is re-allocated to other sub-nodes with sufficient computing resources and the smallest distance value, and then enters the inspection process, wherein the distance value refers to the distance between the physical address of the user uploading the data to be inspected and the physical address of the sub-node.

[0024] When the computing resource on the sub-node is sufficient, the quality inspection task of the to-be-inspected data is started to perform the quality inspection process. Specifically, refer to Figure 2 When the resource of the sub-node is sufficient, the quality inspection task of the to-be-inspected data is started, the quality inspection task is put into a quality inspection queue, and communication with a quality inspection program is started to enter the quality inspection process of the to-be-inspected data through the quality inspection program.

[0025] In an embodiment of the present application, when the computing resource on the sub-node is sufficient, the quality inspection task is normally started to enter the quality inspection process, including: A quality inspection task is created, and the quality inspection task includes a pre-checking scheme, to-be-inspected data and a quality inspection scheme; The to-be-inspected data is pre-checked according to the pre-checking scheme; After the pre-checking passes, a quality inspection program is started, the to-be-inspected data is quality inspected according to the quality inspection scheme, and quality inspection result data is obtained.

[0026] Refer to Figure 3 When the computing resource on the sub-node is sufficient, the quality inspection task is scheduled, the quality inspection resource is reasonably allocated, and the smooth and orderly performance of the quality inspection task is ensured. Meanwhile, the module is also responsible for establishing and maintaining a service end of a multi-channel quality inspection message channel processing program, maintaining the message intercommunication between the communication channel client of the quality inspection rendering module and the quality inspection processing program. The reasonable allocation of the resource includes the management of the computing node, the architecture design of one master and multiple sub-nodes, etc.

[0027] Then, a multi-channel quality inspection task communication program is started, a quality inspection program is started to enter the data quality inspection process.

[0028] Refer to Figure 4 For a single quality inspection task, the quality inspection task includes a pre-checking scheme, to-be-inspected data and a quality inspection scheme, and rule matching is performed on the to-be-inspected data, the pre-checking scheme and the data packet quality inspection scheme (that is, one special to-be-inspected data will have one pre-checking scheme and one quality inspection scheme corresponding thereto). In the pre-checking scheme, it is defined which pre-checking needs to be performed on the to-be-inspected data. When the pre-checking of the to-be-inspected data passes, the subsequent quality inspection process is performed.

[0029] The pre-checking includes data format correctness checking of the to-be-inspected data, table space and attribute table connection testing, data space coordinate system checking, etc. In the data quality inspection scheme, a quality inspection factor sequence is defined, that is, which factors of the to-be-inspected data are quality inspected.

[0030] The quality inspection factor sequence includes but is not limited to: layer attribute field type / length / decimal place inspection, element code consistency, numerical range compliance, number uniqueness, field mandatory item, intra-layer attribute consistency, inter-layer attribute consistency, spatial graph topological relationship, broken line inspection, broken surface inspection, geometry inspection, table integrity, table data structure consistency, table data code consistency, table data range compliance, field mandatory item, and the like.

[0031] According to the quality inspection factor sequence defined by the quality inspection scheme, the quality inspection of multiple quality inspection factors of the to-be-inspected data is performed in parallel through multiple processes, and quality inspection result data is obtained.

[0032] In step S3, when the task quality inspection is successful, the quality inspection result data is stored in the sub-node result database, and the quality inspection result data in all sub-node result databases is aggregated into the master node result database.

[0033] It can be understood that after the quality inspection of each group of to-be-inspected data is completed on each sub-node, the quality inspection result data after the quality inspection is stored in the result database of the sub-node, and then the quality inspection result data in the result database of each sub-node is aggregated into the result database of the master node to form a master-slave architecture.

[0034] In step S4, when map display is needed, the quality inspection result data is extracted from the master node result database, and is rendered and displayed according to the corresponding rendering type.

[0035] It can be understood that in the field of map display application, when these data need to be rendered into map display, the data after the quality inspection need to be rendered. In an embodiment of the present application, in step S4, when map display is needed, the quality inspection result data is extracted from the master node result database, and is rendered and displayed according to the corresponding rendering type, including: When map display is needed, the quality inspection result data is extracted from the master node result database, and the corresponding rendering type is determined according to the data condition of the quality inspection result data, wherein the rendering type includes GIS service rendering, java service rendering, and WEB front-end rendering. The quality inspection result data is rendered and displayed based on the rendering type.

[0036] The corresponding rendering type is determined according to the data condition of the quality inspection result data, including: When the data amount of the quality inspection result data exceeds a preset data amount, GIS service rendering is adopted; When the data amount of the quality inspection result data does not exceed the preset data amount, java service rendering is adopted; For the quality inspection result data that does not need to be stored in the database, WEB front-end rendering is adopted.

[0037] Referring to Figure 5 , a flowchart for different types of rendering of quality inspection result data is shown, mainly including GIS service rendering, java service rendering and WEB front-end rendering.

[0038] Specifically, when rendered as a GIS service, the rendering and display of the quality inspection result data based on the rendering type includes: obtaining corresponding quality inspection result data as GIS geographic data to be rendered based on a rendering request; WEBGL tile rendering is performed on the GIS geographic data to be rendered; When rendered as a java service, the rendering and display of the quality inspection result data based on the rendering type includes: obtaining corresponding quality inspection result data as data to be rendered according to a rendering request; vectorization and pooling are performed on the data to be rendered, and tile data after pooling is extracted for WEBGL tile rendering; When rendered as a WEB front-end, the rendering and display of the quality inspection result data based on the rendering type includes: obtaining corresponding quality inspection result data as data to be rendered according to a rendering request; vectorization and hierarchical gridding are performed on the data to be rendered, and hierarchical tiles are cached; extracting hierarchical tiles and performing WEBGL vector data rendering on the hierarchical tiles.

[0039] It can be understood that the rendering type depends on the data, for example: when the collective volume of a single quality inspection data is more than 200,000 nodes, GIS service rendering is used. Here, GIS service is a rendering method that relies on the GIS platform capability to quickly feedback the data, such as in image or binary format, which relies more on the server's image processing capability.

[0040] When there are less than 200,000 nodes, Java service rendering can be used. Here, it mainly relies on Java service to do some data optimization processing, such as sparse nodes for small-scale rendering, and fine for large-scale, which relies more on the server CPU processing performance.

[0041] For data that does not need to be stored in the database (i.e. process data), WEB front-end rendering is used. Here, the front-end Shader shader is used to draw and color each node position color, which relies more on the client GPU performance According to the data quality inspection, the rendering type is determined, the quality inspection result data is rendered and displayed based on the rendering type, and the map data is displayed.

[0042] It can be understood that if map display of data is required, corresponding quality inspection result data is extracted from the master node database, and the quality inspection result data is rendered through different types of rendering to realize drawing and display of the map.

[0043] Referring to Figure 6 Another embodiment of the application provides a data quality inspection system based on a master and multiple sub-nodes, which comprises: The division module 601 is configured to group the data to be inspected to obtain multiple groups of data to be inspected, and send each group of data to be inspected to a corresponding sub-node. The judgment module 602 is configured to judge whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, and if so, execute task scheduling and task execution; if not, wait or perform resource control, and then execute task scheduling and task execution. The storage module 603 is configured to store the quality inspection result data in the sub-node result database when the task quality inspection is successful, and aggregate the quality inspection result data in all sub-node result databases into the master node result database. The rendering module 604 is configured to extract the quality inspection result data from the master node result database when map display is required, and render and display according to the corresponding rendering type.

[0044] It can be understood that the data quality inspection system based on a master and multiple sub-nodes provided by the application corresponds to the data quality inspection method based on a master and multiple sub-nodes provided by the above-mentioned embodiments, and the related technical features of the data quality inspection system based on a master and multiple sub-nodes can refer to the related technical features of the data quality inspection method based on a master and multiple sub-nodes, which will not be described here.

[0045] The data quality inspection method and system based on a master and multiple sub-nodes provided by the embodiments of the application divide the data to be inspected into multiple groups, send each group of data to be inspected to a corresponding sub-node, judge whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, execute task scheduling and task execution if so, wait or perform resource control if not, and then execute task scheduling and task execution, aggregate the quality inspection result data into the master node, extract the corresponding quality inspection result data when map display is required, and render and display according to the corresponding rendering type. The data to be inspected is divided into groups, and the quality inspection work is performed in parallel on multiple sub-nodes, the data quality inspection method based on a master and multiple sub-nodes improves the data quality inspection efficiency, and the data that passes the quality inspection is used for rendering to realize map display, so that the displayed map is more accurate.

[0046] It should be noted that the descriptions of the various embodiments are each given with emphasis on certain features of the embodiments. The descriptions of the various embodiments are not meant to be taken literally.

[0047] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0048] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0049] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0051] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0052] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended that the present application cover all such changes and modifications that are within its scope.

Claims

1. A method for data quality inspection based on a master multi-node, characterized in that, The method comprises the following steps: S1, grouping the data to be inspected to obtain multiple groups of data to be inspected, and sending each group of data to be inspected to a corresponding sub-node; S2, judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, if sufficient, executing task scheduling and task execution; if not sufficient, waiting or resource regulation, and then executing task scheduling and task execution; S3, when the task inspection is successful, storing the inspection result data in the sub-node result database, and gathering the inspection result data in all sub-node result databases into the main node result database; S4, when map display is needed, extracting the inspection result data from the main node result database, and rendering and displaying according to the corresponding rendering type.

2. The data quality inspection method of claim 1, wherein, In the step S1, the data to be inspected is grouped to obtain multiple groups of data to be inspected, which comprises: According to the division, organization and spatial position of the uploading user of the data to be inspected, clustering and grouping all user-uploaded data to be inspected to obtain multiple groups of data to be inspected; In the step of sending each group of data to be inspected to a corresponding sub-node, it comprises: According to the physical position of each group of data to be inspected, obtaining multiple sub-nodes within the adjacent range of each group of data to be inspected; Assigning a serial number w to each group of data to be inspected, and generating a corresponding hash value according to the serial number w; According to the hash value corresponding to each group of data to be inspected, assigning each group of data to be inspected to a corresponding sub-node, wherein the correspondence between the hash value of the data to be inspected and the serial number of the sub-node is established in advance.

3. The data quality inspection method of claim 2, wherein, In the step S2, judging whether the computing resources on the sub-node are sufficient according to the data volume of the data to be inspected on each sub-node, which comprises: According to the core number and the current available memory of the sub-node, calculating the maximum process number that the sub-node can support at present; According to the process number required by the data to be inspected on the sub-node and the maximum process number that the sub-node can support at present, judging whether the computing resources on the sub-node are sufficient.

4. The data quality inspection method of claim 3, wherein, According to the core number and the current available memory of the sub-node, calculating the maximum process number that the sub-node can support at present, which comprises: P max = MIN(P cpu ,P mem ) Wherein, P cpu is the maximum number of processes that the CPU of the sub-node can support, P mem is the maximum number of processes that the available memory of the sub-node can support; P cpu = ； CPU 核数 CPU 安全阈值 CPU per CPU P mem = ； Wherein, MEM 内存 is the available memory of the node, MEM 缓冲内存 is the preset buffer memory ratio, MEM per is the memory required by a single process; According to the process number required by the data to be inspected on the sub-node and the maximum process number that the sub-node can support at present, judging whether the computing resources on the sub-node are sufficient, which comprises: When P task < P max , the computing resources on the sub-node are sufficient, and the quality inspection task is normally started to enter the quality inspection process; when P task ≥ P max , the computing resources on the sub-node are insufficient, and the quality inspection task is put into a waiting queue to wait for the release of the sub-node resources, and then enters the quality inspection process, or the sub-node resources are re-assigned to the quality inspection task, and then enters the quality inspection process.

5. The data quality inspection method of claim 4, wherein, The when P task ≥ P max When the computing resources on the sub-node are insufficient, the quality inspection task is put into a waiting queue to wait for the release of the sub-node resources, and then enters the quality inspection process, or the sub-node resources are re-assigned to the quality inspection task, and then enters the quality inspection process, including: When the computing resources on the sub-node are not sufficient, obtaining the urgency of the inspection task, if the urgency is less than the preset urgency, putting the inspection task into the waiting queue to wait for the release of the sub-node resources, and then entering the inspection process; If the urgency is greater than or equal to the preset urgency, reassigning the inspection task to other sub-nodes with sufficient computing resources and the minimum distance value, and entering the inspection process, wherein the distance value refers to the distance between the physical address of the user uploading the data to be inspected and the physical address of the sub-node.

6. The data quality inspection method of claim 4, wherein, When the computing resources on the sub-node are sufficient, normally starting the inspection task to enter the inspection process, which comprises: Creating an inspection task, wherein the inspection task comprises a pre-check scheme, data to be inspected and an inspection scheme; According to the pre-check scheme, pre-checking the data to be inspected; After the pre-examination is passed, a quality inspection procedure is started, the quality inspection is performed on the to-be-inspected data according to the quality inspection scheme, and quality inspection result data is acquired.

7. The data quality inspection method of claim 5, wherein, The pre-examination scheme includes data format correctness inspection, table space and attribute library connection test and data space coordinate system inspection, the quality inspection scheme defines a quality inspection factor sequence, and the quality inspection is performed on the to-be-inspected data according to the quality inspection scheme, and quality inspection result data is acquired, including: According to the quality inspection factor sequence defined by the quality inspection scheme, the quality inspection of multiple quality inspection factors of the to-be-inspected data is performed in parallel through multiple processes, and quality inspection result data is acquired.

8. The data quality inspection method of claim 1, wherein, In step S4, when map display is needed, the quality inspection result data is extracted from the master node result database, and is rendered and displayed according to the corresponding rendering type, including: When map display is needed, the quality inspection result data is extracted from the master node result database, and the corresponding rendering type is determined according to the data condition of the quality inspection result data, and the rendering type includes GIS service rendering, java service rendering and WEB front-end rendering; Based on the rendering type, the quality inspection result data is rendered and displayed; The corresponding rendering type is determined according to the data condition of the quality inspection result data, including: When the data amount of the quality inspection result data exceeds the preset data amount, GIS service rendering is adopted; When the data amount of the quality inspection result data does not exceed the preset data amount, java service rendering is adopted; For the quality inspection result data that does not need to be stored in the database, WEB front-end rendering is adopted.

9. The data quality inspection method of claim 8, wherein, When GIS service rendering is adopted, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is acquired as to-be-rendered GIS geographic data based on a rendering request; The to-be-rendered GIS geographic data is WEBGL tile rendered; When java service rendering is adopted, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is acquired as to-be-rendered data based on a rendering request; The to-be-rendered data is vectorized and pooled, and the pooled tile data is extracted for WEBGL tile rendering; When WEB front-end rendering is adopted, the rendering and display of the quality inspection result data based on the rendering type includes: The corresponding quality inspection result data is acquired as to-be-rendered data based on a rendering request; The to-be-rendered data is vectorized and hierarchical gridded, and hierarchical tiles are cached; The hierarchical tiles are extracted, and the hierarchical tiles are WEBGL vector data rendered.

10. A data quality inspection system based on a master multi-node, characterized in that, including: The division module is configured to group and divide the to-be-inspected data to obtain multiple groups of to-be-inspected data, and send each group of to-be-inspected data to a corresponding sub-node; The judgment module is configured to determine whether the computing resources on the sub-node are sufficient according to the data amount of the to-be-inspected data on each sub-node, and if sufficient, perform task scheduling and task execution; If not, wait or perform resource regulation, and then perform task scheduling and task execution; A storage module is configured to store the quality inspection result data into the sub-node result database when the task quality inspection is successful, and to gather the quality inspection result data in all the sub-node result databases into the master node result database; A rendering module is configured to extract the quality inspection result data from the master node result database when map display is needed, and to perform rendering display according to the corresponding rendering type.

Images