Data centralized processing method, system and equipment applied to government affair hall and medium

Abstract

The invention discloses a centralized data processing method, system and equipment applied to a government affair hall and a medium, belongs to the technical field of data processing, and aims to solve the technical problems of low efficiency, low accuracy and low expansibility of a traditional data processing method. According to the technical scheme, a distributed computing framework is built, wherein the distributed computing framework comprises a data interface, computing nodes and communication information; data receiving: receiving a data set to be processed through a data interface; data preprocessing: carrying out preprocessing operations of data cleaning, data conversion and data standardization on the received data set; data segmentation: segmenting the preprocessed data set into a plurality of data blocks, wherein each data block comprises a part of data; distributed calculation: distributing the data blocks segmented by the distributed framework member to a plurality of calculation nodes for parallel processing; processing data; combining results; and outputting data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, system, device and medium for centralized data processing applied to a government affairs hall. Background Art

[0002] With the rapid development of information technology, data has become an indispensable resource in modern society. Data processing technology is playing an increasingly prominent role in fields such as big data analysis, machine learning, and artificial intelligence. However, traditional data processing methods often suffer from inefficiency, low accuracy, and limited scalability when processing large datasets. Summary of the Invention

[0003] The technical task of the present invention is to provide a data centralized processing method, system, equipment and medium applied to government halls to solve the problems faced by traditional data processing methods such as low efficiency, low accuracy and poor scalability.

[0004] The technical task of the present invention is achieved in the following manner: a method for centralized data processing applied to a government affairs hall, the method being specifically as follows:

[0005] Build a distributed computing framework: The distributed computing framework includes a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the computing nodes are used to perform data processing tasks; and the communication protocol is used to ensure the correct transmission of data between nodes.

[0006] Data reception: receiving the data set to be processed through the data interface;

[0007] Data preprocessing: Perform data cleaning, data conversion, and data standardization on the received data sets to improve data quality;

[0008] Data segmentation: Split the preprocessed data set into multiple data blocks, each containing a portion of the data;

[0009] Distributed computing: Use the distributed framework to distribute the data blocks after segmentation to multiple computing nodes for parallel processing;

[0010] Data processing: Perform data processing tasks such as data aggregation, data analysis, and data mining on computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally;

[0011] Result merging: Collect and merge the processing results of each computing node to obtain the final processing result. The merging operation can be implemented through specific algorithms or tools to ensure the accuracy and consistency of the results.

[0012] Data output: The final processing results are output through the data interface for subsequent analysis or application.

[0013] Preferably, the data interface supports CSV, JSON and XML data formats.

[0014] Preferably, the number of data blocks is dynamically adjusted according to system resources and processing requirements.

[0015] Preferably, the computing nodes are physical machines, virtual machines or containers.

[0016] Preferably, the method also introduces an intelligent data filtering mechanism for screening and filtering data during data preprocessing operations to improve the accuracy and efficiency of data processing; wherein the intelligent data filtering mechanism is based on a machine learning algorithm, which automatically identifies and filters out invalid or redundant data by learning the characteristics and rules of the data set.

[0017] Better yet, the method also builds a scalable data processing architecture, which achieves flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation, ensuring that it can cope with various complex data processing needs and improve the efficiency and accuracy of data processing.

[0018] A data centralized processing system applied to a government affairs hall, the system comprising:

[0019] A distributed computing framework building module is used to build a distributed computing framework, including a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the data interface supports CSV, JSON, and XML data formats; computing nodes are used to perform data processing tasks; computing nodes use physical machines, virtual machines, or containers; and communication protocols are used to ensure the correct transmission of data between nodes.

[0020] A data receiving module, used for receiving a data set to be processed through a data interface;

[0021] The data preprocessing module is used to perform data cleaning, data conversion and data standardization preprocessing operations on the received data set to improve data quality;

[0022] The data segmentation module is used to segment the preprocessed data set into multiple data blocks, each of which contains a portion of the data; the number of data blocks is dynamically adjusted according to system resources and processing requirements;

[0023] Distributed computing module, used to distribute the data blocks divided by the distributed framework to multiple computing nodes for parallel processing;

[0024] The data processing module is used to perform data processing tasks such as data aggregation, data analysis, and data mining on the computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally;

[0025] The result merging module is used to collect and merge the processing results of each computing node to obtain the final processing result. The merging operation can be implemented through specific algorithms or tools to ensure the accuracy and consistency of the results;

[0026] The data output module is used to output the final processing results through the data interface for subsequent analysis or application.

[0027] Preferably, the system further comprises:

[0028] The intelligent data filtering mechanism module is used to screen and filter data during data preprocessing operations to improve the accuracy and efficiency of data processing. The intelligent data filtering mechanism is based on machine learning algorithms, which automatically identify and filter invalid or redundant data by learning the characteristics and patterns of data sets.

[0029] The scalable data processing architecture building module is used to achieve flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation, ensuring that it can cope with various complex data processing needs and improve the efficiency and accuracy of data processing. It can cope with various complex data processing needs and improve the efficiency and accuracy of data processing.

[0030] An electronic device comprising: a memory and at least one processor;

[0031] Wherein, the memory stores a computer program;

[0032] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the above-mentioned data centralized processing method applied to the government hall.

[0033] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned data centralized processing method applied to the government hall.

[0034] The data centralized processing method, system, device and medium applied to the government affairs hall of the present invention have the following advantages:

[0035] (1) The present invention achieves efficient, accurate, and scalable data processing by introducing a distributed computing framework and an intelligent data filtering mechanism. The implementation of the present invention is simple and easy to implement, and can be widely applied to data processing tasks in fields such as big data analysis, machine learning, and artificial intelligence. Furthermore, the present invention has the advantages of strong scalability and high degree of visualization, providing new ideas and methods for the development of data processing technology.

[0036] (2) The present invention improves data processing efficiency: by introducing a distributed computing framework and parallel processing technology, system resources can be fully utilized to achieve efficient data processing. Compared with traditional serial processing methods, the present invention has a faster processing speed and can obtain processing results more quickly.

[0037] (3) The present invention improves data processing accuracy: by introducing an intelligent data filtering mechanism, it can automatically identify and filter out invalid or redundant data, reduce the interference of erroneous data, and improve the accuracy of data processing. At the same time, the present invention also ensures the reliability and stability of data processing through the fault-tolerant mechanism of the distributed computing framework;

[0038] (4) The present invention improves scalability: The present invention provides a scalable data processing architecture that can flexibly respond to data sets of varying sizes and processing requirements by dynamically adjusting the number of computing nodes and resource allocation. This scalability enables the present invention to handle a variety of complex data processing scenarios and improves data processing efficiency and accuracy.

[0039] (5) The present invention also provides a visual display method for the results of data processing and analysis, which can be displayed in the form of charts, reports, etc., to help users better understand the process and results of data processing and improve the transparency and credibility of data processing;

[0040] (6) The present invention achieves efficient, accurate and scalable data processing by introducing a distributed computing framework and an intelligent data filtering mechanism, effectively solving the problems of low efficiency, low accuracy and poor scalability in existing data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] Attachment Figure 1 This is a flowchart of the data centralized processing method applied in the government hall. DETAILED DESCRIPTION

[0043] The data centralized processing method, system, equipment and medium applied to the government hall of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0044] Example 1:

[0045] As attached Figure 1 As shown, this embodiment provides a data centralized processing method applied to the government hall, and the method is specifically as follows:

[0046] S1. Build a distributed computing framework: The distributed computing framework includes a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the computing nodes are used to perform data processing tasks; and the communication protocol is used to ensure the correct transmission of data between nodes.

[0047] S2. Data reception: receiving the data set to be processed through the data interface;

[0048] S3. Data preprocessing: Perform data cleaning, data conversion, and data standardization on the received data set to improve data quality;

[0049] S4, data segmentation: split the preprocessed data set into multiple data blocks, each data block contains a part of the data;

[0050] S5, Distributed computing: Use the distributed framework to distribute the data blocks after segmentation to multiple computing nodes for parallel processing;

[0051] S6. Data processing: Perform data processing tasks such as data aggregation, data analysis, and data mining on the computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally;

[0052] S7. Result merging: Collect and merge the processing results of each computing node to obtain the final processing result. The merging operation can be implemented using a specific algorithm or tool to ensure the accuracy and consistency of the results.

[0053] S8. Data output: Output the final processing results through the data interface for subsequent analysis or application.

[0054] The data interface in this embodiment supports CSV, JSON and XML data formats.

[0055] In this embodiment, the number of data blocks is dynamically adjusted according to system resources and processing requirements.

[0056] The computing nodes in this embodiment are physical machines, virtual machines or containers.

[0057] This embodiment also introduces an intelligent data filtering mechanism for screening and filtering data during data preprocessing operations to improve the accuracy and efficiency of data processing; wherein, the intelligent data filtering mechanism is based on a machine learning algorithm, which automatically identifies and filters out invalid or redundant data by learning the characteristics and patterns of the data set.

[0058] This embodiment also builds a scalable data processing architecture, which achieves flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation, ensuring that it can cope with various complex data processing needs and improve the efficiency and accuracy of data processing.

[0059] Example 2:

[0060] This embodiment provides a data centralized processing system for a government affairs hall, the system comprising:

[0061] A distributed computing framework building module is used to build a distributed computing framework, including a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the data interface supports CSV, JSON, and XML data formats; computing nodes are used to perform data processing tasks; computing nodes use physical machines, virtual machines, or containers; and communication protocols are used to ensure the correct transmission of data between nodes.

[0062] A data receiving module, used for receiving a data set to be processed through a data interface;

[0063] The data preprocessing module is used to perform data cleaning, data conversion and data standardization preprocessing operations on the received data set to improve data quality;

[0064] The data segmentation module is used to segment the preprocessed data set into multiple data blocks, each of which contains a portion of the data; the number of data blocks is dynamically adjusted according to system resources and processing requirements;

[0065] Distributed computing module, used to distribute the data blocks divided by the distributed framework to multiple computing nodes for parallel processing;

[0066] The data processing module is used to perform data processing tasks such as data aggregation, data analysis, and data mining on the computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally;

[0067] The result merging module is used to collect and merge the processing results of each computing node to obtain the final processing result. The merging operation can be implemented through specific algorithms or tools to ensure the accuracy and consistency of the results;

[0068] The data output module is used to output the final processing results through the data interface for subsequent analysis or application.

[0069] This embodiment also includes:

[0070] The intelligent data filtering mechanism module is used to screen and filter data during data preprocessing operations to improve the accuracy and efficiency of data processing. The intelligent data filtering mechanism is based on machine learning algorithms, which automatically identify and filter invalid or redundant data by learning the characteristics and patterns of data sets.

[0071] The scalable data processing architecture building module is used to achieve flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation, ensuring that it can cope with various complex data processing needs and improve the efficiency and accuracy of data processing. It can cope with various complex data processing needs and improve the efficiency and accuracy of data processing.

[0072] Example 3:

[0073] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0074] wherein the memory stores computer-executable instructions;

[0075] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the data centralized processing method applied to the government hall in any embodiment of the present invention.

[0076] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0077] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0078] Example 4:

[0079] This embodiment further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the centralized data processing method for a government affairs hall according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0080] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0081] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0082] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0083] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data centralized processing method applied to a government affairs hall, characterized in that: The method is as follows: Build a distributed computing framework: The distributed computing framework includes a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the computing nodes are used to perform data processing tasks; and the communication protocol is used to ensure the correct transmission of data between nodes. Data reception: receiving the data set to be processed through the data interface; Data preprocessing: Perform data cleaning, data conversion, and data standardization preprocessing operations on the received data set; Data segmentation: Split the preprocessed data set into multiple data blocks, each containing a portion of the data; Distributed computing: The data blocks after segmentation are distributed to multiple computing nodes for parallel processing using the distributed framework; Data processing: Perform data processing tasks such as data aggregation, data analysis, and data mining on computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally; Result merging: Collect and merge the processing results of each computing node to obtain the final processing result; Data output: The final processing results are output through the data interface for subsequent analysis or application.

2. The data centralized processing method applied to the government hall according to claim 1 is characterized in that: The data interface supports CSV, JSON and XML data formats.

3. The data centralized processing method applied to the government hall according to claim 1 is characterized in that: The number of data blocks is dynamically adjusted based on system resources and processing requirements.

4. The data centralized processing method applied to the government hall according to claim 1 is characterized in that: Compute nodes use physical machines, virtual machines, or containers.

5. The data centralized processing method applied to the government hall according to claim 1 is characterized in that: The method also introduces an intelligent data filtering mechanism to screen and filter data during data preprocessing operations; the intelligent data filtering mechanism is based on a machine learning algorithm, which automatically identifies and filters out invalid or redundant data by learning the characteristics and patterns of the data set.

6. The method for centralized data processing applied to a government affairs hall according to any one of claims 1 to 5, characterized in that: This method also builds a scalable data processing architecture, which achieves flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation.

7. A data centralized processing system applied to government affairs hall, characterized in that: The system includes: A distributed computing framework building module is used to build a distributed computing framework, including a data interface, computing nodes, and a distributed computing framework for communication information. The data interface is used to receive and output data; the data interface supports CSV, JSON, and XML data formats; computing nodes are used to perform data processing tasks; computing nodes use physical machines, virtual machines, or containers; and communication protocols are used to ensure the correct transmission of data between nodes. A data receiving module, used for receiving a data set to be processed through a data interface; The data preprocessing module is used to perform data cleaning, data conversion and data standardization preprocessing operations on the received data set; The data segmentation module is used to segment the preprocessed data set into multiple data blocks, each of which contains a portion of the data; the number of data blocks is dynamically adjusted according to system resources and processing requirements; Distributed computing module, used to distribute the data blocks divided by the distributed framework to multiple computing nodes for parallel processing; The data processing module is used to perform data processing tasks such as data aggregation, data analysis, and data mining on the computing nodes, obtain processing results, and transmit the processing results to other computing nodes through communication protocols or store them locally; The result merging module is used to collect and merge the processing results of each computing node to obtain the final processing result; The data output module is used to output the final processing results through the data interface for subsequent analysis or application.

8. The data centralized processing system applied to the government hall according to claim 7 is characterized in that: The system also includes: An intelligent data filtering mechanism module is used to screen and filter data during data preprocessing operations. The intelligent data filtering mechanism is based on a machine learning algorithm that automatically identifies and filters out invalid or redundant data by learning the characteristics and patterns of the data set. A scalable data processing architecture building module that is used to achieve flexibility for data sets of different sizes by dynamically adjusting the number of computing nodes and resource allocation.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data centralized processing method applied to the government hall as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data centralized processing method applied to the government hall as described in any one of claims 1 to 6.

Images