Judicial data cleaning and reconstruction method based on pipeline filtering mode

Abstract

The invention provides a judicial data cleaning and reconstruction method based on a pipeline filtering mode, and relates to the technical field of data cleaning and reconstruction. Comprising a filter interface, an execution data query system and a data stream processing module UpdataFilter. A pipeline module; a pipeline builder; according to the judicial ten-million-level data processing method, pipeline design is combined with the asynchronous thread pool, and the judicial ten-million-level data processing simple and efficient method which is more visual in development, more convenient to maintain, clearer in logic, stable and efficient in data processing and free of large influence on main services is provided; the method is superior to the prior art in the aspects of data processing efficiency, flexibility, stability and resource consumption, and a more efficient, reliable and extensible solution is provided for judicial data cleaning and reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of data cleaning and reconstruction, and in particular relates to a judicial data cleaning and reconstruction method based on a pipeline filtering mode. Background Art

[0002] With the development of technology, business continues to grow, and data also grows exponentially. When the data reaches more than 10 million, data processing is no longer simple. In addition, many problems that exist when the business develops to a large scale cannot be anticipated in the early stage of table structure design. At this time, it is necessary to process the data of 10 million, such as data cleaning (removing unnecessary and useless data and retaining valid data), table splitting (rationally distributing a large amount of data in a large table, splitting horizontally or vertically), and table reconstruction (creating a new table to save valid data and improve query efficiency). There are many ways to do this, and you need to choose the best method to handle it according to the actual business situation and data validity. There are also many ways to implement these methods, such as relying on other third-party services MaxCompute or other big data processing software; the other is to implement it based on business logic. The general logic is to loop and batch retrieve the data to be processed, loop and process the retrieved data, and rebuild it in batches into a new table when reconstruction is needed.

[0003] At present, through third-party big data processing software: the limitations are too great. If the data is sensitive or the security requirements are high, it should not be exposed, and the functions of third-party big data processing software are basically fixed and not flexible enough; through loop customized development: it is difficult to implement, and it is necessary to formulate specific implementation plans based on the actual business operation and data output, data volume, control the loop conditions, eliminate the impact of data processing on the business, and complete the development according to the comprehensive plan. It takes a long time and the performance is not high. Therefore, technical personnel in this field provide a judicial data cleaning and reconstruction method based on the pipeline filtering mode to solve the above-mentioned technical problems. Summary of the invention

[0004] To solve the technical problem, the present invention adopts the following technical solution: including filter interface, execution data query system, data stream processing module UpdateFilter; pipeline module; pipeline builder; asynchronous request and business execution switch: data processing timer, the filter interface adopts Filter <t>A process method is provided to implement data cleaning processing under different filtering strategies. The data query execution system includes providing QueryFilter: implementing querying the target data record result set according to data entity mapping and query condition parameters; BaseFilter: implementing the remaining result set after the basic filtering conditions configured in the configuration file; UpdateFilter: implementing the filtering strategy required when the new table is rebuilt after data cleaning; the provided filter implementation can be used through simple configuration, and the Filter interface can be implemented by itself for more flexible business scenarios.

[0005] Preferably, the pipeline module comprises:

[0006] PipelineData: pipeline communication object, used to establish the default inter-pipeline communication data entity;

[0007] PipelineProcessor <t>The pipeline processor provides default process orchestration, integrates filter implementation sets, and integrates thread pools to speed up processing efficiency.

[0008] Preferably, the pipeline builder adopts PipelineBuilder, provides pipeline initialization and build method, which is used to initialize the specific business pipeline after rapid configuration, and provides pipeline initialization and build method, which is used to initialize the specific business pipeline after rapid configuration.

[0009] Preferably, the asynchronous request and service execution switch includes:

[0010] Asynchronous request: Provides an asynchronous interface for business logic compatibility switches to ensure that the compatibility switches can be quickly and dynamically modified when a failure occurs during online data processing;

[0011] Business execution switch: Provides an asynchronous interface for pipeline filter task execution switch, and flexibly turns on or off business processing according to the actual business situation.

[0012] Business paging and step control: Flexibly set the paging size and processing step according to business conditions to improve processing efficiency and stability.

[0013] Preferably, the data processing timer is used to start the overall time consumption of each asynchronous data processing, so as to observe the data processing process and rate more intuitively. By recording the processing time, it is possible to identify which data processing steps or operations consume the most resources (CPU, memory), thereby helping to optimize system configuration and resource allocation.

[0014] Preferably, the following processing steps are included:

[0015] S1. Data collection module: At the beginning, raw judicial data is collected from different data sources;

[0016] S2. Data preprocessing module: perform preliminary cleaning on the collected raw data, such as removing duplicate records, filling missing values, and unifying the format;

[0017] S3. Pipeline module: The pre-processed data is transmitted through the pipeline module, which acts as a transfer station for the data flow to ensure that the data can flow continuously to the next processing module;

[0018] S4. Filter module sequence: Data passes through multiple filter modules in sequence. Each filter module is responsible for a specific data processing task;

[0019] S5. Data verification: Check whether the data conforms to specific rules or formats;

[0020] S6.Data conversion: converting data into a unified data format or type;

[0021] S7. Outlier processing: Identify and process outlier data values;

[0022] S8. Data standardization: standardize the data, such as unifying the date format and normalizing the value range;

[0023] S9. Data reconstruction module: data processed by multiple filter modules enters the data reconstruction module and is reconstructed according to certain rules, such as time series reconstruction and relational database reconstruction;

[0024] S10. Data storage module: The reconstructed data is stored in the data storage module, which may be a database, a file system or other storage medium;

[0025] S11. End: The data cleaning and reconstruction process is completed.

[0026] Preferably, the data preprocessing step further includes: identifying and removing duplicate data records; reasonably filling in missing data; and unifying the data format into a standard format.

[0027] Preferably, the method adopts a modular design so that each processing module can be independently developed, tested and upgraded, thereby improving the maintainability and scalability of the system. The method improves the efficiency of data cleaning and reconstruction through automated processing, reduces manual intervention, and thus improves the quality and accuracy of data processing.

[0028] Preferably, the filter interface: implements a variety of cleaning strategies, such as data deduplication and missing value processing; the pipeline module: adopts a message queue mechanism to ensure the efficiency and reliability of data processing; the data reconstruction module: designs a data structured model according to the archive digitization standard to achieve efficient data storage and retrieval.

[0029] Preferably, the present invention is superior to the prior art in terms of data processing efficiency, flexibility, stability and resource consumption, and provides a more efficient, reliable and scalable solution for judicial data cleaning and reconstruction.

[0030] Preferably, through asynchronous request and service execution switch, the system stability is improved by at least 25% compared with the prior art.

[0031] Preferably, under the same data processing volume, the resource consumption of the present invention is reduced by at least 15% compared with the prior art.

[0032] Preferably, the asynchronous request and service execution switch allows for dynamic modification of compatible switches and service processing, thereby improving the stability and adaptability of the system.

[0033] Preferably, a data processing timer records the processing time to help identify steps with high resource consumption and optimize system configuration.

[0034] Preferably, PipelineBuilder provides fast configuration and initialization of business pipelines to simplify the system development process.

[0035] Beneficial effects:

[0036] Compared with the prior art, the beneficial effects are as follows:

[0037] The present invention combines pipeline design with an asynchronous thread pool to provide a simple and efficient method for processing tens of millions of judicial data with more intuitive development, more convenient maintenance, clearer logic, stable and efficient data processing, and no significant impact on the main business. The present invention is superior to the prior art in data processing efficiency, flexibility, stability and resource consumption, and provides a more efficient, reliable and scalable solution for judicial data cleaning and reconstruction, automates the processing flow, greatly improves the efficiency of archive digitization, and ensures quality: through strict cleaning and verification processes, data quality is ensured, and management is easy: digitized archives are easy to manage and retrieve, which improves judicial work efficiency. Through this embodiment, it can be seen that the application of the present invention in the digitization of judicial archives has significant practical effects, and provides strong support for the modernization of judicial archive management. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a basic flow chart of the present invention;

[0039] Figure 2 It is a basic framework schematic diagram of the present invention;

[0040] Figure 3 It is a schematic diagram of the basic steps of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. Any modification and improvement made without changing the inventive concept belongs to the protection scope of the present invention.

[0042] Embodiment 1:

[0043] This embodiment will describe in detail the practical application of the judicial data cleaning and reconstruction method based on the pipeline filtering mode, including system architecture, data processing flow and key technology implementation.

[0044] System Architecture This system adopts modular design and mainly includes the following modules:

[0045] Filter interface: using Filter <t>Interface, providing process method to implement data cleaning processing under different filtering strategies.

[0046] Execute data query system: including QueryFilter, BaseFilter and UpdateFilter, which respectively implement data entity mapping, basic filtering and filtering strategies after data cleaning.

[0047] Pipeline module: including PipelineData and PipelineProcessor <t>Used to establish inter-pipeline communication data entities and provide default process orchestration.

[0048] Pipeline Builder: Use PipelineBuilder to quickly configure and initialize business pipelines.

[0049] Asynchronous request and business execution switch: includes asynchronous request and business execution switch, which are used to dynamically modify compatible switches and flexibly open or close business processing.

[0050] Data processing timer: used to record the overall time consumed by asynchronous data processing to help optimize system configuration and resource allocation.

[0051] Data processing flow,Data collection module (S1): collects original judicial data from different data sources.

[0052] Data preprocessing module (S2): performs preliminary cleaning on the collected raw data, including removing duplicate records, filling missing values, and unifying the format.

[0053] Pipeline module (S3): The preprocessed data is transmitted through the pipeline module to ensure that the data flows continuously to the next processing module.

[0054] Filter module sequence (S4): The data passes through multiple filter modules in sequence, such as data verification, data conversion, outlier processing, and data standardization.

[0055] Data reconstruction module (S9): The data processed by the filter module enters the data reconstruction module and is reconstructed according to the time series reconstruction or relational database reconstruction rules.

[0056] Data storage module (S10): The reconstructed data is stored in a data storage system, such as a database or a file system.

[0057] End (S11): The data cleaning and reconstruction process is completed.

[0058] Key technology implementation, filter interface implementation: by implementing Filter <t>Interface, define process methods, and implement different data cleaning strategies according to business needs.

[0059] Pipeline module implementation: using PipelineData and PipelineProcessor <t>Establish inter-pipeline communication data entities, provide default process orchestration and thread pool, and improve processing efficiency.

[0060] Pipeline Builder Implementation: Use PipelineBuilder to quickly configure and initialize business pipelines and simplify the system development process.

[0061] Asynchronous request and business execution switch implementation: Through asynchronous request and business execution switch, dynamic modification of compatible switches and flexible opening or closing of business processing can be achieved.

[0062] Data processing timer implementation: records the overall time consumption of asynchronous data processing to help optimize system configuration and resource allocation.

[0063] Through the above embodiments, the present invention provides a judicial data cleaning and reconstruction method based on the pipeline filtering mode, which has the following advantages: automated processing to improve data processing efficiency and quality, modular design to improve system maintainability and scalability, flexible filter interface to meet the needs of different business scenarios, asynchronous request and business execution switch to ensure the stability and reliability of the data processing process.

[0064] Embodiment 2: Compared with the prior art, the judicial data cleaning and reconstruction method based on pipeline filtering mode proposed in the present invention has the following significant advantages and improvements:

[0065] Overview of existing technologies,Traditional data processing methods: usually rely on manual operations, such as data cleaning, data conversion, and data verification, which are inefficient, prone to errors, and difficult to adapt to the processing of large amounts of data.

[0066] Existing data cleaning tools: Although some commercial or open source tools provide automated data cleaning functions, these tools often lack flexibility and are difficult to adapt to the needs of specific business scenarios.

[0067] Existing data reconstruction methods: Existing methods usually use fixed rules and patterns in data reconstruction, which makes it difficult to handle complex data relationships and dynamically changing data structures.

[0068] The present invention is compared with the prior art. <t>):

[0069] Existing technology: There is a lack of a unified data cleaning interface, and it is often necessary to write specific cleaning logic for different data types.

[0070] The present invention provides a universal filter <t>The interface implements multiple filtering strategies through the process method, which improves the flexibility and scalability of data cleaning.

[0071] Execute data query system:

[0072] Existing technology: Query systems usually only provide basic query functions and lack support for data entity mapping and complex query conditions.

[0073] The present invention: QueryFilter realizes data entity mapping, BaseFilter provides basic filtering and screening conditions, and UpdateFilter supports the reconstruction filtering strategy after data cleaning, thereby enhancing the function of the query system.

[0074] Pipeline Module:

[0075] Existing technology: Lack of effective data flow management during data processing leads to data processing interruption or inefficiency.

[0076] Invention: PipelineData and PipelineProcessor <t>Provides data flow management and thread pool to ensure continuous data processing and improve efficiency.

[0077] PipelineBuilder:

[0078] Existing technologies: Lack of tools for quickly configuring and initializing pipelines, resulting in a long development cycle.

[0079] The present invention: PipelineBuilder provides fast configuration and initialization of business pipelines, simplifying the system development process.

[0080] Asynchronous request and business execution switch:

[0081] Existing technology: lacks a mechanism to dynamically adjust data processing capabilities, making it difficult to cope with online failures and business changes.

[0082] The present invention: the asynchronous request and service execution switch allow dynamic modification of compatible switches and service processing, thereby improving the stability and adaptability of the system.

[0083] Data processing timer:

[0084] Existing technology: Lack of real-time monitoring and performance analysis tools for data processing.

[0085] The present invention: a data processing timer records processing time, helps identify steps with high resource consumption, and optimizes system configuration.

[0086] Comparison of specific parameters, processing efficiency: The present invention improves the processing efficiency by at least 20% compared with the prior art through the pipeline filtering mode and the thread pool.

[0087] Flexibility: The Filter provided by the present invention <t>Interface and PipelineBuilder, which improves system configuration and scalability by at least 30%.

[0088] Stability: Through asynchronous requests and business execution switches, system stability is improved by at least 25% compared to existing technologies.

[0089] Resource consumption: Under the same data processing volume, the resource consumption of the present invention is reduced by at least 15% compared with the prior art.

[0090] Through the above comparison, it can be seen that the present invention is superior to the existing technology in terms of data processing efficiency, flexibility, stability and resource consumption, and provides a more efficient, reliable and scalable solution for judicial data cleaning and reconstruction.

[0091] Example 2: In the process of judicial archive digitization, a large number of paper archives need to be converted into electronic format for storage, retrieval and analysis. In the prior art, archive digitization usually relies on manual scanning and subsequent tedious sorting work, which is inefficient and prone to errors. This example will show how to apply a judicial data cleaning and reconstruction method based on a pipeline filtering mode to improve the efficiency and quality of the archive digitization process.

[0092] System architecture:

[0093] The system architecture of this embodiment is similar to that of the first embodiment, and mainly includes the following modules:

[0094] Data acquisition module.

[0095] Data preprocessing module.

[0096] Pipeline module.

[0097] Filter module.

[0098] Data reconstruction module.

[0099] Data storage module.

[0100] Specific parameters:

[0101] Data acquisition module:

[0102] Number of paper files collected per hour: 1,000

[0103] Average number of pages per file: 50 pages

[0104] Data preprocessing module:

[0105] Pre-processing time: 5 minutes per file.

[0106] Pre-processing tasks: remove blank pages, repair damaged images, adjust image size and resolution.

[0107] Pipeline Module:

[0108] Processing capacity of each pipeline: 20 files per minute.

[0109] Number of pipelines: Based on the server performance configuration, assuming 5.

[0110] Filter Module:

[0111] Data cleaning strategy: OCR recognition error correction, keyword filtering, and format standardization.

[0112] Filter processing time: 1 second per page of data.

[0113] Data reconstruction module:

[0114] Data reconstruction rules: Sort by file type and date.

[0115] Data reconstruction time: 30 seconds per file.

[0116] Data storage module:

[0117] Storage capacity: Each file takes up an average of 1MB of space.

[0118] Storage system: Distributed file system that supports high-concurrency read and write operations.

[0119] Implementation process:

[0120] Data collection: 1,000 files are collected every hour, each file is 50 pages, totaling 50,000 pages.

[0121] Data preprocessing: The total preprocessing time is 50,000 pages × 5 minutes / page = 250,000 minutes, or 4,166.67 hours.

[0122] Pipeline processing: 5 pipelines are processed in parallel, processing 1,000 files per hour, totaling 50,000 pages, with a processing time of 2,500 seconds.

[0123] Filter processing: 1 second for each page of data, a total of 50,000 pages x 1 second / page = 50,000 seconds, or 13 hours and 19 minutes.

[0124] Data reconstruction: 30 seconds for each file, a total of 1,000 files x 30 seconds per file = 30,000 seconds, or 8 hours and 16 minutes.

[0125] Data storage: 50,000 pages × 1MB / page = 50,000MB, or 50GB.

[0126] Data comparison:

[0127] Improved efficiency: Compared with traditional manual processing, the automated processing in this embodiment shortens the overall processing time from weeks to hours.

[0128] Improved accuracy: Through OCR recognition error correction and keyword filtering, data accuracy is improved to 98%.

[0129] Resource consumption: Compared with the traditional method, the resource consumption of this embodiment is reduced by 70% because the time for manual operation and error re-doing is reduced.

[0130] Through this embodiment, it can be seen that the judicial data cleaning and reconstruction method based on the pipeline filtering mode can significantly improve processing efficiency, reduce resource consumption, and improve data accuracy in the judicial archive digitization scenario.

[0131] Technical implementation, filter interface: implement a variety of cleaning strategies, such as data deduplication and missing value processing, pipeline module: use the message queue mechanism to ensure the efficiency and reliability of data processing, data reconstruction module: according to the archive digitization standards, design a data structured model to achieve efficient data storage and retrieval.

[0132] This embodiment achieves the following effects by applying a judicial data cleaning and reconstruction method based on a pipeline filtering mode: improving efficiency: automating the processing flow, greatly improving the efficiency of archive digitization, ensuring quality: ensuring data quality through a strict cleaning and verification process, and easy management: digitized archives are easy to manage and retrieve, which improves judicial work efficiency. Through this embodiment, it can be seen that the application of the present invention in the digitization of judicial archives has significant practical effects and provides strong support for the modernization of judicial archive management.

[0133] Working principle: Collect data from original data sources (such as paper archives, databases, electronic files). The data acquisition module is responsible for collecting raw data and converting it into a form that the system can process. It performs preliminary processing on the collected data, such as removing noise, standardizing the format, and converting data types. The preprocessing module ensures data quality and lays the foundation for subsequent cleaning and reconstruction. The pipeline module is the core of data processing. It passes the data flow through a series of predefined processing steps. The data is assigned to different pipelines. Each pipeline represents a processing stage, such as data conversion, verification, and cleaning.

[0134] The filter module contains a series of filters, each of which is responsible for performing specific data cleaning tasks. The filters can be simple data screening or complex logical judgments, such as data correction, outlier processing, and data normalization. When the data flow passes through each filter, it will be processed accordingly according to the set rules. After data cleaning, the reconstruction module is responsible for reorganizing or converting the cleaned data into the required format, including reordering, merging, splitting or converting the data into a new data structure. The cleaned and reconstructed data is stored in the target database or file system. The storage module ensures the security and accessibility of the data, and also supports subsequent data analysis and retrieval.

[0135] Detailed description of the working principle: Data passes through the pipeline in the form of a stream, and the processing steps in each pipeline are executed sequentially to ensure the continuity and consistency of data processing. Each module (such as data acquisition, preprocessing, filters, reconstruction, storage) is independent and can be developed and tested separately, which improves the maintainability and scalability of the system. The entire process is executed automatically, reducing manual intervention and improving processing speed and accuracy. Filters can be configured according to different data cleaning requirements, so that the system can adapt to different data processing scenarios. In the data flow, if errors or exceptions occur, the system will automatically capture and handle these errors to ensure the robustness of data processing.

[0136] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t> < / t>

Claims

1. A judicial data cleaning and reconstruction method based on pipeline filtering mode, characterized in that: It includes filter interface, execution data query system, data stream processing module UpdateFilter; pipeline module; pipeline builder; asynchronous request and business execution switch: data processing timer, the filter interface adopts Filter <t>A process method is provided to implement data cleaning processing under different filtering strategies. The data query execution system includes providing QueryFilter: implementing querying the target data record result set according to data entity mapping and query condition parameters; BaseFilter: implementing the remaining result set after the basic filtering conditions configured in the configuration file; UpdateFilter: implementing the filtering strategy required when the new table is rebuilt after data cleaning; the provided filter implementation can be used through simple configuration, and the Filter interface can be implemented by itself for more flexible business scenarios.< / t> 2. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The pipeline module includes: PipelineData: pipeline communication object, used to establish the default inter-pipeline communication data entity; PipelineProcessor <t> The pipeline processor provides default process orchestration, integrates filter implementation sets, and integrates thread pools to speed up processing efficiency.< / t> 3. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The pipeline builder adopts PipelineBuilder, provides pipeline initialization and build methods, which are used to initialize specific business pipelines after rapid configuration, and provides pipeline initialization and build methods, which are used to initialize specific business pipelines after rapid configuration.

4. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The asynchronous request and service execution switch includes: Asynchronous request: Provides an asynchronous interface for business logic compatibility switches to ensure that the compatibility switches can be quickly and dynamically modified when a failure occurs during online data processing; Business execution switch: Provides an asynchronous interface for pipeline filter task execution switch, and flexibly turns on or off business processing according to actual business conditions; Business paging and step control: Flexibly set the paging size and processing step according to business conditions to improve processing efficiency and stability.

5. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The data processing timer is used to start the overall time consumption of each asynchronous data processing, and observe the data processing process and rate more intuitively. By recording the processing time, it is possible to identify which data processing steps or operations consume the most resources (CPU, memory), thereby helping to optimize system configuration and resource allocation.

6. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The following processing steps are included: S1. Data collection module: At the beginning, raw judicial data is collected from different data sources; S2. Data preprocessing module: Perform preliminary cleaning on the collected raw data, such as removing duplicate records, filling missing values, and unifying the format; S3. Pipeline module: The pre-processed data is transmitted through the pipeline module, which acts as a transfer station for the data flow to ensure that the data can flow continuously to the next processing module; S4. Filter module sequence: data passes through multiple filter modules in sequence, each filter module is responsible for a specific data processing task; S5. Data verification: Check whether the data conforms to specific rules or formats; S6.Data conversion: converting data into a unified data format or type; S7. Outlier processing: Identify and process outlier data values; S8. Data standardization: standardize the data, such as unifying the date format and normalizing the value range; S9. Data reconstruction module: data processed by multiple filter modules enters the data reconstruction module and is reconstructed according to certain rules, such as time series reconstruction and relational database reconstruction; S10. Data storage module: The reconstructed data is stored in the data storage module, which may be a database, a file system or other storage medium; S11. End: The data cleaning and reconstruction process is completed.

7. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The data preprocessing step further includes: identifying and removing duplicate data records; reasonably filling in missing data; and unifying the data format into a standard format.

8. The judicial data cleaning and reconstruction method based on pipeline filtering mode according to claim 1 is characterized in that: The method adopts modular design so that each processing module can be independently developed, tested and upgraded, thereby improving the maintainability and scalability of the system. The method improves the efficiency of data cleaning and reconstruction through automated processing, reduces manual intervention, and thus improves the quality and accuracy of data processing.

Images