Data processing method and device based on rule dynamic matching, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device based on rule dynamic matching, electronic equipment and a storage medium. The method comprises the steps that firstly, a data file to be stored is received and analyzed to extract features, and then the data file is matched with a preset rule base; if yes, a preprocessing process defined by the rule is automatically executed, archiving is completed, and processing automation is achieved; and if the data type is not matched with the current data type, generating a data feature report, guiding a user to configure a new rule based on the report and apply the new rule to the current data in real time, and meanwhile, persisting the new rule to expand the system processing capability. The problem that a traditional scheme is insufficient in flexibility and expansibility when facing unknown or changed data types is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device, and storage medium based on rule-based dynamic matching. Background Technology

[0002] With the widespread application of information technology across various industries, data entry has become a ubiquitous and crucial step in information systems. Traditional data entry solutions are typically designed for known and structurally stable data types, based on pre-defined parsing rules and fixed processing steps. These solutions can achieve efficient and reliable data storage and management in business scenarios where data types are fixed and processing flows remain unchanged.

[0003] However, in practical applications, especially in dynamic fields such as emergency management and results sharing, systems often need to access new, structurally unknown, or heterogeneous data sources, and data preprocessing requirements may be frequently adjusted as business changes occur. Traditional methods, lacking the ability to dynamically configure rules and processes, struggle to flexibly adapt to new data types and cannot easily adjust preprocessing steps such as cleaning, transformation, and validation. When new data formats or processing requirements emerge, manual intervention, code modification, or even system refactoring is often necessary, resulting in high development and maintenance costs, long response cycles, and poor system scalability.

[0004] Therefore, designing a data entry system and method that supports dynamic data type recognition, flexible configuration of entry rules, and free arrangement of preprocessing processes has become a key technical issue for improving data access flexibility, reducing system operation and maintenance complexity, and adapting to rapidly changing business needs. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a data processing method, apparatus, electronic device, and storage medium based on rule-based dynamic matching, so as to dynamically adapt to data processing needs and improve the flexibility of data entry scheme.

[0006] In a first aspect, embodiments of the present invention provide a data processing method based on rule-based dynamic matching, the method comprising: Receive data files to be imported into the database; Perform feature parsing on the data file to extract the corresponding data features; The data features are matched with a pre-defined database of rules. Each rule in the database defines the identification features of a specific data type, optional preprocessing rules, and archiving methods. If a target rule is matched, the data is processed according to the preprocessing rules defined by the target rule, and the data features are stored in the database according to the archiving method of the target rule. If no rule is matched, a data feature report is generated, and a new rule is created based on the data feature report and user input. After the new rule is stored in the rule base, the current data is processed and stored in the rule base according to the new rule.

[0007] In conjunction with the first aspect, the steps of matching data features with a pre-defined database of import rules include: The data features are compared with the multi-dimensional identification feature constraints defined in each rule. The dimensions include at least one of the following: file name pattern based on regular expressions, file composition structure based on specified file type and quantity relationship, content keywords based on scanning non-binary file content, and features based on specified format of inbound list file. Based on the preset matching strategy, the comparison results of various dimensions are combined to determine whether there is a matching target rule.

[0008] In conjunction with the first aspect, the steps for processing data according to the preprocessing rules defined in the target rules include: Obtain the preprocessing rule configuration information defined by the target rule definition. The configuration information defines the execution order and dependencies of multiple preprocessing algorithm nodes. Based on the configuration information, each preprocessing algorithm node is executed sequentially to obtain the preprocessed data features.

[0009] In conjunction with the first aspect, the preprocessing rule configuration information is generated through the visual workflow orchestration interface provided by the data processing system; wherein, the data processing system is configured with a preprocessing algorithm library, and each preprocessing algorithm in the preprocessing algorithm library is associated with metadata describing its function and compatible data types; The method also includes: In the visual workflow orchestration interface, algorithm components are recommended from the algorithm library based on the data type corresponding to the target rule and on the metadata. In response to drag-and-drop and connection operations performed by users through the visual process orchestration interface, a directed acyclic graph (DAG) structure is generated that defines the execution order and dependencies of each algorithm node, and the DAG is persisted as preprocessing rule configuration information.

[0010] In conjunction with the first aspect, the steps for generating a data feature report include: Perform deep analysis on the data file to extract at least one feature from the following: data format, field type, data structure, statistical features, and pattern recognition features; The extracted features are used to generate a data feature report in a structured format.

[0011] In conjunction with the first aspect, the steps for creating new rules based on data feature reports and user input include: The rule configuration interface is generated based on the data feature report. The rule configuration interface provides configuration items for defining the identification features, preprocessing rules and archiving methods of new rules. Based on the analysis of historical rule bases, data feature reports are analyzed to recommend rule templates that match the data files to be added to the database to users; In response to the user's configuration operation on the configuration interface, a new entry rule corresponding to the new data type is generated and persisted. The new entry rule includes at least data identification features defined according to data characteristics, optional preprocessing process configuration, and archiving constraints.

[0012] In conjunction with the first aspect, the steps of each preprocessing algorithm node are executed sequentially based on the configuration information, including: The configuration information defines multiple preprocessing algorithm nodes and their dependencies, which are then decomposed into multiple independently schedulable computational tasks. A distributed task scheduling framework is used to allocate multiple computing tasks to different nodes in the computing cluster for parallel or serial execution. Monitor the execution status of each computing task, and after all computing tasks are successfully completed, summarize the preprocessed data features.

[0013] Secondly, embodiments of this application also provide a data processing apparatus based on rule-based dynamic matching, the apparatus comprising: The receiving module is used to receive data files to be imported into the database. The extraction module is used to perform feature parsing on the data file and extract the data features corresponding to the data file. The matching module is used to match data features with a pre-defined database of rules. Each rule in the database defines the identification features of a specified data type, optional preprocessing rules, and archiving methods. The data processing and archiving module is used to process the data according to the preprocessing rules defined by the target rule if a target rule is matched, and to store the data features into the database according to the archiving method of the target rule. The creation of the database module is used to generate a data feature report if no rule is matched, and to create a new rule based on the data feature report and user input. After the new rule is stored in the rule base, the current data is processed and stored in the database according to the new rule.

[0014] Thirdly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the methods described above.

[0015] Fourthly, this application provides a storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0016] The embodiments of this invention bring the following beneficial effects: This application provides a data processing method, apparatus, electronic device, and storage medium based on rule-based dynamic matching. The method first receives and parses the data file to be stored to extract features, and then matches it with a preset rule base. If a match is successful, the preprocessing flow defined by the rule is automatically executed and archiving is completed, achieving automated processing. If no match is found, a data feature report is generated, guiding the user to configure new rules based on the report and apply them to the current data immediately. Simultaneously, the new rules are persisted to expand the system's processing capacity. This method, through rule-based dynamic matching and online expansion mechanisms, effectively solves the problem of insufficient flexibility and scalability in traditional solutions when dealing with unknown or changing data types, while ensuring processing efficiency for known data types.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the rule-based dynamic matching data processing method provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the flow principle of the rule-based dynamic matching data processing method provided as an example of the present invention; Figure 3 A schematic diagram of a rule-based dynamic matching data processing device provided as an example of the present invention; Figure 4 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention.

[0021] Figure label: 10 - Receiving module, 20 - Extraction module, 30 - Matching module, 40 - Processing archive module, 50 - Creating inbound module; 130 - Processor, 131 - Memory, 132 - Bus, 133 - Communication interface. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To facilitate understanding of this embodiment, the application scenarios and design concepts of this application embodiment will be briefly introduced below.

[0024] Traditional data entry solutions are based on pre-defined rules and fixed steps. When faced with dynamic changes in data types and frequent adjustments to preprocessing requirements, they suffer from poor flexibility, high expansion costs, and long response cycles, making them unable to meet the needs of efficient and flexible data access and management.

[0025] Based on this, this application provides a data processing method, apparatus, electronic device, and storage medium based on rule-based dynamic matching to solve the problems of poor flexibility and high expansion costs of traditional data entry schemes, and to achieve dynamic adaptation, scalability, and efficient management of data types and processing flows.

[0026] Example 1 This application provides a data processing method based on rule-based dynamic matching, combined with... Figure 1 As shown, the method includes: S110 receives the data file to be imported into the database.

[0027] S120: Perform feature parsing on the data file and extract the data features corresponding to the data file.

[0028] S130 matches data features with a pre-defined inbound rule base. Each rule in the rule base defines the identification features of a specified data type, optional preprocessing rules, and archiving methods.

[0029] S140, if a target rule is matched, the data is processed according to the preprocessing rules defined by the target rule, and the data features are stored in the database according to the archiving method of the target rule.

[0030] S150 If no rule is matched, a data feature report is generated, and a new rule is created based on the data feature report and user input. After the new rule is stored in the rule base, the current data is processed and stored in the database according to the new rule.

[0031] The data processing method based on rule-based dynamic matching provided in this application achieves intelligent and pipelined processing of known data types through a main process of feature parsing, rule matching, and automatic execution. Simultaneously, through sub-loops of feature reporting and rule expansion, it addresses the adaptive and self-learning capabilities required for unknown data types. This allows for a unified framework that simultaneously satisfies the automation and standardization of the processing flow while maintaining the flexibility and scalability required by business needs. It fundamentally overcomes the rigidity and high maintenance costs of traditional solutions, achieving a synergistic evolution between data processing capabilities and dynamic business changes. By constructing an intelligent and scalable data processing pipeline, it effectively addresses the business challenges of diverse data types and ever-changing processing flows.

[0032] This method is applied to a data processing system, which includes: a dynamic rule management module, a data parsing and matching module, a process execution engine, and a rule extension module.

[0033] The rule dynamic management module is used to maintain an extensible inbound rule base, where each inbound rule uniquely corresponds to a data type and includes feature constraints for identifying that data type, optional preprocessing configuration, and archiving constraints.

[0034] The data parsing and matching module is used to parse the features of the received data files to be entered into the database, and match the parsed features with the entry rules in the rule base to determine the applicable target rules.

[0035] The process execution engine is used to process the data by calling the corresponding preprocessing algorithm according to the preprocessing process configuration in the target rule when a match is successful, and to archive the processed data features to the database according to the archiving constraints.

[0036] The rule extension module is used to generate a feature report based on the data parsing results when a match fails, and to add the newly defined data type and its corresponding entry rule to the rule library in response to user configuration instructions.

[0037] The system receives data files from different sources and in various formats through a unified interface. In step S120, the system automatically and deeply scans the files using its built-in parsing engine, extracting key features from multiple dimensions such as filename, content, and structure. This process transforms unstructured raw data into structured feature descriptions, providing accurate judgment criteria for subsequent rule matching, thereby extracting the data features corresponding to the data files.

[0038] In conjunction with the first aspect, step S130, which involves matching data features with a pre-set database of import rules, includes: S131 compares the data features with the multi-dimensional identification feature constraints defined in each rule. The dimensions include at least one of the following: file name pattern, file composition structure, and file content keywords.

[0039] S132, based on the preset matching strategy, comprehensively compare the results of various dimensions to determine whether there is a matching target rule.

[0040] Steps S131 and S132 together constitute the intelligent rule matching engine. This engine aims to overcome the limitations of traditional solutions that rely on single, static features for data identification. By introducing a multi-dimensional, configurable composite matching mechanism, it achieves accurate, flexible, and reliable automated identification and routing of complex and variable data types.

[0041] Specifically, this matching mechanism first performs refined feature deconstruction and comparison in S131 (feature comparison stage). It doesn't just check surface information like filenames or extensions, but conducts a multi-faceted, in-depth scan of the data entity, verifying each element according to predefined constraints in the rules, such as filename patterns (e.g., regular expressions), file structure, and keywords in the file content. This process is like creating a complete "feature fingerprint" for each type of data, significantly improving the specificity and anti-interference capability of the identification. Even if the data differs in non-critical features (such as temporary filename changes), it can still be effectively captured through core features.

[0042] Subsequently, in S132 (strategy decision-making phase), instead of mechanically requiring all dimensions to be completely identical, a pre-defined, business-configurable matching strategy is used to intelligently and comprehensively judge the comparison results across multiple dimensions. This strategy can be a "strict full match" requiring all dimensions to match perfectly, a "key dimension priority match" emphasizing key dimensions (such as content keywords), or a "weighted scoring match" calculated using weights. This gives the matching process extremely high flexibility and adaptability, enabling the system to balance the accuracy and coverage of identification based on the importance of different data types and business tolerance.

[0043] Overall, steps S131 and S132 form a closed-loop decision-making process of feature acquisition and strategy adjudication. This not only upgrades data identification from a single, rigid judgment to a multi-dimensional, comprehensive feature evaluation, but also, through a configurable strategy layer, enables the technical solution to be deeply integrated with diverse and dynamically evolving business needs. Thus, it forms the cornerstone for dynamic expansion of data access and flexible adaptation of processing flows, ensuring that when facing unknown or heterogeneous data, the system can maintain high efficiency and stability in processing known data types, while also providing accurate and reliable judgment criteria for the intelligent expansion of rules.

[0044] Among them, the multi-dimensional identification feature constraints are the "data ID card" or "fingerprint features" defined for each rule entering the rule base. Unlike existing technologies that may rely solely on file extensions or single keywords for simple judgment, this application's embodiments employ multi-dimensional, combined feature descriptions, significantly improving the accuracy and anti-interference capability of rule identification. Specific dimensions include: Filename pattern recognition based on regular expressions focuses on the naming regularity and surface identifiers of data files. By using regular expressions, a powerful pattern description tool, the system can define highly flexible and precise filename matching rules. For example, a rule can be described as matching all files named in the pattern "Survey_Report_202[0-9]{4}.pdf", thus identifying survey reports for all years in the 2020s at once, without needing to create separate rules for each year. This effectively solves the recognition challenges caused by variable parts in file names such as dates, version numbers, and region codes, enabling batch, efficient, and accurate identification of files with regular names.

[0045] Based on the file composition structure of specified file types and quantities, dimensionality transcends the limitations of individual files, identifying data from the overall compositional relationship of a file set. It defines the components and conditions that a complete data submission (or "data package") must include. For example, a valid "satellite imagery package" might be defined as containing at least one .tiff format image file, a .xml metadata file with the same name, and an optional thumbnail.jpg thumbnail file. This identification method is particularly suitable for scenarios such as results archiving, project submission, and inter-system data exchange, ensuring data integrity and standardization, preventing the loss of key components, and defining data types at a higher logical level.

[0046] Based on content keywords scanned from non-binary file content, this dimension delves into the internal semantic content of the file for substantial information extraction and judgment. The system scans the content of non-binary files such as text, CSV, XML, JSON, and logs, searching for predefined keywords, specific fields, or data structures. For example, it identifies the presence of the key-value pair "sensorType": "temperature" in JSON data, or checks whether the header row of a CSV file contains the columns "EmployeeID" and "Department". This is the most direct and powerful semantic-level recognition method, ensuring that the data meets expectations in terms of business meaning, and can be correctly identified even if its filename or external form changes.

[0047] Based on the characteristics of the inbound manifest file in a specified format, this dimension is a metadata-driven, declarative identification method. It does not directly examine the target data file itself, but relies on a structured manifest file (such as an XML or JSON manifest) that accompanies the submission. This manifest file explicitly lists the paths, types, versions, descriptions, and other metadata of all files included in this inbound process. By parsing this manifest file, the system can authoritatively and accurately understand the complete composition of the data to be processed. This approach transforms the identification logic from "guessing what the data is" to "reading authoritative descriptions," making it particularly suitable for batch data inbound scenarios involving a large number of files, complex relationships, and strict version management, greatly improving identification accuracy and processing efficiency.

[0048] Understandably, these four dimensions can be arbitrarily combined according to the rules. This multi-dimensional, configurable feature recognition system enables the system to handle various recognition needs, from simple single files to complex data packets, and from format matching to semantic verification. It lays a solid and flexible foundation for subsequent rule matching and intelligent processing, and is a key technical support for achieving high adaptability.

[0049] The preset matching strategy in step S132 is a built-in decision logic template that users or administrators can choose based on business scenarios. This step first uses the detailed results generated by the multi-dimensional feature comparison in the early stage of step S131 (such as filename patterns, file structure, content keywords, and database list features), and then introduces a pre-defined matching strategy for global judgment. These strategies are not simple "yes / no" logics, but rather decision models that can be flexibly selected according to business scenarios. For example, strict full matching that requires all dimensions to match completely, priority matching that focuses on key dimensions (such as content semantics), or weighted scoring matching that uses weights and thresholds for fine calculation. By using such strategies, the system can intelligently handle common non-key feature deviations in real-world data, significantly improving its tolerance and adaptability to non-standard or complex data while ensuring recognition accuracy. Ultimately, this mechanism transforms technical feature matching into an understandable and controllable business logic, enabling the system not only to determine whether a match is made, but also to provide a reasonable explanation of why a match is made in a way that fits actual needs, thus establishing a crucial bridge between automated processing and business flexibility.

[0050] Common strategy types include strict full-match strategy, key dimension priority (or veto) strategy, or weighted scoring decision.

[0051] A strict full-match strategy requires that data must fully meet the dimensional constraints defined by the rules in order to be considered a match. It is suitable for scenarios with extremely strict requirements on data format that cannot tolerate any deviation, such as the storage of financial transaction records and legally formatted documents. It ensures absolute accuracy in processing.

[0052] The key dimension priority strategy sets priorities for different dimensions. Typically, one or more dimensions (such as "file content keywords" or "inbound list") are designated as key dimensions. If a key dimension matches successfully, it is considered a match even if other non-key dimensions (such as filename patterns) do not match; conversely, if a key dimension fails to match, it is directly considered a non-match. This strategy is suitable for scenarios that focus more on the core semantics of the data than its surface form. For example, if the data content indicates "earthquake monitoring data," it should be correctly routed even if the filename is not standard.

[0053] The weighted scoring strategy assigns a weighted score to each comparison dimension. The system calculates a total score based on the actual matching performance of each dimension (e.g., a perfect match receives full marks, a partial match receives partial marks). If the final total score exceeds a preset threshold (e.g., 80 points), it is considered a match. This is the most refined and flexible strategy. It is suitable for complex scenarios that require balancing multiple recognition features and allow for some fuzzy matching. Users can finely control the "tightness" of the matching by adjusting the weights and thresholds.

[0054] For example, step S131 generates detailed comparison results for each dimension (e.g., dimension A - successful match; dimension B - failed match; dimension C - partial match). These discrete results are then synthesized according to the algorithm of the selected strategy (e.g., AND logic, key dimension logic, weighted calculation logic). A clear Boolean value is output, indicating whether a matching target rule exists or not, and if it exists, the specific rule is identified.

[0055] In conjunction with the first aspect, step S130 processes the data according to the preprocessing rules defined in the target rules, specifically including: S133, obtain the preprocessing rule configuration information defined by the target rule definition. The configuration information defines the execution order and dependency relationship of multiple preprocessing algorithm nodes.

[0056] S134, based on the configuration information, execute each preprocessing algorithm node in sequence to obtain the preprocessed data features.

[0057] Steps S133-S134 together constitute the process execution engine in the aforementioned system. These two steps transform the "processing blueprint" (i.e., preprocessing rule configuration information) previously determined through rule matching into actual data processing actions, enabling flexible definition, orderly scheduling, and automatic execution of data processing logic. Through the standardized configuration information obtained in step S133, a complex processing task that may involve multiple stages such as data cleaning, format conversion, integrity verification, and data anonymization is decomposed into a set of clearly defined, well-defined, and clearly interfaced independent algorithm nodes and their execution logic. Step S134 then acts like an intelligent pipeline controller, automatically and reliably driving these nodes to work sequentially, strictly following the configured order and dependencies.

[0058] In step S133, based on the matched target rule, the associated preprocessing rule configuration information is located and read. This configuration information is not a simple list of algorithms, but a structured process definition, which can generally be understood as a data processing work instruction manual or an abstract description of a directed acyclic graph (DAG). This configuration information explicitly defines multiple preprocessing algorithm nodes and their execution order and dependencies, thereby transforming the intuitive flowchart arranged by the user through the visual interface into a structured instruction set that the engine can accurately understand and execute, laying the foundation for automated execution. Among them, each preprocessing algorithm node refers to an executable unit with a specific function (such as "space trimming", "date format standardization", "outlier filtering"); the structured instruction set for execution specifies the order of nodes and the flow of data. For example, the input of node B depends on the output of node A, which forces the business logic that "data cleaning (node ​​A) must be completed before statistical calculation (node ​​B) can be performed."

[0059] Step S134 instantiates and starts the algorithm nodes sequentially according to the configured order. It rigorously checks dependencies; a subsequent node is only triggered after the preceding node has successfully executed and produced output, ensuring the data flow is correct. Furthermore, nodes exchange data through predefined interfaces. Typically, the output of the previous node, stored in memory or temporary storage, is automatically packaged as the input of the next node. This setup enables seamless data flow in the pipeline without manual intervention.

[0060] Furthermore, the execution of each node is atomic, and its success or failure can be independently captured and logged. This provides fine-grained fault tolerance and control for the process. For example, the failure of a node may not cause the entire process to crash, but instead trigger a predefined retry or bypass mechanism. Thus, decomposing complex tasks into manageable small units and executing them sequentially not only reduces system complexity but also greatly improves the reliability, maintainability, and execution efficiency of data processing (especially in extended designs that support parallel execution), transforming static, configurable rule definitions into dynamic, automated, and reliable data processing capabilities.

[0061] Combination Figure 2 As shown, the system first receives data, then performs feature recognition to extract its multi-dimensional features and matches them with a pre-set rule base. If a matching rule exists, it further determines whether the rule is associated with the preprocessing process. If associated, a pre-set preprocessing algorithm chain (such as algorithm 1, 2...N) is executed sequentially to complete data cleaning and transformation. If not associated or after preprocessing, the data is finally archived and stored. If no existing rule is matched, the system outputs a structured data feature report, providing a basis for subsequent manual configuration of new rules. This ensures automated and pipelined processing of known data types while providing a standard and scalable interface for the system to handle unknown data types.

[0062] In conjunction with the first aspect, the preprocessing rule configuration information is generated through the visual workflow orchestration interface provided by the data processing system; wherein, the system is configured with a preprocessing algorithm library, and each preprocessing algorithm in the preprocessing algorithm library is associated with metadata describing its function and compatible data types.

[0063] The method also includes: S210, in the visual process orchestration interface, recommends algorithm components from the algorithm library based on the data type corresponding to the target rule and on metadata.

[0064] S220, in response to drag-and-drop and connection operations performed by the user through the visual process orchestration interface, generates a directed acyclic graph structure that defines the execution order and dependencies of each algorithm node, and persists the directed acyclic graph as preprocessing rule configuration information.

[0065] Preprocessing rule configuration information is generated through the system's visual workflow orchestration interface. This interface acts as a graphical workflow design workbench, allowing users to construct and adjust complex data processing logic using intuitive graphical elements (such as nodes and connections) without requiring professional programming knowledge. This significantly reduces communication barriers between business experts and technical personnel and enables rapid response to changes in business requirements. The system's preprocessing algorithm library is not a simple collection of code, but a standardized, encapsulated, and metadata-described pluggable component repository. The metadata of each algorithm (such as input / output data formats, processing function descriptions, applicable data types, performance parameters, etc.) provides precise explanatory information for the system's automated management and intelligent recommendation, enabling algorithms to be flexibly invoked and combined as standardized "formulas."

[0066] When a user orchestrates a process for a specific data type, the system proactively analyzes the characteristics of that data type and, based on information such as "compatible data types" in the algorithm's metadata, automatically filters and highlights the most likely applicable algorithm components. This significantly reduces the time users spend blindly searching through a large number of algorithms, avoids configuration errors caused by selecting incompatible algorithms, and improves the efficiency and accuracy of orchestration. Users create "nodes" by dragging and dropping algorithm components onto the canvas and define the data flow and execution dependencies between nodes through connection operations. The system responds to user operations in real time, automatically building and maintaining a "Directed Acyclic Graph" (DAG) in the background. The DAG structure ensures that the process is logically free of cyclic dependencies, thus clarifying the execution order of each step, making it an ideal mathematical model for expressing complex parallel and serial hybrid processes.

[0067] The system serializes the DAG structure built in the graphical interface into a structured configuration file (such as JSON, XML, or YAML format) that can be read and parsed by the process execution engine. This persistence process allows the user-designed visual process to be saved, reused, and version-managed, and bound to specific data entry rules. When data processing is required, the execution engine only needs to load this configuration file to accurately recreate and run the entire preset process.

[0068] In conjunction with the first aspect, step S140, which involves creating new rules based on data feature reports and user input, specifically includes: S141, Generate a rule configuration interface based on the data feature report. The interface provides configuration items for defining the identification features, preprocessing rules and archiving methods for new rules.

[0069] Based on the data feature report, a user interface containing specific input items and components is automatically generated. This interface creates the abstract elements required for a complete rule: data identification features, preprocessing logic, and archiving methods. These elements are decomposed and mapped into specific, fillable configuration items (such as feature constraint forms, algorithm selection lists, and archiving field mapping tables). This step, through a pre-defined configuration framework, constrains the open rule definition task within a standardized and systematically processed path, ensuring the completeness and standardization of user input and providing a standardized input source for subsequent automatic generation.

[0070] S142, based on the historical rule base, analyzes the data feature report and recommends rule templates that match the data files to be added to the database to the user.

[0071] Specifically, the system calls upon the historical rule base and uses a pre-defined similarity analysis algorithm to compare the current data feature report with the identification features of existing rules. By calculating the distance between feature vectors or matching key patterns, the system selects one or more existing rules that are structurally or semantically closest and recommends them to the user as modifiable templates. This step, by reusing validated rule structures, aims to improve configuration efficiency, reduce redundant design, and promote consistency in processing logic among different rules.

[0072] S143 responds to the user's configuration operation on the configuration interface, generates and persists a new entry rule corresponding to the new data type. The new entry rule includes at least data identification features defined according to data characteristics, optional preprocessing process configuration, and archiving constraints.

[0073] The system receives and parses the final settings completed by the user on the configuration interface. Based on a predefined rule data model, it serializes discrete configuration items (such as selected regular expressions, orchestrated algorithm node sequences, and specified database fields) into a complete, internally interpretable new rule object. Subsequently, the system writes this rule object to persistent storage (such as a database or configuration file), completing an incremental update to the rule base. The completion of this step signifies that the system has officially expanded its capabilities, adding a new data processing logic that can be matched and executed, and can typically be immediately applied to the data file to be added to the database that triggered this creation.

[0074] In conjunction with the first aspect, step S134 executes each preprocessing algorithm node sequentially based on the configuration information, including: S1341 decomposes the multiple preprocessing algorithm nodes and their dependencies as defined by the configuration information into multiple independently schedulable computational tasks.

[0075] The abstract, dependency-based algorithm node graph in the process configuration information is transformed into a series of concrete, clearly defined, independently manageable and schedulable computational task units. This decomposition process requires resolving the data dependencies between nodes to ensure that after task partitioning, the dependencies can be correctly converted into priority constraints for task scheduling, laying the foundation for subsequent distributed parallelization.

[0076] S1342 uses a distributed task scheduling framework to distribute multiple computing tasks to different nodes in a computing cluster for parallel or serial execution.

[0077] Through a distributed task scheduling framework, the task set decomposed from S1341 is dynamically allocated to multiple physical or virtual computing nodes in the computing cluster based on their dependencies, resource requirements, and the load of each node in the cluster. Tasks with dependencies are executed sequentially, while tasks without dependencies are allocated to different nodes for parallel execution, thereby making full use of the cluster's computing power and shortening the overall process execution time.

[0078] S1343 monitors the execution status of each computing task and summarizes the preprocessed data features after all computing tasks have been successfully completed.

[0079] The framework or monitoring component continuously tracks the execution status of each distributed computing task, including success, failure, running, or waiting states. This monitoring capability is crucial for ensuring process reliability. Only after all tasks have successfully completed according to their dependencies will the system trigger result aggregation, integrating the intermediate or final data features generated by each task according to process logic to generate complete preprocessed data features for subsequent archiving. If any task fails, the system can retry or handle the overall process failure according to its strategy.

[0080] Steps S1341-S1343 upgrade the preprocessing workflow from a linear execution mode on a single node to a cluster-oriented, parallel job execution mode. Through task decomposition, distributed scheduling, and unified monitoring, it achieves horizontal scaling of processing capacity, resolving performance bottlenecks in complex data processing workflows, thus enabling it to handle large-scale, high-performance demands.

[0081] Secondly, embodiments of this application provide a data processing apparatus based on rule-based dynamic matching, such as... Figure 3 As shown, the device includes: a receiving module 10, an extraction module 20, a matching module 30, a processing and archiving module 40, and a creation and entry module 50.

[0082] The receiving module 10 is used to receive data files to be stored in the database.

[0083] The extraction module 20 is used to perform feature parsing on the data file and extract the data features corresponding to the data file.

[0084] The matching module 30 is used to match data features with a pre-set database of rules. Each rule in the database defines the identification features of a specified data type, optional preprocessing rules, and archiving methods.

[0085] The processing and archiving module 40 is used to process the data according to the preprocessing rules defined by the target rule if a target rule is matched, and to store the data features into the database according to the archiving method of the target rule.

[0086] The creation module 50 is used to generate a data feature report if no rule is matched, and to create a new rule based on the data feature report and user input. After storing the new rule in the rule base, the current data is processed and stored in the database according to the new rule.

[0087] Thirdly, embodiments of this application provide an electronic device, combined with Figure 4 As shown, the electronic device includes a memory 131 and a processor 130. The memory 131 stores a computer program, and the processor 130 runs the computer program to make the electronic device perform the above-described method.

[0088] Furthermore, combined Figure 4 The electronic device shown also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0089] The memory 131 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0090] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0091] Fourthly, embodiments of this application provide a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0093] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0094] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0096] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method based on rule-based dynamic matching, characterized in that, The method includes: Receive data files to be imported into the database; The data file is subjected to feature parsing to extract the data features corresponding to the data file; The data features are matched with a pre-set database of rules, where each rule defines the identification features of a specific data type, optional preprocessing rules, and archiving methods. If a target rule is matched, the data is processed according to the preprocessing rules defined by the target rule, and the data features are stored in the database according to the archiving method of the target rule. If no rule is matched, a data feature report is generated, and a new rule is created based on the data feature report and user input. After the new rule is stored in the rule base, the current data is processed and stored in the database according to the new rule.

2. The method according to claim 1, characterized in that, The step of matching the data features with a pre-set database of import rules includes: The data features are compared with the multi-dimensional identification feature constraints defined in each rule. The dimensions include at least one of the following: file name pattern based on regular expressions, file composition structure based on specified file type and quantity relationship, content keywords based on scanning non-binary file content, and features based on specified format of inbound list file. Based on the preset matching strategy, the comparison results of various dimensions are combined to determine whether there is a matching target rule.

3. The method according to claim 1 or 2, characterized in that, The steps of processing data according to the preprocessing rules defined in the target rule include: Obtain the preprocessing rule configuration information defined by the target rule definition, wherein the configuration information defines the execution order and dependency relationship of multiple preprocessing algorithm nodes; Based on the configuration information, each of the preprocessing algorithm nodes is executed sequentially to obtain the preprocessed data features.

4. The method according to claim 3, characterized in that, The preprocessing rule configuration information is generated through a visual workflow orchestration interface provided by the data processing system; wherein, the data processing system is configured with a preprocessing algorithm library, and each preprocessing algorithm in the preprocessing algorithm library is associated with metadata describing its function and compatible data types; The method further includes: In the visual process orchestration interface, algorithm components are recommended from the algorithm library based on the data type corresponding to the target rule and the metadata. In response to the drag-and-drop and connection operations performed by the user through the visual process orchestration interface, a directed acyclic graph structure defining the execution order and dependencies of each algorithm node is generated, and the directed acyclic graph is persisted as the preprocessing rule configuration information.

5. The method according to claim 1, characterized in that, The step of generating a data feature report includes: The data file is subjected to deep analysis to extract at least one feature from the following: data format, field type, data structure, statistical features, and pattern recognition features. The extracted features are used to generate the data feature report in a structured form.

6. The method according to claim 5, characterized in that, The steps for creating new rules based on data feature reports and user input include: A rule configuration interface is generated based on the data feature report. The rule configuration interface provides configuration items for defining the identification features, preprocessing rules, and archiving methods of new rules. Based on the analysis of the data feature report using the historical rule base, the system recommends rule templates that match the data files to be added to the database to the user. In response to the user's configuration operation on the configuration interface, a new entry rule corresponding to the new data type is generated and persisted. The new entry rule includes at least data identification features defined according to the data characteristics, optional preprocessing process configuration, and archiving constraints.

7. The method according to claim 3, characterized in that, The step of sequentially executing each of the preprocessing algorithm nodes based on the configuration information includes: The configuration information defines multiple preprocessing algorithm nodes and their dependencies, which are then decomposed into multiple independently schedulable computational tasks. The distributed task scheduling framework allocates the multiple computing tasks to different nodes in the computing cluster for parallel or serial execution. The execution status of each computing task is monitored, and the preprocessed data features are summarized after all computing tasks are successfully completed.

8. A data processing device based on rule-based dynamic matching, characterized in that, The device includes: The receiving module is used to receive data files to be imported into the database. The extraction module is used to perform feature parsing on the data file and extract the data features corresponding to the data file; The matching module is used to match the data features with a preset database of rules. Each rule in the database defines the identification features of a specified data type, optional preprocessing rules, and archiving methods. The data processing and archiving module is used to process the data according to the preprocessing rules defined by the target rule if a target rule is matched, and to store the data features into the database according to the archiving method of the target rule. An inbound module is created to generate a data feature report if no rule is matched, and to create a new rule based on the data feature report and user input. After the new rule is stored in the rule library, the current data is processed and inbounded according to the new rule.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program and the processor running the computer program to cause the electronic device to perform the method of any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when read and executed by a processor, perform the method described in any one of claims 1 to 7.

Images