A data processing method, system, device and storage medium

Abstract

The application provides a data processing method, system, device and storage medium, and relates to the technical field of data processing. The method comprises the following steps: in response to a data operation instruction sent by a user, generating an operation log, wherein the operation log at least comprises the data operation instruction; analyzing key parameters in the operation log, and determining target data identifier information and an operation type in the data operation instruction; according to the target data identifier information, capturing target data, wherein the target data represents the whole data of a data entry corresponding to the target data identifier information; in response to a configuration specification rule set by the user, and according to the configuration specification rule and the operation type, converting the target data into structured data; and in response to a data filtering rule set by the user, filtering the structured data to obtain standardized data. The technical problem that manual setting of configuration item parameters is complicated in the prior art is solved, and the degree of complexity of data processing is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, system, device and storage medium. Background Technology

[0002] Existing data processing systems use database management systems such as SQL Server, MySQL, and Oracle to store and manage data, but lack effective data verification mechanisms, affecting data consistency and accuracy. Changes in business needs lead to frequent adjustments to the data structure, increasing the complexity of data processing.

[0003] To address the data validation problem, researchers have proposed a rule-based and transformation-based data validation method. This method improves data quality and reliability by defining rules and transformation logic for preprocessing and standardizing data. However, existing technologies face challenges in implementation, particularly due to the cumbersome and complex configuration options, requiring users to manually set parameters and options, which increases operational complexity and error rates.

[0004] In existing technologies, the cumbersome and complex process of manually setting configuration parameters is a major problem. Furthermore, the serial processing method becomes a performance bottleneck. Customized implementation solutions for data platforms rely on fixed processes and rules, lacking flexibility and automation. Deficiencies exist in data synchronization, filtering and screening, and log processing, making it difficult to meet complex and ever-changing business needs. These issues limit the efficiency and accuracy of data processing systems, affecting the performance of data platforms in complex business environments. Summary of the Invention

[0005] In view of this, this application provides a data processing method, system, device and storage medium to solve the technical problem of cumbersome and complicated manual setting of configuration parameters in the prior art.

[0006] According to a first aspect of this application, a data processing method is provided, comprising:

[0007] In response to a data operation command sent by the user, an operation log is generated, wherein the data operation command represents the data processing rule selected by the user from a pre-set configuration item;

[0008] Analyze the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction;

[0009] Target data is retrieved based on the target data identifier information, whereby the target data represents the entire data entry corresponding to the target data identifier information.

[0010] In response to a user-defined configuration rule, and based on the configuration rule and the operation type, the target data is converted into structured data, wherein the configuration rule represents at least one configuration condition selected from pre-defined configuration items;

[0011] In response to user-defined data filtering rules, the structured data is filtered to obtain standardized data, wherein the data filtering rules refer to at least one data filtering condition selected from the pre-defined configuration items.

[0012] Preferably, the step of parsing the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction includes the following steps:

[0013] Analyze the operation log to confirm the command trigger scenario;

[0014] When the instruction triggering scenario is the first run trigger, a temporary data table with the same structure as the original data table is created. The first run trigger indicates that the operation log is generated for the first time. The temporary data table is used to temporarily store and process the data in the original data table.

[0015] The identifier information of all data in the temporary data table is confirmed as the target data identifier information, and the operation type is confirmed as adding all data in the temporary data table.

[0016] Preferably, the data operation instruction includes at least one sub-transaction instruction, and each sub-transaction instruction includes at least a sub-transaction operation type, a sub-transaction operation object, and a sub-transaction operation content;

[0017] The step of parsing the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction includes the following steps:

[0018] Analyze the key parameters in the operation log to determine the sub-transaction instruction;

[0019] Based on the sub-transaction operation type, the sub-transaction instructions are classified to generate at least one sub-transaction group;

[0020] At least based on the sub-transaction operation object and the sub-transaction operation content, analyze the relationship between the sub-transaction instructions in the sub-transaction group;

[0021] Based on the relationships between the sub-transaction instructions in the sub-transaction group, a sub-transaction merging strategy is generated;

[0022] According to the sub-transaction merging strategy, the sub-transaction instructions in each sub-transaction instruction group are merged to generate an instruction merge set;

[0023] Analyze the instruction set to determine the target data identifier information and the operation type.

[0024] Preferably, the step of responding to user-defined configuration rules and converting the target data into structured data according to the configuration rules and the operation type includes the following steps:

[0025] Based on the operation log and the target data, a structured data framework is constructed, which includes at least the identifier information of the operation log, the identifier information of the target data, the operation type, and the operation time.

[0026] In response to the rules specified in the configuration, the specified fields in the target data are identified and deleted to generate data to be structured;

[0027] Extract all fields and the current value information of each field from the data to be structured;

[0028] The structured data is generated by filling all the fields and the current value information of each field into the structured data frame.

[0029] Preferably, after constructing the structured data framework based on the operation log and the target data, the method further includes the following steps:

[0030] When the operation type is to update target data, a first influencing field is extracted from the target data, wherein the first influencing field represents the field directly associated with the updated target data;

[0031] Identify the second influence field in the target data that is associated with the first influence field;

[0032] The first impact field and the second impact field are merged to generate an impact field list, which includes at least the first impact field and the second impact field.

[0033] Preferably, after filtering the structured data in response to user-defined data filtering rules to obtain standardized data, the method further includes:

[0034] Obtain filtered data from the structured data, wherein the filtered data represents data that has been filtered by the data filtering rules;

[0035] Write the filtered data into the filtered data table;

[0036] Record operation data related to the filtered data, the operation data including at least the data filtering rules and the time information when the filtered data was filtered.

[0037] Preferably, the method further includes:

[0038] If the incremental data table does not exist, create the incremental data table;

[0039] If an incremental data table exists, the standardized data is written into the incremental data table according to the standardized data writing rules set by the user. The incremental data table represents the incremental data records.

[0040] The standardized data is mapped to the incremental data table.

[0041] According to a second aspect of this application, a data processing system is provided, comprising:

[0042] The log generation module is used to generate operation logs in response to data operation instructions sent by the user. The data operation instructions represent the data processing rules selected by the user from the pre-set configuration items.

[0043] The parsing module is used to parse the key parameters in the operation log and determine the target data identifier information and operation type in the data operation instruction;

[0044] The data capture module is used to capture target data based on the target data identifier information, wherein the target data represents the entire data of the data entry corresponding to the target data identifier information;

[0045] A data conversion module is used to respond to a user-defined configuration rule and convert the target data into structured data according to the configuration rule and the operation type. The configuration rule represents at least one configuration condition selected from pre-defined configuration items.

[0046] A filtering module is used to filter the structured data in response to user-defined data filtering rules to obtain standardized data. The data filtering rules refer to at least one data filtering condition selected from the pre-defined configuration items.

[0047] The writing module is used to write the standardized data into an incremental data table, which represents incremental data records.

[0048] According to a third aspect of this application, an electronic device is provided, comprising:

[0049] processor;

[0050] Memory for storing processor-executable instructions;

[0051] The processor is configured to execute the instructions to implement the data processing method described above.

[0052] According to a fourth aspect of this application, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of a terminal, enables the terminal to perform the data processing method described above.

[0053] Compared with the prior art, this application has the following advantages:

[0054] Users first select at least one configuration condition from pre-set configuration items as a configuration rule, and at least one data filtering condition as a data filtering rule, based on actual needs. By parsing key parameters in the operation log, the target data identifier information and operation type in the data operation instruction are determined. The target data identifier information is used to extract the entire data entry representing the corresponding data item. According to the configuration rule and data filtering rule, the target data is processed to generate standardized data. This approach allows for flexible selection of configuration rules and data filtering rules based on actual needs, providing excellent flexibility for data processing. It allows for customization of configuration conditions and data filtering conditions according to different business scenarios, thereby achieving effective processing of raw data generated under different business scenarios. This solves the cumbersome and complex technical problem of manually setting configuration parameters in existing technologies, reducing the complexity of data processing. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the steps of a data processing method provided in an embodiment of this application;

[0056] Figure 2 This is a flowchart of a method for parsing operation logs provided in an embodiment of this application;

[0057] Figure 3 This is a flowchart of a method for merging sub-transaction instructions provided in an embodiment of this application;

[0058] Figure 4 This is a flowchart of a method for generating structured data according to an embodiment of this application;

[0059] Figure 5 This is a flowchart of a method for generating a list of influencing fields according to an embodiment of this application;

[0060] Figure 6 This is a flowchart of a method for writing filtered data into a filtered data table according to an embodiment of this application;

[0061] Figure 7 This is a flowchart of a method for creating an incremental data table when the incremental data table does not exist, provided in an embodiment of this application.

[0062] Figure 8This is a flowchart of a method for mapping standardized data to an incremental data table according to an embodiment of this application;

[0063] Figure 9 This is a first example diagram of the settings interface for standardized data writing rules provided in an embodiment of this application;

[0064] Figure 10 This is a second example diagram of the settings interface for standardized data writing rules provided in an embodiment of this application;

[0065] Figure 11 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0066] Figure 12 This is a structural block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0067] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] The following is a relevant background introduction:

[0069] Existing data processing systems typically employ database management systems such as SQL Server, MySQL, and Oracle to store and manage massive amounts of data. However, in practical applications, these systems often reveal a lack of effective data validation mechanisms, failing to guarantee data consistency and accuracy. Due to the absence of data validation, data errors and outliers are difficult to detect and handle in a timely manner, thus impacting the accuracy of data analysis and business decisions.

[0070] Furthermore, as business needs constantly evolve, the structure and format of data also frequently change. This dynamism not only increases the complexity of data processing but also places higher demands on the adaptability of data management systems. Traditional database management systems often fall short in the face of such frequent changes, struggling to provide flexible and efficient data processing solutions.

[0071] To address this issue, some researchers have proposed rule-based and transformation-based data validation methods. The core of this approach lies in preprocessing and standardizing the input data by defining a series of sophisticated rules and transformation logic. During this process, the system can intelligently identify and handle common problems such as outliers and missing values, thereby significantly improving data quality and reliability. However, despite its improved data processing capabilities to some extent, this method still faces several challenges.

[0072] First, the configuration options for many existing systems are quite cumbersome and complex, requiring users to manually set various parameters and options. This manual operation not only increases the complexity of the process but is also prone to configuration errors due to human error. Second, existing systems often lack a unified data reading interface for different types of data sources (such as table structures, views, XML files, etc.). This inconsistency makes system integration extremely difficult, requiring a significant investment of time and effort for adaptation and integration.

[0073] Finally, when processing large-scale datasets, traditional serial processing methods often become a performance bottleneck. Because serial processing cannot fully utilize the parallelism of modern computing resources, processing speed is limited, thus affecting the overall system efficiency.

[0074] As a core component of enterprise data management and analysis, the data platform bears the heavy responsibility of processing large amounts of data from different data sources. With the increasing complexity and diversification of enterprise businesses, the demand for customized functions of the data platform is becoming increasingly urgent. However, existing solutions for customized data platform functions typically rely on fixed data processing flows and rules, resulting in a lack of sufficient flexibility and automation when facing ever-changing business needs.

[0075] In existing technologies, the closest solutions include data synchronization, data filtering and screening, and log processing. However, these solutions all have significant shortcomings. Regarding data synchronization, existing technologies typically employ full or incremental synchronization methods, but lack fine-grained control at the data field level, making it difficult to meet complex and ever-changing business needs. In data filtering and screening, existing solutions rely on relatively fixed rules, making it difficult to adapt to rapidly changing market environments and business requirements. Regarding log processing, existing technologies often require manual intervention for log analysis and processing, lacking automated and intelligent processing workflows.

[0076] In summary, existing data processing and data platform technologies have significant limitations in terms of flexibility, automation, and compatibility. These issues restrict the efficiency and accuracy of data processing systems and also affect the performance of data platforms in complex business environments. Therefore, developing more flexible, efficient, and intelligent data processing and data platform technologies has become an urgent problem to be solved.

[0077] Example 1

[0078] See Figure 1 As shown, the data processing method provided in this application includes:

[0079] S101. In response to the data operation command sent by the user, an operation log is generated. The data operation command represents the data processing rule selected by the user from the pre-set configuration items.

[0080] Specifically, data operation instructions refer to the actual operation instructions executed by the user, including at least the addition, deletion, and modification of data. Operation logs are detailed documents recording user data operation behaviors, including key parameters such as operation time, user, instruction, and result. Operation time records the specific time the operation occurred, helping to understand the sequence of operations and the system's activity at different times. The user indicates the user's identity, crucial for access control and accountability. Operation instructions are the core of the log, recording the actual commands or requests executed by the user. During parsing, the specific content of the instructions needs to be examined to determine the user's intent. Target data identifiers specify the specific object of the operation, such as a record ID in a database or a pathname in a file. These identifiers need to be identified and extracted during log parsing to understand the specific target of the operation. Operation results record whether the operation was successfully executed and the resulting status, helping to determine the validity of the operation and take appropriate remedial measures when necessary.

[0081] For example, a data operation instruction represents a data processing rule selected by the user from pre-set configuration items. This data processing rule includes at least configuration-specified rules, data filtering rules, and standardized data writing rules. Configuration-specified rules represent rules for converting target data into structured data; data filtering rules represent specific rules for filtering structured data; and standardized data writing rules represent specific rules for writing standardized data after the target data has been processed into standardized data. Pre-set configuration items include source database, source data table, exclusion field list, data filtering rules, target database, target data table, and filtering data processing flow. The source database and source data table represent the database and table used to store the target data, while the target database and target data table represent the database and table where the standardized data obtained after processing the target data is written. Before the user sends an operation instruction, the operation instruction is generated by selecting a data processing rule from the pre-set configuration items. Setting data processing rules through pre-set configuration items avoids the need for manual resetting each time data processing is performed due to different data processing requirements. Generating data processing rules by selecting specific options from pre-set configuration items improves data processing efficiency.

[0082] S102. Parse the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction.

[0083] Specifically, the operation type refers to the specific action performed by the user. Determining the operation type helps understand the user's intent when parsing operation logs. This process begins by identifying the data operation instructions in the log. By searching for specific keywords or commands, the data operation instructions are identified. Based on the specific content of the instructions and predefined operation rules, the corresponding operation type is determined. For example, when the data operation instruction is INSERT, the operation type is determined to be a data insertion operation; when the instruction is UPDATE, the operation type is determined to be a data update operation.

[0084] S103. Retrieve target data based on target data identifier information. The target data represents the entire data of the data entry corresponding to the target data identifier information.

[0085] Specifically, target data identifier information serves as a unique symbol or code to identify target data. This identifier information is used to locate the target data within a database or data system. For example, by comparing the parsed target data identifier information with records in the database, the location of the target data can be determined, thereby enabling the retrieval of the target data.

[0086] The captured target data includes all relevant information about the data entries, such as field values ​​and related data. Each data entry typically contains multiple fields, each used to store specific information. When capturing target data, it is necessary to ensure that all field values ​​of the data entries are obtained to guarantee that the target data is fully represented.

[0087] S104. In response to the configuration specification rule set by the user, and based on the configuration specification rule and the operation type, the target data is converted into structured data, wherein the configuration specification rule represents at least one configuration condition selected from the pre-set configuration items.

[0088] Specifically, configuration rules are set by users according to their actual needs. They are a series of conditions and operations that guide how to transform target data into structured data. Structured data refers to data with a fixed format and a finite set, which has the advantages of being easy to query and integrate. Configuration rules are usually based on pre-set configuration items, from which users can select at least one configuration condition to build rules. Pre-set configuration items include at least data type, data format, and transformation logic. Specific data types include at least text, numeric, and date; data formats include at least JSON, CSV, and XML; and transformation logic includes at least data cleaning, data filtering, data splitting, and data mapping.

[0089] Based on the user-defined configuration rules, transformation rules are generated to convert target data into structured data. These rules guide how to process the target data and achieve the conversion. Before converting the target data into structured data, the target data is preprocessed to ensure its completeness, accuracy, and suitability, thus guaranteeing a smooth conversion process. During the conversion, the target data is cleaned, mapped, and split according to the transformation rules and operation type to generate structured data. For example, structured data includes at least the operation log ID, target data identifier information, operation type, operation time, and current values ​​of each field.

[0090] S105. In response to user-defined data filtering rules, filter structured data to obtain standardized data. The data filtering rules refer to at least one data filtering condition selected from pre-defined configuration items.

[0091] Specifically, the data filtering rules set by the user are set by the user according to actual needs. Standardized data refers to datasets that conform to specific standards or specifications, used to ensure data consistency, accuracy, and comparability. Data filtering rules are selected by the user from the pre-set configuration items. According to the data filtering rules set by the user, the filtering conditions are specified, and the structured data is filtered to obtain standardized data.

[0092] For example, pre-configured items include at least database read rules, data table read rules, a list of fields to be excluded, data filtering conditions, database write rules, data table write rules, and data processing flow rules. More specifically, data filtering conditions include at least data type filtering conditions, data format filtering conditions, specific value filtering conditions, data quality filtering conditions, and business logic filtering conditions. Data type filtering conditions refer to conditions that filter based on the essential type of data, including filtering rules for different types of data such as dirty data, ignored data, and confidential data. Data format filtering conditions are based on the data's storage format. Specific value filtering conditions are based on the values ​​of specific fields in the data, such as retaining or excluding data containing specific keywords, numerical ranges, or date intervals. Business logic filtering is set according to business logic or rules. Data table write rules include valid data table write rules and filtered data table write rules. Valid data tables are used to write standardized data after data processing, and filtered data tables are used to write data that has been filtered out. Data processing flow rules include at least the target responsible person, configured specified rules, and trigger time. The target responsible person refers to the list of users authorized to send data operation instructions.

[0093] In this embodiment, the user first selects at least one configuration condition from pre-set configuration items as a configuration specification rule and at least one data filtering condition as a data filtering rule, based on actual needs. By parsing key parameters in the operation log, the target data identifier information and operation type in the data operation instruction are determined. The target data identifier information captures the entire data representing the data entry corresponding to the target data identifier information. According to the configuration specification rule and data filtering rule, the target data is processed to generate standardized data. The user can flexibly select configuration specification rules and data filtering rules according to actual needs to achieve data processing, providing good flexibility. Configuration conditions and data filtering conditions can be customized according to different business scenarios, thereby achieving effective processing of raw data generated under different business scenarios. This solves the technical problem of cumbersome and complex manual setting of configuration item parameters in existing technologies, reducing the complexity of data processing.

[0094] To better understand this application, the following detailed description is provided in conjunction with specific implementation methods.

[0095] See Figure 2 As shown, S101, parse the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction, including:

[0096] S201. Analyze the operation log and confirm the command trigger scenario.

[0097] Specifically, the command triggering scenarios include initial operation and daily real-time triggering. By parsing the operation logs, we can determine the specific time when the data operation command sent by the user was responded to, and understand under what circumstances the data operation command was triggered, which helps to determine the subsequent processing flow.

[0098] S202. When the instruction triggering scenario is the first run trigger, a temporary data table with the same structure as the original data table is created. The first run trigger indicates the first generation of the operation log. The temporary data table is used to temporarily store and process the data in the original data table.

[0099] Specifically, when the confirmation command is triggered for the first time, a data table synchronization initialization will be performed first. This involves creating a temporary data table with the same structure as the original data table to temporarily store and process the data from the original data table. Specifically, the temporary data table must include at least the same fields, data types, and indexes defined as those in the original data table. By using a temporary data table, operations that could cause problems can be avoided by directly executing them on the original data table, thereby reducing the risk of data loss or corruption and preventing direct impact on the original data table.

[0100] S203. Confirm the identifier information of all data in the temporary data table as the target data identifier information, and confirm the operation type as all data in the newly added temporary data table.

[0101] Specifically, after the temporary data table is created and populated with data, it is necessary to determine the identifier information of all data in the temporary data table, and confirm the identifier information of all data as the target data identifier information. The operation type is confirmed as adding all data in the temporary data table as new data, that is, adding all data in the temporary data table to the target location.

[0102] Confirming the target data identifier information helps ensure the uniqueness and accuracy of the data, avoid data duplication and conflicts. In the case of the instruction triggering scenario being the first time, confirming the operation type as new data provides clear guidance for subsequent data processing, ensuring the accuracy and effectiveness of data processing.

[0103] See Figure 3 As shown, a data operation instruction includes at least one sub-transaction instruction, and each sub-transaction instruction includes at least the sub-transaction operation type, the sub-transaction operation object, and the sub-transaction operation content.

[0104] S101. Parse the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction, including the following steps:

[0105] S301. Analyze the key parameters in the operation log to determine the sub-transaction instructions.

[0106] Specifically, each data operation instruction includes at least one sub-transaction instruction. Each sub-transaction instruction records a specific data operation. By parsing the key parameters in the operation log, the sub-transaction instructions are identified, providing support for subsequent classification, correlation analysis, and merging.

[0107] S302. Classify the sub-transaction instructions according to the sub-transaction operation type and generate at least one sub-transaction group.

[0108] Specifically, different sub-transaction instructions may belong to different operation types. By classifying them, instructions with similar operation types are grouped together, which facilitates subsequent analysis and processing.

[0109] S303. At least based on the sub-transaction operation object and the sub-transaction operation content, analyze the relationship between the sub-transaction instructions in the sub-transaction group.

[0110] S304. Generate a sub-transaction merging strategy based on the relationships between the sub-transaction instructions in the sub-transaction group.

[0111] S305. According to the sub-transaction merging strategy, merge the sub-transaction instructions in each sub-transaction instruction group to generate an instruction merge set.

[0112] S306. Analyze the instruction merge set to determine the target data identifier information and operation type.

[0113] Specifically, by analyzing the relationships between the sub-transaction instructions within a sub-transaction group, a sub-transaction merging strategy is formulated. Based on this strategy, sub-transaction instructions that meet the merging criteria are merged to form a merged instruction set. The operation type and target identifier information of the merged instruction set are then determined. Using a sub-transaction merging strategy to combine logically related or sequentially consecutive sub-transaction instructions into a larger operation unit simplifies operation logs, reduces data operation redundancy, avoids performance overhead caused by frequent database writes, reduces disk I / O operations, and improves overall throughput and data processing efficiency.

[0114] See Figure 4 As shown, S104, responding to the user-defined configuration rules and converting the target data into structured data according to the configured rules and operation type, includes the following steps:

[0115] S401. Based on the operation log and target data, construct a structured data framework. The structured data framework shall include at least the identifier information of the operation log, the identifier information of the target data, the operation type, and the operation time.

[0116] Specifically, in constructing a structured data framework, the first step is to define the fields within the framework based on the data operation instructions and target data in the operation log. These fields should cover all key information from both the operation log and the target data. Next, a suitable data type is determined for each field to ensure data accuracy and consistency. For example, the operation time field uses a date and time type, while the operation type uses an enumeration type. Finally, the relationships between multiple fields are clarified, generating the structured data framework. By constructing a structured data framework, a unified data model is provided, enabling the operation log and target data to be stored and managed in a standardized manner, which facilitates subsequent data processing and analysis.

[0117] S402. In response to the configured rules, identify and delete the specified fields in the target data to generate the data to be structured.

[0118] Specifically, the specified fields in the target data represent unnecessary fields. The specified rules are selected based on preset configuration items. Specifically, the configuration items are generated in advance, and the specified rules are determined by factors such as field name, data type, and value range. After deleting the specified fields, the remaining data is the data to be structured.

[0119] By deleting specific fields from the target data, data redundancy and complexity can be reduced, ensuring that only relevant and valuable data for analysis is retained, thus improving the efficiency and accuracy of data processing.

[0120] S403. Extract all fields and the current value information of each field from the data to be structured.

[0121] Specifically, by traversing the data to be structured, the name and corresponding current value information of each field are extracted, and the current value information of all fields and each field is extracted to populate the corresponding fields in the structured data framework, thus converting the data to be structured into a structured form and ensuring the accuracy and consistency of the data.

[0122] S404. Fill all fields and the current value information of each field into the structured data frame to generate structured data.

[0123] Specifically, the name and value of each field are populated into a structured data frame to generate structured data. Using structured data ensures that data is stored and presented in a standardized, easily managed, and analyzable manner, facilitating further analysis, querying, and report generation. With structured data, users can more easily understand the meaning and structure of the data.

[0124] See Figure 5 As shown, after constructing the structured data framework based on the operation log and target data in S401, the following steps are also included:

[0125] S501. When the operation type is to update target data, extract the first influencing field from the target data. The first influencing field represents the field directly related to the updated target data.

[0126] Specifically, when the operation type is to update target data, the first impact field that is directly related to the updated target data is extracted from the target data. Identifying the first impact field can clarify which data has changed directly in the update operation, which is helpful for subsequent data processing and analysis.

[0127] S502, Identify the second influence field in the target data that is associated with the first influence field.

[0128] Specifically, by analyzing the relationships between various fields, a second influencing field is determined. For example, the relationships between various fields are determined based on the data model or business logic. By determining the second influencing field, it is ensured that all fields that have been indirectly changed due to update operations are taken into consideration, further improving the accuracy and completeness of data processing.

[0129] S503. Merge the first impact field and the second impact field to generate an impact field list, which shall include at least the first impact field and the second impact field.

[0130] Specifically, during the merging of the first and second affected fields, it is necessary to ensure that there are no duplicate fields in the list of affected fields. By generating a list of affected fields, it is possible to show which fields are affected when the target data is updated, providing data support for subsequent data verification, auditing, and report generation, and ensuring that the accuracy and integrity of the data are effectively maintained during the update operation.

[0131] See Figure 6 As shown, after S105, in response to the user-defined data filtering rules, filtering the structured data to obtain standardized data, the process also includes:

[0132] S601. Obtain the filtered data from the structured data. The filtered data represents the data that has been filtered by the data filtering rules.

[0133] Specifically, data filtering rules are applied to query or process structured data, identifying data records that meet the filtering criteria. The filtered data is then separated to form the filtered data. Filtering rules are customized according to actual needs to retain high-quality data useful for processing or analysis, ensuring the accuracy of subsequent processing or analysis.

[0134] S602. Write the filtered data into the filtered data table.

[0135] Specifically, the filtered data table is used to store the filtered data. The structure of the filtered data table is compatible with the format of the filtered data. By writing the filtered data into the filtered data table, it is convenient to review or analyze the filtered data, ensuring the transparency of the filtering operation during data processing, and facilitating auditing and traceability.

[0136] S603. Record operation data related to filtering data. The operation data shall include at least the data filtering rules and the time information when the filtered data is filtered.

[0137] Specifically, by recording operational data related to the filtered data, data governance, compliance, and audit requirements are met. In the event of data quality issues or system failures, the operational data related to the filtered data serves as an important basis for diagnosis and analysis. By analyzing the operational data, bottlenecks and deficiencies in the data processing flow can be identified, guiding the optimization and improvement of the data processing flow.

[0138] See Figures 7-10 As shown, it also includes:

[0139] S701. Create an incremental data table if it does not exist.

[0140] Specifically, in the absence of an incremental data table, the fields and data types of the incremental data table are designed according to business needs and data specifications. The table's name, field names, data types, and constraints are then created. Incremental data tables are used to store data records that are added or updated over time, facilitating effective data management and analysis. Creating dedicated incremental data tables ensures data integrity and consistency, preventing confusion with original data or data from other processing steps.

[0141] S801. If the incremental data table exists, write the standardized data into the incremental data table according to the standardized data writing rules set by the user. The incremental data table represents the incremental data records.

[0142] Specifically, standardized data writing rules refer to the rules for writing standardized data into incremental data tables according to certain formats, standards, and specifications. Incremental data tables are used to store standardized data. Writing standardized data to incremental data tables according to these rules allows for real-time or periodic updates of data records, reflecting the latest data status. Furthermore, data in incremental data tables can be integrated with original data or other data, supporting more complex data analysis and processing tasks. By setting standardized data writing rules, standardized data can be written to incremental data tables efficiently and accurately, supporting real-time data updates and integration, improving data consistency and processing efficiency, and providing strong support for complex data analysis and processing tasks.

[0143] For example, Figure 9 and Figure 10 The interface for setting up standardized data writing rules is shown. The source table refers to the standardized data, and the target table represents the table where the standardized data will be written. The specific process for users to set up standardized data writing rules includes selecting the target database and target table, configuring the incremental data table, defining the fields of the incremental data table, and setting the data writing rules. In the process of selecting the target database and target table, users first determine the database type to which the standardized data will be written. Next, they select the target table for that database type. When there is no incremental data table for that database type, the interface for setting up standardized data writing rules provides an option to automatically create the table. By selecting to automatically create the incremental data table and naming the incremental data table to be created, the storage location of the standardized data can be clearly specified.

[0144] After determining the incremental data table, the fields of the incremental data table are defined to determine the information that the incremental data table needs to record. When the fields of the standardized data and the incremental data table are inconsistent, a mapping method is used to map the standardized data to the fields of the incremental data table one-to-one. In addition, the standardized data can also be integrated to generate the corresponding fields of the incremental data table. After setting the database type, incremental data table, and fields, standardized data writing rules are generated by setting the timestamp, the type of standardized data change, and the specific content of the standardized data.

[0145] S802, Map standardized data to incremental data tables.

[0146] Specifically, the mapping process first clarifies the mapping relationship between each field of the standardized data and its corresponding field in the incremental data table. This mapping relationship includes at least matching fields or data type conversion. By defining and executing data mapping rules, it ensures that standardized data can be accurately and consistently written into the incremental data table, avoiding data loss or errors. Through proper mapping, it ensures that the data in the incremental data table is easier to understand and analyze, supporting more efficient data processing tasks.

[0147] Example 2

[0148] See Figure 11 As shown, this application provides a data processing system, including:

[0149] The log generation module 400 is used to generate operation logs in response to data operation instructions sent by the user. The data operation instructions represent the data processing rules selected by the user from the pre-set configuration items.

[0150] The parsing module 500 is used to parse key parameters in the operation log to determine the target data identifier information and operation type in the data operation instructions.

[0151] The data capture module 600 is used to capture target data based on target data identifier information. The target data represents the entire data of the data entry corresponding to the target data identifier information.

[0152] The data conversion module 700 is used to respond to the configuration-specified rules set by the user and convert the target data into structured data according to the configuration-specified rules and the operation type. The configuration-specified rules represent at least one configuration condition selected from the pre-set configuration items.

[0153] The filtering module 800 is used to filter structured data and obtain standardized data in response to user-defined data filtering rules. The data filtering rules refer to at least one data filtering condition selected from pre-set configuration items.

[0154] In some embodiments, the data capture module 600 includes:

[0155] The parsing unit is used to parse the operation log and confirm the command trigger scenario.

[0156] The creation unit is used to create a temporary data table with the same structure as the original data table when the instruction trigger scenario is the first run trigger. The first run trigger indicates the first generation of the operation log. The temporary data table is used to temporarily store and process the data in the original data table.

[0157] The confirmation unit is used to confirm the identifier information of all data in the temporary data table as the target data identifier information, and to confirm the operation type as all data in the newly added temporary data table.

[0158] In some embodiments, a data manipulation instruction includes at least one sub-transaction instruction, and each sub-transaction instruction includes at least a sub-transaction operation type, a sub-transaction operation object, and a sub-transaction operation content.

[0159] The data capture module 600 also includes: parsing key parameters in the operation log to determine sub-transaction instructions.

[0160] The classification unit is used to classify sub-transaction instructions according to the sub-transaction operation type and generate at least one sub-transaction group.

[0161] The analysis unit is used to analyze the relationships between the sub-transaction instructions in the sub-transaction group, based at least on the sub-transaction operation object and the sub-transaction operation content.

[0162] The strategy generation unit is used to generate a sub-transaction merging strategy based on the relationship between the sub-transaction instructions in the sub-transaction group.

[0163] The merging unit is used to merge the sub-transaction instructions in each sub-transaction instruction group according to the sub-transaction merging strategy, and generate an instruction merge set.

[0164] The determination unit is used to analyze the instruction merge set and determine the target data identifier information and operation type.

[0165] In some embodiments, the data conversion module 700 includes:

[0166] The building unit is used to construct a structured data framework based on the operation log and the target data. The structured data framework includes at least the identifier information of the operation log, the identifier information of the target data, the operation type, and the operation time.

[0167] The delete unit is used to identify and delete specified fields in the target data in response to configured rules, generating data to be structured.

[0168] The extraction unit is used to extract all fields and the current value information of each field from the data to be structured.

[0169] Structured data units are used to populate a structured data frame with all fields and the current value information of each field to generate structured data.

[0170] In some embodiments, it also includes:

[0171] The associated field module is used to extract the first influencing field from the target data when the operation type is to update the target data. The first influencing field represents the field that is directly associated with the updated target data.

[0172] The identification module is used to identify the second influence field in the target data that is associated with the first influence field.

[0173] The Influence Field List module is used to merge the first influence field and the second influence field to generate an influence field list, which includes at least the first influence field and the second influence field.

[0174] In some embodiments, it also includes:

[0175] The data filtering module is used to obtain filtered data from structured data. Filtered data represents data that has been filtered out according to data filtering rules.

[0176] The write module is used to write the filtered data into the filtered data table.

[0177] The recording module is used to record operational data related to the filtered data. The operational data includes at least the data filtering rules and the time information when the filtered data was filtered.

[0178] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0179] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0180] Example 3

[0181] Please see Figure 12 This application also provides an electronic device, including:

[0182] processor.

[0183] Memory is used to store processor-executable instructions.

[0184] The processor is configured to execute instructions to implement any data processing method.

[0185] In this embodiment, the computer device includes a processor, memory, and network interface connected via a system bus.

[0186] The processor of this computer device provides computing and control capabilities. The memory of this computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of this computer device is used to store data samples. The network interface of this computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements any data processing method.

[0187] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] Example 4

[0189] This application also provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a terminal, enables the terminal to perform any data processing method.

[0190] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0191] Optionally, a readable storage medium can be coupled to the processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0192] Example 5

[0193] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any data processing method.

[0194] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0199] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0200] The data processing method, system, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, Includes the following steps: In response to a data operation instruction sent by a user, an operation log is generated. The data operation instruction represents the data processing rule selected by the user from a pre-set configuration item. The data operation instruction includes at least one sub-transaction instruction, and each sub-transaction instruction includes at least a sub-transaction operation type, a sub-transaction operation object, and a sub-transaction operation content. Parsing the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction includes the following steps: The process involves parsing key parameters in the operation log to determine the sub-transaction instructions; classifying the sub-transaction instructions according to their operation types to generate at least one sub-transaction group; analyzing the relationships between the sub-transaction instructions within each sub-transaction group based on at least the sub-transaction operation object and the sub-transaction operation content; generating a sub-transaction merging strategy based on the relationships between the sub-transaction instructions within each sub-transaction group; merging the sub-transaction instructions in each sub-transaction instruction group according to the sub-transaction merging strategy to generate an instruction merge set; and analyzing the instruction merge set to determine the target data identifier information and the operation type. Target data is retrieved based on the target data identifier information, whereby the target data represents the entire data entry corresponding to the target data identifier information. In response to a user-defined configuration rule, and based on the configuration rule and the operation type, the target data is converted into structured data, wherein the configuration rule represents at least one configuration condition selected from pre-defined configuration items; In response to user-defined data filtering rules, the structured data is filtered to obtain standardized data, wherein the data filtering rules refer to at least one data filtering condition selected from the pre-defined configuration items.

2. The data processing method as described in claim 1, characterized in that, The step of parsing the key parameters in the operation log to determine the target data identifier information and operation type in the data operation instruction includes the following steps: Analyze the operation log to confirm the command trigger scenario; When the instruction triggering scenario is the first run trigger, a temporary data table with the same structure as the original data table is created. The first run trigger indicates that the operation log is generated for the first time. The temporary data table is used to temporarily store and process the data in the original data table. The identifier information of all data in the temporary data table is confirmed as the target data identifier information, and the operation type is confirmed as adding all data in the temporary data table.

3. The data processing method as described in claim 1, characterized in that, The step of responding to user-defined configuration rules and converting the target data into structured data according to the configuration rules and the operation type includes the following steps: Based on the operation log and the target data, a structured data framework is constructed, which includes at least the identifier information of the operation log, the identifier information of the target data, the operation type, and the operation time. In response to the rules specified in the configuration, the specified fields in the target data are identified and deleted to generate data to be structured; Extract all fields and the current value information of each field from the data to be structured; The structured data is generated by filling all the fields and the current value information of each field into the structured data frame.

4. The data processing method as described in claim 3, characterized in that, After constructing the structured data framework based on the operation logs and the target data, the following steps are also included: When the operation type is to update target data, a first influencing field is extracted from the target data, wherein the first influencing field represents the field directly associated with the updated target data; Identify the second influence field in the target data that is associated with the first influence field; The first impact field and the second impact field are merged to generate an impact field list, which includes at least the first impact field and the second impact field.

5. The data processing method as described in claim 1, characterized in that, After filtering the structured data in response to user-defined data filtering rules to obtain standardized data, the process further includes: Obtain filtered data from the structured data, wherein the filtered data represents data that has been filtered by the data filtering rules; Write the filtered data into the filtered data table; Record operation data related to the filtered data, the operation data including at least the data filtering rules and the time information when the filtered data was filtered.

6. The data processing method as described in claim 1, characterized in that, The method further includes: If the incremental data table does not exist, create the incremental data table; If an incremental data table exists, the standardized data is written into the incremental data table according to the standardized data writing rules set by the user. The incremental data table represents the incremental data records. The standardized data is mapped to the incremental data table.

7. A data processing system, characterized in that, include: The log generation module is used to generate operation logs in response to data operation instructions sent by the user. The data operation instructions represent the data processing rules selected by the user from the pre-set configuration items. The data operation instructions include at least one sub-transaction instruction, and each sub-transaction instruction includes at least the sub-transaction operation type, the sub-transaction operation object, and the sub-transaction operation content. The parsing module is used to parse the key parameters in the operation log and determine the target data identifier information and operation type in the data operation instruction, including the following steps: The process involves parsing key parameters in the operation log to determine the sub-transaction instructions; classifying the sub-transaction instructions according to their operation types to generate at least one sub-transaction group; analyzing the relationships between the sub-transaction instructions within each sub-transaction group based on at least the sub-transaction operation object and the sub-transaction operation content; generating a sub-transaction merging strategy based on the relationships between the sub-transaction instructions within each sub-transaction group; merging the sub-transaction instructions in each sub-transaction instruction group according to the sub-transaction merging strategy to generate an instruction merge set; and analyzing the instruction merge set to determine the target data identifier information and the operation type. The data capture module is used to capture target data based on the target data identifier information, wherein the target data represents the entire data of the data entry corresponding to the target data identifier information; A data conversion module is used to respond to a user-defined configuration rule and convert the target data into structured data according to the configuration rule and the operation type. The configuration rule represents at least one configuration condition selected from pre-defined configuration items. A filtering module is used to filter the structured data in response to user-defined data filtering rules to obtain standardized data, wherein the data filtering rules refer to at least one data filtering condition selected from the pre-defined configuration items.

8. An electronic device, characterized in that, include: processor; Memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the terminal, the terminal is able to perform the data processing method according to any one of claims 1 to 6.

Images