A data processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a data processing method, device, equipment and medium, the method comprises: performing data governance operation on to-be-evaluated data through current data governance rules to obtain governed data, wherein the data governance operation at least includes parsing and storing the to-be-evaluated data; performing data quality inspection on the governed data to obtain a quality evaluation result of the governed data; in the case that the quality evaluation result does not meet a preset requirement, optimizing and updating the current data governance rules; repeating the above steps until the quality evaluation result meets the preset requirement, and saving the current data governance rules. Through some embodiments of the present application, the quality and efficiency of data governance can be improved through continuous optimization of data governance rules.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data security, and in particular, to a data processing method and device, equipment and medium. BACKGROUND

[0002] In the data governance process, data collection, data processing, data sharing and data quality assessment are performed by respective modules, and there is a problem of fragmentation of data processing and data quality assessment, and there is a lack of effective linkage optimization mechanism. In related technologies, a self-defined engine and a model are applied to assess data quality, but after processing, the user still needs to manually perform data processing optimization, resulting in reduced quality and efficiency of data governance.

[0003] Therefore, how to improve the quality and efficiency of data governance has become a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a data processing method, device, equipment and medium, which can at least improve the quality and efficiency of data governance by continuously optimizing data governance rules through some embodiments of the present application.

[0005] In a first aspect, the present application provides a data processing method, comprising: performing a data governance operation on to-be-evaluated data through a current data governance rule to obtain governed data, wherein the data governance operation at least includes parsing and storing the to-be-evaluated data; performing data quality checking on the governed data to obtain a quality evaluation result of the governed data; in the case that the quality evaluation result does not meet a preset requirement, optimizing and updating the current data governance rule; repeating the above steps until the quality evaluation result meets the preset requirement, and saving the current data governance rule.

[0006] Therefore, unlike the method in related technologies that requires the user to continue to manually perform data processing optimization, the embodiments of the present application can automatically optimize and update the current data governance rule in the case that the evaluation result does not meet the preset requirement, so that the parameters in the data governance rule can be adjusted to be optimal, thereby improving the quality and efficiency of data governance.

[0007] In an implementation of the first aspect, the current data governance rule comprises a data processing rule and a data sharing rule; and the optimizing the current data governance rule in the case that the quality evaluation result does not meet the preset requirement comprises: in the case that the quality evaluation result does not meet the preset requirement, obtaining the current data processing rule parameter and the current data sharing rule parameter; adjusting the values of the current data processing rule parameter and the current data sharing rule parameter to obtain an adjusted data processing rule parameter and an adjusted data sharing rule parameter; calculating a change rate of the adjusted data processing rule parameter and the adjusted data sharing rule parameter, wherein the change rate is obtained by calculating the number of times of parameter value adjustment in a period of time; and in the case that the change rate meets a change rate threshold, confirming that the optimizing the current data governance rule is completed in the current cycle.

[0008] Therefore, by calculating the change rate of parameter adjustment, the frequency of parameter adjustment can be controlled, and the influence of data quality evaluation caused by too fast or too slow parameter adjustment can be prevented.

[0009] In an implementation of the first aspect, the adjusting the values of the current data processing rule parameter and the current data sharing rule parameter to obtain an adjusted data processing rule parameter and an adjusted data sharing rule parameter comprises: multiplying the current data processing rule parameter and the current data sharing rule parameter by corresponding preset proportions respectively to obtain a sharing rule proportion adjustment parameter and a processing rule proportion adjustment parameter; and adding the sharing rule proportion adjustment parameter and the processing rule proportion adjustment parameter to corresponding preset values to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0010] In an implementation of the first aspect, the adjusting the values of the current data processing rule parameter and the current data sharing rule parameter to obtain an adjusted data processing rule parameter and an adjusted data sharing rule parameter comprises: adding the current data processing rule parameter and the current data sharing rule parameter to corresponding preset values respectively to obtain a sharing rule value adjustment parameter and a processing rule value adjustment parameter; and multiplying the sharing rule value adjustment parameter and the processing rule value adjustment parameter by corresponding preset proportions respectively to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0011] Therefore, the embodiments of the present application can continuously try the arrangement and combination of the rule parameters and the size of the numerical value by adjusting the rule parameters by multiplying a coefficient and adding a numerical value, so that the optimal rule parameters can be found.

[0012] In combination with the first aspect, in an embodiment of the present application, before the data management operation on the to-be-evaluated data is performed by using the current data management rule to obtain the managed data, the method further comprises: collecting the to-be-evaluated data by using a current data collection rule; and the optimization and updating of the current data management rule in the case where the quality evaluation result does not meet the preset requirement comprises: optimization and updating of the current data management rule and the current data collection rule in the case where the quality evaluation result does not meet the preset requirement.

[0013] Therefore, the embodiments of the present application can optimize the data collection process and improve the data collection quality by updating the current data collection rule.

[0014] In the second aspect, the present application provides a data processing apparatus, comprising: a data management module configured to perform a data management operation on to-be-evaluated data by using a current data management rule to obtain managed data, wherein the data management operation at least comprises data analysis and storage of the to-be-evaluated data; a data quality checking module configured to perform data quality checking on the managed data to obtain a quality evaluation result of the managed data; a rule optimization module configured to optimize and update the current data management rule in the case where the quality evaluation result does not meet a preset requirement; and a rule storage module configured to repeat the above steps until the quality evaluation result meets the preset requirement and save the current data management rule.

[0015] In combination with the second aspect, in an embodiment of the present application, the current data management rule comprises a data processing rule and a data sharing rule; and the rule optimization module is further configured to: in the case where the quality evaluation result does not meet the preset requirement, obtain the current data processing rule parameter and the current data sharing rule parameter; adjust the numerical value of the current data processing rule parameter and the current data sharing rule parameter to obtain an adjusted data processing rule parameter and an adjusted data sharing rule parameter; calculate a change rate of the adjusted data processing rule parameter and the adjusted data sharing rule parameter, wherein the change rate is obtained by calculating the number of times of parameter value adjustment in a period of time; and in the case where the change rate meets a change rate threshold, it is confirmed that the optimization of the current data management rule in the current cycle is completed.

[0016] In combination with the second aspect, in an embodiment of the present application, the rule optimization module is further configured to: multiply the current data processing rule parameter and the current data sharing rule parameter by corresponding preset ratios respectively to obtain a sharing rule ratio adjustment parameter and a processing rule ratio adjustment parameter; and add the sharing rule ratio adjustment parameter and the processing rule ratio adjustment parameter to corresponding preset values to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0017] In combination with the second aspect, in an embodiment of the present application, the rule optimization module is further configured to: add the current data processing rule parameter and the current data sharing rule parameter to corresponding preset values respectively to obtain a sharing rule value adjustment parameter and a processing rule value adjustment parameter; and multiply the sharing rule value adjustment parameter and the processing rule value adjustment parameter by corresponding preset ratios respectively to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0018] In combination with the second aspect, in an embodiment of the present application, the data governance module is further configured to: collect to-be-evaluated data through a current data collection rule; and optimize and update the current data governance rule and the current data collection rule in a case where the quality evaluation result does not meet the preset requirement.

[0019] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory and a bus; the processor is connected with the memory through the bus, the memory stores a computer program, and the computer program is executed by the processor to realize the method according to any embodiment of the first aspect.

[0020] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to realize the method according to any embodiment of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A system composition schematic diagram of data processing according to an embodiment of the present application is shown;

[0022] Figure 2 One of method flowcharts of data processing according to an embodiment of the present application is shown;

[0023] Figure 3 Another of method flowcharts of data processing according to an embodiment of the present application is shown;

[0024] Figure 4 The third of method flowcharts of data processing according to an embodiment of the present application is shown;

[0025] Figure 5 The fourth method flowchart of data processing shown in the embodiments of the present application;

[0026] Figure 6 The fifth method flowchart of data processing shown in the embodiments of the present application;

[0027] Figure 7 The sixth method flowchart of data processing shown in the embodiments of the present application;

[0028] Figure 8 The seventh method flowchart of data processing shown in the embodiments of the present application;

[0029] Figure 9 The device composition of data processing shown in the embodiments of the present application;

[0030] Figure 10 The schematic diagram of electronic equipment composition shown in the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0032] The embodiments of the present application can be applied to the scene of data processing and optimization of data processing rules. In order to improve the problems in the background art, in some embodiments of the present application, the current data governance rule is optimized and updated in the case that the quality evaluation result does not meet the preset requirement. For example, in some embodiments of the present application, in the case that the quality evaluation result does not meet the preset requirement, the current data processing rule parameter, the current data sharing rule parameter and the current data collection rule parameter are adjusted, and in the case that the change rate meets the change rate threshold, it is confirmed that the current cycle optimization is completed.

[0033] The method steps in the embodiments of the present application will be described in detail below with reference to the drawings.

[0034] Figure 1A structural diagram of a system for data processing in some embodiments of the present application is provided, which includes a client 110 and a server 120. The client 110 sends to-be-evaluated data to the server 120, and the server 120 performs quality evaluation on the to-be-evaluated data after data processing. If the quality evaluation result does not meet the preset requirement, the current data processing rule parameter, the current data sharing rule parameter, and the current data collection rule parameter are adjusted, and the current cycle optimization is confirmed to be completed if the change rate meets the change rate threshold.

[0035] Unlike the embodiments of the present application, in the related art, the data quality is evaluated by using a custom engine and a model, but the user still needs to manually optimize the data processing after processing, which reduces the quality and efficiency of data governance. Therefore, the related art lacks an effective data quality evaluation result feedback data processing mechanism, and cannot realize optimization of the data processing strategy based on the data quality evaluation result. In the embodiments of the present application, the rules of data governance are automatically optimized when the data after governance does not meet the quality requirement.

[0036] Therefore, the present application focuses on further optimizing the data processing strategy based on the data quality evaluation result, and applying the data processing strategy, to realize the closed-loop effect of data collection, data processing, quality evaluation, and secondary optimization of data processing. The data processing method proposed in the present application can be flexibly configured and adaptively applied, which improves the data processing effect, so that the optimal data governance rule parameters can be obtained without manual adjustment of parameters.

[0037] It should be noted that the to-be-evaluated data in the embodiments of the present application can be any data in any scenario. For example, the data processing method provided by the present application can be applied to a network security scenario to process network flow data or log data.

[0038] A data processing method performed by a server in the embodiments of the present application will be described below. It can be understood that the data processing method of the embodiments of the present application can be applied to any server.

[0039] At least to solve the problems in the background art, as shown in Figure 2 Some embodiments of the present application provide a data processing method, which includes:

[0040] S210, performing a data governance operation on the to-be-evaluated data by using the current data governance rule to obtain data after governance.

[0041] In an embodiment of the present application, before S210, the method further includes: collecting the to-be-evaluated data by using a current data collection rule.

[0042] In other words, the data to be evaluated is collected by the server according to the current data collection rules. Specifically, system, application, or security logs are collected through active or passive collection methods, and preprocessed according to preset data specifications to enable real-time correlation analysis and data analysis of the data to be evaluated (e.g., network traffic data). Data is collected from various business application systems, devices, servers, terminals, and other devices within the network using a log collector, and is collected through various methods such as system logs, Simple Network Management Protocol (SMMP), network monitoring (Netflow), API interfaces, mirrored traffic, and files.

[0043] In one embodiment of this application, the current data governance rules include data processing rules and data sharing rules. The data processing rules are used to perform data processing operations such as parsing and filtering on the received data to be evaluated, while the data sharing rules are used to share and forward the processed data to be evaluated. In this embodiment, after obtaining the data to be evaluated through the current data collection rules, firstly, the data to be evaluated is processed using the current data processing rules to obtain the processed data to be evaluated. Then, the processed data to be evaluated is stored. Finally, the processed data to be evaluated is shared using the data sharing rules. It is understood that data sharing is achieved through data sharing applications, enabling data forwarding, dedicated data sharing, API interface management, and forwarding quality monitoring.

[0044] As a specific embodiment of this application, such as Figure 3 As shown, the process of managing data processing rules includes: data parsing rule management 301, data filtering rule management 302, data enrichment rule management 303, and data tagging rule management 304. For example, processing rules are configured for different data collectors to process the collected data to be evaluated. For logs of a specific data source type, data processing rules are constructed by formulating and applying parsing rules, filtering rules, enrichment rules, and tagging rules. The logs of this type are then parsed, filtered, enriched, and tagged before being stored in the database. Existing data processing rules are edited and modified according to changes in data type and data source. Simultaneously, processing control is achieved through enabling and disabling log collectors.

[0045] In other words, this application manages and applies combination rules, parsing rules, filtering rules, enrichment rules, and tagging rules in data processing rules through rule management. The data processing rule module enables centralized management of processing rules for all data sources, allowing for basic management through editing, batch deletion, importing, and exporting. Furthermore, this application manages the order of parsing, filtering, enrichment, and tagging rules, as well as the selection of different rules, within the organization and management of data processing rules, thus enabling the application of rule strategies.

[0046] As a specific embodiment of this application, such as Figure 4 As shown, the execution order of the rules in the data processing rules is set. The specific process includes: first, using data parsing rule 401; then using data filtering rule 402; next, using data enrichment rule 403; and finally using data labeling rule 404. Afterwards, application rule 405 is executed to apply each rule to the system. In other words, the data processing process for the data to be evaluated is as follows: first, data parsing rule 401 is used to parse the data; then, data filtering rule 402 is used to filter the parsed data; next, data enrichment rule 403 is used to enrich the filtered data; and finally, data labeling rule 404 is used to combine the enriched data to obtain the processed data to be evaluated.

[0047] Specifically, this application parses log data using data parsing rules, including log data collected through various methods such as system log collection, simple network management protocol collection, text format collection, database collection, and system plugin collection. Based on data samples, target fields are extracted using regular expressions, delimiters, key-value pairs, and JSON (JavaScript Object Notation). By defining pre-filtering rules, logs that do not meet the requirements are quickly filtered out, improving parsing speed.

[0048] This application defines various semantics such as AND, NOT, and OR through data filtering rules, which can further match the feature fields parsed by the data parsing rules. For example, if the type field is parsed according to the data parsing rules, the data filtering rules can be configured to match log type A when type=1, and match log type B when type=2.

[0049] This application uses data enrichment rules to generate new fields from the source fields parsed according to data parsing rules, thereby enriching the data content. For example, it automatically generates country, city, latitude and longitude based on the IP field.

[0050] This application uses data labeling rules to assign different defined labels to different types of data, enabling applications such as data classification and grading. Examples include the location of the data, data ownership, data identity tags, data source, and data identifiers.

[0051] S220 involves conducting a data quality check on the treated data to obtain a quality assessment result for the treated data.

[0052] In one embodiment of this application, data quality inspection involves a comprehensive quantitative analysis of the quality of secure data across multiple dimensions, including data comprehensiveness, data accuracy, data timeliness, data sharing, and data collection. This analysis automatically generates a quality report and incorporates historical data to provide data quality trends.

[0053] In other words, such as Figure 5 As shown, the process of checking the data quality of the treated data first executes the S510 quality rule definition, then executes the S520 data quality model calculation, then executes the S530 quality assessment report data, and finally executes the S540 rectification application.

[0054] Specifically, a quality rule engine is used to define data quality rules, build quality models to check data quality, and then evaluate the data based on algorithms to generate quality scores. The quality rule engine enables the definition, editing, and submission of data quality rules, as well as monitoring the execution status and aggregating results. Monitoring allows for keyword search queries along rule and table dimensions. A quality model is defined, allowing for customized management based on different project environments. Quality assessment calculates quality scores by combining task execution results with the factor algorithms and weights defined in the quality model.

[0055] S230: If the quality assessment results do not meet the preset requirements, optimize and update the current data governance rules.

[0056] Understandably, quality assessment results can be represented by quality scores, and preset requirements for quality scores can be set in advance. For example, a quality score greater than or equal to 60 points indicates that the current quality assessment result meets the preset requirements, or a quality score less than 60 points indicates that the current quality assessment result does not meet the preset requirements.

[0057] In one embodiment of this application, the specific implementation process for optimizing and updating the current data governance rules includes:

[0058] S2301, if the quality assessment result does not meet the preset requirements, obtain the current data processing rule parameters, the current data sharing rule parameters, and the current data acquisition rule parameters.

[0059] It should be noted that the data processing rule parameters include: the execution order of each rule in the data processing rule, the identifier of the rule used for data parsing (e.g., identifier 1 corresponds to using the first method for data parsing, and identifier 2 corresponds to using the second method for data parsing), the identifier of the content enriched during the data enrichment process (e.g., identifier 1 corresponds to using the region to enrich the data, and identifier 2 corresponds to using latitude and longitude to enrich the data), and the identifier of the label used for data labeling (e.g., identifier c corresponds to using the data source to label the data), etc.

[0060] It should be noted that the data sharing rule parameters include: the identifier corresponding to the data sharing method, for example, the identifier corresponding to data sharing through the first method is 3; and the identifier corresponding to the protocol used for data sharing, etc.

[0061] It should be noted that the data collection rule parameters include: the identifier corresponding to the data collection path, the identifier corresponding to the data collection method, etc.

[0062] It is understood that the parameter types mentioned above are merely examples, and this application does not limit the specific types of data governance rule parameters.

[0063] S2302, adjust the values ​​of the current data processing rule parameters and the current data sharing rule parameters to obtain the adjusted data processing rule parameters and the adjusted data sharing rule parameters.

[0064] In one embodiment of this application, the current data processing rule parameters, the current data sharing rule parameters, and the current data acquisition rule parameters are first multiplied by their respective preset ratios to obtain the sharing rule ratio adjustment parameters, the processing rule ratio adjustment parameters, and the acquisition rule ratio adjustment parameters.

[0065] For example, if the preset ratio is 2, and the current data processing rule parameters include the identifier 1 corresponding to using the first method for data parsing and the identifier 2 for enriching the data using latitude and longitude, then the preset ratio is multiplied by the identifier 1 corresponding to using the first method for data parsing and the identifier 2 for enriching the data using latitude and longitude to obtain the processing rule ratio adjustment parameters, which include the identifier 2 corresponding to data parsing and the identifier 4 for enriching the data.

[0066] Then, the shared rule ratio adjustment parameters, processing rule ratio adjustment parameters, and acquisition rule ratio adjustment parameters are added to the corresponding preset values ​​to obtain the adjusted data processing rule parameters, adjusted data sharing rule parameters, and adjusted data acquisition rule parameters.

[0067] For example, with a preset value of 1, the processing rule ratio adjustment parameters include: identifier 2 corresponding to data parsing and identifier 4 for data enrichment. The processing rule ratio adjustment parameters are added to the preset value 1 to obtain identifier 3 corresponding to data parsing and identifier 5 for data enrichment. The adjusted data processing rule parameters are then obtained, including identifier 3 corresponding to data parsing and identifier 5 for data enrichment.

[0068] In another embodiment of this application, firstly, the current data processing rule parameters, the current data sharing rule parameters, and the current data acquisition rule parameters are added to their corresponding preset values ​​to obtain the sharing rule value adjustment parameters, the processing rule value adjustment parameters, and the acquisition rule value adjustment parameters.

[0069] For example, if the preset value is 1, and the current data processing rule parameters include the identifier 1 corresponding to using the first method for data parsing and the identifier 2 for enriching the data using latitude and longitude, then the preset value 1 is added to the identifier 1 corresponding to using the first method for data parsing and the identifier 2 for enriching the data using latitude and longitude to obtain the processing rule value adjustment parameters, including the identifier 2 corresponding to data parsing and the identifier 3 for enriching the data.

[0070] Then, the shared rule value adjustment parameters, processing rule value adjustment parameters, and acquisition rule value adjustment parameters are multiplied by their respective preset ratios to obtain the adjusted data processing rule parameters, adjusted data sharing rule parameters, and adjusted data acquisition rule parameters.

[0071] For example, with a preset ratio of 2, the processing rule numerical adjustment parameters include: identifier 2 corresponding to data parsing and identifier 3 for data enrichment. The processing rule numerical adjustment parameters are multiplied by the preset ratio 2 to obtain identifier 4 corresponding to data parsing and identifier 6 for data enrichment. The adjusted data processing rule parameters are then obtained, including identifier 4 corresponding to data parsing and identifier 6 for data enrichment.

[0072] In other words, the data quality model analyzes the assessed data quality, collection quality, and sharing quality scores, and performs secondary optimization on collection, processing, and sharing rules with low scores. This includes combining regular expression knowledge with different rules, adaptively updating the optimized rules, and applying the updated rules to data quality, collection quality, and sharing quality assessments. The system automatically records each optimized rule. Automated model optimization includes parameter tuning of the rule expressions for data comparison, data association, data cleaning, and data extraction according to algorithmic rules, while also adaptively combining and tuning data processing rules.

[0073] For example, such as Figure 6As shown, if the quality assessment result does not meet the preset requirements, S610 is executed to start adjusting the parameters, then S620 is executed to check the parameter range set by the rules, S630 applies the built-in operation rules to traverse the parameters in sequence, S640 evaluates the data quality of the traversed parameters in sequence, S650 determines the optimal parameter set based on the evaluation results, and S660 saves the optimal parameter set and applies it.

[0074] S2303, calculate the rate of change of the adjusted data processing rule parameters and the adjusted data sharing rule parameters.

[0075] It is understandable that the rate of change is calculated by the number of times the parameter value is adjusted over a period of time.

[0076] For example, after adjusting various rule parameters, the number of sets of adjusted data processing rule parameters and adjusted data sharing rule parameters obtained within 5 minutes is counted. The number of sets of rule parameters obtained is the number of parameter adjustments. Then, dividing the number of parameter adjustments by 5 minutes gives the rate of change. When the rate of change is too high or too low, affecting the evaluation of the effect, periodic feedback is performed. The feedback period can be set as a parameter to adjust the rate of change.

[0077] S2304, if the rate of change meets the rate of change threshold, confirm that the optimization of the current data governance rule is completed in the current loop.

[0078] S240, Repeat the above steps until the quality assessment results meet the preset requirements, and save the current data governance rules.

[0079] In other words, optimization is complete when the rate of change meets the rate of change threshold and the quality assessment result meets the preset requirements (e.g., adjusting more than once per minute meets the rate of change threshold). If the quality assessment result does not meet the preset requirements, the above steps must be repeated for adjustment.

[0080] For example, such as Figure 7As shown, Ki represents the data acquisition rule parameters, Ki+1 represents the data processing rule parameters, Ki+2 represents the data sharing rule parameters, and Ki+3 represents the data storage rule parameters. The rules other than the data acquisition rules are permuted and combined to obtain cKi. A represents the proportional adjustment of rule parameters, and a0, a1, a2, a3...aN are proportional coefficients, corresponding to Ki, Ki+1, Ki+2, Ki+3,...Ki+N respectively. The parameters are adjusted according to a proportional relationship, with the proportions following a normal distribution of 1, and the proportions are applied adaptively. T represents parameter addition, and S represents the value added to the parameters, ranging from 0 to N. S1, S2, S3, and S4 correspond to Ki, Ki+1, Ki+2, and Ki+3 respectively. B represents the rate of change, b0, b1, b2, b3...bN represent the frequency of change of a0, a1, a2, a3...aN respectively, and S represents the added value's change frequency. When the rate of change is too high or too low, affecting the effect evaluation, periodic feedback is performed. The feedback period can be set as a parameter to adjust the rate of change. A, T, and B are adjusted for rule parameter ratios, addition, and rate of change respectively. Based on the adjustment effect, data is output to combine rule C.

[0081] In other words, the rule base contains a series of rules for data collection, data comparison, data association, data cleaning, and data extraction. Each rule has specific parameters; for example, data collection parameters include collection frequency and collection protocol, while data cleaning parameters include the type of data to be cleaned and frequency parameters. Then, input data (various rule parameters) is taken, and the execution order of each rule is arranged and combined. Rule optimization is performed based on a parameter optimization function. First, the proportions of various rule parameters are optimized, applying a manually set normal distribution of 0-2. Parameters are adjusted around 1, and then adjusted by adding the parameters together. The parameter range is a set range, and the range can be adaptively adjusted each time. Finally, the rule change rate is adjusted based on the feedback results of parameter adjustments, ultimately outputting a combined rule C. Data collection and processing are performed based on combined rule C. The data governance effect is evaluated, and the evaluation results are fed back as A, T, and B. A, T, and B are adaptively adjusted based on the feedback results.

[0082] This application can also perform human-machine collaborative optimization. After multiple rounds of automated model parameter tuning, humans can perform secondary optimization on the system-tuned rules in a visualization system based on experience, ultimately achieving optimization of data processing rules and data processing strategies.

[0083] As a specific embodiment of this application, firstly, S801 collects the data to be evaluated, S802 processes the data, S803 stores the data, S804 shares the data, S805 checks the data quality, and S806 checks whether the quality requirements are met. If yes, then S807 saves the rules; if not, then S808 optimizes the data processing rules and S813 updates the data processing rules, S809 optimizes the data collection rules and S812 updates the data collection rules, and S810 optimizes the data sharing rules and S811 updates the data sharing rules.

[0084] Specifically, this application involves secondary optimization after collecting and quality-assessing security data. The specific operations are as follows: Connecting to the logs of auditing devices and interfacing with the target data via the Syslog protocol. Applying rules and policies to automatically perform data filtering, parsing, and enrichment, the processed data is then stored in a designated database. A data quality assessment task is created, evaluating the completeness, consistency, and timeliness of the collected data based on an assessment engine and model, and deriving an assessment score through an assessment algorithm. A score baseline is set; when unqualified data assessment scores appear, adaptive optimization and human-machine collaborative optimization are performed on the unqualified rules and policies to achieve rule updates. The updated rules and policies are applied to data processing, and automatic evaluation is performed after processing. Once the assessment reaches the set baseline score, the rules and policies are not updated, and data collection, processing, and sharing continue.

[0085] Therefore, this application establishes a data quality assessment model, engine, and assessment algorithm. It implements closed-loop data governance, enabling data assessment quality to provide feedback and optimize the data processing process. This improves the quality of data governance, especially the quality of data processing, and enhances data governance efficiency based on process-oriented data governance.

[0086] The above describes the specific implementation process of a data processing method provided by this application. The following will describe a schematic diagram of the data processing apparatus provided by this application.

[0087] like Figure 9 As shown, some embodiments of this application provide a data processing apparatus including: a data governance module 910, a data quality inspection module 920, a rule optimization module 930, and a rule storage module 940.

[0088] The data governance module 910 is configured to perform data governance operations on the data to be evaluated according to the current data governance rules to obtain the governed data. The data governance operations include at least parsing and storing the data to be evaluated. The data quality inspection module 920 is configured to perform data quality inspection on the governed data to obtain the quality assessment result of the governed data. The rule optimization module 930 is configured to optimize and update the current data governance rules if the quality assessment result does not meet the preset requirements. The rule storage module 940 is configured to repeat the above steps until the quality assessment result meets the preset requirements and save the current data governance rules.

[0089] In one embodiment of this application, the current data governance rule includes data processing rules and data sharing rules; the rule optimization module 930 is further configured to: when the quality assessment result does not meet preset requirements, obtain the current data processing rule parameters and the current data sharing rule parameters; adjust the values ​​of the current data processing rule parameters and the current data sharing rule parameters to obtain adjusted data processing rule parameters and adjusted data sharing rule parameters; calculate the change rate of the adjusted data processing rule parameters and the adjusted data sharing rule parameters, wherein the change rate is calculated by the number of times the parameter values ​​are adjusted within a certain period of time; and when the change rate meets the change rate threshold, confirm that the optimization of the current data governance rule is completed in the current loop.

[0090] In one embodiment of this application, the rule optimization module 930 is further configured to: multiply the current data processing rule parameter and the current data sharing rule parameter by corresponding preset ratios to obtain a sharing rule ratio adjustment parameter and a processing rule ratio adjustment parameter; and add the sharing rule ratio adjustment parameter and the processing rule ratio adjustment parameter to corresponding preset values ​​to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0091] In one embodiment of this application, the rule optimization module 930 is further configured to: add the current data processing rule parameter and the current data sharing rule parameter to corresponding preset values ​​respectively to obtain a sharing rule value adjustment parameter and a processing rule value adjustment parameter; and multiply the sharing rule value adjustment parameter and the processing rule value adjustment parameter to corresponding preset ratios respectively to obtain the adjusted data processing rule parameter and the adjusted data sharing rule parameter.

[0092] In one embodiment of this application, the data governance module 910 is further configured to: collect data to be evaluated using the current data collection rules; and optimize and update the current data governance rules and the current data collection rules if the quality evaluation result does not meet the preset requirements.

[0093] In the embodiments of this application, Figure 9 The module shown can achieve Figures 1 to 8 Each process in the method embodiment. Figure 9 The operations and / or functions of each module in the document are respectively designed to achieve... Figures 1 to 8 The corresponding processes in the method embodiments are described above. For details, please refer to the descriptions in the above method embodiments; to avoid repetition, detailed descriptions are omitted here.

[0094] like Figure 10 As shown, this application provides an electronic device 100, including: a processor 101, a memory 102 and a bus 103. The processor is connected to the memory via the bus. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the method as described in any one of the above embodiments. For details, please refer to the description in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0095] The bus is used to enable direct communication between these components. In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an On-Premises Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor.

[0096] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores computer-readable instructions, which, when executed by the processor, can perform the methods described in the above embodiments.

[0097] Understandable. Figure 10 The structure shown is for illustrative purposes only and may include structures larger than [other structures]. Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown. Figure 10 The components shown can be implemented using hardware, software, or a combination thereof.

[0098] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a server, it implements any of the methods described in all the above embodiments. For details, please refer to the descriptions in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0099] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The data governance operation is performed on the data to be evaluated according to the current data governance rules to obtain the governed data. The data governance operation includes at least parsing and storing the data to be evaluated. Perform a data quality check on the treated data to obtain the quality assessment results of the treated data; If the quality assessment results do not meet the preset requirements, the current data governance rules will be optimized and updated. Repeat the above steps until the quality assessment results meet the preset requirements, and then save the current data governance rules.

2. The method according to claim 1, characterized in that, The current data governance rules include data processing rules and data sharing rules; The optimization of the current data governance rules when the quality assessment results do not meet the preset requirements includes: If the quality assessment result does not meet the preset requirements, obtain the current data processing rule parameters and the current data sharing rule parameters; The values ​​of the current data processing rule parameters and the current data sharing rule parameters are adjusted to obtain the adjusted data processing rule parameters and the adjusted data sharing rule parameters; Calculate the rate of change of the adjusted data processing rule parameters and the adjusted data sharing rule parameters, wherein the rate of change is obtained by calculating the number of times the parameter values ​​are adjusted within a certain period of time; If the rate of change meets the rate of change threshold, it is confirmed that the optimization of the current data governance rule has been completed in the current loop.

3. The method according to claim 2, characterized in that, The step of adjusting the values ​​of the current data processing rule parameters and the current data sharing rule parameters to obtain the adjusted data processing rule parameters and the adjusted data sharing rule parameters includes: The current data processing rule parameters and the current data sharing rule parameters are multiplied by their respective preset ratios to obtain the sharing rule ratio adjustment parameters and the processing rule ratio adjustment parameters. The adjusted data processing rule parameters and the adjusted data sharing rule parameters are obtained by adding the shared rule ratio adjustment parameters and the processing rule ratio adjustment parameters to their corresponding preset values.

4. The method according to claim 2, characterized in that, The step of adjusting the values ​​of the current data processing rule parameters and the current data sharing rule parameters to obtain the adjusted data processing rule parameters and the adjusted data sharing rule parameters includes: The current data processing rule parameter and the current data sharing rule parameter are added to their corresponding preset values ​​to obtain the sharing rule value adjustment parameter and the processing rule value adjustment parameter. The adjusted data processing rule parameters and the adjusted data sharing rule parameters are obtained by multiplying the shared rule numerical adjustment parameters and the processing rule numerical adjustment parameters by their respective preset ratios.

5. The method according to any one of claims 1-4, characterized in that, Before performing data governance operations on the data to be evaluated according to the current data governance rules to obtain the governed data, the method further includes: Collect the data to be evaluated using the current data collection rules; The step of optimizing and updating the current data governance rules when the quality assessment results do not meet the preset requirements includes: If the quality assessment results do not meet the preset requirements, the current data governance rules and the current data collection rules shall be optimized and updated.

6. A data processing apparatus, characterized in that, The device includes: The data governance module is configured to perform data governance operations on the data to be evaluated according to the current data governance rules to obtain the governed data. The data governance operations include at least parsing and storing the data to be evaluated. The data quality inspection module is configured to perform data quality inspection on the treated data and obtain the quality assessment result of the treated data. The rule optimization module is configured to optimize and update the current data governance rules when the quality assessment results do not meet the preset requirements. The rule storage module is configured to repeat the above steps until the quality assessment result meets the preset requirements, and then save the current data governance rules.

7. The apparatus according to claim 6, characterized in that, The current data governance rules include data processing rules and data sharing rules; The rule optimization module is also configured to: If the quality assessment result does not meet the preset requirements, obtain the current data processing rule parameters and the current data sharing rule parameters; The values ​​of the current data processing rule parameters and the current data sharing rule parameters are adjusted to obtain the adjusted data processing rule parameters and the adjusted data sharing rule parameters; Calculate the rate of change of the adjusted data processing rule parameters and the adjusted data sharing rule parameters, wherein the rate of change is obtained by calculating the number of times the parameter values ​​are adjusted within a certain period of time; If the rate of change meets the rate of change threshold, it is confirmed that the optimization of the current data governance rule has been completed in the current loop.

8. The apparatus according to claim 7, characterized in that, The rule optimization module is also configured to: The current data processing rule parameters and the current data sharing rule parameters are multiplied by their respective preset ratios to obtain the sharing rule ratio adjustment parameters and the processing rule ratio adjustment parameters. The adjusted data processing rule parameters and the adjusted data sharing rule parameters are obtained by adding the shared rule ratio adjustment parameters and the processing rule ratio adjustment parameters to their corresponding preset values.

9. An electronic device, characterized in that, include: Processor, memory, and bus; The processor is connected to the memory via the bus, and the memory stores a computer program that, when executed by the processor, can implement the method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1-5.

Images