Problem data processing method and device and electronic equipment
By obtaining and executing data check tasks and determining and updating the status of the problem data table, the problem of how to accurately monitor and handle problem data is solved, and timely tracking and updating the status of the problem data is achieved.
Patent Information
- Application Number
- CN202411402761.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-02-18
AI Technical Summary
In the context of big data, how to accurately monitor and process problem data has become an urgent problem.
Provide a method for processing problem data, including obtaining data verification tasks, performing data verification tasks to determine problem data, determining problem data table based on problem data, and updating the status in the problem data table after receiving the processing message.
Accurate monitoring and processing of problem data is realized, and the status of problem data can be updated in a timely manner, thereby effectively monitoring the status of problem data.
Smart Images
Figure CN119474068A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device and electronic device for processing problematic data. Background Art
[0002] Data quality verification is to verify the standardization, completeness, accuracy, validity, uniqueness and timeliness of databases, data tables and data based on certain rules in the context of big data. After the data table is verified based on certain rules, problematic data will be found. How to accurately monitor problematic data is an urgent problem to be solved. Summary of the invention
[0003] Embodiments of the present disclosure provide a method, device and electronic device for processing problem data.
[0004] In a first aspect, an embodiment of the present disclosure provides a method for processing problem data, including: obtaining a data verification task; in response to determining that a task execution condition is met, executing the data verification task and determining the problem data; based on the problem data, determining a problem data table; in response to receiving a processing message for the problem data table, updating the status of each problem data in the problem data table.
[0005] In the second aspect, an embodiment of the present disclosure provides a problem data processing device, including: a task acquisition unit, configured to acquire a data verification task; a task execution unit, configured to execute the data verification task and determine the problem data in response to determining that the task execution conditions are met; a data storage unit, configured to determine a problem data table based on the problem data; and a data update unit, configured to update the status of each problem data in the problem data table in response to receiving a processing message for the problem data table.
[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the problem data processing method described in the first aspect when executing the computer program.
[0007] By applying the technical solution of the present disclosure, problem data can be checked and the status of the problem data can be updated in a timely manner, thereby enabling monitoring of the status of the problem data.
[0008] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0010] Figure 1 An exemplary system architecture diagram is shown in which an embodiment of the problem data processing method disclosed herein can be applied;
[0011] Figure 2 A flowchart of an embodiment of the problem data processing method disclosed in the present invention;
[0012] Figure 3 A flowchart of another embodiment of the problem data processing method disclosed in the present invention;
[0013] Figure 4 A schematic diagram of the structure of an embodiment of a problem data processing device disclosed in the present invention;
[0014] Figure 5 A schematic diagram of the structure of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION
[0015] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.
[0016] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0017] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.
[0018] In order to make the technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0019] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the problem data processing method or problem data processing apparatus of the present disclosure can be applied.
[0020] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a server 102, a network 103, and a database 104. The network 103 is used to provide a medium for a communication link between the terminal device 101, the server 102, and the database 104. The network 103 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0021] The user can use the terminal device 101 to interact with the server 102 and the database 104 through the network 103 to receive or send messages, or add, delete, modify or remove data. The terminal device 101 can be installed with various communication client applications, such as browser applications.
[0022] The terminal device 101 can be hardware or software. When the terminal device 101 is hardware, it can be various electronic devices, including but not limited to smart phones, tablet computers, car computers, laptop computers, desktop computers, etc. When the terminal device 101 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0023] The server 102 may be a server that provides various services, such as a backend server that performs verification services for data in a database. The backend server may feed back the verified problematic data to the terminal device 101, and the user may process the problematic data through the terminal device 101.
[0024] It should be noted that the server 102 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 102 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0025] The database 104 may be any type of database for storing data, including a plurality of data tables. The server 102 may access the data in the database 104 , perform verification, and store the obtained problematic data in the database 104 .
[0026] It should be noted that the problem data processing method provided in the embodiment of the present disclosure is generally executed by the server 102. Accordingly, the problem data processing device is generally set in the server 102.
[0027] It should be understood that Figure 1The number of terminal devices, servers, networks and databases in the embodiment is only for illustration. Any number of terminal devices, servers, networks and databases may be provided as required.
[0028] Figure 2 FIG. 2 shows a process 200 of an embodiment of the problem data processing method of the present disclosure. Figure 2 As shown, the problem data processing method of this embodiment may include the following steps:
[0029] Step 201, obtaining a data verification task.
[0030] In this embodiment, the execution subject of the problem data processing method (for example Figure 1 The server 102 shown in FIG. 10A can obtain a data verification task. Here, the execution subject can obtain the data verification task at a fixed time, or can obtain the data verification task when receiving an instruction. The data verification task may include a target data table, a target field, a field type, and a verification rule. The execution subject can verify the target field and the field type in the target data table according to the verification rule.
[0031] Step 202, in response to determining that the task execution condition is met, executing the data verification task to determine the problematic data.
[0032] In this embodiment, after obtaining the data verification task, the execution subject will continue to detect whether the task execution conditions are met. If the task execution conditions are met, the data verification task can be executed to determine the problematic data. The above-mentioned task execution conditions may include but are not limited to: the preset time has arrived, the preset time has passed since the last data verification task was completed, and so on. When executing the data verification task, an instance for executing the task can be created first. The above-mentioned instance can check the data table according to the verification rules to obtain the problematic data. The problematic data here may include: data with sensitive information, data with incorrect data types, and so on.
[0033] Step 203, determining a problem data table according to the problem data.
[0034] After the problem data is obtained through verification, it can be stored in a data table to obtain a problem data table. Specifically, it can be stored according to the primary key, field, field type, etc. of the problem data. Alternatively, it can be stored according to the fields in the source data table to which the problem data belongs. Alternatively, new fields can be added on the basis of the source data table fields for storage. The problem data table can be stored under a specified path in the database. Users can view the problem data through this path, and can use the problem data processing tool to process the problem data in the problem data table. The processing here may include but is not limited to: data cleaning, data rectification, etc. After the problem data is processed, a processing message can be sent to the execution entity. The processing message may include the location and content of the problem data, the content of the modified problem data, the modification time, etc.
[0035] Step 204: in response to receiving a processing message for the problem data table, updating the status of each problem data in the problem data table.
[0036] After receiving the processing message for the problem data table, the execution subject can parse the processing message to determine the processed problem data, thereby further updating the status of each problem data in the problem data table. The status here may include processed or updated, etc.
[0037] The problem data processing method provided by the above-mentioned embodiment of the present disclosure can check the data, determine the problem data, and timely update the status of each problem data according to the processing message of the problem data table, so as to effectively monitor the status of the problem data.
[0038] Continue to see Figure 3 , which shows a process 300 of another embodiment of the problem data processing method according to the present disclosure. Figure 3 As shown, the method of this embodiment may include the following steps:
[0039] Step 301, obtaining a data verification task.
[0040] In this embodiment, the user can set the data verification task through the specified page. Specifically, the name, field and other information of the data table to be verified can be selected. The data verification task includes the target data table, target field, field type and verification rules. The target data table can be any data table stored in the database. The target field can be a field in the target data table. The field type can indicate the type of the field. The verification rule can indicate the format that a certain type of data should meet, and the format can be used to determine whether the data is problematic data. Specifically, the verification rules may include: postal code format verification, regular expression verification, date verification, email verification, etc.
[0041] Step 302, in response to determining that the task execution condition is met, the target field in the target data table is checked according to the check rule to determine the check type of the target field; and the problem data is determined according to the check type and the field type.
[0042] In this embodiment, when it is determined that the task execution condition is met, the target field in the target data table can be checked according to the check rule to determine the check type of the target field. Then, the check type is compared with the field type. If the two are consistent, the data is considered to be problematic data.
[0043] In some specific practices, a data verification task instance is generated, and then the data verification task instance is executed. When generating a data verification instance, it is necessary to save the specific information of each rule configuration to the instance log table (ZL_ZYSL_RZ), including: id, job instance id, rule id, rule name, data table English name, field English name, problem data sql, creator, creation time, execution status, problem data volume, etc. The ZL_ZYSL_RZ data table structure is shown in Table 1.
[0044] For example, if you check the natural person basic information table zrr_jbxx, the ID card field sfz_bm, select the field non-empty rule and the ID card verification rule, then execute the task, and the corresponding storage data in the running log table ZL_ZYSL_RZ is shown in Table 2.
[0045] Table 1 Instance log table ZL_ZYSL_RZ table structure
[0046]
[0047]
[0048] Table 2 ZL_ZYSL_RZ table storage example
[0049]
[0050] Step 303: Create a question data table according to the fields of the question data table that match the target data table; and store the question data in the question data table.
[0051] After obtaining the problem data, the fields of the problem data table that the target data table matches can be first determined. Then, based on the determined fields, a problem data table is created, and the problem data is stored in the problem data table. Specifically, different data tables can correspond to different fields, and the correspondence can be determined by the type of data included in the target data table. For example, if the data included in the target data table is the user's vehicle information, the corresponding field may include the purchase time of the vehicle. If the data included in the target data table is the user's ID card information, the corresponding field may include the validity period of the ID card.
[0052] In some optional implementations of this embodiment, when storing the problem data in the problem data table, the field value corresponding to the problem data can be determined according to the problem data. For the preset fields in the problem data table, they can be initialized to default values. The preset fields here may include: the rectification time of the problem data, the rectification person, etc.
[0053] In some specific practices, after executing the problem data sql, the data found by the problem data sql is saved to the problem data table according to the structure of the problem data table. The problem data table (ZL_WTSJ) includes: id, primary key id of the check table, English name of the check table, English name of the check field, current value of the check field, check rule id, check task instance id, creation time, whether to rectify, rectification time, remarks and other information. The ZL_WTSJ data table structure is shown in Table 3.
[0054] Table 3 Problem data table ZL_WTSJ
[0055]
[0056] Execute the problem data query sql, save the found problem data into the problem data table according to the key information of the problem data table, and initialize the "Whether to rectify" field to "No".
[0057] Based on the above check of the natural person basic information table zrr_jbxx, the ID card field sfz_bm, the check rule: the field is not empty, the ID card is verified, and the information stored in the problem data table ZL_WTSJ is shown in Table 4.
[0058] Table 4 ZL_WTSJ storage information
[0059]
[0060]
[0061] Step 304: output the problem data table and output tool options for processing the problem data in the problem data table.
[0062] After obtaining the problem data table, the problem data table can be output. At the same time, the tool options for processing the problem data in the problem data table can also be output. Through this option, the user can select a data processing tool to process the problem data. The above options can be displayed through a floating window, an icon, or a drop-down box.
[0063] In some specific practices, users can use the above tools to perform data governance on problematic data in problematic data tables, or they can directly modify data in target data tables offline.
[0064] Step 305, in response to receiving a processing message for the problem data table, determining the processed problem data; and updating the preset fields corresponding to the processed problem data in the problem data table to preset values.
[0065] In this embodiment, if a processing message for the problem data table is received, the processed problem data is determined. Then, the preset field corresponding to the processed problem data in the problem data table is updated to a preset value. Specifically, the rectification status field corresponding to the processed problem data can be updated to a value indicating that rectification has been completed.
[0066] In some optional implementations of this embodiment, the above update may be performed when a processing message is received, or may be performed periodically.
[0067] In some optional implementations of this embodiment, the updating may be to query the problem data table to determine the processed problem data, and then update the preset fields corresponding to the processed problem data.
[0068] In some specific practices, it is necessary to mark the data that the user has managed and update the status of the corresponding problem data in the problem data table. Specifically, the English name of the verification data table, the English name of the verification field, and the verification rule ID can be used as conditions to update the status of the problem data in the problem data table.
[0069] When updating, you can first query the problem data table, group by the English name of the check table, the check task instance ID, and the check rule, and query the data table and rule information that need to be reviewed. Then, generate the SQL of the problem data. Use the query data table and rule information that need to be reviewed as conditions, and query the SQL of the problem data in the instance execution log table. Use the English name of the data table, the check instance ID, and the check rule as conditions.
[0070] Through the above query, the problem data sql found cannot be executed directly. It is necessary to narrow the query scope and limit the data that has been checked to be problematic. It is necessary to add the condition where zj_id in (specific primary key id of problematic data) to wt_sql. Then, query the primary key id of the problematic data, and find the primary key id corresponding to the unrectified problematic data in the problem data table ZL_WTSJ with the verification task instance id, rule id, and English name of the verification table as conditions.
[0071] Then, execute the problem data sql to find out the problem data ids that still exist. If they still exist, it means that the data has not been rectified. If they do not exist, it means that the data has been rectified. Find all the primary key ids that have not been rectified. Finally, according to the English name of the check table, the primary key id, and the rule id, update the corresponding data in the problem table to the rectified state.
[0072] Through the above update operation, when the problematic data is queried again, the rectified data will not be found.
[0073] The problem data processing method provided by the above-mentioned embodiment of the present disclosure can timely update the status of the problem data, thereby realizing the status monitoring of the problem data.
[0074] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a problem data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0075] like Figure 4 As shown, the problem data processing device 400 of this embodiment includes: a task acquisition unit 401 , a task execution unit 402 , a data storage unit 403 and a data updating unit 404 .
[0076] The task acquisition unit 401 is configured to acquire a data verification task.
[0077] The task execution unit 402 is configured to execute a data verification task and determine problematic data in response to determining that a task execution condition is satisfied.
[0078] The data storage unit 403 is configured to determine a problem data table according to the problem data.
[0079] The data updating unit 404 is configured to update the status of each problem data in the problem data table in response to receiving a processing message for the problem data table.
[0080] In addition, in the technical solution of the present application, an electronic device is also proposed.
[0081] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.
[0082] like Figure 5 As shown, the electronic device may include a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501, wherein the processor 501 and the memory 502 communicate with each other via the bus 503. When the processor 501 executes the computer program, the steps of implementing the above method include, for example: obtaining a data verification task; in response to determining that the task execution condition is satisfied, executing the data verification task and determining problem data; determining a problem data table based on the problem data; in response to receiving a processing message for the problem data table, updating the status of each problem data in the problem data table.
[0083] In addition, in one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: obtaining a data verification task; in response to determining that the task execution conditions are met, executing the data verification task and determining the problem data; based on the problem data, determining a problem data table; in response to receiving a processing message for the problem data table, updating the status of each problem data in the problem data table.
[0084] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.
Claims
1. A method for processing problem data, comprising: Get data verification tasks; In response to determining that the task execution condition is satisfied, executing the data verification task to determine the problematic data; Determine a problem data table according to the problem data; In response to receiving a processing message for the problem data table, the status of each problem data in the problem data table is updated.
2. The method according to claim 1, wherein: The data verification task includes a target data table, a target field, a field type, and a verification rule; The performing of the data checking task to determine the problematic data includes: According to the checking rule, checking the target field in the target data table to determine the checking type of the target field; Determine problem data according to the check type and the field type.
3. The method according to claim 1, wherein: Determining a problem data table according to the problem data includes: Creating a problem data table according to the fields of the problem data table matched by the target data table; The question data is stored in the question data table.
4. The method according to claim 3, wherein: The storing the problem data into the problem data table comprises: According to the question data, determining a field value corresponding to the question data; Initialize the preset fields in the problem data table to default values.
5. The method according to claim 1, wherein: The method further comprises: The problem data table and tool options for processing the problem data in the problem data table are output.
6. The method according to claim 4, wherein: In response to receiving a processing message for the problem data table, updating the status of each problem data in the problem data table includes: In response to receiving a processing message for the problem data table, determining processed problem data; In the problem data table, the preset fields corresponding to the processed problem data are updated to preset values.
7. The method according to claim 6, wherein: The updating of the preset fields corresponding to the processed question data in the question data table to preset values includes: When receiving the processing message, performing the updating; or; The updating is performed regularly.
8. The method according to claim 7, wherein: The updating of the preset fields corresponding to the processed question data in the question data table to preset values includes: Query the problem data table to determine processed problem data; Update the preset fields corresponding to the processed question data.
9. A problem data processing device, comprising: A task acquisition unit, configured to acquire a data verification task; A task execution unit, configured to execute the data verification task and determine problematic data in response to determining that the task execution condition is satisfied; A data storage unit is configured to determine a problem data table according to the problem data; The data updating unit is configured to update the status of each problem data in the problem data table in response to receiving a processing message for the problem data table.
10. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the problem data processing method according to any one of claims 1 to 8 is implemented.