Data processing method and device, equipment, storage medium and program product
By obtaining the lineage mapping relationship of the target table, the missing fields in the source table are automatically traced and filled, which solves the problem of low efficiency in generating test data in the existing technology and realizes an efficient and accurate data filling process.
Patent Information
- Application Number
- CN202511776691.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-01-06
AI Technical Summary
Existing technologies are inefficient at generating test data, especially when the data chain is complex or there are many table fields, requiring a lot of manual time.
By obtaining the lineage mapping relationship of the target table, tracing the origin of the target table, locating the missing fields in the source table, performing semantic analysis to obtain field identifiers and attribute information, and generating target field values based on the lineage mapping relationship and value association information, the target field values are automatically filled into the missing positions in the source table.
It reduces human intervention, improves the efficiency of data generation and population, and ensures the accuracy and integrity of the data.
Smart Images
Figure CN121277941A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of databases, and particularly relates to a data processing method and device, equipment, a storage medium and a program product. BACKGROUND
[0002] In the absence of real business data, logical test data needs to be generated to verify whether the data processing flow and system function can normally run.
[0003] In related technologies, a tester manually reads a data processing script, analyzes the source and processing logic of a missing field, and then manually writes a script to insert test data into a database one by one.
[0004] This data generation method completely relies on manual understanding to write a script, and a large amount of time is consumed in the case of complex data links or a large number of table fields, so the data generation efficiency is low. SUMMARY
[0005] The embodiments of the present application provide a data processing method, device, equipment, storage medium and program product, which can solve the problem of low data generation efficiency in the existing data processing method.
[0006] In a first aspect, the embodiments of the present application provide a data processing method, which comprises: obtaining a target table and a blood relationship mapping relationship corresponding to the target table; In the case that a first field with missing field values exists in the target table, tracing the source of the target table according to the blood relationship mapping relationship to determine a source table corresponding to the target table; locating a field position corresponding to the first field in the source table to obtain a target field corresponding to the first field in the source table, wherein the target field has missing field values; performing semantic analysis on the target field to obtain field identification and field attribute information of the target field; finding value association information corresponding to the target field according to at least one of the field identification, the field attribute information and the blood relationship mapping relationship, wherein the value association information is a constraint condition of the value in the target field or a value of the target field; generating a target field value according to the value association information; filling the target field value into a position corresponding to the target field in the source table.
[0007] In some embodiments, before determining the source table corresponding to the target table by tracing the origin of the target table based on the lineage mapping relationship in the case where a first field with a missing value exists in the target table, the method further includes: Determine the processing script corresponding to the target table; The processing script is parsed to determine the field mapping instructions in the processing script; The source table and intermediate table corresponding to the target table are determined according to the field mapping instruction; The processing link corresponding to the target table is determined based on the dependencies between the intermediate table, the source table, and the target table. The processing script is analyzed based on the processing link to obtain the source table, the intermediate table, and the target table, and the lineage mapping relationship corresponding to the target table is determined.
[0008] In some embodiments, the value association information includes field value information and data generation rules. The step of finding the value association information corresponding to the target field based on at least one of the field identifier, field attribute information, and lineage mapping relationship, and generating the target field value based on the value association information, includes: If a field value information matching the field identifier exists in the information technology management platform, the interface of the information technology management platform is called to query the field value information corresponding to the field identifier based on the field identifier, and the target field value is determined from the set of legal values in the field value information. The information technology management platform is used to store multiple field identifiers and the field value information corresponding to each field identifier. If no field value information matching the field identifier exists in the information technology management platform, the data generation rule corresponding to the field identifier is obtained, and the target field value conforming to the data generation rule is generated according to the data generation rule.
[0009] In some embodiments, the value association information includes data samples, and the step of finding the value association information corresponding to the target field based on at least one of the field identifier, field attribute information, and lineage mapping relationship, and generating the target field value based on the value association information, includes: The associated fields of the target field are determined based on the bloodline mapping relationship; Construct a query statement based on the associated fields; Execute the query statement in the database containing the source table to obtain multiple data samples; Determine the target field value in the data sample.
[0010] In some embodiments, the value association information includes field type and field length, and the step of generating the target field based on the field attribute information includes: Obtain the database system table of the database where the source table is located; Query the field attribute information corresponding to the target field from the database system table. The field attribute information includes the field type and field length. Generate target field values that conform to the field attribute information based on the field type and field length.
[0011] In some embodiments, after filling the target field value into the corresponding position of the target field in the source table, the method further includes: The target field value is processed according to the processing link corresponding to the source table to obtain the processed second field value; Load the value of the second field into the position corresponding to the first field in the target table; If the value of the second field loaded into the target table is abnormal, the second field value is traced according to the lineage mapping relationship to determine the third field value in the target field value that caused the abnormality of the second field value, and the third field value in the target field value in the source table is modified.
[0012] In some embodiments, after filling the target field value into the corresponding position of the target field in the source table, the method further includes: Display the target field in the source table; Receive input to modify the target field; In response to the modified input, update the target field.
[0013] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: The first acquisition module is used to acquire the target table and the bloodline mapping relationship corresponding to the target table; The first determining module is used to determine the source table corresponding to the target table by tracing the source of the target table according to the lineage mapping relationship when a first field with a missing field value exists in the target table. The first positioning module is used to locate the field position corresponding to the first field in the source table, and obtain the target field corresponding to the first field in the source table, wherein the field value of the target field is missing. The analysis module is used to perform semantic analysis on the target field to obtain the field identifier and field attribute information of the target field; The search module is used to search for the value association information corresponding to the target field based on at least one of the field identifier, field attribute information and bloodline mapping relationship, wherein the value association information is the constraint condition of the value in the target field or the value of the target field; The generation module is used to generate target field values based on the value association information; The fill module is used to fill the target field value into the position corresponding to the target field in the source table.
[0014] Thirdly, embodiments of this application provide a data processing apparatus, the apparatus including: a processor and a memory storing computer program instructions; The processor executes the above data processing method when executing computer program instructions.
[0015] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the data processing method described above.
[0016] Fifthly, embodiments of this application provide a computer program product, the computer program product including computer program instructions, which, when executed by a processor, implement the data processing method described above.
[0017] In this application, a source table can be traced through a lineage mapping relationship to identify the origin of a field in the target table. The corresponding target field in the source table for the first field with a missing value is located within the source table. Semantic analysis is then performed on the target field in the source table to obtain its identifier and attribute information. Based on the field identifier, field attribute information, and at least one of the lineage mapping relationships, a target field value is generated and filled into the corresponding position in the source table. This automated field tracing, analysis, and filling process effectively reduces manual intervention and improves the efficiency of data generation and filling. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of a data processing device provided in an embodiment of this application. Detailed Implementation
[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.
[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0022] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.
[0023] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.
[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.
[0025] Specifically, in order to solve the problems of the prior art, embodiments of this application provide a data processing method, apparatus, device, storage medium, and program product. The data processing method provided in the embodiments of this application will be described first below.
[0026] Figure 1A flowchart illustrating a data processing method according to an embodiment of this application is shown. The method includes the following steps: S110, Obtain the target table and the bloodline mapping relationship corresponding to the target table.
[0027] In this embodiment, the lineage mapping relationship corresponding to the target table refers to the mapping of the source path, processing rules, and dependencies between each field in the target table and its upstream source table fields. It describes how the fields in the target table are generated from the fields in the source table through ETL (Extract Transform Load). The target table is the data table that ultimately carries the processing results after multiple extractions, transformations, and calculations throughout the entire ETL processing chain; it is the final output object for data generation and verification.
[0028] S120, if the target table has a first field with a missing field value, trace the source of the target table according to the lineage mapping relationship to determine the source table corresponding to the target table.
[0029] In this embodiment, when a first field with a missing value is detected in the target table, the source of the target table can be traced back according to the lineage mapping relationship, thereby locating the corresponding upstream source table.
[0030] S130, locate the field position corresponding to the first field in the source table to obtain the target field corresponding to the first field in the source table, wherein the field value of the target field is missing.
[0031] In this embodiment, after locating the source table, the position of the field corresponding to the first field can be found in the source table to confirm the target field corresponding to the first field in the source table. Since the field value of the first field is missing, and the field value of the first field should be obtained by processing the field value of the target field, the field value of the target field is also missing.
[0032] S140, Perform semantic analysis on the target field to obtain the field identifier and field attribute information of the target field; In this embodiment, after determining the target field in the source table, semantic analysis can be performed on the target field to extract its field identifier and field attribute information. The field identifier may include metadata such as the field name, the name of the source table to which it belongs, and the field's meaning. Field attribute information includes the field type (e.g., string, integer, floating-point), field length, value range limits, and whether null values are allowed. During semantic analysis, the system combines metadata from the data standardization platform, explicit or implicit field conversion logic in the ETL script, and field definitions in the database system tables. For example, if a field is of type VARCHAR and its length is 20, the system will record this constraint to ensure that subsequent data filling will not exceed the length limit.
[0033] S150, based on at least one of the field identifier, field attribute information and bloodline mapping relationship, find the value association information corresponding to the target field, wherein the value association information is the constraint condition of the value in the target field or the value of the target field; In this embodiment, after determining the field identifier and field attribute information of the target field, the system can search for the corresponding value association information of the target field from multiple sources based on at least one of the field identifier, field attribute information, and lineage mapping relationship. The value association information may include constraints that the target field must meet, such as business enumeration values, field domain values, a set of optional values formed by inter-table relationships, or real values sampled from the source table. For example, if the field is a currency field, the system may obtain legal values such as "RMB, USD, EUR" from the information technology management platform; if the field is a customer number, the system may generate "starting with test + 14 digits" according to length rules; if the field has no standard definition but historical samples exist in the source table, the system extracts real data as the value source by constructing an SQL query statement.
[0034] S160, Generate the target field value based on the value association information; S170, fill the target field value into the position corresponding to the target field in the source table.
[0035] In this embodiment, after determining the value association information, a target field value can be generated based on the value association information. The generated target field value can then be filled into the corresponding position of the target field in the source table, restoring the integrity of the source table data. After filling the source table, the ETL processing chain can be further triggered to recalculate, so that the data in the completed source table is processed by the processing logic to generate a new field value for the first field in the target table, thereby verifying whether the data flows correctly and the accuracy of the target field value.
[0036] In this embodiment, the source table from which a field in the target table originates can be traced through a lineage mapping relationship. The corresponding target field in the source table for the first field with a missing value is located within the source table. Semantic analysis is then performed on the target field in the source table to obtain its identifier and attribute information. Based on the field identifier, field attribute information, and at least one of the lineage mapping relationships, a target field value is generated and filled into the corresponding position in the source table. This automated field tracing, analysis, and filling process effectively reduces manual intervention and improves the efficiency of data generation and filling.
[0037] As an optional embodiment, before determining the source table corresponding to the target table by tracing the origin of the target table according to the lineage mapping relationship in the case where a first field with a missing value exists in the target table, the method further includes: Determine the processing script corresponding to the target table; The processing script is parsed to determine the field mapping instructions in the processing script; The source table and intermediate table corresponding to the target table are determined according to the field mapping instruction; The processing link corresponding to the target table is determined based on the dependencies between the intermediate table, the source table, and the target table. The processing script is analyzed based on the processing link to obtain the source table, the intermediate table, and the target table, and the lineage mapping relationship corresponding to the target table is determined.
[0038] In this embodiment, before tracing missing fields, the processing script corresponding to the target table is first determined to accurately identify the data processing logic that generated the target table. The processing script can be an ETL script, where ETL stands for Extract, Transform, and Load. This script contains field mapping instructions such as field mapping, joins, data filtering, aggregation calculations, and business rule transformations. By performing structured parsing of the processing script, the field mapping instructions in the script can be identified. These instructions explicitly describe the correspondence between fields in the target table and fields in the upstream source table. For example, fields in the target table can be obtained through direct assignment, calculation, or generated based on multi-table joins. By parsing these field mapping instructions, the system can extract the source table and intermediate table involved in the target table from the processing script. The source table is the original business table, and the intermediate table is a transitional table in the data processing process, used to receive intermediate results after cleaning, filtering, or aggregation.
[0039] After identifying the source and intermediate tables, the system further determines the processing path of the target table based on the reference relationships between them. The processing path describes the path of data flow from the source table to the target table. Finally, by combining the field mapping logic in the processing path, the processing script can be analyzed to construct a field-level lineage mapping relationship between the source, intermediate, and target tables, ensuring that each field in the target table can find its corresponding source path and processing rule, thus forming a lineage mapping relationship.
[0040] In this embodiment, by parsing and organizing the processing script, the lineage mapping relationship of the target table can be accurately constructed, making subsequent field tracing, missing data location and automatic data generation more accurate and reliable, and avoiding errors and omissions caused by manual script analysis.
[0041] As an optional embodiment, the value association information includes field value information and data generation rules. Generating a target field value based on the value association information includes: If a field value information matching the field identifier exists in the information technology management platform, the interface of the information technology management platform is called to query the field value information corresponding to the field identifier based on the field identifier, and the target field value is determined from the set of legal values in the field value information. The information technology management platform is used to store multiple field identifiers and the field value information corresponding to each field identifier. If no field value information matching the field identifier exists in the information technology management platform, the data generation rule corresponding to the field identifier is obtained, and the target field value conforming to the data generation rule is generated according to the data generation rule.
[0042] In this embodiment, after identifying the field identifier, field attribute information, and lineage mapping relationship of the target field, the system further searches for the corresponding value association information based on this information to generate a target field value that conforms to business rules and data format requirements. The value association information can include two categories: field domain value information and data generation rules. The field domain value information is stored and maintained by the information technology management platform, which is used to uniformly manage the business meaning, value range, legal enumeration value set, and coding specifications of fields. For example, a currency field may correspond to fixed enumeration values such as RMB, USD, and EUR, and the five-level loan classification may include predefined values such as normal, watchlist, substandard, doubtful, and loss. After finding the field domain value information, the system can directly select or combine a value from the legal value set as the target field value.
[0043] When a target field is not registered in the domain value system of the IT management platform, a data generation rule corresponding to the field identifier can be used to construct the target field value. Data generation rules can include field length requirements, prefix structures, random generation methods, or incremental methods. For example, a customer number can be defined as a 14-digit number followed by the prefix "test," and an organization number can be defined as a 9-digit numeric string. The system reads the rules for the corresponding field and then automatically generates a target field value that conforms to the rules based on the field type, field length, character format, or constraints.
[0044] The above approach enables the generation of target field values from two paths: field value information and data generation rules, ensuring the accuracy and comprehensiveness of the generated data.
[0045] As an optional embodiment, the value association information includes data samples, and the step of generating target field values based on the value association information includes: The associated fields of the target field are determined based on the bloodline mapping relationship; Construct a query statement based on the associated fields; Execute the query statement in the database containing the source table to obtain multiple data samples; Determine the target field value in the data sample.
[0046] In this embodiment, when a target field cannot obtain a suitable value through field value information or preset data generation rules, the system will adopt a value association method based on data samples to provide data content for the target field that is closer to the actual business. First, the related fields of the target field can be determined according to the lineage mapping relationship. These related fields are similar fields of the target field in the source table.
[0047] For example, if the target field is "customer name", the system may identify fields with similar meanings such as "contact person name" and "account holder name" from similar business tables; if the target field is "transaction amount", the system may find alternative fields such as "repayment amount" and "payment amount".
[0048] The system then constructs a query based on the associated fields to obtain candidate data samples for generating the target field values. This query can be written in Structured Query Language (SQL) to retrieve data samples from the database that satisfy the definition of bloodline relationship.
[0049] After executing the query, the system retrieves multiple data samples from the database. These data samples represent typical values of the target field in real-world business scenarios. One or more samples can be selected from these multiple data samples to determine the final target field value. Selection methods can include random sampling, frequency-based selection, or selection based on contextual conditions.
[0050] By using the above methods to generate target field values based on data samples, we can not only generate data with correct format and in compliance with field constraints, but also ensure the authenticity of the data in terms of business semantics, thereby improving the representativeness and effectiveness of the test data.
[0051] As an optional embodiment, the value association information includes field type and field length, and the step of generating the target field value based on the field attribute information includes: Obtain the database system table of the database where the source table is located; Query the field attribute information corresponding to the target field from the database system table. The field attribute information includes the field type and field length. Generate target field values that conform to the field attribute information based on the field type and field length.
[0052] In this embodiment, when a target field cannot obtain a valid value through field value information, nor can it obtain suitable data through similar field sampling or rule generation, the system can also generate a target field value based on the field type and field length to ensure that the field structurally meets the database constraint requirements.
[0053] Specifically, the first step is to obtain the database system table of the database containing the source table. The database system table is an internal table used by the database to store metadata information. This metadata describes the structural definitions of each business table, including field names, field types, field lengths, default values, field constraints, and so on. By reading the database system table, the system can accurately obtain the structural information of a specific field without relying on business scripts or external rules.
[0054] After reading the database system table, the system queries the field attribute information based on the target field's field name. This field attribute information includes the field type and field length. The field type determines the acceptable data format, such as string, numeric, or date types. The field length limits the size of the data or the number of characters the field can hold. For example, a VARCHAR field may allow a maximum of twenty characters, while an integer field may only hold values with a specific width. By determining the field type and field length, the system can determine the structural constraints required to generate the field value. For example, if the field is a string type with a length of twenty, the generated field value must control the number of characters to not exceed that length; if the field is a date type, data conforming to a date format must be generated; if the field is a numeric type, a value conforming to the required range and precision must be generated.
[0055] After obtaining the field type and field length, target field values that meet the field structure requirements can be generated based on these attributes.
[0056] By using this field value generation method based on field type and field length, the system can still generate structurally compliant data even when there is a lack of any business-level value information, ensuring the legality of fields at the database level and making the data filling process more stable.
[0057] As an optional embodiment, after filling the target field value into the corresponding position of the target field in the source table, the method further includes: The target field value is processed according to the processing link corresponding to the source table to obtain the processed second field value; Load the value of the second field into the position corresponding to the first field in the target table; If the value of the second field loaded into the target table is abnormal, the second field value is traced according to the lineage mapping relationship to determine the third field value in the target field value that caused the abnormality of the second field value, and the third field value in the target field value in the source table is modified.
[0058] In this embodiment, after the system fills the corresponding field position in the source table with the generated target field value, it will further process the field value based on the processing link corresponding to the source table to generate the second field value.
[0059] After obtaining the processed second field value, the system loads this value into the corresponding first field position in the target table and then validates the target table. If an anomaly is detected in the loaded second field value—for example, if the value is empty, incorrectly formatted, outside a reasonable range, or inconsistent with expected logic—the system will trace back the second field value based on its lineage mapping to find the third field value that caused the anomaly, i.e., the specific field value in the source table that triggered the anomaly. The third field value will then be modified.
[0060] The above solution can automatically locate the root cause of the problem and correct it when data anomalies are detected, effectively improving the reliability of test data processing and significantly reducing the time cost required for manual error investigation.
[0061] As an optional embodiment, after filling the target field value into the corresponding position of the target field in the source table, the method further includes: Display the target field in the source table; Receive input to modify the target field; In response to the modified input, update the target field.
[0062] In this embodiment, after the system automatically generates and fills the target field values into the corresponding field positions in the source table, it also provides the ability to visualize and manually verify the data, ensuring that the final filled values not only meet the field constraints but also the business expectations of the testers. The system first displays the target fields in the source table in a graphical interface, allowing users to intuitively view the automatically generated data. This display typically presents the field names, field types, and current filled values in a table format, enabling users to quickly determine whether the generated data is reasonable. For example, for fields with business semantics such as name, ID number, currency, and amount, manual review is more likely to promptly identify anomalies that are difficult to detect in the automatic generation logic.
[0063] After displaying the target field, the system receives user input to modify it, allowing testers to further adjust the field value based on the auto-fill. Input modifications can take the form of directly changing the text box content, selecting candidate values from a dropdown list, or clicking a button to randomly regenerate the value. Different field types offer different input methods; for example, date fields can use date controls, enumeration fields can offer fixed options, and amount fields can have upper and lower limit validations. This design ensures that user-modified values still conform to the field's format and business rules, helping to improve the business accuracy of the data on top of automation.
[0064] Upon receiving a user's modification request, the system immediately updates the target field in the source table, rewriting the modified field value into the database. This update operation is typically accompanied by a real-time validation mechanism to check whether the newly populated value meets basic constraints such as field type, field length, and NOT NULL restrictions. If necessary, it may also trigger a re-execution of the processing chain to ensure that the modified source field is correctly processed and affects the target table field. For example, when a user modifies the loan amount field, the system will re-execute the corresponding business calculation chain to ensure that the balance field in the target table is updated synchronously. This not only guarantees the correctness of the data processing chain but also enhances the consistency of test data.
[0065] By displaying source table fields, allowing manual modification, and updating field values in real time, this embodiment establishes a mechanism that combines automatic data generation with manual verification. This enables testers to make personalized adjustments based on automatically generated data, thereby further improving the accuracy, controllability, and business adaptability of the data.
[0066] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of the data processing apparatus. Please refer to the following embodiments.
[0067] First see Figure 2 The data processing apparatus 200 provided in this application embodiment includes the following modules: The first acquisition module 201 is used to acquire a target table and the bloodline mapping relationship corresponding to the target table; The first determining module 202 is used to determine the source table corresponding to the target table by tracing the source of the target table according to the lineage mapping relationship when there is a first field with a missing field value in the target table; The first positioning module 203 is used to locate the field position corresponding to the first field in the source table and obtain the target field corresponding to the first field in the source table, wherein the field value of the target field is missing. The first analysis module 204 is used to perform semantic analysis on the target field to obtain the field identifier and field attribute information of the target field; The lookup module 205 is used to look up the value association information corresponding to the target field based on at least one of the field identifier, field attribute information and bloodline mapping relationship, wherein the value association information is either the constraint condition of the value in the target field or the value of the target field. Generation module 206 is used to generate target field values based on the value association information; The fill module 207 is used to fill the target field value into the position corresponding to the target field in the source table.
[0068] In this embodiment, the source table from which a field in the target table originates can be traced through a lineage mapping relationship. The corresponding target field in the source table for the first field with a missing value is located within the source table. Semantic analysis is then performed on the target field in the source table to obtain its identifier and attribute information. Based on the field identifier, field attribute information, and at least one of the lineage mapping relationships, a target field value is generated and filled into the corresponding position in the source table. This automated field tracing, analysis, and filling process effectively reduces manual intervention and improves the efficiency of data generation and filling.
[0069] As one implementation of this application, the data processing apparatus 200 may further include: The second determining module is used to determine the processing script corresponding to the target table; The parsing module is used to parse the processing script and determine the field mapping instructions in the processing script; The third determining module is used to determine the source table and intermediate table corresponding to the target table according to the field mapping instruction; The fourth determining module is used to determine the processing link corresponding to the target table based on the dependency relationship between the intermediate table, the source table, and the target table; The second analysis module is used to analyze the processing script according to the processing link, obtain the source table, the intermediate table and the target table, and determine the lineage mapping relationship corresponding to the target table.
[0070] As one implementation of this application, the above-mentioned generation module 206 may further include: The first generation unit is used to call the interface of the information technology management platform when there is field value information in the information technology management platform that matches the field identifier, query the field value information corresponding to the field identifier according to the field identifier, and determine the target field value from the set of legal values in the field value information. The information technology management platform is used to store multiple field identifiers and field value information corresponding to each field identifier. The second generation unit is used to obtain the data generation rule corresponding to the field identifier when there is no field value information matching the field identifier in the information technology management platform, and generate the target field value that conforms to the data generation rule according to the data generation rule.
[0071] As one implementation of this application, the above-mentioned generation module 206 can also be used for: The associated fields of the target field are determined based on the bloodline mapping relationship; Construct a query statement based on the associated fields; Execute the query statement in the database containing the source table to obtain multiple data samples; Determine the target field value in the data sample.
[0072] As one implementation of this application, the above-mentioned generation module 206 can also be used for: Obtain the database system table of the database where the source table is located; Query the field attribute information corresponding to the target field from the database system table. The field attribute information includes the field type and field length. Generate target field values that conform to the field attribute information based on the field type and field length.
[0073] As one implementation of this application, the data processing apparatus 200 may further include: The processing module is used to process the target field value according to the processing link corresponding to the source table to obtain the processed second field value; A loading module is used to load the value of the second field into the position corresponding to the first field in the target table; The modification module is used to, in the event that the value of the second field loaded into the target table is abnormal, trace the value of the second field according to the lineage mapping relationship, determine the third field value in the target field value that caused the abnormality of the second field value, and modify the third field value in the target field value in the source table.
[0074] As one implementation of this application, the data processing apparatus 200 may further include: A display module is used to display the target field in the source table; A receiving module is used to receive modification inputs for the target field; An update module is used to update the target field in response to the modified input.
[0075] The data processing apparatus provided in this embodiment of the invention can implement the various steps in the above method embodiments, and will not be repeated here to avoid repetition.
[0076] Figure 3 A schematic diagram of the hardware structure of the data processing device provided in an embodiment of this application is shown.
[0077] The data processing device may include a processor 1001 and a memory 1002 storing computer program instructions.
[0078] Specifically, the processor 1001 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0079] Memory 1002 may include mass storage for data or instructions. For example, and not limitingly, memory 1002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1002 may include removable or non-removable (or fixed) media. Where appropriate, memory 1002 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1002 is non-volatile solid-state memory.
[0080] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0081] The processor 1001 reads and executes computer program instructions stored in the memory 1002 to implement any of the data processing methods in the above embodiments.
[0082] In one example, the data processing device may further include a communication interface 1003 and a bus 1010. Wherein, for example... Figure 3 As shown, the processor 1001, memory 1002, and communication interface 1003 are connected through bus 1010 and complete communication with each other.
[0083] The communication interface 1003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0084] Bus 1010 includes hardware, software, or both, that couples components of a data processing device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1010 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0085] The data processing device can be based on the above embodiments to realize the data processing method and apparatus combined with the above.
[0086] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.
[0087] In addition, this application also provides a computer program product, including computer program instructions, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.
[0088] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0089] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0090] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0091] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0092] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method of processing data, characterized by, The method comprises: obtaining a target table and a blood relationship corresponding to the target table; in the case that there is a first field with missing field values in the target table, tracing the source of the target table according to the blood relationship to determine the source table corresponding to the target table; locating the field position corresponding to the first field in the source table to obtain the target field corresponding to the first field in the source table, wherein the target field has missing field values; performing semantic analysis on the target field to obtain the field identifier and field attribute information of the target field; finding the value association information corresponding to the target field according to at least one of the field identifier, the field attribute information, and the blood relationship, wherein the value association information is the constraint condition of the value in the target field or the value of the target field; generating a target field value according to the value association information; filling the target field value into the position corresponding to the target field in the source table.
2. The data processing method according to claim 1, characterized in that, Before the case that there is a first field with missing field values in the target table, the method further comprises: determining the processing script corresponding to the target table; parsing the processing script to determine the field mapping instruction in the processing script; determining the source table and the intermediate table corresponding to the target table according to the field mapping instruction; determining the processing link corresponding to the target table according to the dependency relationship between the intermediate table, the source table and the target table; analyzing the processing script according to the processing link to obtain the blood relationship corresponding to the target table.
3. The data processing method according to claim 1, characterized in that, The value association information includes field domain value information and data generation rules, and generating a target field value according to the value association information comprises: in the case that there is field domain value information matching the field identifier in the information technology management platform, calling the interface of the information technology management platform to query the field domain value information corresponding to the field identifier from the legal value set in the field domain value information, wherein the information technology management platform is used to store a plurality of field identifiers and the field domain value information corresponding to each field identifier; in the case that there is no field domain value information matching the field identifier in the information technology management platform, obtaining the data generation rule corresponding to the field identifier, and generating the target field value conforming to the data generation rule according to the data generation rule.
4. The data processing method according to claim 1, characterized in that, The value association information includes data samples, and generating a target field value according to the value association information comprises: determining the associated field of the target field according to the blood relationship; constructing a query statement according to the associated field; executing the query statement in the database where the source table is located to obtain a plurality of data samples; determining the target field value in the data samples.
5. The data processing method according to claim 1, characterized in that, The value association information includes field type and field length, and generating a target field value according to the value association information comprises: acquire a database system table of a database where the source table is located; query field attribute information corresponding to the target field from the database system table, the field attribute information including a field type and a field length; generate a target field value conforming to the field attribute information according to the field type and the field length.
6. The data processing method according to claim 1, characterized by, After the target field value is filled into the position corresponding to the target field in the source table, the method further includes: processing the target field value according to a processing link corresponding to the source table to obtain a second field value after processing; loading the second field value into a position corresponding to the first field in the target table; in a case where the second field value loaded into the target table is abnormal, tracing the second field value according to the blood relationship mapping relationship to determine a third field value in the target field value that causes the abnormality of the second field value, and modifying the third field value in the target field value in the source table.
7. The data processing method according to claim 1, characterized in that, After the target field value is filled into the position corresponding to the target field in the source table, the method further includes: displaying the target field in the source table; receiving a modification input for the target field; updating the target field in response to the modification input.
8. A data processing device, characterized by comprising: The apparatus includes: a first acquisition module configured to acquire a target table and a blood relationship mapping relationship corresponding to the target table; a first determination module configured to, in a case where a first field with a missing field value exists in the target table, determine a source table corresponding to the target table according to a trace of a source of the target table based on the blood relationship mapping relationship; a first positioning module configured to locate a field position corresponding to the first field in the source table to obtain a target field corresponding to the first field in the source table, wherein the target field has a missing field value; a first analysis module configured to perform semantic analysis on the target field to obtain a field identifier and field attribute information of the target field; a search module configured to search for value association information corresponding to the target field according to at least one of the field identifier, the field attribute information, and the blood relationship mapping relationship, wherein the value association information is a constraint condition for a value in the target field or a value of the target field; a generation module configured to generate a target field value according to the value association information; a filling module configured to fill the target field value into a position corresponding to the target field in the source table.
9. A data processing device, characterized by The data processing device includes a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the data processing method of any one of claims 1-7.
10. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the data processing method of any one of claims 1-7.
11. A computer program product, characterised in that, The computer program product includes computer program instructions, and the computer program instructions are executed by a processor to implement the data processing method of any one of claims 1-7.