Heterogeneous data processing method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for processing heterogeneous data. The present application relates to the field of data processing technology. The method obtains heterogeneous data to be processed and extracts field identifiers in the heterogeneous data. Then, based on a pre-trained semantic recognition model, the field identifiers are semantically matched with target fields in a preset database to generate a field-level mapping relationship. Then, based on the field-level mapping relationship, a step-by-step processing logic adapted to the heterogeneous data format is generated. Then, a verification operation is performed on the step-by-step processing logic in a code sandbox, and the output target field data is saved to a structured database after the verification is passed. The present application improves the accuracy of heterogeneous data processing by extracting field identifiers and performing semantic matching, and then combining the code sandbox to verify the extracted target fields.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, and storage medium for processing heterogeneous data. Background Art

[0002] Currently, integrating heterogeneous data typically relies on technicians manually analyzing the source data, classifying the identified fields, and associating tags. They then compare the source data with the database data based on pre-set rules, identify matching data, and store them in the database. However, when dealing with large or frequently changing data sources, this reliance on manual identification and classification can lead to data mapping errors due to subjective misunderstandings or oversights, resulting in low accuracy in subsequent data processing and storage.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The present application provides a method, device and storage medium for processing heterogeneous data, aiming to solve the problem of low accuracy of data processing and warehousing in traditional solutions.

[0005] To achieve the above objectives, the present application provides a method for processing heterogeneous data, which includes the following steps:

[0006] Acquire heterogeneous data to be processed, and extract field identifiers from the heterogeneous data;

[0007] Based on the pre-trained semantic recognition model, the field identifier is semantically matched with the target field in the preset database to generate a field-level mapping relationship;

[0008] Generating step-by-step processing logic adapted to heterogeneous data formats based on the field-level mapping relationship;

[0009] A verification operation is performed on the step-by-step processing logic in a code sandbox, and after the verification passes, the output target field data is saved to a structure database.

[0010] In one embodiment, the step of semantically matching the field identifier with the target field in the preset database based on the pre-trained semantic recognition model to generate a field-level mapping relationship includes:

[0011] Identify the semantic description and associated fields corresponding to the target field from the preset database;

[0012] Determining the similarity between the field identifier and the target field based on the semantic description and the associated fields, and generating a semantic similarity list;

[0013] The field-level mapping relationship between the field identifier and the target field is determined from a mapping rule set according to the semantic similarity list.

[0014] In one embodiment, the step of generating step-by-step processing logic adapted to the heterogeneous data format based on the field-level mapping relationship includes:

[0015] determining the heterogeneous data format of the heterogeneous data;

[0016] Determining a matching processing logic template according to the field-level mapping relationship, and selecting a matching target processing logic template according to the heterogeneous data format;

[0017] According to the target processing logic template, a step-by-step processing logic for the heterogeneous data is generated.

[0018] In one embodiment, the step of performing a verification operation on the step-by-step processing logic in a code sandbox and saving the target field data to a structure database after the verification passes includes:

[0019] Loading the code sandbox and allocating a corresponding target container in the code sandbox to each processing node in the step-by-step processing logic;

[0020] Executing the operation corresponding to the processing node in the target container, and extracting the target field data output by the target container;

[0021] performing the verification operation on the target field data;

[0022] After verification is passed, the target field data is saved to the structure database.

[0023] In one embodiment, the step of performing the verification operation on the target field data includes:

[0024] Extracting a time field, a value field, and a text field from the target field data;

[0025] Verify the format and time range of the time field according to data type rules;

[0026] The units of the numerical fields are uniformly converted, and the fields in the text fields that match the standardized terms are replaced with each other.

[0027] In one embodiment, after the step of performing the verification operation on the target field data, the method further includes:

[0028] When an operation corresponding to any of the processing nodes fails, an error code and original data context corresponding to the operation are marked and a verification report is generated;

[0029] Parse the error code in the verification report to determine the defect type of the step-by-step processing logic:

[0030] generating an optimized instruction set according to the verification report and the defect type, and reconstructing the step-by-step processing logic based on the optimized instruction set to generate a target processing logic;

[0031] The reconstructed target processing logic is executed and verified in the code sandbox.

[0032] In one embodiment, before the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after the verification passes, the step further includes:

[0033] generating lineage metadata of the heterogeneous data based on the field-level mapping relationship;

[0034] Associating and storing the lineage metadata with the extracted target field data, wherein the lineage metadata includes a source file, a field mapping version, and a parsed logical fingerprint;

[0035] When new target heterogeneous data is added, the analytical logic fingerprint in the historical lineage metadata is obtained;

[0036] The target logical fingerprint of the target heterogeneous data is compared, and if the fingerprint matching degree exceeds a threshold, the associated step-by-step processing logic is called to process the target heterogeneous data.

[0037] In one embodiment, after the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after the verification passes, the method further includes:

[0038] Obtain the user's query requirements and identify and parse the corresponding heterogeneous query statements through natural language models;

[0039] Perform database retrieval based on the heterogeneous query statements, obtain required data and perform report normalization processing, the normalization processing including data aggregation, calculation and sorting;

[0040] The normalized demand data is processed based on a preset report template to generate a data view, and the data view is visually output through a data visualization tool.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a heterogeneous data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the heterogeneous data processing method as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for processing heterogeneous data as described above are implemented.

[0043] The present application provides a method for processing heterogeneous data, a processing device for heterogeneous data, and a storage medium. The method obtains the heterogeneous data to be processed and extracts the field identifiers in the heterogeneous data. Then, based on a pre-trained semantic recognition model, the field identifiers are semantically matched with the target fields in a preset database to generate a field-level mapping relationship. Then, based on the field-level mapping relationship, a step-by-step processing logic adapted to the heterogeneous data format is generated. Then, a verification operation is performed on the step-by-step processing logic in a code sandbox, and the output target field data is saved to a structured database after the verification is passed. The present application improves the accuracy of heterogeneous data processing by extracting field identifiers and performing semantic matching, and then combining the code sandbox to verify the extracted target fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flow chart of the first embodiment of the method for processing heterogeneous data of the present application;

[0047] Figure 2 This is a flow chart of a second embodiment of the method for processing heterogeneous data of the present application;

[0048] Figure 3 This is a flow chart of a third embodiment of the method for processing heterogeneous data of the present application;

[0049] Figure 4 This is a flowchart of a fourth embodiment of the method for processing heterogeneous data of the present application;

[0050] Figure 5 This is a schematic diagram of the architecture of the hardware operating environment of the heterogeneous data processing device involved in the embodiments of the present application.

[0051] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0053] To better understand the above technical solutions, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0054] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0055] The main solution of this application is to obtain heterogeneous data to be processed and extract field identifiers from the heterogeneous data;

[0056] Based on the pre-trained semantic recognition model, the field identifier is semantically matched with the target field in the preset database to generate a field-level mapping relationship;

[0057] Generating step-by-step processing logic adapted to heterogeneous data formats based on the field-level mapping relationship;

[0058] A verification operation is performed on the step-by-step processing logic in a code sandbox, and after the verification passes, the output target field data is saved to a structure database.

[0059] Currently, integrating heterogeneous data typically relies on technicians manually analyzing the source data, classifying the identified fields, and associating tags. They then compare the source data with the database data based on pre-set rules, identify matching data, and store them in the database. However, when dealing with large or frequently changing data sources, this reliance on manual identification and classification can lead to data mapping errors due to subjective misunderstandings or oversights, resulting in low accuracy in subsequent data processing and storage.

[0060] By obtaining the heterogeneous data to be processed and extracting the field identifiers in the heterogeneous data, and then based on the pre-trained semantic recognition model, semantically matching the field identifiers with the target fields in the preset database, generating a field-level mapping relationship, and then generating a step-by-step processing logic adapted to the heterogeneous data format based on the field-level mapping relationship, and then performing a verification operation on the step-by-step processing logic in the code sandbox, and saving the output target field data to the structure database after the verification is passed. This application improves the accuracy of heterogeneous data processing by extracting field identifiers and performing semantic matching, and then combining the code sandbox to verify the extracted target fields.

[0061] Example 1

[0062] Based on this, the present invention provides a method for processing heterogeneous data. Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for processing heterogeneous data of the present application. The method for processing heterogeneous data includes steps S10 to S40:

[0063] Step S10: Obtain heterogeneous data to be processed, and extract field identifiers from the heterogeneous data.

[0064] In this embodiment, the data processing system performs processing actions. Heterogeneous data refers to data of different sources, formats, and structures. This includes structured data, such as SQL table / CSV column name parsing, which is obtained through primary key identification and data type inference; semi-structured data, such as JSON / XML / log files, which can be obtained through tree structure parsing and path expressions (XPath); and unstructured data, including text / images / PDFs, which is extracted through NLP entity recognition and OCR image-text separation. Field identifiers are names used to identify data content. For example, in a preset database, "number," "name," and "occurrence time" are all field identifiers.

[0065] Alternatively, heterogeneous data can be obtained by acquiring heterogeneous verification data from internet vendors and then performing structured extraction based on the heterogeneous verification data, including online interface APIs, online databases, FTP resources, local document OCR recognition, voice data extraction, etc. Thus, heterogeneous data is obtained and delivered to the data processing system.

[0066] Furthermore, when acquiring heterogeneous verification data from internet vendors, the data source identification module uses a knowledge base to retrieve data source information relevant to the user's needs, including the data source's location, type (e.g., SQL database, CSV file, Excel spreadsheet, etc.), and structure. For example, if a user queries for call records associated with a certain number, the system retrieves the telecom operator's database from the knowledge base as the relevant data source, located on a specific server and typed as an SQL database. It then retrieves a pre-set data specification standard, which corresponds to the data type and specifies the format requirements for different data types, such as text, numbers, and dates. For example, date data is specified in the "YYYY-MM-DD" format. Based on this data specification standard, the raw verification data is parsed to extract field identifiers. For heterogeneous data, field identifiers are extracted through column name parsing, which involves obtaining the names of each column in the table or file. For example, in an SQL table, column names such as "User ID," "Call Duration," and "Call Number" serve as field identifiers.

[0067] It should be noted that during the collection process of private data such as phone records, to which the above technical solution applies, a corresponding restriction process is implemented to prevent privacy violations. This allows the corresponding data to be obtained through legal channels. Furthermore, the above restriction process specifically requires that when obtaining private data such as phone records, a permission request for access to the data is sent back to the user's corresponding terminal. Only after the user authorizes the access can the private data be obtained. If the user does not authorize the access, the private data will not be obtained.

[0068] Step S20: Based on the pre-trained semantic recognition model, semantic matching is performed between the field identifier and the target field in the preset database to generate a field-level mapping relationship.

[0069] In this embodiment, the pre-trained semantic recognition model is a natural language processing model based on deep learning. By pre-training on large-scale text data, it learns the common semantic features and representations of language. Semantic matching refers to the process of determining whether two texts match semantically by calculating the semantic similarity between the field identifiers of the text. The field-level mapping relationship refers to the correspondence established between the field identifiers in the source data and the field identifiers in the target data, indicating which field in the target database a certain field in the source data semantically corresponds to.

[0070] Specifically, semantic features are first extracted using a pre-trained semantic recognition model, first for the field identifiers of the heterogeneous data and the target field names in the preset database. The pre-trained semantic recognition model encodes each field identifier text into a fixed-length vector representing the semantic features of the text. For example, for the field identifier "processing time" in the source data, the model encodes it into a vector of length 768, which contains the semantic information of the concept "processing time."

[0071] Then, using methods such as cosine similarity, the similarity between the semantic vector of the source data field identifier and the semantic vector of the target field name is calculated. Cosine similarity measures the degree of directional similarity between two vectors, with values ​​ranging from -1 to 1. Values ​​closer to 1 indicate greater directional similarity, and therefore greater semantic similarity. For example, the cosine similarity between the semantic vectors of the heterogeneous data field identifier "Processing Time" and the target field "Occurrence Time" in the pre-set database is 0.85, indicating semantic similarity between the two.

[0072] As an optional implementation, in this embodiment, step S20 includes:

[0073] Identify the semantic description and associated fields corresponding to the target field from the preset database; determine the similarity between the field identifier and the target field based on the semantic description and associated fields, and generate a semantic similarity list; based on the semantic similarity list, determine the field-level mapping relationship between the field identifier and the target field from a mapping rule set.

[0074] Specifically, semantic recognition is performed on the data source and destination, i.e., the table names and field names of the preset data, to obtain the data source semantics and destination semantics. A similarity comparison is performed between the semantics of each field in each data source and the semantics of all fields in the destination, to obtain a semantic similarity list for each field in the corresponding data source. Based on this semantic similarity list, the mapping relationship between the data source and the destination is determined from a mapping rule set. All mapping relationships are stored in a mapping relationship library.

[0075] Optionally, a similarity threshold can be set based on the similarity results. When the similarity between the source data field identifier and the target field name exceeds the threshold, a field-level mapping relationship is determined to exist between the two. For example, in the above example, since the similarity of 0.85 is greater than 0.7, a field-level mapping relationship is determined to exist between "Processing Time" and "Occurrence Time", and this mapping relationship is stored in the mapping relationship library.

[0076] Step S30: Generate step-by-step processing logic adapted to heterogeneous data formats based on the field-level mapping relationship.

[0077] In this embodiment, step-by-step processing logic refers to breaking down the data processing process into multiple steps, each of which defines the rules and procedures for performing specific operations on the data. Heterogeneous data formats refer to the format characteristics of heterogeneous data, such as the table structure of a relational database or the row and column structure of a CSV file, which include data organization, field types, and data relationships. Distributed processing logic can also dynamically select process processing nodes or generate code processing nodes based on a large model based on semantic mapping.

[0078] Specifically, the identified field-level mapping relationships are analyzed to understand the correspondence between source and target data fields, including whether data types are consistent and field lengths match. For example, the analysis reveals that the "Processing Amount" field in the source data semantically matches the "Related Amount" field in the target data, but the "Processing Amount" field's data type is a floating point number, while the "Related Amount" field's data type is an integer. Based on the mapping relationship analysis results, data processing rules are determined for each field. If the source and target fields differ in data type, data type conversion is required; if the field lengths differ, truncation or padding is required. For example, for the "Processing Amount" field, it is determined that it should be converted to an integer before being mapped to the "Related Amount" field. Based on the identified processing rules and incorporating data standards, step-by-step processing logic adapted to heterogeneous data formats is generated. This includes defining the operation sequence and input and output data formats for each processing step. For example, the generated processing logic is: first convert the format of the "processing time" field to make it conform to the date format requirements of the "occurrence time" field in the preset database; then convert the data type and perform numerical processing on the "processing amount" field, and finally insert the processed data into the target field.

[0079] Optionally, in this embodiment, step S30 includes:

[0080] Determine the heterogeneous data format of the heterogeneous data; determine a matching processing logic template based on the field-level mapping relationship, and select a matching target processing logic template based on the heterogeneous data format; generate a step-by-step processing logic for the heterogeneous data based on the target processing logic template.

[0081] Specifically, a processing logic template is a set of predefined data processing rules and process templates. It specifies data conversion, cleansing, and integration operations for different types of heterogeneous data formats and field-level mapping relationships. A target processing logic template is the template that best matches the current heterogeneous data formats and field-level mapping relationships among multiple processing logic templates, ensuring efficient and accurate data processing.

[0082] The acquired heterogeneous data is scanned through the data source identification module, and its file header information or database metadata is read to preliminarily determine the data format type. For example, when scanning the header information of a file, if it is found to start with "<" and contain a tag structure, it is preliminarily determined to be an XML file; if it is found to start with "{" and contain a key-value pair structure, it is preliminarily determined to be a JSON file. Further analysis of the content characteristics of the data, such as the data organization structure, delimiters, data type definitions, etc., is performed to accurately determine the heterogeneous data format. For Excel files, the worksheet structure and cell format are analyzed; for CSV files, the data delimiters, such as commas and tabs, are determined, as well as whether header information is included. Then, the preliminarily determined heterogeneous data format is verified using preset format validation rules to ensure the accuracy of the judgment result.

[0083] In the preset processing logic template library, search based on the keywords of the field-level mapping relationship to filter out the processing logic templates related to the mapping relationship. Then, analyze the filtered processing logic templates, compare the heterogeneous data format adapted by the template with the currently determined heterogeneous data format, and calculate the degree of match between them. Based on the matching results, select the processing logic template with the highest matching degree as the target processing logic template. If there are multiple templates with similar matching degrees, comprehensively consider factors such as the template's processing efficiency and versatility to make the selection. For example, among multiple processing logic templates suitable for JSON format data, select the template with the highest processing efficiency and good versatility as the target processing logic template. Replace the universal parameters in the target processing logic template with the specific parameters of the current heterogeneous data, such as the data source path, field name, data type, etc. Based on the actual situation of the heterogeneous data and the field-level mapping relationship, refine the processing flow in the template and clarify the specific operation method and sequence of each processing step.

[0084] Step S40: performing a verification operation on the step-by-step processing logic in the code sandbox, and saving the output target field data to the structure database after the verification passes.

[0085] In this embodiment, the code sandbox is a secure code execution environment that limits code access to system resources, preventing malicious or defective code from causing damage to the system. Verification is the process of checking and testing the correctness and security of step-by-step processing logic according to predefined rules and standards. A structured database is a database used to store heterogeneous data and has a fixed table structure and data type definitions.

[0086] Specifically, first define the rules used to validate the step-by-step processing logic in the code sandbox, including data type accuracy verification, such as checking whether the converted data meets the data type requirements of the target field, data integrity verification, semantic correctness verification, etc. The step-by-step processing logic is then placed in the code sandbox for execution, and the output of each step is checked according to the verification rules during execution. For example, after executing each step of the processing logic, check whether the intermediate results meet the expected data type and format, such as checking whether the converted date format of the "processing time field" is correct. If the verification operation passes, that is, all check results meet the validation rules, the processed target field data is saved to the structure database. The saving process includes establishing a connection with the structure database and inserting the processed data into the corresponding tables and fields according to the database's insert statement specifications. For example, the processed field data such as "processing amount" and "processing time" are inserted into the corresponding tables in the preset database through SQL insert statements.

[0087] In the technical solution provided in this embodiment, heterogeneous data to be processed is obtained and field identifiers in the heterogeneous data are extracted. Then, based on a pre-trained semantic recognition model, the field identifiers are semantically matched with target fields in a preset database to generate a field-level mapping relationship. Then, based on the field-level mapping relationship, a step-by-step processing logic adapted to the heterogeneous data format is generated. Then, a verification operation is performed on the step-by-step processing logic in a code sandbox, and after the verification passes, the output target field data is saved to a structure database. This embodiment improves the accuracy of heterogeneous data processing by extracting field identifiers and performing semantic matching, and then combining the code sandbox to verify the extracted target fields.

[0088] Furthermore, through a series of actions involving data source identification, data specification acquisition, and field identifier extraction, field identifiers can be accurately extracted from various heterogeneous data sources, improving the accuracy and specificity of data processing. Semantic matching using a pre-trained semantic recognition model accurately identifies the semantic correspondence between source data field identifiers and preset database target fields, facilitating the effective integration of heterogeneous data from different sources and formats into the preset database, thereby improving the accuracy and efficiency of data integration. Through the steps of mapping relationship analysis, processing rule determination, and processing logic generation, the accuracy and consistency of data during conversion and insertion into the target database are ensured, avoiding data errors and incompatibilities. By performing verification operations in a code sandbox, the correctness and security of the step-by-step processing logic can be effectively tested, preventing damage to the database from erroneous data or malicious code, and ensuring data integrity and accuracy.

[0089] Example 2

[0090] Based on any embodiment, the present application also provides a second embodiment, referring to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the method for processing heterogeneous data of the present application, wherein step S40 includes steps S41 to S44:

[0091] Step S41: loading the code sandbox and allocating a corresponding target container in the code sandbox to each processing node in the step-by-step processing logic.

[0092] Step S42: executing the operation corresponding to the processing node in the target container, and extracting the target field data output by the target container.

[0093] Step S43: performing the verification operation on the target field data.

[0094] Step S44: After verification, the target field data is saved to the structure database.

[0095] In this embodiment, the code sandbox is an isolated code execution environment that can limit the code's access rights to system resources and prevent malicious code or defective code from causing damage to the system. The target container is an independent operating environment allocated to each processing node in the code sandbox. Each target container has independent resource space and execution permissions. The target field data output by the target container refers to the data that meets the expected format and content requirements after the processing node operation. The extraction operation refers to the process of obtaining these target field data from the output results of the target container. The verification operation is the process of checking the accuracy, completeness and consistency of the target field data according to predefined rules and standards. The verification rules are specific criteria used to determine whether the data meets the requirements, including data type check, data range check, data format check, and data correlation check.

[0096] Optionally, first start the code sandbox service and load the configuration parameters of the sandbox, including resource restrictions such as memory, CPU usage, and security policies such as prohibiting access to external networks and restricting file operation permissions. Then parse the step-by-step processing logic to identify the various processing nodes and their relationships. For example, in the step-by-step processing logic, identify processing nodes such as "data type conversion", "field mapping", and "data aggregation", and determine their execution order and dependencies. According to the number and type of processing nodes, dynamically allocate corresponding target containers in the code sandbox. Each target container configures the corresponding operating environment according to the needs of the processing node, such as the required programming language to run, such as Python interpreter, Java virtual machine, library files, etc. For example, a target container is allocated to the "data type conversion" processing node, in which the Python 3.8 version interpreter and related data processing libraries are installed.

[0097] According to the execution order defined by the step-by-step processing logic, the operations of the processing nodes in each target container are triggered sequentially. For example, during the execution of processing node operations, data is transferred between target containers as needed. For example, the data results after the "Time Format Conversion" operation are transferred as input to the "Amount Unit Conversion" target container. During this transfer process, the data format needs to be converted to meet the input requirements of the next processing node. After each target container completes its operation, its output results are captured and the target field data is extracted. This can be achieved by reading the target container's standard output stream, a designated output file, or through inter-container communication mechanisms. Next, validation rules relevant to the current data processing task are loaded from the validation rule library. For example, the loaded rules specify that the data type of the "Occurrence Time" field is a date with a format of "YYYY-MM-DD" and that the data type of the "Related Amount" field is a numeric value with a value range between 0 and 100 million yuan.

[0098] The extracted target field data is then compared against validation rules one by one, performing validation operations. For example, the value of the "Occurrence Time" field is checked to ensure it conforms to the required date format and can be correctly converted to a date object; the value of the "Related Amount" field is checked to ensure it is numeric and within the permitted range. If the data does not conform to the validation rules, detailed error information is recorded, including the error type, location, and an example of the error data. Based on the severity of the error, a decision is made as to whether to terminate the entire data processing process or attempt data repair. Finally, a connection to the structured database is established using preconfigured database connection parameters, such as the database address, port, username, and password. Based on the structured database's table structure and field definitions, an SQL insert statement is generated to insert the target field data into the corresponding table. The SQL insert statement is executed, committing the data to the structured database. A database transaction is also initiated to ensure the atomicity and integrity of the data insertion operation. If an error occurs during the insertion process, such as a violation of a unique constraint or foreign key constraint, the transaction is rolled back to prevent inconsistent data in the database.

[0099] Optionally, in this embodiment, the step of performing the verification operation on the target field data includes:

[0100] Extracting a time field, a numerical field, and a text field from the target field data; validating the format and time range of the time field according to data type rules; performing a uniform unit conversion on the numerical field, and replacing fields in the text field that match standardized terms with each other.

[0101] Specifically, each field in the target field data is identified using a predefined data type dictionary. The data type dictionary includes common time formats, numeric formats, and text features. Based on the identification results, the fields in the target field data are separated into three categories: time fields, numeric fields, and text fields. This classification can be achieved through data structures in programming languages. Validation rules and processing logic are then prepared for each separated field category. The format validation rules for the time field are loaded from the data type rule library, including allowed date separators and the order of date components. The time field value is then checked against the format validation rules. Based on business logic and data context, a reasonable range for the time field is defined, and the time value is verified to be within this range. For example, if the "occurrence time" cannot be earlier than "2020-01-01," assuming it is the start time of recording in a preset database, it cannot be later than the current system time. If a record's "occurrence time" is "2025-01-01" and the current system time is "2024-10-10," the time is considered to be outside the reasonable range.

[0102] Next, the unit conversion rules for the numeric field are loaded from the data processing rule library. These rules define the conversion relationships and conversion factors between different units. The data in the numeric field is then converted based on the loaded unit conversion rules. Finally, the standardized terminology dictionary for the text field is loaded. This dictionary contains common terms and their corresponding standardized replacement terms. The content in the text field is scanned, and any matches found with terms in the standardized terminology dictionary are replaced. For example, if the value of the "Target User Relationship" field in a record is "target user's brother," the dictionary will convert "target user" to "target user's brother."

[0103] Furthermore, in this embodiment, after the step of performing the verification operation on the target field data, the method further includes:

[0104] When the operation corresponding to any of the processing nodes fails, the error code and original data context corresponding to the operation are marked and a verification report is generated; the error code in the verification report is parsed to determine the defect type of the step-by-step processing logic; an optimized instruction set is generated according to the verification report and the defect type, and the distributed processing logic is reconstructed based on the optimized instruction set to generate a target processing logic; the reconstructed target processing logic is executed in the code sandbox and verified.

[0105] Specifically, the error code is a predefined set of codes used to identify specific error types and causes. The original data context refers to the original data environment related to the operation when the processing node operation fails, including input data, processing parameters, etc., which is used to restore the specific situation when the error occurs. The verification report is a document that records the results of the verification operation, including error information, error location, error code, etc. The defect type refers to the category of problems existing in the step-by-step processing logic, such as data format mismatch, data missing, logical errors, etc. The parsing operation refers to the process of analyzing the error codes in the verification report and mapping them to the corresponding defect types. The optimization instruction set is a correction instruction for defects in the step-by-step processing logic, including specific correction steps and parameter adjustments. Reconstruction refers to modifying and improving the original step-by-step processing logic according to the optimization instruction set to generate a new target processing logic.

[0106] In this embodiment, after each processing node operation is executed, an exception handling mechanism detects whether an error has occurred. Based on the captured exception type, a matching error code is searched in a predefined error code table. The original data context at the time of the failed operation is recorded, including the input time string and processing node configuration parameters. The error code, original data context, and other relevant error information, such as the time the error occurred and the processing node name, are integrated to generate a verification report. The error code in the verification report is read and its meaning analyzed. Based on the mapping between error codes and defect types, the defect type of the step-by-step processing logic is determined. The defect type corresponding to each processing node is recorded. Based on the defect type and the information in the verification report, an optimization strategy is formulated. The optimization strategy is then converted into a specific optimization instruction set, defining the operation content and execution order of each instruction. Based on the optimized instruction set, the original step-by-step processing logic is modified. The reconstructed target processing logic is loaded into a code sandbox to ensure its execution in an isolated environment. Finally, test data is prepared, including the original data that previously caused the error. The reconstructed target processing logic is executed in the code sandbox to verify that it can correctly process the test data.

[0107] By flagging errors and generating reports when processing node operations fail, we can quickly locate problems and determine the type of defect. Combined with the generation of optimized instruction sets and the reconfiguration of processing logic, we achieve continuous optimization and improvement of the data processing pipeline. In practical applications, this mechanism can effectively address various data processing issues, improve the robustness and adaptability of the system, ensure the efficient operation of the data processing pipeline, provide more stable and high-quality data support, and enhance the system's maintainability and scalability.

[0108] The technical solution provided in this embodiment ensures the security and accuracy of data processing by isolating and rigorously verifying each processing node of the step-by-step processing logic within a code sandbox. In practical applications, this effectively prevents malicious code or erroneous logic from damaging the system, while ensuring that only high-quality data that has undergone rigorous verification can enter the structured database. This not only improves data security and credibility, but also provides a data foundation for data analysis and query, enhancing the data processing capabilities and business value of the entire application platform.

[0109] Example 3

[0110] Based on any embodiment, the present application also provides a third embodiment, referring to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the method for processing heterogeneous data of the present application. Before step S40, steps S50 to S80 are also included:

[0111] Step S50: Generate lineage metadata of the heterogeneous data based on the field-level mapping relationship.

[0112] Step S60: storing the lineage metadata in association with the extracted target field data, wherein the lineage metadata includes a source file, a field mapping version, and a parsed logical fingerprint.

[0113] Step S70: When target heterogeneous data is newly added, the parsed logical fingerprint in the historical lineage metadata is obtained.

[0114] Step S80: comparing the target logical fingerprint of the target heterogeneous data, and if the fingerprint matching degree exceeds a threshold, calling the associated step-by-step processing logic to process the target heterogeneous data.

[0115] In this embodiment, lineage metadata refers to metadata that describes the data source, processing process, and conversion relationship, and is used to track the entire lifecycle of the data. It typically includes information about the data's source, processing steps, conversion logic, and other content. Field-level mapping relationships refer to the correspondence between source data fields and target data fields. The source file refers to the original storage location of the data, the field mapping version refers to the version number of the field-level mapping relationship, and the parsing logic fingerprint is a unique identifier for the parsing logic, such as a hash value or checksum.

[0116] Specifically, by analyzing the field-level mapping relationship, the data source and conversion process of each target field are determined. And the generation rules of lineage metadata are defined, including the information that needs to be recorded and the generation method. Specifically, it is stipulated that the lineage metadata should record the source data file name, field mapping version number, and the hash value of the parsing logic. Then, based on the field-level mapping relationship and generation rules, the lineage metadata of heterogeneous data is generated. Then, the storage structure is designed to associate the lineage metadata with the target field data for storage. And the association relationship between the lineage metadata and the target field data is established. In the database table, the source file, field mapping version, and parsing logic fingerprint of the lineage metadata are used as foreign keys or additional fields of the target field data. The target field data and the associated lineage metadata are stored in the database.

[0117] Finally, the parsing logic fingerprint of the newly added heterogeneous data is generated. The parsing logic code for the newly added data is hashed to generate the fingerprint "def456." The target logical fingerprint is then compared with the parsing logic fingerprint in the historical lineage metadata. Specifically, the similarity between the target fingerprint "def456" and the historical fingerprint "abc123" is calculated. Based on the comparison results, if the match exceeds a threshold, the step-by-step processing logic associated with the historical lineage metadata is invoked to process the newly added heterogeneous data. For example, if the calculated match is 85%, exceeding the threshold of 80%, the step-by-step processing logic that previously processed the data is automatically invoked to process the newly added heterogeneous data.

[0118] The technical solution provided in this embodiment effectively tracks and manages the sources and processing of heterogeneous data by generating and utilizing lineage metadata. In practical applications, this solution helps reduce the workload of repetitively developing processing logic, improves the efficiency and quality of data processing, and facilitates long-term data maintenance and management, providing reliable and efficient data support for work.

[0119] Example 4

[0120] Based on any embodiment, the present application also provides a fourth embodiment, referring to Figure 4 , Figure 4 This is a flow chart of the fourth embodiment of the method for processing heterogeneous data of the present application. After step S40, steps S90 to S110 are further included:

[0121] Step S90: Obtain the user's query requirements, and identify and parse the corresponding heterogeneous query statements through a natural language model.

[0122] Step S100: performing a database search based on the heterogeneous query statement, obtaining the required data and performing report normalization processing, wherein the normalization processing includes data aggregation, calculation and sorting.

[0123] Step S110: Processing the normalized demand data based on a preset report template to generate a data view, and visually outputting the data view through a data visualization tool.

[0124] In this embodiment, a query requirement refers to a user's request for data expressed in natural language. A natural language model utilizes deep learning technology to process and understand natural language, converting the user's natural language query into a machine-understandable form. Heterogeneous query statements refer to query statements applicable to different data structures and data sources, such as SQL (Structured Query Language) for relational databases and XPath for XML data. Report normalization refers to the organization and transformation of retrieved data to meet report generation requirements, including operations such as data aggregation, calculation, and sorting. A report template is a pre-designed report format and layout that defines how data is presented in the report, including table styles and chart types. A data view is an intermediate data representation generated based on a report template, used for final visualization. Data visualization tools are software tools that display data graphically, such as bar charts, line charts, and pie charts.

[0125] Specifically, the user's natural language query input is received through the user interface. A pre-trained natural language model is used to semantically parse the query, extracting key information such as the time range, type, and query target. The model uses word embedding and sequence encoding to understand the semantics of the user input. Based on the parsing results and the type of the target data source, a corresponding heterogeneous query statement is generated. The generated heterogeneous query statement is sent to the corresponding database for execution to retrieve the raw data. The retrieved data is then aggregated and sorted. Based on the user's query requirements, a report template is selected, specifically one containing a bar chart or a data table to display quantitative statistics. Finally, the normalized data is populated into the report template, with monthly transaction volume data populated into the bar chart's data series and growth rate data populated into the corresponding columns of the data table. A data view is then generated, including setting chart axes, legends, titles, and other elements. The generated data view is then graphically presented to the user using data visualization tools.

[0126] In the technical solution provided in this embodiment, through natural language processing and report generation technology, intelligent analysis of user query needs and visual display of data are achieved, which improves the efficiency of user query and data analysis, enables users to quickly obtain the required information, and improves the availability and ease of use of data.

[0127] The present application provides a device for processing heterogeneous data, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for processing heterogeneous data in the above-mentioned embodiment 1.

[0128] Reference below Figure 5 , which shows a schematic diagram of the structure of a heterogeneous data processing device suitable for implementing the embodiments of the present application. The heterogeneous data processing device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The heterogeneous data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 5As shown, the heterogeneous data processing device may include a processing device 1001 (e.g., a core processor, a graphics processor, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the heterogeneous data processing device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication devices 1009 may allow the heterogeneous data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a heterogeneous data processing device with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0130] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0131] The heterogeneous data processing device provided in this application, employing the heterogeneous data processing method described in the aforementioned embodiment, can address the technical issues associated with low accuracy in data processing and storage in conventional solutions. Compared to the prior art, the heterogeneous data processing device provided in this application offers the same beneficial effects as the heterogeneous data processing method described in the aforementioned embodiment. Other technical features of the heterogeneous data processing device are the same as those disclosed in the aforementioned embodiment and are not further elaborated upon here.

[0132] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the heterogeneous data processing method in the above-mentioned embodiment.

[0135] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0136] The computer-readable storage medium may be included in a heterogeneous data processing device, or may exist independently without being incorporated into the heterogeneous data processing device.

[0137] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a heterogeneous data processing device, the heterogeneous data processing device is enabled to: obtain the heterogeneous data to be processed and extract the field identifiers in the heterogeneous data; based on a pre-trained semantic recognition model, semantically match the field identifiers with the target fields in a preset database to generate a field-level mapping relationship; based on the field-level mapping relationship, generate a step-by-step processing logic adapted to the heterogeneous data format; perform a verification operation on the step-by-step processing logic in a code sandbox, and save the output target field data to a structure database after the verification is passed.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0140] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0141] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for processing heterogeneous data. This computer-readable storage medium can address the technical issues associated with low accuracy in data processing and storage in traditional solutions. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the heterogeneous data processing method provided in the aforementioned embodiments and are not further elaborated here.

[0142] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned method for processing heterogeneous data when executed by a processor.

[0143] The computer program product provided in this application can solve the technical problem of low accuracy in data processing and storage in traditional solutions. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the heterogeneous data processing method provided in the above embodiments, and will not be repeated here.

[0144] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A method for processing heterogeneous data, characterized in that: The method for processing heterogeneous data comprises the following steps: Acquire heterogeneous data to be processed, and extract field identifiers from the heterogeneous data; Based on the pre-trained semantic recognition model, the field identifier is semantically matched with the target field in the preset database to generate a field-level mapping relationship; Generating step-by-step processing logic adapted to heterogeneous data formats based on the field-level mapping relationship; Loading a code sandbox, and allocating a corresponding target container in the code sandbox to each processing node in the step-by-step processing logic; Executing the operation corresponding to the processing node in the target container, and extracting the target field data output by the target container; Performing a verification operation on the target field data; When an operation corresponding to any of the processing nodes fails, an error code and original data context corresponding to the operation are marked and a verification report is generated; Parse the error code in the verification report to determine the defect type of the step-by-step processing logic: generating an optimized instruction set according to the verification report and the defect type, and reconstructing the step-by-step processing logic based on the optimized instruction set to generate a target processing logic; Executing and verifying the reconstructed target processing logic in the code sandbox; After verification, the target field data is saved in the structure database.

2. The method according to claim 1, wherein The step of semantically matching the field identifier with the target field in the preset database based on the pre-trained semantic recognition model to generate a field-level mapping relationship includes: Identify the semantic description and associated fields corresponding to the target field from the preset database; Determining the similarity between the field identifier and the target field based on the semantic description and the associated fields, and generating a semantic similarity list; The field-level mapping relationship between the field identifier and the target field is determined from a mapping rule set according to the semantic similarity list.

3. The method according to claim 1, wherein The step of generating step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship includes: determining the heterogeneous data format of the heterogeneous data; Determining a matching processing logic template according to the field-level mapping relationship, and selecting a matching target processing logic template according to the heterogeneous data format; According to the target processing logic template, a step-by-step processing logic for the heterogeneous data is generated.

4. The method according to claim 1, wherein The step of performing a verification operation on the target field data includes: Extracting a time field, a value field, and a text field from the target field data; Verify the format and time range of the time field according to data type rules; The units of the numerical fields are uniformly converted, and the fields in the text fields that match the standardized terms are replaced with each other.

5. The method according to claim 1, wherein Before the steps of loading the code sandbox and allocating the corresponding target container in the code sandbox to each processing node in the step-by-step processing logic, the method further includes: generating lineage metadata of the heterogeneous data based on the field-level mapping relationship; Associating and storing the lineage metadata with the extracted target field data, wherein the lineage metadata includes a source file, a field mapping version, and a parsed logical fingerprint; When new target heterogeneous data is added, the analytical logic fingerprint in the historical lineage metadata is obtained; The target logical fingerprint of the target heterogeneous data is compared, and if the fingerprint matching degree exceeds a threshold, the associated step-by-step processing logic is called to process the target heterogeneous data.

6. The method according to claim 1, wherein After the step of saving the target field data to the structure database after the verification is passed, the method further includes: Obtain the user's query requirements and identify and parse the corresponding heterogeneous query statements through natural language models; Perform database retrieval based on the heterogeneous query statements, obtain required data and perform report normalization processing, the normalization processing including data aggregation, calculation and sorting; The normalized demand data is processed based on a preset report template to generate a data view, and the data view is visually output through a data visualization tool.

7. A device for processing heterogeneous data, characterized in that: The heterogeneous data processing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the heterogeneous data processing method according to any one of claims 1 to 6.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for processing heterogeneous data according to any one of claims 1 to 6 are implemented.

Images