Heterogeneous data processing method and device and storage medium

Abstract

The invention discloses a heterogeneous data processing method and device and a storage medium, and relates to the technical field of data processing.The method comprises the steps that to-be-processed heterogeneous data are obtained, field identifiers in the heterogeneous data are extracted, and then the field identifiers in the heterogeneous data are identified on the basis of a pre-trained semantic identification model; performing semantic matching on the field identifier and a target field in a preset database to generate a field-level mapping relationship, generating a step-by-step processing logic adaptive to a heterogeneous data format according to the field-level mapping relationship, executing verification operation on the step-by-step processing logic in a code sandbox, and performing verification operation on the step-by-step processing logic in the code sandbox. And after the verification is passed, storing the output target field data to a structure database. According to the heterogeneous data processing method and device, the extracted field identification is subjected to semantic matching, and the extracted target field is verified in combination with the code sandbox, so that the accuracy of heterogeneous data processing is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a method, device, and storage medium for processing heterogeneous data. Background Art

[0002] Currently, when integrating heterogeneous data, it is usually dependent on technicians to manually analyze the source data, then classify the identified fields and associate tags. Then, according to the preset rules, compare the source data with the data in the database one by one to find the matching data and store it in the database. However, in the above method, when dealing with a large amount of or frequently changing data sources, relying on manual identification and classification search, data mapping errors may occur due to subjective understanding deviations or negligence, which may lead to low accuracy in subsequent data processing and storage.

[0003] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] This application provides a method, device, and storage medium for processing heterogeneous data, aiming to solve the problem of low accuracy in data processing and storage in the traditional solution.

[0005] To achieve the above object, a method for processing heterogeneous data provided by this application includes the following steps: Obtain the heterogeneous data to be processed and extract the field identifiers in the heterogeneous data; Based on a pre-trained semantic recognition model, semantically match the field identifiers with the target fields in a preset database to generate a field-level mapping relationship; According to the field-level mapping relationship, generate a step-by-step processing logic adapted to the heterogeneous data format; Perform a verification operation on the step-by-step processing logic in a code sandbox, and save the output target field data to a structured database after the verification passes.

[0006] In one embodiment, the step of based on a pre-trained semantic recognition model, semantically match the field identifiers with the target fields in a preset database to generate a field-level mapping relationship includes: Identify the semantic descriptions and associated fields corresponding to the target fields from the preset database; According to the semantic descriptions and associated fields, determine the similarity between the field identifiers and the target fields, and generate a semantic similarity list; According to the semantic similarity list, determine the field-level mapping relationship between the field identifiers and the target fields from a mapping rule set.

[0007] In one embodiment, the step of generating step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship includes: Determine the heterogeneous data format of the heterogeneous data; Determine a matching processing logic template according to the field-level mapping relationship, and select a matching target processing logic template according to the heterogeneous data format; Generate step-by-step processing logic for the heterogeneous data according to the target processing logic template.

[0008] In one embodiment, the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the target field data to the structure database after the verification passes includes: Load the code sandbox, and allocate corresponding target containers in the code sandbox to each processing node in the step-by-step processing logic; Perform the operations corresponding to the processing nodes in the target containers, and extract the target field data output by the target containers; Perform the verification operation on the target field data; After the verification passes, save the target field data to the structure database.

[0009] In one embodiment, the step of performing the verification operation on the target field data includes: Extract the time field, numerical field, and text field from the target field data; Verify the format and time range of the time field according to the data type rules; Perform unit conversion on the numerical field in a unified manner, and replace the fields in the text field that match the standardized terms with each other.

[0010] In one embodiment, after the step of performing the verification operation on the target field data, it further includes: When the operation corresponding to any one of the processing nodes fails, mark the error code and the original data context corresponding to the operation and generate a verification report; Parse the error code in the verification report to determine the defect type of the step-by-step processing logic: Generate an optimization instruction set according to the verification report and the defect type, and reconstruct the step-by-step processing logic based on the optimization instruction set to generate target processing logic; Execute the reconstructed target processing logic in the code sandbox and verify it.

[0011] In one embodiment, before the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after the verification passes, the following steps are further included: Generate lineage metadata for the heterogeneous data based on the field-level mapping relationship; Associatively store the lineage metadata with the extracted target field data, where the lineage metadata includes the source file, the field mapping version, and the parsing logic fingerprint; When new target heterogeneous data is added, obtain the parsing logic fingerprint in the historical lineage metadata; Compare the target logic fingerprint of the target heterogeneous data. If the fingerprint matching degree exceeds the threshold, call the associated step-by-step processing logic to process the target heterogeneous data.

[0012] In one embodiment, after the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after the verification passes, the following steps are further included: Obtain the user's query requirement and identify and parse it through a natural language model to determine the corresponding heterogeneous query statement; Perform a database retrieval based on the heterogeneous query statement, obtain the required data and perform report normalization processing, where the normalization processing includes data aggregation, calculation, and sorting; Process the normalized required data based on a preset report template to generate a data view, and visually output the data view through a data visualization tool.

[0013] In addition, to achieve the above object, the present application further provides a processing device for heterogeneous data. The processing device for heterogeneous data includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the method for processing heterogeneous data as described above.

[0014] In addition, to achieve the above object, the present application further provides a storage medium. The storage medium is a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the method for processing heterogeneous data as described above are implemented.

[0015] The present application provides a method for processing heterogeneous data, a device for processing heterogeneous data, and a storage medium. By obtaining the heterogeneous data to be processed, extracting the field identifiers in the heterogeneous data, then based on a pre-trained semantic recognition model, semantically matching the field identifiers with the target fields in a preset database to generate a field-level mapping relationship, and then generating a step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship, then performing a verification operation on the step-by-step processing logic in a code sandbox, and saving the output target field data to a structure database after the verification passes. The present application improves the accuracy of heterogeneous data processing by extracting field identifiers, performing semantic matching, and then verifying the extracted target fields in combination with a code sandbox. Brief Description of the Drawings

[0016] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of the first embodiment of the method for processing heterogeneous data of the present application; Figure 2 It is a schematic flowchart of the second embodiment of the method for processing heterogeneous data of the present application; Figure 3 It is a schematic flowchart of the third embodiment of the method for processing heterogeneous data of the present application; Figure 4 It is a schematic flowchart of the fourth embodiment of the method for processing heterogeneous data of the present application; Figure 5 It is a schematic architecture diagram of the hardware operating environment of the device for processing heterogeneous data involved in the embodiments of the present application.

[0019] The implementation, functional features, and advantages of the objectives of the present application will be further described in combination with the embodiments with reference to the drawings. Detailed Embodiments

[0020] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] To better understand the above technical solution, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully conveyed to those skilled in the art.

[0022] To better understand the above technical solution, the above technical solution will be described in detail below in combination with the accompanying drawings of the specification and specific implementation manners.

[0023] The main solution of the present application is: obtaining heterogeneous data to be processed and extracting field identifiers in the heterogeneous data; Based on a pre-trained semantic recognition model, semantically matching the field identifiers with target fields in a preset database to generate a field-level mapping relationship; Generating step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship; Performing a verification operation on the step-by-step processing logic in a code sandbox and saving the output target field data to a structure database after the verification passes.

[0024] Currently, when integrating heterogeneous data, it usually relies on technicians to manually analyze the source data, then classify the identified fields and associate tags. Then, according to the preset rules, compare the source data with the data in the database one by one and find the matching data, and store it in the database. However, in the above method, when dealing with a large amount of or frequently changing data sources, relying on manual identification and classification search, data mapping errors may occur due to subjective understanding deviations or negligence, which will lead to low accuracy in subsequent data processing and warehousing.

[0025] By obtaining heterogeneous data to be processed, extracting field identifiers in the heterogeneous data, then based on a pre-trained semantic recognition model, semantically matching the field identifiers with target fields in a preset database to generate a field-level mapping relationship, then generating step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship, then performing a verification operation on the step-by-step processing logic in a code sandbox, and saving the output target field data to a structure database after the verification passes. The present application improves the accuracy of heterogeneous data processing by extracting field identifiers, performing semantic matching, and combining code sandbox verification of the extracted target fields.

[0026] Embodiment 1 Based on this, an embodiment of the present application provides a method for processing heterogeneous data, referring to Figure 1 , Figure 1It is a schematic flowchart of the first embodiment of the method for processing heterogeneous data of this application. The method for processing heterogeneous data includes steps S10 to S40: Step S10: Obtain the heterogeneous data to be processed, and extract the field identifiers in the heterogeneous data.

[0027] In this embodiment, the processing actions are performed by a data processing system. Heterogeneous data refers to data from different sources, formats, and structures, including structured data, such as SQL table / CSV column name parsing, specifically obtaining the above data through primary key identification and data type inference, semi-structured data, such as JSON / XML / log files, which can be obtained through tree structure parsing and path expressions (XPath), and unstructured data, including text / images / PDFs, extracting unstructured data through NLP entity recognition and OCR image and text separation. A field identifier is a name used to identify the data content. For example, in a preset database, "number", "name", "occurrence time", etc. are all field identifiers.

[0028] Optionally, the heterogeneous data can be obtained by acquiring heterogeneous investigation data from Internet manufacturers, and then performing structured extraction based on the heterogeneous investigation data, including online interface APIs, online databases, FTP resources, local document OCR recognition, voice data extraction, etc. Thus, heterogeneous data is obtained and passed to the data processing system.

[0029] Furthermore, when acquiring the heterogeneous investigation data from Internet manufacturers, through a data source identification module, relevant data source information related to the user's needs is retrieved using a knowledge base, including the location, type (such as SQL database, CSV file, Excel spreadsheet, etc.), structure, and other data of the data source. For example, if the user queries the phone records related to a certain number, the system retrieves the telecommunications operator's database as the relevant data source through the knowledge base. Its location is the database on a specific server, and the type is an SQL database. Then, the preset data specification standard is obtained, which corresponds to the data type and stipulates the format requirements for different data types, such as text, number, date, etc. For example, for date type data, it is stipulated that its format should be "YYYY-MM-DD". Then, based on the data specification standard, the original investigation data is parsed to extract the field identifiers. For heterogeneous data, the field identifiers are extracted through column name parsing, that is, obtaining the names of each column in the table or file. For example, in an SQL table, the names of each column, such as "user ID", "call duration", "call number", etc., are field identifiers.

[0030] It should be noted that in the process of collecting privacy data such as call records applied in the above technical solution, corresponding restriction processes for preventing privacy infringement are set up, so that the corresponding data can be obtained through legal channels. In addition, the above restriction process is specifically as follows: when obtaining privacy data such as call records, a permission request for obtaining data needs to be fed back to the user's corresponding terminal, and after the user authorizes, the above privacy data is obtained. If the user does not authorize, the privacy data is not obtained.

[0031] Step S20: Based on the pre-trained semantic recognition model, semantically match the field identifier with the target field in the preset database to generate a field-level mapping relationship.

[0032] In this embodiment, the pre-trained semantic recognition model is a natural language processing model based on deep learning. Through pre-training on a large-scale text data, it has learned the general semantic features and representations of the language. Semantic matching refers to the process of judging whether two texts or field identifiers are semantically matched by calculating the semantic similarity between them. The field-level mapping relationship refers to the corresponding relationship established between the field identifier in the source data and the field identifier in the target data, indicating which field in the target database the certain field in the source data corresponds to semantically.

[0033] Specifically, first extract semantic features through the pre-trained semantic recognition model. First, extract semantic features from the field identifiers of heterogeneous data and the target field names in the preset database respectively. The pre-trained semantic recognition model encodes each field identifier text into a vector with a fixed length, and this vector represents the semantic features of the text. For example, for the field identifier "processing time" in the source data, the model encodes it into a vector with a length of 768, and this vector contains the semantic information of the concept of "processing time".

[0034] Then, by methods such as cosine similarity, calculate the similarity between the semantic vector of the source data field identifier and the semantic vector of the target field name. Cosine similarity is used to measure the similarity degree of two vectors in direction, and its value range is between -1 and 1. The closer the value is to 1, the more similar the directions of the two vectors are, that is, the more similar the semantics are. For example, calculate the cosine similarity of the semantic vectors of the heterogeneous data field identifier "processing time" and the target field "occurrence time" in the preset database, and the result is 0.85, which indicates that they are semantically similar.

[0035] As an optional implementation manner, in this embodiment, step S20 includes: Identify the semantic description corresponding to the target field and the associated fields from the preset database; determine the similarity between the field identifier and the target field according to the semantic description and the associated fields, and generate a semantic similarity list; determine the field-level mapping relationship between the field identifier and the target field from the mapping rule set according to the semantic similarity list.

[0036] Specifically, perform semantic recognition on the data source and the destination, that is, the table names and field names of the preset data, to obtain the data source semantics and the destination semantics. Compare the semantics of each field of each data source with the semantics of all fields of the destination to obtain the semantic similarity list of each field of the corresponding data source; determine the mapping relationship between the data source and the destination from the mapping rule set according to the semantic similarity list; store all mapping relationships in the mapping relationship library.

[0037] Optionally, a similarity threshold can be set according to the similarity result. When the similarity between the source data field identifier and the target field name is greater than the threshold, it is determined that there is a field-level mapping relationship between the two. For example, in the above example, since the similarity 0.85 is greater than 0.7, it is determined that there is a field-level mapping relationship between "processing time" and "occurrence time", and this mapping relationship is stored in the mapping relationship library.

[0038] Step S30: Generate step-by-step processing logic adapted to heterogeneous data formats according to the field-level mapping relationship.

[0039] In this embodiment, the step-by-step processing logic refers to decomposing the data processing process into multiple steps, and each step defines the rules and processes for performing specific operations on the data. The heterogeneous data format refers to the format characteristics of heterogeneous data, such as the table structure of a relational database, the row-column structure of a CSV file, etc., which includes the data organization method, field type, and data relationship. The distributed processing logic can also be to dynamically select process processing nodes or generate code processing nodes based on semantic mapping based on a large model.

[0040] Specifically, analyze the determined field-level mapping relationships to understand the correspondence between source data fields and target fields, including whether the data types are consistent and whether the field lengths match. For example, it is found through analysis that the "processing amount" field in the source data matches the "associated amount" field in the target field semantically, but the data type of the "processing amount" field is floating-point, while the data type of the "associated amount" field is integer. Then, based on the analysis results of the mapping relationships, determine the data processing rules for each field. If the data types of the source data field and the target field are inconsistent, data type conversion is required; if the field lengths are different, operations such as truncation or padding are needed. For example, for the above-mentioned "processing amount" field, it is determined to convert it to an integer type and then map it to the "associated amount" field. Next, based on the determined processing rules and combined with the data specification standards, generate step-by-step processing logic adapted to heterogeneous data formats. This includes defining the operation sequence of each processing step, the input and output data formats, etc. For example, the generated processing logic is: first, perform format conversion on the "processing time" field to make it conform to the date format requirements of the "occurrence time" field in the preset database; then, perform data type conversion and numerical processing on the "processing amount" field, and finally insert the processed data into the target field.

[0041] Optionally, in this embodiment, step S30 includes: Determine the heterogeneous data format of the heterogeneous data; determine a matching processing logic template according to the field-level mapping relationship, and select a matching target processing logic template according to the heterogeneous data format; generate step-by-step processing logic for the heterogeneous data according to the target processing logic template.

[0042] Specifically, a processing logic template is a template of a set of predefined data processing rules and processes, which stipulates corresponding data conversion, cleaning, integration, etc. operation methods for different types of heterogeneous data formats and field-level mapping relationships. A target processing logic template refers to the template that best matches the current heterogeneous data format and field-level mapping relationship among numerous processing logic templates, which can ensure the efficiency and accuracy of data processing.

[0043] The heterogeneous data obtained is scanned by the data source identification module, and its file header information or database metadata is read to preliminarily determine the data format type. For example, when scanning the header information of a file, if it is found that it starts with "<" and contains a tag structure, it is preliminarily determined to be an XML file; if it is found that it starts with "{" and contains a key-value pair structure, it is preliminarily determined to be a JSON file. The content characteristics of the data are further analyzed, such as the organizational structure of the data, delimiters, data type definitions, etc., to accurately determine the heterogeneous data format. For Excel files, analyze their worksheet structure, cell formats, etc.; for CSV files, determine the data delimiters, such as commas, tab characters, etc., and whether they contain header information. Then, using the preset format verification rules, the preliminarily determined heterogeneous data format is verified to ensure the accuracy of the judgment result.

[0044] In the preset processing logic template library, retrieve according to the keywords of the field-level mapping relationship, and filter out the processing logic templates related to this mapping relationship. Then analyze the filtered processing logic templates, compare the heterogeneous data formats adapted by the templates with the currently determined heterogeneous data formats, and calculate the matching degree between them. According to the matching degree result, select the processing logic template with the highest matching degree as the target processing logic template. If there are multiple templates with similar matching degrees, factors such as the processing efficiency and generality of the templates are comprehensively considered for selection. For example, among multiple processing logic templates applicable to JSON format data, select the template with the highest processing efficiency and better generality as the target processing logic template. Replace the general parameters in the target processing logic template with the specific parameters of the current heterogeneous data, such as the data source path, field name, data type, etc. According to the actual situation of the heterogeneous data and the field-level mapping relationship, refine the processing flow in the template, and clarify the specific operation methods and sequences of each processing step.

[0045] Step S40: Perform a verification operation on the step-by-step processing logic in the code sandbox, and save the output target field data to the structure database after the verification passes.

[0046] In this embodiment, the code sandbox is a secure code execution environment that restricts the access rights of the code to system resources and prevents malicious code or defective code from damaging the system. The verification operation refers to the process of checking and testing the correctness and security of the step-by-step processing logic according to predefined rules and standards. The structure database is a database used to store heterogeneous data, with a fixed table structure and data type definitions.

[0047] Specifically, first define the rules for verifying the step-by-step processing logic in the code sandbox, including the verification of the accuracy of data types, such as checking whether the converted data meets the data type requirements of the target field, the verification of data integrity, the verification of semantic correctness, etc. Then place the step-by-step processing logic in the code sandbox for execution, and check the output of each step according to the verification rules during the execution process. For example, after executing each step of the processing logic, check whether the intermediate result meets the expected data type and format, such as checking whether the date format after converting the "processing time field" is correct. If the verification operation passes, that is, all the check results meet the verification rules, then save the processed target field data to the structured database. The saving process includes establishing a connection with the structured database and inserting the processed data into the corresponding tables and fields according to the database insertion statement specification. For example, insert the processed data of fields such as "processing amount" and "processing time" into the corresponding tables of the preset database through SQL insertion statements.

[0048] In the technical solution provided in this embodiment, by obtaining the heterogeneous data to be processed, extracting the field identifiers in the heterogeneous data, then based on the pre-trained semantic recognition model, semantically matching the field identifiers with the target fields in the preset database to generate a field-level mapping relationship, and then according to the field-level mapping relationship, generating a step-by-step processing logic adapted to the heterogeneous data format, then performing a verification operation on the step-by-step processing logic in the code sandbox, and saving the output target field data to the structured database after the verification passes. This embodiment improves the accuracy of heterogeneous data processing by extracting field identifiers and performing semantic matching, and then combining the code sandbox to verify the extracted target fields.

[0049] In addition, through a series of actions such as data source identification, data specification acquisition, and field identifier extraction, the field identifiers can be accurately extracted from various heterogeneous data sources, improving the accuracy and pertinence of data processing. Through semantic matching by the pre-trained semantic recognition model, the semantic correspondence between the source data field identifiers and the target fields in the preset database can be accurately identified, which helps to effectively integrate heterogeneous data from different sources and formats into the preset database, improving the accuracy and efficiency of data integration. Through steps such as mapping relationship analysis, processing rule determination, and processing logic generation, the accuracy and consistency of data during conversion and insertion into the target database are ensured, avoiding data errors and incompatibility problems. By performing a verification operation in the code sandbox, the correctness and security of the step-by-step processing logic can be effectively detected, avoiding the damage to the database caused by incorrect data or malicious code, and ensuring the integrity and accuracy of the data.

[0050] Embodiment Two Based on any one of the embodiments, the present application also provides a second embodiment. Refer to Figure 2 ,Figure 2 This is a schematic flowchart of the second embodiment of the method for processing heterogeneous data in this application. Step S40 includes steps S41 to S44: Step S41: Load the code sandbox and allocate corresponding target containers in the code sandbox to each processing node in the step-by-step processing logic.

[0051] Step S42: Execute the operation corresponding to the processing node in the target container and extract the target field data output by the target container.

[0052] Step S43: Perform the verification operation on the target field data.

[0053] Step S44: After the verification passes, save the target field data to the structure database.

[0054] In this embodiment, the code sandbox is an isolated code execution environment that can restrict the access rights of code to system resources and prevent malicious code or defective code from damaging the system. The target container is an independent running environment allocated for each processing node in the code sandbox, and each target container has an independent resource space and execution permission. The target field data output by the target container refers to the data that meets the expected format and content requirements after the operation of the processing node. The extraction operation refers to the process of obtaining this target field data from the output result of the target container. The verification operation is a process of checking the accuracy, integrity, and consistency of the target field data according to predefined rules and standards. The verification rules are specific criteria for judging whether the data meets the requirements, including data type checking, data range checking, data format checking, and data correlation checking.

[0055] Optionally, first start the code sandbox service and load the configuration parameters of the sandbox, including resource limitations such as memory and CPU usage rate, and security policies such as prohibiting access to external networks and restricting file operation permissions. Then parse the step-by-step processing logic to identify each processing node and its mutual relationship. For example, in the step-by-step processing logic, identify processing nodes such as "data type conversion", "field mapping", and "data aggregation", and determine their execution order and dependencies. According to the number and type of processing nodes, dynamically allocate corresponding target containers in the code sandbox. Each target container configures the corresponding running environment according to the requirements of the processing node, such as running the required programming language, such as Python interpreter, Java virtual machine, library files, etc. For example, allocate a target container for the "data type conversion" processing node, and install the Python 3.8 version interpreter and related data processing libraries in this container.

[0056] Trigger the operations of the processing nodes in each target container in sequence according to the execution order defined by the step-by-step processing logic. For example, during the execution of the processing node operations, transfer data between target containers as needed. For example, the data result after "time format conversion" is passed as input to the "amount unit conversion" target container. During the transfer process, data format conversion is required to meet the input requirements of the next processing node. After each target container completes the operation, capture its output result and extract the target field data from it. This can be achieved by reading the standard output stream of the target container, the specified output file, or through the container intercommunication mechanism. Then load the validation rules related to the current data processing task from the validation rule library. For example, the loaded rule stipulates that the data type of the "occurrence time" field is the date type, and the format is "YYYY-MM-DD"; the data type of the "associated amount" field is the numerical type, and the value range is between 0 and 100 million yuan.

[0057] Then, compare the extracted target field data with the validation rules one by one and perform the verification operation. For example, check whether the value of the "occurrence time" field conforms to the date format requirement and can be correctly converted into a date object; check whether the value of the "associated amount" field is of the numerical type and within the allowed value range. If it is found that the data does not conform to the validation rules, record the detailed error information, including the error type, occurrence location, error data example, etc. At the same time, according to the severity of the error, decide whether to terminate the entire data processing flow or attempt to repair the data. Finally, use the pre-configured database connection parameters, such as the database address, port, username, password, etc., to establish a connection with the structure database. According to the table structure and field definition of the structure database, generate an SQL insert statement to insert the target field data into the corresponding table. Execute the SQL insert statement to submit the data to the structure database. At the same time, start a database transaction to ensure the atomicity and integrity of the data insertion operation. If an error occurs during the insertion process, such as violating the unique constraint, foreign key constraint, etc., roll back the transaction to avoid inconsistent data in the database.

[0058] Optionally, in this embodiment, the step of performing the verification operation on the target field data includes: Extract the time field, numerical field, and text field from the target field data; verify the format and time range of the time field according to the data type rule; perform unit conversion on the numerical field, and replace the fields in the text field that match the standardized terms with each other.

[0059] Specifically, through a predefined data type dictionary, the type of each field in the target field data is identified. The data type dictionary includes common time formats, numerical formats, and text features. According to the identification results, the fields in the target field data are separated into three categories: time fields, numerical fields, and text fields. Classification storage can be implemented through data structures in programming languages. Then, corresponding validation rules and processing logics are prepared for each separated field category. The format validation rules for time fields are loaded from the data type rule library, including allowed date separators, the order of date components, etc. The value of the time field is matched and checked against the format validation rules. According to the business logic and data background, a reasonable range for the time field is defined, and it is verified whether the time value is within this range. For example, it is stipulated that the "occurrence time" cannot be earlier than "2020-01-01", assumed to be the time when a certain preset database starts recording, and it cannot be later than the current system time. If the "occurrence time" of a certain record is "2025-01-01" and the current system time is "2024-10-10", then it is determined that this time exceeds the reasonable range.

[0060] Then, the unit conversion rules for numerical fields are loaded from the data processing rule library, and the conversion relationships and conversion factors between different units are defined in the rules. Then, according to the loaded unit conversion rules, conversion operations are performed on the data in the numerical fields. Finally, the standardized term dictionary for text fields is loaded, and the dictionary contains common terms and their corresponding standardized replacement terms. And the content of the text field is scanned, and when a match is found with the entries in the standardized term dictionary, a replacement operation is performed. For example, the value of the "target user relationship" field in a certain record is "the brother of the target user", and according to the dictionary, "target user" is changed to "the brother of the target user".

[0061] Furthermore, in this embodiment, after the step of performing the validation operation on the target field data, the following steps are further included: When the operation corresponding to any one of the processing nodes fails, mark the error code and the original data context corresponding to the operation and generate a validation report; analyze the error code in the validation report to determine the defect type of the step-by-step processing logic: generate an optimization instruction set according to the validation report and the defect type, and reconstruct the distributed processing logic based on the optimization instruction set to generate the target processing logic; execute the reconstructed target processing logic in the code sandbox and verify it.

[0062] Specifically, an error code is a predefined set of codes used to identify specific error types and causes. The original data context refers to the original data environment related to an operation when a processing node operation fails, including input data, processing parameters, etc., and is used to restore the specific situation when the error occurred. A verification report is a document recording the results of verification operations, including error information, error locations, error codes, etc. A defect type refers to the category of problems existing in the step-by-step processing logic, such as data format mismatch, data missing, logical errors, etc. The parsing operation is the process of mapping the error code in the verification report to the corresponding defect type through analysis. The optimization instruction set is the correction instruction for defects in the step-by-step processing logic, including specific correction steps and parameter adjustments. Reconstruction refers to modifying and improving the original step-by-step processing logic according to the optimization instruction set to generate a new target processing logic.

[0063] In this embodiment, after each processing node operation is executed, the exception handling mechanism is used to detect whether an error occurs. And according to the captured exception type, a matching error code is searched in the predefined error code table. The original data context at the time of operation failure is recorded, including the input time string, configuration parameters of the processing node, etc. The error code, the original data context, and other relevant error information, such as the time when the error occurred and the name of the processing node, are integrated to generate a verification report. The error code in the verification report is read and its meaning is analyzed. Then, according to the mapping relationship between the error code and the defect type, the defect type of the step-by-step processing logic is determined, the defect type corresponding to each processing node is recorded, and an optimization strategy is formulated based on the defect type and the information in the verification report. Then the optimization strategy is transformed into a specific optimization instruction set, and the operation content and execution order of each instruction are defined. According to the optimization instruction set, the original step-by-step processing logic is modified. The reconstructed target processing logic is loaded into the code sandbox to ensure its operation in an isolated environment. Finally, test data is prepared, including the original data that caused the error before, and the reconstructed target processing logic is executed in the code sandbox to check whether it can correctly process the test data.

[0064] By generating error markings and reports when a processing node operation fails, the problem can be quickly located and the defect type can be determined. Combining the generation of the optimization instruction set and the reconstruction of the processing logic realizes the continuous optimization and improvement of the data processing flow. In practical applications, this mechanism can effectively handle various problems in data processing, improve the robustness and adaptability of the system, ensure the efficient operation of the data processing flow, provide more stable and high-quality data support for work, and enhance the maintainability and scalability of the system.

[0065] In the technical solution provided in this embodiment, by isolating and strictly verifying each processing node of the step-by-step processing logic in the code sandbox, the security and accuracy of data processing are ensured. In practical applications, it can effectively prevent malicious code or incorrect logic from damaging the system, and at the same time ensure that high-quality data that has passed strict verification can enter the structured database. This not only improves the security and credibility of the data, but also provides a data basis for data analysis and query, enhancing the data processing ability and business value of the entire application platform.

[0066] Embodiment III Based on any one of the embodiments, the present application further provides a third embodiment. Refer to Figure 3 , Figure 3 which is a schematic flowchart of the third embodiment of the method for processing heterogeneous data of the present application. Before step S40, steps S50 to S80 are further included: Step S50: Generate lineage metadata of the heterogeneous data based on the field-level mapping relationship.

[0067] Step S60: Associatively store the lineage metadata with the extracted target field data, where the lineage metadata includes the source file, the field mapping version, and the parsing logic fingerprint.

[0068] Step S70: When new target heterogeneous data is added, obtain the parsing logic fingerprint in the historical lineage metadata.

[0069] Step S80: Compare the target logic fingerprint of the target heterogeneous data. If the fingerprint matching degree exceeds the threshold, call the associated step-by-step processing logic to process the target heterogeneous data.

[0070] In this embodiment, the lineage metadata refers to metadata that describes the data source, processing process, and transformation relationship, and is used to trace the entire life cycle of the data. It usually includes information such as the data source, processing steps, and transformation logic. The field-level mapping relationship refers to the correspondence between the source data field and the target data field. The source file refers to the original storage location of the data, the field mapping version refers to the version number of the field-level mapping relationship, and the parsing logic fingerprint is the unique identifier of the parsing logic, such as a hash value, checksum, etc.

[0071] Specifically, by analyzing the field-level mapping relationships, determine the data sources and conversion processes for each target field. Also, define the generation rules for lineage metadata, including the information to be recorded and the generation methods. Specifically, it is stipulated that the lineage metadata should record the source data file name, the field mapping version number, and the hash value of the parsing logic. Then, based on the field-level mapping relationships and the generation rules, generate the lineage metadata for heterogeneous data. Next, design a storage structure to store the lineage metadata and the target field data in an associated manner. And establish the association relationship between the lineage metadata and the target field data. In the database table, use the source file of the lineage metadata, the field mapping version, and the parsing logic fingerprint as foreign keys or additional fields for the target field data. Store the target field data and the associated lineage metadata in the database.

[0072] Finally, generate the parsing logic fingerprint for the newly added heterogeneous data, perform a hash calculation on the parsing logic code of the newly added data to generate the fingerprint "def456". Compare the target logic fingerprint with the parsing logic fingerprints in the historical lineage metadata. Specifically, calculate the similarity between the target fingerprint "def456" and the historical fingerprint "abc123". According to the comparison result, if the matching degree exceeds the threshold, call the step-by-step processing logic associated with the historical lineage metadata to process the newly added heterogeneous data. For example, the matching degree is calculated as 85%, which exceeds the threshold of 80%, and automatically call the previous step-by-step processing logic for this data to process the newly added heterogeneous data.

[0073] In the technical solution provided in this embodiment, by generating and utilizing lineage metadata, it is possible to effectively trace and manage the sources and processing processes of heterogeneous data. In practical applications, the above solution helps to reduce the workload of repeatedly developing processing logic, improve the efficiency and quality of data processing, and is also conducive to the long-term maintenance and management of data, providing reliable and efficient data support for work.

[0074] Embodiment Four Based on any of the embodiments, the present application also provides a fourth embodiment. Refer to Figure 4 , Figure 4 which is a schematic flowchart of the fourth embodiment of the method for processing heterogeneous data of the present application. After step S40, it further includes steps S90 to S110: Step S90: Obtain the user's query requirements and identify and parse them through a natural language model to determine the corresponding heterogeneous query statement.

[0075] Step S100: Perform a database retrieval based on the heterogeneous query statement, obtain the required data, and perform report normalization processing, where the normalization processing includes data aggregation, calculation, and sorting.

[0076] Step S110: Process the normalized requirement data based on a preset report template to generate a data view, and visually output the data view through a data visualization tool.

[0077] In this embodiment, the query requirement is the query requirement for data expressed by the user in natural language. The natural language model is a model that uses deep learning technology to process and understand natural language, and can convert the user's natural language query into a machine-understandable form. Heterogeneous query statements refer to query statements applicable to different data structures and data sources. For example, SQL (Structured Query Language) is used for relational databases, and XPath is used for XML data, etc. Report normalization processing refers to the collation and transformation of the retrieved data to make it meet the requirements of report generation, including operations such as data aggregation, calculation, and sorting. The report template is a pre-designed report format and layout that defines how data is displayed in the report, including table styles, chart types, etc. The data view refers to the intermediate data representation generated according to the report template for final visual display. The data visualization tool refers to a software tool that displays data in a graphical manner, such as bar charts, line charts, pie charts, etc.

[0078] Specifically, receive the user's natural language query input through the user interface, use the pre-trained natural language model to perform semantic parsing on the user's input query, extract key information such as time range, type, and query target. This model understands the semantics of the user input through word embedding and sequence encoding. Then, according to the parsing result and the type of the target data source, generate the corresponding heterogeneous query statement. Send the generated heterogeneous query statement to the corresponding database for execution to obtain the original data. Then perform aggregation operations on the retrieved data, perform sorting operations on the data, and select the corresponding report template according to the user's query requirements. Specifically, a report template containing a bar chart and a data table can be selected to display the quantity statistics. Finally, fill the normalized data into the report template, fill the monthly business quantity data into the data series of the bar chart, and fill the growth rate data into the corresponding columns of the data table. And generate a data view, including setting elements such as the coordinate axes, legends, and titles of the chart. And use the data visualization tool to display the generated data view in a graphical manner to the user.

[0079] In the technical solution provided in this embodiment, through natural language processing and report generation technologies, the intelligent parsing of the user's query requirements and the visual display of data are realized, improving the efficiency of the user's querying and analyzing data, enabling the user to quickly obtain the required information, and enhancing the availability and usability of the data.

[0080] The present application provides a processing device for heterogeneous data. The processing device for heterogeneous data includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for processing heterogeneous data in the first embodiment above.

[0081] Reference is made below Figure 5 , which shows a schematic structural diagram of a processing device for heterogeneous data suitable for implementing the embodiments of the present application. The processing device for heterogeneous data in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The processing device for heterogeneous data shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0082] As Figure 5As shown in the figure, the processing device for heterogeneous data may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the processing device for heterogeneous data are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the processing device for heterogeneous data to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a processing device for heterogeneous data having various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0083] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0084] The processing device for heterogeneous data provided by the present application adopts the method for processing heterogeneous data in the above embodiments, and can solve the technical problem of relatively low accuracy in data processing and warehousing in the traditional solution. Compared with the prior art, the beneficial effects of the processing device for heterogeneous data provided by the present application are the same as those of the method for processing heterogeneous data provided by the above embodiments, and other technical features in the processing device for heterogeneous data are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0085] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0086] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0087] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the heterogeneous data processing method in the above embodiments.

[0088] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0089] The above computer-readable storage medium can be included in the heterogeneous data processing device; it can also exist separately without being assembled into the heterogeneous data processing device.

[0090] The above computer-readable storage medium carries one or more programs, which, when executed by a heterogeneous data processing device, cause the heterogeneous data processing device to: obtain the heterogeneous data to be processed and extract the field identifiers in the heterogeneous data; based on a pre-trained semantic recognition model, semantically match the field identifiers with the target fields in a preset database to generate a field-level mapping relationship; generate step-by-step processing logic adapted to the heterogeneous data format according to the field-level mapping relationship; perform a verification operation on the step-by-step processing logic in a code sandbox, and save the output target field data to a structure database after the verification passes.

[0091] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through an Internet service provider using the Internet).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0093] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0094] The readable storage medium provided by the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned heterogeneous data processing method, which can solve the technical problem of low accuracy in data processing and warehousing in the traditional solution. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the heterogeneous data processing method provided by the above embodiments, and will not be elaborated here.

[0095] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned heterogeneous data processing method are implemented.

[0096] The computer program product provided by the present application can solve the technical problem of low accuracy in data processing and warehousing in the traditional solution. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the heterogeneous data processing method provided by the above embodiments, and will not be elaborated here.

[0097] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the description and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent scope of the present application.

Claims

1. A method for processing heterogeneous data, characterized in that The method for processing heterogeneous data includes the following steps: Obtain the heterogeneous data to be processed, and extract the field identifiers in the heterogeneous data; Based on a pre-trained semantic recognition model, semantically match the field identifiers with the target fields in a preset database to generate a field-level mapping relationship; According to the field-level mapping relationship, generate step-by-step processing logic adapted to the heterogeneous data format; Execute a verification operation on the step-by-step processing logic in a code sandbox, and save the output target field data to a structure database after the verification passes.

2. The method according to claim 1, wherein The step of, based on a pre-trained semantic recognition model, semantically matching the field identifiers with the target fields in a preset database to generate a field-level mapping relationship includes: Identify the semantic descriptions and associated fields corresponding to the target fields from the preset database; According to the semantic descriptions and associated fields, determine the similarity between the field identifiers and the target fields, and generate a semantic similarity list; According to the semantic similarity list, determine the field-level mapping relationship between the field identifiers and the target fields from a mapping rule set.

3. The method according to claim 1, characterized in that The step of, according to the field-level mapping relationship, generating step-by-step processing logic adapted to the heterogeneous data format includes: Determine the heterogeneous data format of the heterogeneous data; Determine a matching processing logic template according to the field-level mapping relationship, and select a matching target processing logic template according to the heterogeneous data format; According to the target processing logic template, generate step-by-step processing logic for the heterogeneous data.

4. The method according to claim 1, characterized in that, The step of, in a code sandbox, executing a verification operation on the step-by-step processing logic and saving the target field data to a structure database after the verification passes includes: Load the code sandbox, and allocate corresponding target containers in the code sandbox to each processing node in the step-by-step processing logic; Execute the operations corresponding to the processing nodes in the target containers, and extract the target field data output by the target containers; Execute the verification operation on the target field data; After the verification passes, save the target field data to the structure database.

5. The method according to claim 4, characterized in that, The step of, executing the verification operation on the target field data includes: Extract time fields, numerical fields, and text fields from the target field data; Verify the format and time range of the time fields according to data type rules; Perform unit conversion on the numerical fields in a unified manner, and replace the fields in the text fields that match the standardized terms with each other.

6. The method according to claim 4, wherein After the step of, executing the verification operation on the target field data, it further includes: When the operation corresponding to any of the processing nodes fails, mark the error code and the original data context corresponding to the operation and generate a verification report; Parse the error codes in the verification report to determine the defect type of the step-by-step processing logic: Generate an optimization instruction set according to the verification report and the defect type, and reconstruct the step-by-step processing logic based on the optimization instruction set to generate target processing logic; Execute the reconstructed target processing logic in the code sandbox and verify it.

7. The method according to claim 1, wherein Before the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after passing the verification, the following steps are further included: Generate lineage metadata for the heterogeneous data based on the field-level mapping relationship; Associate and store the lineage metadata with the extracted target field data, where the lineage metadata includes source files, field mapping versions, and parsing logic fingerprints; When new target heterogeneous data is added, obtain the parsing logic fingerprints in the historical lineage metadata; Compare the target logic fingerprints of the target heterogeneous data. If the fingerprint matching degree exceeds the threshold, call the associated step-by-step processing logic to process the target heterogeneous data.

8. The method according to claim 1, characterized in that, After the step of performing a verification operation on the step-by-step processing logic in the code sandbox and saving the output target field data to the structure database after passing the verification, the following steps are further included: Obtain the user's query requirements and identify and parse them through a natural language model to determine the corresponding heterogeneous query statement; Perform database retrieval based on the heterogeneous query statement, obtain the required data, and perform report normalization processing, where the normalization processing includes data aggregation, calculation, and sorting; Process the normalized required data based on a preset report template to generate a data view, and visually output the data view through a data visualization tool.

9. A processing device for heterogeneous data, characterized in that, The processing device for the heterogeneous data includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is configured to implement the steps of the method for processing heterogeneous data according to any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the steps of the method for processing heterogeneous data according to any one of claims 1 to 8.

Images