Data fusion method, electronic device, and program product

Abstract

The application relates to the field of data processing, in particular to a data fusion method, an electronic device and a program product. The method comprises the following steps: obtaining a plurality of to-be-fused structured data files; performing analysis on the structured data files to obtain a structured data set, wherein the structured data set comprises a plurality of field identifiers and data content associated with the field identifiers; performing associated matching on the field identifiers according to the data content associated with the field identifiers to obtain associated field identifiers and non-associated field identifiers; establishing an associated mapping according to the associated field identifiers and establishing a newly-added mapping according to the non-associated field identifiers; performing fusion processing on the structured data set by using the associated mapping and the newly-added mapping respectively to obtain a fused data set; and performing structured processing on the fused data set to obtain a target structured data file. The method is used for realizing automatic fusion of multi-source heterogeneous data files and improving data integration efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data fusion method, electronic device, and program product. Background Technology

[0002] The research and testing of intelligent driving assistance systems involve a large number of structured data files, such as diagnostic fault code definition tables, fault diagnosis strategy tables, signal mapping relationship tables, and controller parameter configuration tables. These files are maintained independently by different functional departments, and their field naming rules, column order, and format vary significantly. As the system's functions iterate, the structure and content of these files change frequently, but it is still often necessary to integrate files from different data sources into a single file.

[0003] Existing technologies typically address the need to merge data between two spreadsheets by manually pre-setting specific column numbers for the merge columns and matching rules. Based on the set column numbers, they read key field values ​​from both spreadsheets and compare them. If the comparison results match, a successful match is considered achieved. The data from the pre-defined columns to be copied is then written to the corresponding positions in the target spreadsheet according to the matching results. Before merging the data, this approach requires the user to specify the column numbers for the comparison and copy columns, as well as the type of merge rules. Once configured, this approach is only suitable for merging spreadsheets with a fixed column structure.

[0004] Existing fusion methods cannot automatically identify and match fields based on the semantic information of the fields themselves and the characteristics of the data content they carry. They are difficult to adapt to scenarios involving multiple files and differences in field identifiers between different files, as well as changes in the order of field arrangement. Data fusion work relies heavily on manual configuration of mapping relationships, resulting in low overall processing efficiency. Summary of the Invention

[0005] This application provides a data fusion method, electronic device, and program product to improve data integration efficiency.

[0006] In a first aspect, embodiments of this application provide a data fusion method, including:

[0007] Obtain multiple structured data files to be merged;

[0008] The structured data file to be merged is parsed to obtain a structured dataset, which contains multiple field identifiers and data content associated with the field identifiers;

[0009] Based on the data content associated with the field identifier, the field identifier is matched to obtain associated field identifiers and non-associated field identifiers;

[0010] The structured datasets corresponding to the associated field identifiers are mapped, and the structured datasets corresponding to the non-associated field identifiers are mapped.

[0011] The structured datasets of the association mapping and the newly added mapping are fused using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain the fused dataset.

[0012] The fused dataset is then processed to obtain the target structured data file.

[0013] In one possible implementation, the structured datasets of the association mapping and the newly added mapping are fused using a first fusion method corresponding to the association mapping and a second fusion method corresponding to the newly added mapping, respectively, to obtain a fused dataset, including:

[0014] The first fusion method is used to merge the data content corresponding to the associated fields into the fused dataset;

[0015] The second fusion method is used to add the data content corresponding to the non-related fields to the fused dataset.

[0016] In one possible implementation, a first fusion method is used to merge the data content corresponding to the associated fields into the fused dataset, including:

[0017] Using the first fusion method, the similarity of the data content corresponding to the associated fields is calculated to obtain a similarity value;

[0018] Data with similarity values ​​greater than a preset threshold are merged into the fused dataset.

[0019] In one possible implementation, the parsing of the structured data file to be merged includes:

[0020] Identify the field identifier region and data region in each of the structured data files to be merged, and extract the field identifier from the field identifier region;

[0021] Associate the field identifier with the data content of the data area.

[0022] In one possible implementation, it also includes:

[0023] In the fused dataset, the first matching status information and file source identifier are set based on the data content corresponding to the associated fields, and the second matching status information and file source identifier are set based on the data content corresponding to the non-associated fields, so as to obtain the updated fused dataset.

[0024] In one possible implementation, the field identifier is matched for association based on the data content associated with it to obtain associated and non-associated fields, including:

[0025] Perform consistency verification on the data content associated with the field identifier, and determine the field identifier corresponding to the data content that passes the verification as the associated field;

[0026] Fields in the field identifier that are not identified as associated fields are identified as non-associated fields.

[0027] In one possible implementation, consistency verification of the data content associated with the field identifier includes:

[0028] Based on the format characteristics and numerical range characteristics of the data content corresponding to the field identifier, the consistency of the data content associated with the field identifier is verified.

[0029] In one possible implementation, it also includes:

[0030] Perform semantic similarity analysis on the identifiers of each field;

[0031] Fields with semantic similarity greater than a preset similarity value are identified as similar fields in order to verify the consistency of the data content associated with the similar fields.

[0032] Secondly, embodiments of this application provide a data fusion apparatus, comprising:

[0033] The acquisition module is used to acquire multiple structured data files to be merged;

[0034] The parsing module is used to parse the structured data file to be merged to obtain a structured dataset, which contains multiple field identifiers and data content associated with the field identifiers;

[0035] The association module is used to perform association matching on the field identifiers based on the data content associated with the field identifiers, and obtain associated field identifiers and non-associated field identifiers;

[0036] The association module is also used to perform association mapping on the structured datasets corresponding to the association field identifiers, and to perform new mapping on the structured datasets corresponding to the non-association field identifiers;

[0037] The fusion module is used to fuse the structured datasets of the association mapping and the structured datasets of the newly added mapping using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain a fused dataset.

[0038] The processing module is used to perform structured processing on the fused dataset to obtain a target structured data file.

[0039] In one possible implementation, the fusion module is specifically used for:

[0040] The first fusion method is used to merge the data content corresponding to the associated fields into the fused dataset;

[0041] The second fusion method is used to add the data content corresponding to the non-related fields to the fused dataset.

[0042] In one possible implementation, the fusion module is further configured to:

[0043] Using the first fusion method, the similarity of the data content corresponding to the associated fields is calculated to obtain a similarity value;

[0044] Data with similarity values ​​greater than a preset threshold are merged into the fused dataset.

[0045] In one possible implementation, the parsing module is specifically used for:

[0046] Identify the field identifier region and data region in each of the structured data files to be merged, and extract the field identifier from the field identifier region;

[0047] Associate the field identifier with the data content of the data area.

[0048] In one possible implementation, the processing module is further configured to:

[0049] In the fused dataset, the first matching status information and file source identifier are set based on the data content corresponding to the associated fields, and the second matching status information and file source identifier are set based on the data content corresponding to the non-associated fields, so as to obtain the updated fused dataset.

[0050] In one possible implementation, the association module is specifically used for:

[0051] Perform consistency verification on the data content associated with the field identifier, and determine the field identifier corresponding to the data content that passes the verification as the associated field;

[0052] Fields in the field identifier that are not identified as associated fields are identified as non-associated fields.

[0053] In one possible implementation, the association module is specifically used for:

[0054] Based on the format characteristics and numerical range characteristics of the data content corresponding to the field identifier, the consistency of the data content associated with the field identifier is verified.

[0055] In one possible implementation, the association module is further configured to:

[0056] Perform semantic similarity analysis on the identifiers of each field;

[0057] Fields with semantic similarity greater than a preset similarity value are identified as similar fields in order to verify the consistency of the data content associated with the similar fields.

[0058] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0059] The memory stores computer-executed instructions;

[0060] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0061] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0062] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0063] The data fusion method, electronic device, and program product provided in this application parse multiple structured data files to be fused into structured datasets containing field identifiers and corresponding data content. The field identifiers are matched to distinguish between associated field identifiers used to establish relationships and newly added field identifiers used to supplement data. Association mappings and new mappings are established respectively, and the structured datasets are fused according to the fusion methods corresponding to the two types of mappings. This avoids the cumbersome operation of manually specifying column numbers and matching rules in advance and reconfiguring them after the table structure changes, which is required in the prior art. As a result, it realizes the automatic fusion of multiple data files from different sources and with different structures, and improves the efficiency of data integration. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 Flowchart of the data fusion method provided in this application Figure 1 ;

[0066] Figure 2 Flowchart of the data fusion method provided in this application Figure 2 ;

[0067] Figure 3 Flowchart of the data fusion method provided in this application Figure 3 ;

[0068] Figure 4 Flowchart of the data fusion method provided in this application Figure 4 ;

[0069] Figure 5 Flowchart of the data fusion method provided in this application Figure 5 ;

[0070] Figure 6 A schematic diagram of the data fusion device provided in this application;

[0071] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0074] In existing technologies, to achieve cross-team data collaboration, the common practice is to first obtain multiple tables to be processed, then identify the corresponding fields using scripts or manual methods, use the corresponding columns as the basis for association, and add the contents of the remaining columns into the target table, finally outputting a summary result file. When the column names, column order, or field structure of the tables change, the configured rules cannot be adjusted adaptively, requiring manual reconfiguration. This makes the data fusion operation cumbersome and unable to adapt to scenarios with frequent changes in data structure.

[0075] However, in practical applications, field identifiers in different files often have naming differences, order changes, and header adjustments, making it difficult for existing scripts to continuously adapt. Manual processing also requires repeated confirmation of field relationships, resulting in low efficiency. At the same time, when integrating multiple files consecutively, there is often a lack of unified processing logic for which fields are used for association and which fields should be added as new content, which can easily lead to omissions, mismatches, and inconsistent results.

[0076] The data fusion method, electronic device, and program product provided in this application parse structured data files into structured datasets containing field identifiers and corresponding data content. The field identifiers are matched to distinguish between associated field identifiers used to establish row-level correspondences and non-associated field identifiers used to supplement data content. Association mappings are established based on associated field identifiers, and new mappings are established based on non-associated field identifiers. Merging operations based on similarity judgment are performed on the data content corresponding to associated fields according to the association mappings, and new operations are performed on the data content corresponding to non-associated fields according to the new mappings. This achieves automated fusion of multi-source heterogeneous structured data files, solving the technical problems of poor adaptability and cumbersome operation caused by manual configuration in existing technologies.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0078] Figure 1 A flowchart illustrating the data fusion method provided in this application is shown below. Figure 1 As shown, the method includes:

[0079] S101. Obtain multiple structured data files to be merged.

[0080] Specifically, a structured data file refers to a collection of data organized in a structured format. This collection contains clearly defined field identifiers and the corresponding data content. Specific forms of structured data files include, but are not limited to, spreadsheet files, database export files, and structured text files. Multiple structured data files are obtained through data import or file reading interfaces and used as input data sources for subsequent fusion processing.

[0081] Multiple structured data files to be merged come from different data producers or different stages of data production, each carrying data information in different dimensions. By acquiring these multiple files, a data foundation is provided for subsequent cross-file data fusion.

[0082] In practice, the executing entity first receives the fusion task instruction and establishes a corresponding task context. The task context records at least the task identifier, file source, import time, and output directory. Multiple structured data files to be fused can be imported by users uploading them through a graphical interface, automatically scanned by reading preset folder paths, or pulled from the project document management system through a batch import interface.

[0083] Taking a spreadsheet file as an example, when merging data from multiple worksheets, the target worksheet to be merged can be selected according to worksheet name matching rules, the completeness of the first row fields, or pre-configured tab identifiers. After the file is obtained, the executing entity performs a preliminary confirmation of the file's validity. This preliminary confirmation includes at least whether the file format is parsable, whether the file is corrupted, whether the target worksheet exists, whether the header area is identifiable, and whether it contains the minimum number of record rows. For empty files or files containing only descriptive text without table data, the executing entity marks them as invalid input and removes them from this merging task. For files with merged cells, hidden columns, blank leading rows, or other elements that affect parsing, the executing entity records the corresponding status in the task context and passes it to the subsequent parsing stage for normalization processing. Based on the above processing, multiple structured data files from different maintenance entities, different version cycles, and different naming methods are uniformly included in the same task scope, forming the input file set required for subsequent parsing, and triggering the next step to parse the input file set one by one.

[0084] S102. Parse the structured data file to be merged to obtain a structured dataset. The structured dataset contains multiple field identifiers and data content associated with the field identifiers.

[0085] Specifically, parsing refers to identifying and extracting the format and content of a structured data file. This includes identifying the area in the structured data file that carries field identifiers, which contains the column names of each data column or the row names of each data row, and identifying the data content area corresponding to that field identifier, which contains the specific data value. After parsing, the field identifiers are associated with the data content to generate a structured dataset.

[0086] There is a correspondence between field identifiers and data content in structured data files. By parsing and processing, the above correspondence is extracted and converted into a structured dataset containing field identifiers and corresponding data content, thereby realizing the transformation of the structure and improving data processability.

[0087] S103. Based on the data content associated with the field identifier, perform association matching on the field identifier to obtain associated field identifiers and non-associated field identifiers.

[0088] Specifically, association matching refers to the consistency verification of the data content corresponding to field identifiers in different structured data files. This includes extracting the format features and numerical range features of the data content corresponding to each field identifier, comparing these features across files, identifying field identifiers with consistent format and numerical range features as associated field identifiers, and identifying field identifiers that are not identified as associated field identifiers as non-associated field identifiers.

[0089] Field identifiers with the same or similar semantics typically exhibit similarities in format and numerical range characteristics. By comparing these characteristics across files, field identifiers expressing the same data meaning in different files can be identified, thus distinguishing between related field identifiers used to establish cross-file line-level correspondences and non-related field identifiers used to supplement data content. By establishing hierarchical field correspondences between different structured data files, the automatic identification of hierarchical field relationships can be achieved, avoiding the need for manual specification of field correspondences.

[0090] S104. Map the structured datasets corresponding to the associated field identifiers, and add new mappings to the structured datasets corresponding to the non-associated field identifiers.

[0091] Specifically, association mapping refers to a mapping structure that records the correspondence between associated field identifiers across structured data files. This mapping structure contains the correspondence information between associated field identifiers in the source structured data file and their corresponding associated field identifiers in the target structured data file. Addition mapping refers to a mapping structure that records the source relationship between non-associated field identifiers in different structured data files. This mapping structure contains the source information of non-associated field identifiers and their data content in each structured data file, used to instruct the corresponding data content to be added to the fused dataset during subsequent fusion processes.

[0092] Association mapping and addition mapping correspond to two different types of data fusion operations. Association mapping is used to determine the correspondence between data rows in different files, while addition mapping is used to determine how to supplement data content in different files. The two types of mapping are independent of each other, which improves the transparency and controllability of the fusion process.

[0093] In practice, one of the structured data files can be selected as the baseline dataset. The baseline dataset can be specified by the user or automatically determined based on the highest record completeness, the largest number of fields, or the project's main table identity. For each non-baseline dataset, the associated field pairs with the baseline dataset are extracted, and an association mapping table is constructed. The association mapping table must at least include the source file identifier, target file identifier, source field name, target field name, source column position, target column position, and field matching identifier. If multiple associated fields exist between two files, these fields are combined into a set of association keys. For non-associated fields, the execution entity creates a new mapping table. This new mapping table must at least include the source file of the new field, the original field name, the standardized new field name, the target column position, the data type, and the null value filling rule. When fields with the same name but no association exist in different source files, the execution entity appends the source identifier to the new mapping, forming a target field name with a source prefix to distinguish new content from different sources.

[0094] In one possible embodiment, the executing entity also performs a consistency check on the mapping relationship. The consistency check includes whether the associated fields form a circular conflict, whether the same source field is repeatedly mapped to multiple target fields, and whether the newly added field name causes a naming conflict in the target result. If a conflict exists, the executing entity renames the target field according to preset rules or reverts to the single-file add-and-hold mode, and records the processing result in the task log.

[0095] S105. The structured datasets of the association mapping and the newly added mapping are fused using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain the fused dataset.

[0096] Specifically, the fused dataset refers to the comprehensive data result after field alignment, record merging, and the addition of new fields. The target structured data file refers to the result file written based on the fused dataset, which can be used for subsequent querying, verification, archiving, or reprocessing. Similarity calculations are performed on the data content corresponding to the associated field identifiers according to the association mapping. Data content with a similarity value greater than a preset threshold is considered a successful match and a merge operation is performed. An addition operation is performed on the data content corresponding to the non-associated field identifiers according to the addition mapping. The merge operation involves writing the data content corresponding to the associated field identifiers in the successfully matched data rows to the target position in the fused dataset. The addition operation involves writing the data content corresponding to the non-associated field identifiers to the target position in the fused dataset; this process does not involve similarity determination.

[0097] The data content corresponding to the association mapping establishes the correspondence between data rows through similarity calculation and threshold comparison. This process ensures that row-level association is established only when the similarity between data content meets the preset standard. The data content corresponding to the newly added mapping does not participate in row-level matching and is directly written into the fusion dataset as supplementary data. The two types of mapping correspond to different data fusion operation logics, realize the automatic merging and supplementation of multi-source data content, and ensure the data integrity of the fusion result.

[0098] In one possible embodiment, during the fusion process, if conflicting data exists in related fields, data can be selected based on a preset priority. For example, when fusion data from tables, if the diagnostic logic field value of the first table is voltage anomaly, while the diagnostic logic field value of the second table is signal drift, and the first table has a higher priority than the second table, then the voltage anomaly value from the first table will be used as the final fusion result. The priority rules can be user-defined or dynamically adjusted by training a model using historical data to ensure that the processing of conflicting data meets actual business needs.

[0099] S106. Perform structuring processing on the fused dataset to obtain the target structured data file.

[0100] Specifically, structured processing refers to organizing and outputting the fused dataset according to the format specifications of the target file, so that the data content in the fused dataset is presented in a structured form in the target structured data file. The fused dataset exists in the form of an internal data structure of the program and needs to be converted into a user-readable and subsequently usable structured file format through output processing. By converting the fusion result into the target structured data file format, the structured output of the fusion result is achieved, facilitating subsequent data application and analysis.

[0101] In practice, the executing entity first loads the fields and records of the baseline dataset to establish an initial fused dataset. Then, the executing entity aligns the records in the remaining structured datasets according to the association mappings and writes the corresponding field contents into the fused dataset based on the newly added mappings. For records or fields where no corresponding relationship is established, the executing entity can retain, add, or handle null values ​​according to the task configuration to form a unified fusion result. After all structured datasets are processed, the executing entity obtains the fused dataset and writes it out as a target structured data file; the writing format can be consistent with the input file.

[0102] The data fusion method provided in this application parses multiple structured data files into structured datasets containing field identifiers and corresponding data content. It performs association matching on the field identifiers to distinguish between related and non-related field identifiers and establishes association mappings and new mappings respectively. Based on the association mappings and new mappings, it performs similarity-based merging and adding operations respectively, thereby realizing the automated fusion of multi-source heterogeneous data files, avoiding manual configuration of mapping rules, and improving data fusion efficiency and adaptability.

[0103] Figure 2 Flowchart of the data fusion method provided in this application Figure 2 ,like Figure 2 As shown, this embodiment, based on the above embodiments, provides a detailed description of the data fusion method, which includes:

[0104] S201. Using the first fusion method, the data content corresponding to the associated fields is merged into the fused dataset.

[0105] Specifically, the first fusion method refers to a data merging operation based on association mapping. This operation uses the field correspondence recorded in the association mapping as a basis to integrate the data content corresponding to the associated field identifiers in different structured data files according to the row-level correspondence. During execution, according to the correspondence between the source and target fields in the association mapping, the data content corresponding to the associated field identifiers in the source structured data file is written to the corresponding column position of the corresponding row in the target structured data file.

[0106] The association mapping establishes a correspondence between the associated field identifiers of different structured data files. This correspondence indicates which data rows have a correspondence. The first fusion method uses this correspondence as the basis for data merging to achieve the integration of cross-file associated data. By merging the data content with a relationship in different files according to the field mapping relationship, the integration of cross-file associated data is achieved.

[0107] S202. Using the second fusion method, the data content corresponding to the non-related fields is added to the fusion dataset.

[0108] Specifically, the second fusion method refers to a data addition operation performed based on the new mapping. This operation uses the field source relationship recorded in the new mapping as a basis to write the data content corresponding to the non-related field identifiers in the structured data file as new data into the fused dataset. According to the correspondence between the source fields in the new mapping and the target positions in the fused dataset, the data content corresponding to the non-related field identifiers is written to the corresponding addition position in the fused dataset.

[0109] Non-related fields identify data content that is not associated with any other field; their purpose is to supplement additional information. The second fusion method adds this type of data directly to the fused dataset as an independent data item, without requiring matching or judgment. By adding the data content corresponding to non-related fields to the fused dataset, the data dimensions of the fusion result are supplemented, achieving automatic supplementation of non-related data and ensuring the data completeness of the fused dataset.

[0110] In some alternative implementations, S201 above includes:

[0111] S2011. Using the first fusion method, the similarity of the data content corresponding to the associated fields is calculated to obtain the similarity value.

[0112] Specifically, similarity calculation refers to the process of quantitatively evaluating the degree of consistency between two sets of data. During execution, the system extracts the data content corresponding to the associated field identifiers in the source structured data file and the data content corresponding to the associated field identifiers in the target structured data file. The two sets of data content are compared, and a numerical value is calculated to characterize the degree of similarity between the two sets of data content.

[0113] The content of fields with the same meaning in different structured data files may not be completely consistent due to differences in data entry specifications, format standards, etc. Similarity calculation can quantify these differences, providing a numerical basis for subsequent determination of whether to establish row-level associations; the similarity between data content is quantified into similarity values ​​for subsequent threshold determination, realizing the quantitative expression of the similarity between data content.

[0114] S2012. Merge data with similarity values ​​greater than a preset threshold into the fusion dataset.

[0115] Specifically, the preset threshold is a pre-defined similarity threshold used to determine whether two sets of data content have sufficient similarity to establish a row-level association. The calculated similarity value is compared with the preset threshold. When the similarity value is greater than the preset threshold, it is determined that the two sets of data content correspond, and the data content corresponding to the association field of that row in the source structured data file is written to the target location of the fused dataset.

[0116] A preset threshold is used to distinguish between successful and unsuccessful matches. When the similarity value is greater than the threshold, it indicates that the differences between the two sets of data are within an acceptable range, and a row-level correspondence can be established. By comparing the threshold, the data content that should be merged into the fused dataset is determined, realizing automatic determination of row-level association and reducing matching errors caused by differences in data content.

[0117] In some optional implementations, associated fields with similarity values ​​less than a preset threshold are changed to non-associated fields, and the corresponding data content is added to the fused dataset using a second fusion method.

[0118] Specifically, when the similarity value calculated based on the similarity of the data content corresponding to an associated field is less than or equal to a preset threshold, it indicates that the field cannot establish sufficient correspondence between different structured data files. Therefore, the field does not meet the criteria for an associated field and is classified as a non-associated field. The data content corresponding to non-associated fields will no longer participate in the merging operation of the first fusion method, but will instead be written into the fused dataset as new data content using the second fusion method.

[0119] A similarity value less than or equal to a preset threshold indicates that the correspondence between data contents does not meet the criteria for row-level association. However, the data content itself is still valid and should be retained in the fusion result. By reclassifying the failed matching fields as non-matching fields, the fusion operation is switched from merging to adding, ensuring complete retention of the data content. Alternative processing paths are provided for failed matching data content to ensure that no data is missing from the fused dataset due to matching failures, guaranteeing the data integrity of the fused dataset and avoiding data omissions caused by matching threshold settings.

[0120] This invention calculates similarity values ​​by performing similarity calculations on the data content corresponding to associated fields, merges data content with similarity values ​​greater than a preset threshold into the fusion dataset, and converts associated fields with similarity values ​​less than or equal to the preset threshold into non-associated fields before performing a new addition operation. This achieves automatic switching between merging and adding, avoids data loss due to matching failures, and ensures the integrity and accuracy of the fusion results.

[0121] Figure 3 Flowchart of the data fusion method provided in this application Figure 3 ,like Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed description of the data fusion method, which includes:

[0122] S301. Identify the field identifier region and data region in each structured data file to be merged, and extract the field identifier from the field identifier region.

[0123] Specifically, the field identifier area refers to the location area in the structured data file used to carry field identifiers, which includes the column names of each data column or the row names of each data row; the data area refers to the location area in the structured data file used to carry the specific data content, which includes the data values ​​corresponding to the field identifiers.

[0124] The format structure of structured data files is analyzed to locate the start and end positions of the field identifier area and the data area, and all field identifiers are extracted from the field identifier area. Structured data files store data in an organization that separates field identifiers from data content. The field identifier area stores field identifiers centrally, while the data area stores the data content centrally. By analyzing the file format structure, the positional boundaries of these areas can be identified, enabling automatic extraction of field identifiers.

[0125] S302. Associate the field identifier with the data content of the data area.

[0126] Specifically, association refers to establishing a mapping relationship between field identifiers and their corresponding data content. During execution, based on the arrangement order of field identifiers in the field identifier area and the arrangement order of data content in the data area, field identifiers and data content are paired according to their positional correspondence, generating a structured dataset containing field identifiers and their corresponding data content.

[0127] In structured data files, the order of field identifiers corresponds to the order of data content within the data area. This positional correspondence establishes a relationship between field identifiers and data content. By pairing field identifiers with their corresponding data content, a structured data organization is formed, enabling automatic association between field identifiers and data content, and generating the structured dataset required for subsequent processing.

[0128] This invention enables automatic identification and extraction of structured data by identifying field identifier regions and data regions in structured data files, extracting field identifiers from the field identifier regions, and then associating the field identifiers with the data content of the data regions. This avoids the need for manual specification of data regions and improves data parsing efficiency.

[0129] This embodiment, based on the above embodiments, provides a detailed description of the data fusion method, which further includes:

[0130] In the fused dataset, the first matching status information and file source identifier are set based on the data content corresponding to the associated fields, and the second matching status information and file source identifier are set based on the data content corresponding to the non-associated fields, so as to obtain the updated fused dataset.

[0131] Specifically, matching status information refers to information used to indicate the matching result of data rows during the fusion process, including a successful or failed match identifier and the corresponding similarity value when a match is successful; file source identifier refers to information used to indicate the original source of the data content, including the identifier of the source file, the location of the source row, and the identifier of the source field. The first matching status information corresponds to the data content corresponding to the associated field, and the second matching status information corresponds to the data content corresponding to the non-associated field.

[0132] By adding an additional column to the fused dataset to store matching status information and file source identifiers, and writing the matching status information and file source identifiers into the corresponding additional column positions, an updated fused dataset is generated.

[0133] By adding an additional column to the fused dataset, the matching status and source information of the data content are stored in the same data row as the data content itself. This ensures that the source information of the data content corresponds to the data content, provides a basis for tracing the source of the data content in the fused dataset and explains the matching results, thereby achieving traceability of the fused data and improving the verifiability of the data fusion results.

[0134] This invention improves the credibility of fusion results by setting matching status information and file source identifiers in the fused dataset, thereby enabling traceability of the fused data source and verifiability of the matching results.

[0135] Figure 4 Flowchart of the data fusion method provided in this application Figure 4 ,like Figure 4 As shown, this embodiment, based on the above embodiments, provides a detailed description of the data fusion method, which includes:

[0136] S401. Perform consistency verification on the data content associated with the field identifier, and determine the field identifier corresponding to the data content that passes the verification as the associated field.

[0137] Specifically, consistency verification refers to the process of checking whether the data content corresponding to a field identifier is consistent in terms of format and numerical range. This involves extracting the data content corresponding to the field identifier, extracting and analyzing its format and numerical range characteristics, and comparing the format and numerical range characteristics of the data content corresponding to the same field identifier in different structured data files. If both are consistent in both format and numerical range characteristics, the consistency verification is considered successful, and the field identifier is identified as the associated field.

[0138] Fields with the same meaning in different structured data files should have consistent format and numerical range characteristics. By comparing these characteristics, the correspondence between field identifiers can be verified. Through feature comparison at the data content level, the correspondence between field identifiers is verified, enabling automatic identification and verification of associated fields and improving the accuracy of association matching.

[0139] S402. Identify fields in the field identifier that are not identified as related fields as non-related fields.

[0140] Specifically, a set of all field identifiers is obtained. Field identifiers identified as related fields are removed from this set, and the remaining field identifiers are identified as non-related fields. Field identifiers are divided into two mutually exclusive categories: related fields and non-related fields. Fields not identified as related fields are automatically classified into the non-related field category. Clearly distinguishing between these two different types of field identifiers provides a classification basis for subsequently establishing related mappings and adding new mappings, achieving complete classification of field identifiers and ensuring that all fields are categorized and processed.

[0141] In one specific implementation, consistency verification can be accomplished by combining rule comparison and content validation. The system first establishes candidate matching pairs based on field identifiers, and then performs consistency judgment on the field content in the candidate matching pairs. If the field names are different but the field content shows a stable correspondence in multiple sample records, the verification is deemed successful; if the field content is missing a lot, has large format differences, or cannot form a stable correspondence, the verification is deemed unsuccessful. Through the above method, the association matching results of field identifiers can be stably distinguished into associated fields and non-associated fields, providing input for subsequent association mapping and new mapping establishment.

[0142] In some alternative implementations, S401 above includes:

[0143] Based on the format and numerical range characteristics of the data content corresponding to the field identifier, the consistency of the data content associated with the field identifier is verified.

[0144] Specifically, the data content corresponding to the field identifier is extracted, and the format characteristics of the data content are identified. These format characteristics include the encoding form, data type, and format structure of the data content. The numerical range characteristics of the data content are also identified, including the minimum value, maximum value, and numerical distribution range. The format characteristics and numerical range characteristics of the data content corresponding to the same field identifier in different structured data files are compared. When both the format characteristics and numerical range characteristics are consistent, the consistency verification passes.

[0145] The format and range characteristics of data content are quantitative descriptions of the inherent attributes of the data content. These characteristics do not change due to different file sources. By comparing these characteristics across files, the semantic consistency between field identifiers can be verified. Through feature comparison in two dimensions—format and range—quantitative execution of consistency verification is achieved, improving the accuracy and operability of consistency verification.

[0146] Figure 5 Flowchart of the data fusion method provided in this application Figure 5 ,like Figure 5 As shown, in some optional embodiments, before S401 above, the following step is also included:

[0147] S4001. Perform semantic similarity analysis on the identifiers of each field.

[0148] Specifically, semantic similarity analysis refers to the process of analyzing and calculating the degree of similarity between field identifiers at the lexical and semantic levels. During execution, the field identifiers in each structured data file undergo word segmentation and semantic vectorization, and the semantic similarity between field identifiers in different structured data files is calculated to obtain semantic similarity values.

[0149] Field identifiers with the same meaning may be expressed differently in different structured data files. Semantic similarity analysis can identify field identifiers with different expressions but the same meaning at the lexical level. This narrows down the candidate range for subsequent consistency verification, reduces the number of field identifiers that need to be verified, and improves the overall efficiency of association matching.

[0150] S4002. Fields with semantic similarity greater than a preset similarity value are identified as similar fields in order to verify the consistency of the data content associated with similar fields.

[0151] Specifically, the preset similarity value is a pre-defined semantic similarity threshold used to determine whether two field identifiers are similar fields. The calculated semantic similarity value is compared with the preset similarity value. When the semantic similarity value is greater than the preset similarity value, the group of field identifiers is determined to be similar fields, and then consistency verification is performed on the data content associated with the similar fields.

[0152] By using semantic similarity analysis, semantically similar field identifiers are selected from all field identifiers as a candidate matching set. Then, consistency verification is performed on this candidate set, avoiding consistency verification for all field identifiers and reducing processing resource consumption. By using semantic filtering, the processing scope of consistency verification is narrowed, improving processing efficiency. Under the premise of ensuring matching accuracy, the overall processing efficiency of association matching is improved.

[0153] In practical implementation, each field identifier can first be segmented, synonyms merged, and vectorized to map the field names to semantic vectors. Then, the semantic similarity between each field identifier is calculated based on cosine similarity, word embedding distance, or vector similarity output by a pre-trained language model, and the calculation results are compared with preset similarity values. For field identifiers exceeding a threshold, they can be marked as similar fields and sent to subsequent feature extraction and consistency verification stages. This allows for a comparison of the field's format features and value range features to determine whether they can participate in the fusion process as the same associated field. If the table header corresponding to a field identifier contains mixed Chinese and English characters, abbreviations, synonym substitutions, or order adjustments, semantically similar fields can still be identified through semantic similarity analysis, and their correspondence can be further confirmed in the consistency verification.

[0154] This invention provides a phased approach to association matching by performing semantic similarity analysis on each field identifier, identifying fields with semantic similarity greater than a preset similarity value as similar fields, and then verifying the consistency of the data content associated with the similar fields. This approach improves overall processing efficiency while ensuring matching accuracy.

[0155] The following is a specific example: In the requirements analysis phase of an intelligent driving assistance system, the diagnostic fault code requirement table, diagnostic survey form, and fault level definition table provided by the diagnostic function group need to be integrated to verify the diagnostic logic integrity of each diagnostic fault code requirement and the correctness of the fault level configuration.

[0156] The system retrieves a master table of diagnostic fault codes as the first structured data file, a diagnostic survey table as the second structured data file, and a fault level definition table as the third structured data file. These three tables are then parsed to identify the field identifiers and data areas. For the master table, the identified fields are fault code number, fault name, and fault level. For the survey table, the identified fields are fault code number, diagnostic logic, and judgment conditions. For the fault level definition table, the identified fields are fault level, level description, and response strategy. After parsing, structured datasets for each of the three tables are generated.

[0157] The field identifiers of the three tables were identified. Semantic analysis was performed on the fault code number field in the diagnostic fault code requirement summary table and the fault code number field in the diagnostic survey table; they were semantically consistent and thus identified as related fields. Semantic analysis was also performed on the fault level field in the diagnostic fault code requirement summary table and the fault level definition table; they were semantically consistent and thus identified as another set of related fields. The diagnostic logic and judgment conditions fields in the diagnostic survey table were identified as non-related fields with a relationship to the diagnostic fault code requirement summary table. The level description and response strategy fields in the fault level definition table were identified as non-related fields with a relationship to the diagnostic fault code requirement summary table.

[0158] Based on the associated fields, establish relationship mappings and add-to-add mappings between the tables. According to the relationship mappings, calculate the similarity between each value in the fault code number column of the diagnostic fault code requirement master table and each value in the fault code number column of the diagnostic survey table. Rows with a similarity greater than a preset threshold are considered a successful match. Perform an exact match between each value in the fault level column of the diagnostic fault code requirement master table and each value in the fault level definition table. Rows with perfect matches are considered a successful match. According to the add-to-add mappings, copy the data content of the diagnostic logic and judgment condition columns of the successfully matched rows in the diagnostic survey table to the new position of the corresponding fault code number data row in the diagnostic fault code requirement master table. Copy the data content of the level description and response strategy columns of the successfully matched rows in the fault level definition table to the new position of the corresponding fault level data row in the diagnostic fault code requirement master table.

[0159] A fused dataset is generated, containing the original data from the diagnostic fault code requirement master table, along with supplementary diagnostic logic, judgment conditions, level descriptions, and response strategies. Based on this fused dataset, a fused table is generated. This table allows requirements analysts to simultaneously verify, in a single data view, whether each diagnostic fault code requirement has complete diagnostic logic support and whether the fault level configuration is consistent with the level definition table. It also identifies fault code numbers with missing diagnostic logic or incorrect level configurations, and assesses the completeness and consistency of the requirement definitions.

[0160] Figure 6 A schematic diagram of the data fusion device provided in this application is shown below. Figure 6 As shown, the data fusion device 60 provided in this embodiment includes:

[0161] The acquisition module 601 is used to acquire multiple structured data files to be merged;

[0162] The parsing module 602 is used to parse the structured data file to be merged to obtain a structured dataset. The structured dataset contains multiple field identifiers and data content associated with the field identifiers.

[0163] The association module 603 is used to perform association matching on the field identifiers based on the data content associated with the field identifiers, and obtain the associated field identifiers and non-associated field identifiers;

[0164] The association module 603 is also used to perform association mapping on the structured datasets corresponding to the association field identifiers, and to perform new mapping on the structured datasets corresponding to the non-association field identifiers;

[0165] The fusion module 604 is used to fuse the structured dataset of the association mapping and the structured dataset of the newly added mapping using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain a fused dataset.

[0166] Processing module 605 is used to perform structured processing on the fused dataset to obtain the target structured data file.

[0167] In one possible implementation, the fusion module 604 is specifically used for:

[0168] The first fusion method is used to merge the data content corresponding to the associated fields into the fused dataset;

[0169] The second fusion method is adopted to add the data content corresponding to the non-related fields to the fused dataset.

[0170] In one possible implementation, the fusion module 604 is further configured to:

[0171] The first fusion method is used to calculate the similarity of the data content corresponding to the associated fields and obtain the similarity value.

[0172] Data with similarity values ​​greater than a preset threshold are merged into the fusion dataset.

[0173] In one possible implementation, the parsing module 602 is specifically used for:

[0174] Identify the field identifier region and data region in each structured data file to be merged, and extract the field identifier from the field identifier region;

[0175] Associate field identifiers with the data content of the data area.

[0176] In one possible implementation, the processing module 605 is further configured to:

[0177] In the fused dataset, the first matching status information and file source identifier are set based on the data content corresponding to the associated fields, and the second matching status information and file source identifier are set based on the data content corresponding to the non-associated fields, so as to obtain the updated fused dataset.

[0178] In one possible implementation, the association module 603 is specifically used for:

[0179] Perform consistency verification on the data content associated with the field identifier, and determine the field identifier corresponding to the data content that passes the verification as the associated field;

[0180] Fields that are not identified as related fields in the field identifier will be identified as non-related fields.

[0181] In one possible implementation, the association module 603 is specifically used for:

[0182] Based on the format and numerical range characteristics of the data content corresponding to the field identifier, the consistency of the data content associated with the field identifier is verified.

[0183] In one possible implementation, the association module 603 is further configured to:

[0184] Perform semantic similarity analysis on the identifiers of each field;

[0185] Fields with semantic similarity greater than a preset similarity value are identified as similar fields in order to verify the consistency of the data content associated with similar fields.

[0186] The data fusion device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0187] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the electronic device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0188] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above method.

[0189] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0190] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0191] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0192] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0193] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0194] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0195] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0196] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0197] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0200] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0202] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A data fusion method, characterized in that, include: Obtain multiple structured data files to be merged; The structured data file to be merged is parsed to obtain a structured dataset, which contains multiple field identifiers and data content associated with the field identifiers; Based on the data content associated with the field identifier, the field identifier is matched to obtain associated field identifiers and non-associated field identifiers; The structured datasets corresponding to the associated field identifiers are mapped, and the structured datasets corresponding to the non-associated field identifiers are mapped. The structured datasets of the association mapping and the newly added mapping are fused using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain the fused dataset. The fused dataset is then processed to obtain the target structured data file.

2. The method according to claim 1, characterized in that, The structured datasets of the association mapping and the newly added mapping are fused using the first fusion method corresponding to the association mapping and the second fusion method corresponding to the newly added mapping, respectively, to obtain the fused dataset, which includes: The first fusion method is used to merge the data content corresponding to the associated fields into the fused dataset; The second fusion method is used to add the data content corresponding to the non-related fields to the fused dataset.

3. The method according to claim 2, characterized in that, The first fusion method merges the data content corresponding to the associated fields into the fused dataset, including: Using the first fusion method, the similarity of the data content corresponding to the associated fields is calculated to obtain a similarity value; Data with similarity values ​​greater than a preset threshold are merged into the fused dataset.

4. The method according to claim 1, characterized in that, The parsing of the structured data file to be merged includes: Identify the field identifier region and data region in each of the structured data files to be merged, and extract the field identifier from the field identifier region; Associate the field identifier with the data content of the data area.

5. The method according to claim 1, characterized in that, Also includes: In the fused dataset, the first matching status information and file source identifier are set based on the data content corresponding to the associated fields, and the second matching status information and file source identifier are set based on the data content corresponding to the non-associated fields, so as to obtain the updated fused dataset.

6. The method according to claim 1, characterized in that, Based on the data content associated with the aforementioned field identifier, the field identifier is matched to obtain associated and non-associated fields, including: Perform consistency verification on the data content associated with the field identifier, and determine the field identifier corresponding to the data content that passes the verification as the associated field; Fields in the field identifier that are not identified as associated fields are identified as non-associated fields.

7. The method according to claim 6, characterized in that, Perform consistency verification on the data content associated with the field identifier, including: Based on the format characteristics and numerical range characteristics of the data content corresponding to the field identifier, the consistency of the data content associated with the field identifier is verified.

8. The method according to claim 6, characterized in that, Also includes: Perform semantic similarity analysis on the identifiers of each field; Fields with semantic similarity greater than a preset similarity value are identified as similar fields in order to verify the consistency of the data content associated with the similar fields.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images