A deep learning-based group business data collection method

Abstract

This invention discloses a deep learning-based method for aggregating group business data, comprising the following steps: acquiring and preprocessing raw business data to obtain data in a unified format; constructing cross-system business links based on business flow relationships and dividing adjacent data pairs; constructing dual-view training samples based on adjacent data pairs; performing relevant representation learning and progressive updating using DCCA to generate a second mapping relationship; performing data merging based on the second mapping relationship to obtain initial aggregated data; constructing a master data standard view and performing closed-loop correction to correct the mapping relationship; updating the aggregated data based on the correction result and writing it into storage. This invention achieves efficient aggregation of multi-source business data and improves data matching accuracy and consistency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for aggregating group business data based on deep learning. Background Technology

[0002] As enterprises continuously improve their IT capabilities, large corporations typically build multiple business systems to support the data processing needs of different business scenarios. Differences in data structure, field definitions, and storage methods between these systems lead to data fragmentation, heterogeneity, and semantic inconsistencies. Current technologies often integrate multi-source data using rule-based configuration or primary key matching, or align data by manually setting field mapping relationships. These methods rely on predefined rules, making them ill-suited for adapting to dynamically changing data relationships in complex business processes. Furthermore, they struggle to accurately establish correspondences between data when dealing with multi-level, cross-system data flows.

[0003] In recent years, with the development of machine learning and deep learning technologies, some solutions have attempted to use deep models to learn representations of multi-source data and achieve data matching through similarity calculation. However, existing methods are mostly based on single-stage correlation calculations, typically modeling only between two data sources, lacking the ability to model the overall relationships between multiple systems, and struggling to handle the transmission of data relationships during multi-level business flows. After completing data matching, existing methods are usually directly used to generate aggregation results, lacking a reverse verification and correction mechanism for the aggregation results. When there are deviations in the initial matching, it is easy to cause the aggregation results to accumulate errors, affecting the overall data quality.

[0004] Therefore, how to provide a deep learning-based method for group business data aggregation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a group business data aggregation method based on deep learning. This invention fully utilizes deep learning and multi-source data association modeling technology to achieve progressive alignment and closed-loop correction aggregation of cross-system data, and has the advantages of high data matching accuracy, strong aggregation consistency and strong model adaptive optimization capability.

[0006] A method for aggregating group business data based on deep learning according to an embodiment of the present invention includes the following steps: Obtain raw business data, perform preprocessing on the raw business data, and obtain preprocessed data; Link construction involves sorting all business data sources by their connections based on business flow relationships, generating cross-system business links, and dividing the preprocessed data into several adjacent data pairs based on these cross-system business links. Sample generation involves constructing dual-view training samples based on each adjacent data pair. Construct DCCA, perform relevant representation learning on dual-view training samples, output the first mapping relationship corresponding to the current adjacent data pair, use the first mapping relationship corresponding to the current adjacent data pair as the candidate mapping input for the next adjacent data pair, perform constraint matching and relevant representation update segment by segment along the cross-system business link to obtain the second mapping relationship; Based on the second mapping relationship, a unified merging process is performed on the preprocessed data to generate initial aggregated data; Closed-loop calibration involves constructing a master data standard view based on the initial aggregated data, recombining the preprocessed data with the master data standard view into calibration samples, performing secondary correlation representation learning on the calibration samples based on DCCA, outputting the calibration mapping relationship, and correcting the second mapping relationship based on the calibration mapping relationship. The results are updated by updating the initial aggregated data based on the correction mapping relationship, generating the target aggregated data, and writing the target aggregated data into the unified data storage.

[0007] Optionally, the preprocessing specifically includes: Obtain the corresponding raw business data from several business systems respectively, and then unify and aggregate the raw business data from different business systems. Perform data cleaning on the original business data, deleting duplicate data, abnormal data, and data records with missing key fields; perform data format unification processing, converting the field data formats in different business systems into a unified format; perform field standardization processing, unifying fields representing the same business meaning in different business systems into preset field identifiers; and perform data structure alignment processing to obtain preprocessed data.

[0008] Optionally, the link construction specifically includes: Based on the data interaction relationships and business processing sequence between several business systems, a system flowchart is constructed to describe the pre- and post-processing relationships between all business systems. Based on the system flowchart, determine the upstream and downstream systems of each business system, and connect all business systems in a directed manner according to the business processing sequence to form a cross-system business link. For situations where a business system corresponds to more than one upstream system or more than one downstream system, branch nodes or convergence nodes are established in the system flowchart to represent the many-to-many connection relationship between business systems; Based on the system flowchart, the preprocessed data is divided into paths, and the preprocessed data is divided into several adjacent data pairs according to the relationship between adjacent business systems in the cross-system business link, and dual-view training samples are constructed.

[0009] Optionally, the construction of DCCA specifically includes: Construct a first neural network and a second neural network, respectively receiving first view data and second view data from dual-view training samples, performing nonlinear mapping on the first view data and second view data to obtain a first representation vector and a second representation vector, and jointly training the first neural network and the second neural network; In the process of learning the relevant representation, a link-progressive computation structure is introduced for adjacent data pairs in the cross-system business link. The first mapping relationship corresponding to the current adjacent data pair is used as the candidate mapping input for the next adjacent data pair. When learning the relevant representation of the next adjacent data pair, the candidate matching range is constrained, and the relevant representation is updated segment by segment along the cross-system business link. After completing the progressive calculation of all adjacent data pairs in the cross-system business link, the output is a unified related representation result covering multiple business data sources; After generating the initial aggregated data, a closed-loop correction structure is constructed by introducing the master data standard view. The preprocessed data and the master data standard view are then reconstructed into a dual-view input, and secondary correlation representation learning is performed to correct the second mapping relationship.

[0010] Optionally, the link-based progressive calculation structure specifically includes: After performing DCCA related representation learning for the current adjacent data pairs, the corresponding candidate matching samples are extracted based on the first mapping relationship. The first and second representation vectors in the candidate matching samples are concatenated and linearly transformed to generate an intermediate representation vector set. The intermediate representation vector set is fused with the corresponding dual-view training samples of the next adjacent data pair, and the intermediate representation vector set is concatenated into the input features of the first view data and the second view data of the next adjacent data pair to form an enhanced dual-view training sample. Based on the enhanced dual-view training sample input to the first neural network and the second neural network, relevant representation learning is performed on the next adjacent data pair. During the relevant representation learning process, priority labels are set for the data samples corresponding to the intermediate representation vectors from the candidate matching samples, and the corresponding data samples participate in representation alignment first during the relevant representation learning process. After learning the relevant representations of the next adjacent data pair, the intermediate representation vector set is regenerated based on the updated first mapping relationship, and the updated intermediate representation vector set is passed to the next adjacent data pair. The link-progressive calculation steps are repeatedly executed along the cross-system business link, and the intermediate representation vector set is passed segment by segment between multiple adjacent data pairs to participate in the relevant representation update. The fusion process is performed between the input layers and nonlinear mapping layers of the first and second neural networks.

[0011] Optionally, the unified merging process specifically includes: Based on the second mapping relationship, the data sample correspondence between different business systems is extracted, and the data samples with corresponding relationships are divided into several aggregation groups; Generate a unified aggregation identifier for the data samples in each aggregation group, and associate the unified aggregation identifier with all data samples in the aggregation group; Based on the field correspondence determined in the second mapping relationship, field alignment processing is performed on the data samples in the aggregation group to uniformly map the corresponding fields from different business systems; After field alignment, field value selection is performed on the corresponding fields in the aggregation group. When the same field has more than one value, the target field value is selected according to the preset priority rules to generate a unified field result. Perform record integration processing on the data samples in the aggregation group, and combine the results of the unified fields into a single data record; All aggregated groups are processed using a unified data structure. The integrated data records are rearranged according to the preset data structure to generate the initial aggregated data.

[0012] Optionally, the closed-loop correction specifically includes: Extract the merged result records from the initial collected data according to the unified collection identifier, and construct the master data standard view; According to the second mapping relationship, data samples corresponding to each merge result record are extracted from the preprocessed data, and the extracted data samples are paired with the corresponding merge result records to construct correction samples; Perform field alignment processing on the data samples and merge result records in the correction sample, and adjust the field order in the data sample to be consistent with the corresponding merge result record; The data samples after field alignment are used as the first view data, and the corresponding merged result records are used as the second view data. They are then input into the first neural network and the second neural network to perform relevant representation learning. Based on the results of relevant representation learning, the correspondence between data samples and merge result records is determined, and data samples corresponding to the same merge result record are used as a set of correction sample pairs. The corrected sample pairs are input into the first neural network and the second neural network, and the parameters of the first neural network and the second neural network are updated. After the parameter update is completed, the correspondence between data samples is recalculated based on the updated first neural network and second neural network to generate a correction mapping relationship, and the second mapping relationship is corrected based on the correction mapping relationship.

[0013] Optionally, the result update specifically includes: The correspondence between data samples in the preprocessed data is updated based on the corrected mapping relationship, and the data samples are re-divided and grouped according to the updated correspondence. Perform a merge process on the data samples in each aggregation group, and integrate the data samples belonging to the same aggregation group into a single data record; During the merging process, the fields in the grouping are aligned, and the latest field value is selected for fields with more than one value to generate a unified data record. After merging all the data groups, the generated data records are organized according to the preset data structure to form the target data. The target aggregated data is written to a unified data store, and the target aggregated data replaces the initial aggregated data.

[0014] The beneficial effects of this invention are: (1) This invention constructs a cross-system business link and introduces DCCA based on the link progressive calculation structure to learn the segment-by-segment related representation. This makes data matching no longer limited to the static correspondence between single data sources, but can achieve multi-level transmission and dynamic updates along the business flow path, thereby improving the accuracy of the identification of the correspondence between multiple data sources and avoiding the problems of mismatch and information fragmentation caused by local matching in traditional methods.

[0015] (2) By introducing a master data standard view, the present invention performs secondary related representation learning on the initial collection results based on the closed-loop correction mechanism, so that the collection results can participate in the model optimization process in reverse, realize the verification and correction of the correspondence of data samples, reduce the cumulative impact of the initial matching error in the collection process, and improve the consistency and reliability of the collection results.

[0016] (3) This invention uses the mapping relationship as the core intermediate structure throughout the data matching and merging process. In the result update stage, the data merging is re-executed based on the corrected mapping relationship, so that the model output can directly drive the data structure reconstruction, realize the collaborative optimization of the data collection process and the model learning process, and improve the model's adaptability to new business data and the overall data processing efficiency. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a group business data aggregation method based on deep learning proposed in this invention; Figure 2 This is a schematic diagram of the dual-view training sample construction and matching mechanism of a deep learning-based group business data aggregation method proposed in this invention. Figure 3 This is a flowchart illustrating the progressive calculation process of a deep learning-based group business data aggregation method proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-3 A deep learning-based method for group business data aggregation includes the following steps: Obtain raw business data, perform preprocessing on the raw business data, and obtain preprocessed data; Link construction involves sorting all business data sources by their connections based on business flow relationships, generating cross-system business links, and dividing the preprocessed data into several adjacent data pairs based on these cross-system business links. Sample generation involves constructing dual-view training samples based on each adjacent data pair. Construct DCCA, perform relevant representation learning on dual-view training samples, output the first mapping relationship corresponding to the current adjacent data pair, use the first mapping relationship corresponding to the current adjacent data pair as the candidate mapping input for the next adjacent data pair, perform constraint matching and relevant representation update segment by segment along the cross-system business link to obtain the second mapping relationship; Based on the second mapping relationship, a unified merging process is performed on the preprocessed data to generate initial aggregated data; Closed-loop calibration involves constructing a master data standard view based on the initial aggregated data, recombining the preprocessed data with the master data standard view into calibration samples, performing secondary correlation representation learning on the calibration samples based on DCCA, outputting the calibration mapping relationship, and correcting the second mapping relationship based on the calibration mapping relationship. The results are updated by updating the initial aggregated data based on the correction mapping relationship, generating the target aggregated data, and writing the target aggregated data into the unified data storage.

[0020] In this embodiment, the preprocessing specifically includes: Obtain the corresponding raw business data from several business systems respectively, and then unify and aggregate the raw business data from different business systems. Perform data cleaning on the original business data, deleting duplicate data, abnormal data, and data records with missing key fields; perform data format unification processing, converting the field data formats in different business systems into a unified format; perform field standardization processing, unifying fields representing the same business meaning in different business systems into preset field identifiers; and perform data structure alignment processing to obtain preprocessed data.

[0021] Field standardization is achieved by constructing a field mapping dictionary, which contains field name mapping relationships and field semantic tags. Field name similarity is calculated using edit distance. When the similarity is greater than 0.8, a mapping relationship is established. The fields are then semantically validated using a preset semantic tag library, and finally unified with preset field identifiers.

[0022] Data structure alignment is achieved by building a unified data structure template, which rearranges the data from different business systems according to a unified field order, fills missing fields with default values, sets the default value of numeric fields to 0, and sets the default value of character fields to an empty string, thus forming preprocessed data.

[0023] In this embodiment, link construction specifically includes: Based on the data interaction relationships and business processing order between several business systems, a system flowchart is constructed to describe the pre- and post-processing relationships between all business systems. The system flowchart is generated by reading the interface call logs and data transmission records between business systems. The interface call logs include call time, call frequency and call direction. Connections are established between systems with a call frequency greater than a preset threshold of 0.6. The business processing order is sorted according to the data generation time to form a directed process structure. Based on the system flowchart, the upstream and downstream systems of each business system are determined. All business systems are connected in a directed manner according to the business processing order to form a cross-system business link. The upstream system is determined by identifying the data source field, and the downstream system is determined by identifying the data destination field. The in-degree and out-degree of each business system are recorded. When the in-degree is greater than 0, it is marked as having an upstream system. When the out-degree is greater than 0, it is marked as having a downstream system. All business systems are topologically sorted according to the time series to generate a directed link structure. For situations where a business system corresponds to more than one upstream system or more than one downstream system, branch nodes or aggregation nodes are established in the system flowchart to represent the many-to-many connection relationship between business systems. When the in-degree of a business system is detected to be greater than 1, it is marked as an aggregation node in the system flowchart; when the out-degree is detected to be greater than 1, it is marked as a branch node. An independent link path number is generated for each branch node, and a path weight is set for each path. The path weight is calculated based on the proportion of data transmission volume, and paths with a transmission volume proportion of less than 0.1 are filtered out. Based on the system flowchart, the preprocessed data is divided into paths, and the preprocessed data is divided into several adjacent data pairs according to the relationship between adjacent business systems in the cross-system business link, and dual-view training samples are constructed.

[0024] In this embodiment, sample generation extracts corresponding record sets from both sides of each adjacent data pair. The fields in the record sets are then uniformly encoded, converting character fields into vector representations and normalizing numerical fields. Based on the uniformly encoded data, similarity matching is performed on the two record sets. The similarity calculation combines cosine similarity and edit distance. Cosine similarity is used to calculate the similarity between vector representations, while edit distance is used to calculate the difference between field names or identifier fields. The two similarity results are weighted and fused according to a fixed ratio to obtain a comprehensive similarity score. Record pairs with a comprehensive similarity score greater than a preset threshold of 0.8 are marked as candidate matching samples, and record pairs with a comprehensive similarity score lower than 0.4 are marked as non-matching samples. For record pairs with a comprehensive similarity score between 0.4 and 0.8, a set of samples to be verified is generated. The set of samples to be verified is manually verified, and the correspondence between the record pairs is confirmed by manual verification. The results of the manual verification are then added to the candidate matching sample set or the non-matching sample set. The candidate matching sample set and the non-matching sample set are respectively constructed as dual-view training samples, where each set of samples contains the corresponding record vector representation and matching label from both sides of the adjacent data pair.

[0025] In this embodiment, constructing DCCA specifically includes: Construct a first neural network and a second neural network, respectively receiving first view data and second view data from dual-view training samples, performing nonlinear mapping on the first view data and second view data to obtain a first representation vector and a second representation vector, and jointly training the first neural network and the second neural network; In the process of learning the relevant representation, a link-progressive computation structure is introduced for adjacent data pairs in the cross-system business link. The first mapping relationship corresponding to the current adjacent data pair is used as the candidate mapping input for the next adjacent data pair. When learning the relevant representation of the next adjacent data pair, the candidate matching range is constrained, and the relevant representation is updated segment by segment along the cross-system business link. After completing the progressive calculation of all adjacent data pairs in the cross-system business link, the output is a unified related representation result covering multiple business data sources; After generating the initial aggregated data, a closed-loop correction structure is constructed by introducing the master data standard view. The preprocessed data and the master data standard view are then reconstructed into a dual-view input, and secondary correlation representation learning is performed to correct the second mapping relationship.

[0026] In this embodiment, the link-based progressive calculation structure specifically includes: After performing DCCA related representation learning for the current adjacent data pairs, the corresponding candidate matching samples are extracted based on the first mapping relationship. The first and second representation vectors in the candidate matching samples are concatenated and linearly transformed to generate an intermediate representation vector set. The intermediate representation vector set is fused with the corresponding dual-view training samples of the next adjacent data pair. The intermediate representation vector set is then concatenated into the input features of the first and second view data of the next adjacent data pair to form an enhanced dual-view training sample. For data samples that do not match an intermediate representation vector, zero vectors are filled into the corresponding positions. Based on the enhanced dual-view training samples input into the first neural network and the second neural network, relevant representation learning is performed on the next adjacent data pair. During the relevant representation learning process, priority labels are set for the data samples corresponding to the intermediate representation vectors from the candidate matching samples, and the corresponding data samples participate in representation alignment with priority during the relevant representation learning process. During the training process, the data samples with priority labels are repeatedly sampled in each training batch, and the number of resampling times is set to 2 to 4 times to increase their participation weight in relevant representation learning. After learning the relevant representations of the next adjacent data pair, the intermediate representation vector set is regenerated based on the updated first mapping relationship. The updated intermediate representation vector set is then passed to the next adjacent data pair. The original intermediate representation vector set is then overwritten and updated. The link progression calculation steps are repeated sequentially along the cross-system business link. The intermediate representation vector set is passed segment by segment between multiple adjacent data pairs and participates in the relevant representation update. The segment-by-segment transmission is performed in the order of the cross-system business link. Each adjacent data pair only receives the intermediate representation vector set from the upstream adjacent data pair. When the link length is greater than 3, only the intermediate representation vector sets of the two most recent levels are retained, and the rest of the historical intermediate representation vector sets are discarded. The fusion process is performed between the input layers and nonlinear mapping layers of the first and second neural networks.

[0027] In this embodiment, the unified merging process specifically includes: Based on the second mapping relationship, the correspondence between data samples of different business systems is extracted, and the data samples with corresponding relationships are divided into several aggregation groups. The correspondence between data samples is determined by filtering data sample pairs with a correlation greater than 0.75 in the second mapping relationship. The correlation is the output of DCCA. The data samples with corresponding relationships are grouped by disjoint set method to form several aggregation groups. Each aggregation group contains several data samples from different business systems. Generate a unified aggregation identifier for the data samples in each aggregation group, and associate the unified aggregation identifier with all data samples in the aggregation group; Based on the field correspondence determined in the second mapping relationship, field alignment processing is performed on the data samples in the aggregation group to uniformly map the corresponding fields from different business systems. The field correspondence is determined by statistically analyzing the field matching frequency in the second mapping relationship. When the field matching frequency is greater than 0.7, a field mapping relationship is established, and all data samples in the aggregation group are renamed according to the unified field mapping table. After field alignment, field value selection is performed on the corresponding fields in the aggregation group. When the same field has more than one value, the target field value is selected according to the preset priority rule to generate a unified field result. The priority rule is sorted by update time and the latest data value is selected. The data samples in the aggregation group are processed by record integration, which combines the results of the unified fields into a single data record. The record integration process is to form a single data record by splicing the results of the unified fields in the same field order. Each single data record corresponds to a unified aggregation identifier. All aggregated groups are processed using a unified data structure. The integrated data records are rearranged according to the preset data structure to generate the initial aggregated data.

[0028] The unified data structure, defined by a preset data template, rearranges the integrated data records according to the field order, fills missing fields with default values, with numeric fields filled with 0 and character fields filled with empty strings, ultimately generating a standardized initial aggregated data set.

[0029] In this embodiment, closed-loop calibration specifically includes: The merged result records in the initial collected data are extracted according to the unified collection identifier to construct the master data standard view; the merged result records are grouped according to the unified collection identifier, and data records with the same unified collection identifier are integrated into a master data record. All master data records constitute the master data standard view. According to the second mapping relationship, data samples corresponding to each merge result record are extracted from the preprocessed data. The extracted data samples are paired with the corresponding merge result records to construct correction samples. The extraction process is achieved by traversing the second mapping relationship. The set of data samples associated with the same unified collection identifier is extracted and paired with the corresponding merge result records one by one. Each pairing result forms a correction sample. Field alignment is performed on the data samples in the correction sample and the merged result records to adjust the field order in the data samples to be consistent with the corresponding merged result records. Field alignment is accomplished through a field mapping table, which maps the field names in the data samples to the field names in the main data standard view and rearranges them according to a unified field order. For missing fields, preset default values ​​are filled in. The data samples after field alignment are used as the first view data, and the corresponding merged result records are used as the second view data. They are input into the first neural network and the second neural network to perform correlation representation learning. Correlation representation learning is performed by inputting the first view data and the second view data into the first neural network and the second neural network respectively, and obtaining the corresponding representation vectors through multiple nonlinear mappings. Correlation is calculated based on the representation vectors. The correspondence between data samples and merged result records is determined based on the relevant representation learning results. Data samples corresponding to the same merged result record are considered as a set of correction sample pairs. The correspondence is determined by comparing the similarity between representation vectors. Data samples with similarity higher than the threshold are classified as the same set of correction sample pairs with the merged result records. The corrected sample pairs are input into the first and second neural networks, and the parameters of the first and second neural networks are updated. A similarity value is calculated based on the first and second representation vectors, and the similarity value is obtained by cosine similarity. The similarity value is used as the optimization objective, and a loss function is constructed by taking the negative value of the similarity value. In each training batch, the loss function value is calculated for the corrected sample pairs, and backpropagation is performed on the first and second neural networks based on the loss function to update the network weight parameters. After the parameter update is completed, the correspondence between data samples is recalculated based on the updated first neural network and second neural network to generate a correction mapping relationship, and the second mapping relationship is corrected based on the correction mapping relationship.

[0030] The corrected mapping relationship is obtained by recalculating the relevant representation for all data samples, calculating the similarity of the updated representation vectors, and re-establishing the set of correspondences between data samples based on the similarity results. The original relationship in the second mapping relationship is replaced with the set of correspondences.

[0031] In this embodiment, the parameter updates of the calibration samples are performed using the same backpropagation mechanism as the original training processes of the first and second neural networks, without setting up a separate training phase. In each training iteration, the dual-view training samples are first input into the first and second neural networks to perform forward computation and complete one parameter update. Then, in the same training round, the calibration samples are input into the first and second neural networks to perform forward computation, and backpropagation is performed again based on the calculated loss function of the calibration samples to update the same set of network weight parameters. The two backpropagations are executed sequentially in the same iteration, sharing the same network parameters, so that the first and second neural networks can introduce calibration constraints based on the master data standard view while maintaining their original related representation learning capabilities.

[0032] In this embodiment, the result update specifically includes: The correspondence between data samples in the preprocessed data is updated based on the correction mapping relationship, and the data samples are regrouped according to the updated data sample correspondence relationship. The updated data sample correspondence relationship is generated by traversing the correction mapping relationship. Data samples with direct or indirect correspondence relationships are grouped into the same group. The disjoint set method is used to group the data samples so that data samples belonging to the same entity form a group. Merging is performed on the data samples in each aggregation group, integrating data samples belonging to the same aggregation group into a single data record. The merging process is achieved by iterating through the data samples in the aggregation group one by one, extracting the field values ​​in each data sample and summarizing them into the same data record, and merging identical fields. During the merging process, the fields in the grouping are aligned, and the latest field value is selected for fields with more than one value to generate a unified data record. After merging all the data groups, the generated data records are organized according to the preset data structure to form the target data. The target aggregated data is written to a unified data store, and the target aggregated data replaces the initial aggregated data.

[0033] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-business system data collaborative processing scenario. In this scenario, different business systems operate independently, and data flows between multiple systems according to business processes. Due to differences in field naming, data structure, and encoding methods among the systems, the same entity is represented inconsistently in different systems, making it difficult to directly establish a correspondence. Existing technologies typically employ rule-based matching methods, such as aligning data through primary key fields or manually configured field mapping relationships, or using matching methods based on single-time similarity calculations to directly compare data from two systems. These methods can achieve basic matching in simple scenarios, but in multi-level link scenarios, due to the lack of overall modeling capabilities for data transmission relationships, problems such as matching breaks, accumulated mismatches, and inconsistent aggregation results easily arise. This requires significant manual intervention to adjust rules, resulting in low processing efficiency.

[0034] In this scenario, raw business data from multiple business systems is first acquired and preprocessed uniformly to ensure consistency in format and structure across different sources. Then, cross-system business links are constructed based on business flow relationships. Data is divided into adjacent data pairs according to the link sequence, and dual-view training samples are built based on these pairs. During sample construction, high-quality samples are selected through a combination of similarity matching and manual verification. Next, a DCCA model is constructed to perform relevant representation learning on the dual-view training samples, progressively calculating along each segment of the business link. This ensures that the mapping relationship obtained in the previous stage participates in the matching of the next stage, thereby establishing an overall mapping relationship across multiple business systems. Based on this mapping relationship, data merging is performed to generate initial aggregated data. Further, a master data standard view is constructed using the initial aggregated results. This view serves as a unified reference and is recombinated with the original data to form correction samples. These correction samples are then input into the model again for relevant representation learning, validating and adjusting the original mapping relationship to obtain a more accurate one. Finally, merging is performed again based on the corrected mapping relationship to generate the target aggregated data.

[0035] In actual operation, the processing effects of traditional rule-based matching methods, single-shot similarity matching methods, and the method of this invention were compared, and representative indicators were selected for evaluation. It should be noted that the sources of each indicator are as follows: matching accuracy is calculated using manually labeled true correspondences as the standard; aggregation consistency measures the uniqueness of the same entity in the aggregation results, i.e., whether it is split into multiple records; link matching completeness measures whether data can maintain continuous correspondences in multi-level business flows; and the proportion of manual intervention is obtained by statistically analyzing the percentage of samples requiring manual correction. These indicators can comprehensively reflect the data aggregation quality and degree of automation.

[0036] Table 1: Comparison of Data Aggregation Performance

[0037] As shown in Table 1, this invention demonstrates advantages in several key indicators. The improved matching accuracy indicates that through progressive link calculation, the model can more accurately identify the true correspondence between data across systems; the improved aggregation consistency shows that this invention avoids the problem of the same entity being split or repeatedly aggregated; the improved link matching integrity indicates that this invention can maintain the continuity of data relationships during multi-level business flows, while traditional methods are prone to relationship breaks during multi-level transmission; the significant reduction in the proportion of manual intervention reflects the higher degree of automation in this invention, reducing reliance on manual rule adjustments; and the reduction in processing latency indicates that while ensuring accuracy, this invention also has advantages in overall efficiency. In summary, this invention achieves improvements in accuracy, consistency, and automation, and can solve key problems in the aggregation of multi-source data in existing technologies.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep learning-based group business data aggregation method, characterized by, Includes the following steps: Obtain raw business data, perform preprocessing on the raw business data, and obtain preprocessed data; Link construction involves sorting all business data sources by their connections based on business flow relationships, generating cross-system business links, and dividing the preprocessed data into several adjacent data pairs based on these cross-system business links. Sample generation involves constructing dual-view training samples based on each adjacent data pair. Construct DCCA, perform relevant representation learning on dual-view training samples, output the first mapping relationship corresponding to the current adjacent data pair, use the first mapping relationship corresponding to the current adjacent data pair as the candidate mapping input for the next adjacent data pair, perform constraint matching and relevant representation update segment by segment along the cross-system business link to obtain the second mapping relationship; Based on the second mapping relationship, a unified merging process is performed on the preprocessed data to generate initial aggregated data; Closed-loop calibration involves constructing a master data standard view based on the initial aggregated data, recombining the preprocessed data with the master data standard view into calibration samples, performing secondary correlation representation learning on the calibration samples based on DCCA, outputting the calibration mapping relationship, and correcting the second mapping relationship based on the calibration mapping relationship. The results are updated by updating the initial aggregated data based on the correction mapping relationship, generating the target aggregated data, and writing the target aggregated data into the unified data storage. 2.The group business data collection method based on deep learning according to claim 1, characterized in that, The preprocessing specifically includes: Obtain the corresponding raw business data from several business systems respectively, and then unify and aggregate the raw business data from different business systems. Perform data cleaning on the original business data, deleting duplicate data, abnormal data, and data records with missing key fields; perform data format unification processing, converting the field data formats in different business systems into a unified format; perform field standardization processing, unifying fields representing the same business meaning in different business systems into preset field identifiers; and perform data structure alignment processing to obtain preprocessed data.

3. The group business data collection method based on deep learning according to claim 2, characterized in that, The link construction specifically includes: Based on the data interaction relationships and business processing sequence between several business systems, a system flowchart is constructed to describe the pre- and post-processing relationships between all business systems. Based on the system flowchart, determine the upstream and downstream systems of each business system, and connect all business systems in a directed manner according to the business processing sequence to form a cross-system business link. For situations where a business system corresponds to more than one upstream system or more than one downstream system, branch nodes or convergence nodes are established in the system flowchart to represent the many-to-many connection relationship between business systems; Based on the system flowchart, the preprocessed data is divided into paths, and the preprocessed data is divided into several adjacent data pairs according to the relationship between adjacent business systems in the cross-system business link, and dual-view training samples are constructed.

4. The group business data collection method based on deep learning according to claim 3, characterized in that, The construction of DCCA specifically includes: Construct a first neural network and a second neural network, respectively receiving first view data and second view data from dual-view training samples, performing nonlinear mapping on the first view data and second view data to obtain a first representation vector and a second representation vector, and jointly training the first neural network and the second neural network; In the process of learning the relevant representation, a link-progressive computation structure is introduced for adjacent data pairs in the cross-system business link. The first mapping relationship corresponding to the current adjacent data pair is used as the candidate mapping input for the next adjacent data pair. When learning the relevant representation of the next adjacent data pair, the candidate matching range is constrained, and the relevant representation is updated segment by segment along the cross-system business link. After completing the progressive calculation of all adjacent data pairs in the cross-system business link, the output is a unified related representation result covering multiple business data sources; After generating the initial aggregated data, a closed-loop correction structure is constructed by introducing the master data standard view. The preprocessed data and the master data standard view are then reconstructed into a dual-view input, and secondary correlation representation learning is performed to correct the second mapping relationship.

5. The group business data collection method based on deep learning according to claim 4, characterized in that, The link-progressive calculation structure specifically includes: After performing DCCA related representation learning for the current adjacent data pairs, the corresponding candidate matching samples are extracted based on the first mapping relationship. The first and second representation vectors in the candidate matching samples are concatenated and linearly transformed to generate an intermediate representation vector set. The intermediate representation vector set is fused with the corresponding dual-view training samples of the next adjacent data pair, and the intermediate representation vector set is concatenated into the input features of the first view data and the second view data of the next adjacent data pair to form an enhanced dual-view training sample. Based on the enhanced dual-view training sample input to the first neural network and the second neural network, relevant representation learning is performed on the next adjacent data pair. During the relevant representation learning process, priority labels are set for the data samples corresponding to the intermediate representation vectors from the candidate matching samples, and the corresponding data samples participate in representation alignment first during the relevant representation learning process. After learning the relevant representations of the next adjacent data pair, the intermediate representation vector set is regenerated based on the updated first mapping relationship, and the updated intermediate representation vector set is passed to the next adjacent data pair. The link-progressive calculation steps are repeatedly executed along the cross-system business link, and the intermediate representation vector set is passed segment by segment between multiple adjacent data pairs to participate in the relevant representation update; The fusion process is performed between the input layers and nonlinear mapping layers of the first and second neural networks.

6. The group business data collection method based on deep learning according to claim 5, characterized in that, The unified merging process specifically includes: Based on the second mapping relationship, the data sample correspondence between different business systems is extracted, and the data samples with corresponding relationships are divided into several aggregation groups; Generate a unified aggregation identifier for the data samples in each aggregation group, and associate the unified aggregation identifier with all data samples in the aggregation group; Based on the field correspondence determined in the second mapping relationship, field alignment processing is performed on the data samples in the aggregation group to uniformly map the corresponding fields from different business systems; After field alignment, field value selection is performed on the corresponding fields in the aggregation group. When the same field has more than one value, the target field value is selected according to the preset priority rules to generate a unified field result. Perform record integration processing on the data samples in the aggregation group, and combine the results of the unified fields into a single data record; All aggregated groups are processed using a unified data structure. The integrated data records are rearranged according to the preset data structure to generate the initial aggregated data.

7. The group business data collection method based on deep learning according to claim 6, characterized in that, The closed-loop correction specifically includes: Extract the merged result records from the initial collected data according to the unified collection identifier, and construct the master data standard view; According to the second mapping relationship, data samples corresponding to each merge result record are extracted from the preprocessed data, and the extracted data samples are paired with the corresponding merge result records to construct correction samples; Perform field alignment processing on the data samples and merge result records in the correction sample, and adjust the field order in the data sample to be consistent with the corresponding merge result record; The data samples after field alignment are used as the first view data, and the corresponding merged result records are used as the second view data. They are then input into the first neural network and the second neural network to perform relevant representation learning. Based on the results of relevant representation learning, the correspondence between data samples and merge result records is determined, and data samples corresponding to the same merge result record are used as a set of correction sample pairs. The corrected sample pairs are input into the first neural network and the second neural network, and the parameters of the first neural network and the second neural network are updated. After the parameter update is completed, the correspondence between data samples is recalculated based on the updated first neural network and second neural network to generate a correction mapping relationship, and the second mapping relationship is corrected based on the correction mapping relationship.

8. The group business data collection method based on deep learning according to claim 7, characterized in that, The result update specifically includes: The correspondence between data samples in the preprocessed data is updated based on the corrected mapping relationship, and the data samples are re-divided and grouped according to the updated correspondence. Perform a merge process on the data samples in each aggregation group, and integrate the data samples belonging to the same aggregation group into a single data record; During the merging process, the fields in the grouping are aligned, and the latest field value is selected for fields with more than one value to generate a unified data record. After merging all the data groups, the generated data records are organized according to the preset data structure to form the target data. The target aggregated data is written to a unified data store, and the target aggregated data replaces the initial aggregated data.

Images