Methods, apparatus, equipment and storage media for integrating multi-source external data

Abstract

This application provides a method, apparatus, device, and storage medium for integrating multi-source external data, relating to the field of data processing technology. The method for integrating multi-source external data includes: real-time acquisition of source field sets from each data source and the source field values ​​of each source field in the source field sets; taking the union of each source field set to obtain a field union, and using the field union as the standard field set of the standardized model corresponding to each data source, and using the field union as the integration field set of the integration model; standardizing the source field values ​​to obtain the standard values ​​of each standard field in each standard field set; generating ID values ​​based on common identifying fields in each source field set, and using the ID values ​​as unique identifiers for the integration model and each standardized model; and selecting target values ​​from each standard value as the integration field values ​​of each integration field based on the unique identifiers and the preset source selection configuration of each integration field in the integration field set, so as to integrate the data sources into an integration model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for integrating multi-source external data. Background Technology

[0002] In the digital society, data, as a fundamental strategic resource and a key production factor, is increasingly demonstrating its value in enterprise management and strategic decision-making. In recent years, data service providers have been vying to release various high-quality data products. However, due to significant differences in the logic, sequence, and resource allocation of data processing among these providers, similar data products vary in coverage, timeliness, and field saturation, making it difficult for data users to select the best data source that simultaneously meets multiple application needs. Therefore, how to filter and integrate data from multiple sources has become an important research direction in the field of data processing.

[0003] Currently, for multi-source integration of similar data, a row-level granular integration model is often adopted, with a single data source as the primary source and other data sources as secondary sources; that is, the record is used as the unit of integration. This method can improve data coverage and timeliness to some extent.

[0004] However, in the integration model that uses a single data source as the main source and other data sources as supplementary sources, the field null value rate provided by the other data sources after integration is relatively high. Furthermore, the single-source approach has low flexibility in selecting sources for common fields, making it difficult to fully explore and utilize the advantages of other data sources. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, device and storage medium for integrating multi-source external data, aiming to solve the technical problems of low field value rate and low utilization rate of each data source in the integration model.

[0006] To achieve the above objectives, this application provides a method for integrating multi-source external data, the method comprising:

[0007] Real-time acquisition of the source field sets of each data source and the source field values ​​of each source field in the source field sets;

[0008] The standard field set of the standardized model corresponding to each of the source field sets is determined based on the union of the source field sets, and the integrated field set of the integrated model is determined based on the union of the source field sets.

[0009] The values ​​of each source field are standardized to obtain the standard values ​​of each standard field in each set of standard fields.

[0010] The integrated model and each standardized model are generated based on the common identifying fields in each of the source field sets;

[0011] Based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, the integration field value of each integration field is selected from each of the standard values ​​to integrate each of the data sources into the integration model.

[0012] Optionally, in some feasible embodiments, the source selection configuration includes: a pre-agreed model, which includes: weighted source selection;

[0013] The step of selecting the respective integration field value of each integration field from the respective standard values ​​includes:

[0014] If any of the integration fields contains a first field configured as a weighted source selection field, then obtain the weight parameters of each of the standardized models corresponding to the first field.

[0015] If any of the weight parameters contains a target weight greater than a preset weight threshold, then the first standardized model corresponding to the target weight is determined.

[0016] The standard value of the first standard field corresponding to the first field in the first standardized model is used as the integrated field value of the first field.

[0017] Optionally, in some feasible embodiments, the pre-agreed model further includes: auxiliary source selection, which includes: first-in, first-out;

[0018] The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0019] If any of the integration fields has a second field configured as first-in-first-out for source selection, then the second standardized model is determined based on the last update time of each standardized model.

[0020] The standard value of the second standard field corresponding to the second field in the second standardized model is used as the integrated field value of the second field.

[0021] Optionally, in some feasible embodiments, the auxiliary source selection includes: a first non-empty source;

[0022] The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0023] If any of the integration fields contains a third field configured as the first non-empty source, then the third standard field corresponding to the third field in each of the standardization models is determined.

[0024] Identify each non-empty standardized model in which the standard value of the third standard field in each of the standardized models is not empty;

[0025] The third normalization model is determined based on the last update time of each of the aforementioned non-empty normalization models;

[0026] The standard value of the third standard field in the third standardization model is used as the integrated field value of the third field.

[0027] Optionally, in some feasible embodiments, the auxiliary source selection includes: specifying a data source;

[0028] The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0029] If any of the integration fields contains a fourth field configured as a specified data source, then the fourth standardized model corresponding to the specified data source is determined according to the specified data source configuration.

[0030] The standard value of the fourth standard field corresponding to the fourth field in the fourth standardization model is used as the integrated field value of the fourth field.

[0031] Optionally, in some feasible embodiments, the source selection configuration further includes: a post-event monitoring model, which includes: voting for source selection;

[0032] After the step of selecting the respective integrated field value of each of the said standard values, the method further includes:

[0033] When integrating three or more data sources, if there is a fifth field in each of the integration fields that is configured as a voting selection field, and the integration field value of the fifth field has not been updated within a preset time threshold, then the fifth field is determined to be the fifth standard field corresponding to each of the standardization models.

[0034] Determine the mode among the standard values ​​of each of the fifth standard fields, and update the integrated field value of the fifth field based on the mode.

[0035] Optionally, in some feasible embodiments, after the step of generating unique identifiers for the integrated model and each of the standardized models based on common identifying fields in each of the source field sets, the method further includes:

[0036] The mapping relationship between the integrated model and each of the standardized models is determined based on the unique identifier.

[0037] Furthermore, to achieve the above objectives, this application also provides a multi-source external data integration device, which is a virtual device, and includes:

[0038] The source data preparation module is used to obtain the source field set of each data source and the source field value of each source field in the source field set in real time;

[0039] The field standardization module is used to determine the standard field set of the standardization model corresponding to each of the source field sets, and the integrated field set of the integrated model, based on the union of the source field sets.

[0040] The field value standardization module is used to standardize the values ​​of each of the source fields to obtain the standard values ​​of each standard field in the standard field set.

[0041] The unique identifier module is used to generate unique identifiers for the integrated model and each of the standardized models based on the common identifier fields in each of the source field sets;

[0042] The source selection and integration module is used to select the integration field value of each integration field from the standard values ​​based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, so as to integrate the data sources into the integration model.

[0043] In addition, to achieve the above objectives, this application also provides a multi-source external data integration device, which includes: a memory, a processor, and a multi-source external data integration program stored in the memory and executable on the processor. When the multi-source external data integration program is executed by the processor, it implements the steps of the multi-source external data integration method described above.

[0044] This application also provides a storage medium storing a multi-source external data integration program, which, when executed by a processor, implements the steps of the multi-source external data integration method described above.

[0045] This application provides a method, apparatus, device, and storage medium for integrating multi-source external data. The method includes: acquiring in real time the source field sets of each data source and the source field values ​​of each source field in the source field sets; determining the standard field sets of the standardized models corresponding to each data source and the integration field sets of the integrated model based on the union of the source field sets; standardizing the source field values ​​to obtain the standard values ​​of each standard field in the standard field sets; generating unique identifiers for the integrated model and each standardized model based on common identifying fields in the source field sets; and selecting the integration field values ​​of each integration field from the standard values ​​based on the unique identifiers and the preset source selection configurations of each integration field in the integration field sets to integrate the data sources into the integrated model.

[0046] Compared to existing technologies that rely on a single data source for multi-source data integration, this application proposes a multi-source external data integration method based on a unified data model. First, it obtains the source field sets and source field values ​​of each data source. Then, it standardizes the source fields and data of each data source to obtain a standardized model corresponding to each data source. Furthermore, it generates a unique identifier through common identifying fields in each source field set, avoiding interference from other different types of data sources during the source selection process of the integration model. Finally, based on the source selection configurations set by the technical personnel, it selects the integration field values ​​of each integration field in the integration model to integrate the data from each data source into the integration model.

[0047] Thus, this application's method of standardizing multivariate data and then selecting sources for each field in each standardized model, compared to the traditional approach of integrating multiple data sources with a single primary data source and other secondary data sources, does not distinguish between primary and secondary sources and selects sources on a field-by-field basis. On the one hand, it can select the best data source at the field level, thereby improving the field value rate of the integrated model; on the other hand, it can dynamically select sources at the field level based on the actual data performance and application preferences of each data source, thereby fully exploring and utilizing the advantages of each data source. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the structure of a multi-source external data integration device in the hardware operating environment of the device involved in the embodiments of this application;

[0051] Figure 2 This is a schematic diagram illustrating the implementation process of an embodiment of the multi-source external data integration method of this application;

[0052] Figure 3 This is a data standardization diagram illustrating an embodiment of the multi-source external data integration method of this application;

[0053] Figure 4 This is a schematic diagram of the source selection model of an embodiment of the multi-source external data integration method of this application;

[0054] Figure 5 This is a diagram illustrating the overall architecture of a multi-source external data integration method based on a unified model, according to an embodiment of the method for multi-source external data integration in this application.

[0055] Figure 6 This is a schematic diagram of the functional modules of the multi-source external data integration device involved in the embodiments of this application.

[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0058] It is important to note that in the digital society, data, as a fundamental strategic resource and a key production factor, is increasingly valuable in enterprise management and strategic decision-making. Data service providers occupy a central position in the data element industry chain, serving as a bridge between data suppliers and users. In recent years, data service providers have been competing to release various high-quality data products. However, due to significant differences in the logic, sequence, and resource allocation of data processing among different providers, similar data products from different providers vary in terms of coverage, timeliness, and field saturation. This makes it difficult for data users to select the best data source that can simultaneously meet multiple application needs from among several service providers. Therefore, how to filter and integrate data from multiple sources has become an important research direction in the field of data processing.

[0059] Currently, for multi-source integration of similar data, a row-level granular integration model is often adopted, with a single data source as the primary source and other data sources as secondary sources; that is, the record is used as the unit of integration. This method can improve data coverage and timeliness to some extent.

[0060] However, in the integration model that uses a single data source as the main source and other data sources as supplementary sources, the field null value rate provided by the other data sources after integration is relatively high, and the source selection for common fields is less flexible in the single-source-based approach, making it difficult to fully explore and utilize the advantages of other data sources.

[0061] To address the aforementioned issues, this application provides a method, apparatus, device, and storage medium for integrating multi-source external data. The method includes: real-time acquisition of source field sets from each data source and the respective source field values ​​of each source field in the source field sets; determining, based on the union of the source field sets, the standard field sets of the standardized models corresponding to each data source and the integration field set of the integrated model; standardizing the source field values ​​to obtain the standard values ​​of each standard field in each standard field set; generating unique identifiers for the integrated model and each standardized model based on common identifying fields in the source field sets; and selecting the integration field values ​​of each integration field from the standard values ​​based on the unique identifiers and preset source selection configurations for each integration field in the integration field set, to integrate the data sources into the integrated model.

[0062] Compared to existing technologies that rely on a single data source for multi-source data integration, this application proposes a multi-source external data integration method based on a unified data model. First, it obtains the source field sets and source field values ​​of each data source. Then, it standardizes the source fields and data of each data source to obtain a standardized model corresponding to each data source. Furthermore, it generates a unique identifier through common identifying fields in each source field set, avoiding interference from other different types of data sources during the source selection process of the integration model. Finally, based on the source selection configurations set by the technical personnel, it selects the integration field values ​​of each integration field in the integration model to integrate the data from each data source into the integration model.

[0063] Thus, this application's method of standardizing multi-source data and then selecting sources for each field in each standardized model, compared to the traditional approach of integrating multi-source data with a single data source as the primary source and other data sources as secondary sources, does not distinguish between primary and secondary sources and selects sources on a field-by-field basis. On the one hand, it can make optimal data source selection at the field level, thereby improving the field value rate of the integrated model; on the other hand, it can dynamically select sources at the field level based on the actual data performance and application preferences of each data source, thereby fully exploring and utilizing the advantages of each data source.

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0065] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a multi-source external data integration device in the hardware operating environment of the device involved in the embodiments of this application.

[0066] The terminal device in this application embodiment can be a computer, a personal computer, or a mobile phone with data processing capabilities.

[0067] like Figure 1 As shown, the multi-source external data integration device may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to establish communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0068] Optionally, the multi-source external data integration device may also include a user interface 1003, a network interface 1004, a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, etc. The user interface may include a display screen and an input submodule such as a keyboard. Optionally, the user interface may also include a standard wired interface or a wireless interface. The network interface may optionally include a standard wired interface or a wireless interface (such as a WiFi interface).

[0069] Those skilled in the art will understand that Figure 1The multi-source external data integration device structure shown does not constitute a limitation on the multi-source external data integration device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0070] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, and a multi-source external data integration program. The operating system is a program that manages and controls the hardware and software resources of the multi-source external data integration device, supporting the operation of the multi-source external data integration program and other software and / or programs. The network communication module is used to enable communication between the various components within the memory 1005, as well as communication with other hardware and software in the multi-source external data integration device.

[0071] exist Figure 1 In the multi-source external data integration device shown, the processor 1001 is used to execute the multi-source external data integration program stored in the memory 1005 and perform the following operations:

[0072] Real-time acquisition of the source field sets of each data source and the source field values ​​of each source field in the source field sets;

[0073] The standard field set of the standardized model corresponding to each of the source field sets is determined based on the union of the source field sets, and the integrated field set of the integrated model is determined based on the union of the source field sets.

[0074] The values ​​of each source field are standardized to obtain the standard values ​​of each standard field in each set of standard fields.

[0075] The integrated model and each standardized model are generated based on the common identifying fields in each of the source field sets;

[0076] Based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, the integration field value of each integration field is selected from each of the standard values ​​to integrate each of the data sources into the integration model.

[0077] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0078] If any of the integration fields contains a first field configured as a weighted source selection field, then obtain the weight parameters of each of the standardized models corresponding to the first field.

[0079] If any of the weight parameters contains a target weight greater than a preset weight threshold, then the first standardized model corresponding to the target weight is determined.

[0080] The standard value of the first standard field corresponding to the first field in the first standardized model is used as the integrated field value of the first field.

[0081] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0082] If any of the integration fields has a second field configured as first-in-first-out for source selection, then the second standardized model is determined based on the last update time of each standardized model.

[0083] The standard value of the second standard field corresponding to the second field in the second standardized model is used as the integrated field value of the second field.

[0084] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0085] If any of the integration fields contains a third field configured as the first non-empty source, then the third standard field corresponding to the third field in each of the standardization models is determined.

[0086] Identify each non-empty standardized model in which the standard value of the third standard field in each of the standardized models is not empty;

[0087] The third normalization model is determined based on the last update time of each of the aforementioned non-empty normalization models;

[0088] The standard value of the third standard field in the third standardization model is used as the integrated field value of the third field.

[0089] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0090] If any of the integration fields contains a fourth field configured as a specified data source, then the fourth standardized model corresponding to the specified data source is determined according to the specified data source configuration.

[0091] The standard value of the fourth standard field corresponding to the fourth field in the fourth standardization model is used as the integrated field value of the fourth field.

[0092] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0093] When integrating three or more data sources, if there is a fifth field in each of the integration fields that is configured as a voting selection field, and the integration field value of the fifth field has not been updated within a preset time threshold, then the fifth field is determined to be the fifth standard field corresponding to each of the standardization models.

[0094] Determine the mode among the standard values ​​of each of the fifth standard fields, and update the integrated field value of the fifth field based on the mode.

[0095] Furthermore, the processor 1001 can call the multi-source external data integration program stored in the memory 1005 and also perform the following operations:

[0096] The mapping relationship between the integrated model and each of the standardized models is determined based on the unique identifier.

[0097] This application provides a method for integrating multi-source external data. In the first embodiment of this method, please refer to... Figure 2 The multi-source external data integration method includes:

[0098] Step S10: Real-time acquisition of the source field set of each data source and the source field value of each source field in the source field set;

[0099] In this embodiment, the terminal device first obtains the source field sets and source field values ​​of the same type of data from different operators, i.e., different data sources, and updates the data in the data source in real time.

[0100] Step S20: Determine the standard field set of the standardized model corresponding to each of the source field sets, and the integrated field set of the integrated model, based on the union of the source field sets.

[0101] In this embodiment, after the terminal device obtains data from various data sources, it first standardizes the fields, that is, it first takes the union of the various source field sets, and then it also needs to unify the field type and field length of each field in the field union. Then, the field union is used as the standard field set of the standardization model of each data source, and the field union is used as the integration field set of the integration model.

[0102] For example, such as Figure 3 As shown, data source A includes fields a and b, and data source B includes fields a and c. The field set obtained by taking the union of the field sets of data sources A and B is a, b, and c. The field sets of standardized model A and standardized model B after data standardization of data sources A and B are both a, b, and c.

[0103] Step S30: Standardize the values ​​of each source field to obtain the standard values ​​of each standard field in each set of standard fields;

[0104] In this embodiment, after obtaining the standard fields of each standardized model, data standardization processing is performed based on the source field values ​​corresponding to each data source. The standardization processing includes unifying numerical units, code values, and definitions to avoid ambiguity in field values ​​when applying the integrated model. When standardizing numerical values, if multiple data sources share a common field, the field values ​​of that field in each data source are standardized, and then the processed field values ​​are assigned to the standardized models corresponding to the multiple data sources. For a field from a single source, the data source of that field can directly assign the field value to the standardized model corresponding to that data source. If the standard field set of the standardized model corresponding to a data source has more fields than the field set of the data source, the field values ​​of the extra fields are set to null.

[0105] For example, such as Figure 3 As shown, data sources A and B share a common field 'a'. The source field value of 'a' in data source A is A_a, and the source field value of 'a' in data source B is B_a. After data standardization, the standard values ​​of 'a' in data source A and data source B are A_a' and B_a', respectively.

[0106] In standardized model A, field b comes only from data source A. Therefore, the source field value of b in data source A is directly assigned to the standard value of b in standardized model A. The standard value of b in standardized model A is A_b. The standard value of c in standardized model B is the same.

[0107] Since field c does not exist in data source A, the standard value of field c in standardized model A is set to empty and denoted as "NULL". The same applies to the standard value of b in standardized model B.

[0108] The above example illustrates the case with only two data sources. When integrating three or more data sources, the data standardization rules are the same and will not be repeated here.

[0109] Step S40: Generate unique identifiers for the integrated model and each of the standardized models based on the common identifier fields in each of the source field sets;

[0110] In this embodiment, since the integration is performed on similar data from various data sources, each source field set contains a common identifier field. An ID value is generated based on this identifier field, and the ID value is used as a unique identifier for an entity record in each source standardized model and integrated model.

[0111] Step S50: Based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, select the integration field value of each integration field from the standard values ​​to integrate each data source into the integration model.

[0112] In this embodiment, in order to achieve faster, more complete and better data, this application designs a source selection model based on the actual data performance of each data source and business preferences, which is used to solve the problem of how to extract values ​​from the standardized model for each field in the integration model.

[0113] Specifically, the source selection model is as follows: Figure 4 As shown, it includes a pre-defined model and a post-monitoring model. The pre-defined model is divided into weighted source selection and auxiliary source selection. The auxiliary source selection methods include first-in-first-out, first non-empty source, and specified data source. The post-monitoring model includes voting source selection. Technical personnel can configure the source selection of each integration field in the integration model according to the above source selection methods. Then, each integration field selects an appropriate field value as the integration value in each standardized model according to the source selection configuration, thereby integrating the various data sources into the integration model.

[0114] In this embodiment, compared to the existing technology that integrates multiple data sources by using a single data source as the main source and other data sources as auxiliary sources, this application proposes a multi-source external data integration method based on a unified data model. First, the source field sets and source field values ​​of each data source are obtained. Then, the source fields and source data of each data source are standardized to obtain a standardized model corresponding to each data source. A unique identifier is generated through the common identifier fields in each source field set, avoiding interference from other different types of data sources during the source selection process of the integration model. Then, in each standardized model, the integration field value of each integration field in the integration model is selected according to the source selection configuration of each field set by the technician, so as to integrate the data of each data source into the integration model.

[0115] Thus, this application's method of standardizing multi-source data and then selecting sources for each field in each standardized model, compared to the traditional approach of integrating multi-source data with a single data source as the primary source and other data sources as secondary sources, does not distinguish between primary and secondary sources and selects sources on a field-by-field basis. On the one hand, it can make optimal data source selection at the field level, thereby improving the field value rate of the integrated model; on the other hand, it can dynamically select sources at the field level based on the actual data performance and application preferences of each data source, thereby fully exploring and utilizing the advantages of each data source.

[0116] Furthermore, based on the first embodiment of the multi-source external data integration method of this application described above, a second embodiment of the multi-source external data integration method of this application is proposed.

[0117] In a second embodiment of the multi-source external data integration method of this application, the source selection configuration includes: a pre-agreed model, which includes: weighted source selection;

[0118] In step S50 above, the step of selecting the respective integration field value of each integration field from the respective standard values ​​includes:

[0119] Step A10: If there is a first field in each of the integrated fields that is configured as a weighted source selection field, then obtain the weight parameters of each of the standardized models corresponding to the first field.

[0120] Step A20: If there is a target weight among the weight parameters that is greater than a preset weight threshold, then determine the first standardized model corresponding to the target weight.

[0121] Step A30: Take the standard value of the first standard field corresponding to the first field in the first standardized model as the integrated field value of the first field.

[0122] In this embodiment, if one or more of the integration fields are configured as the first field for weighted source selection, then when data analysis or application experience indicates that a certain data source has a significant advantage or a clear business application tendency in these first fields, the data source can be preferentially selected through weight settings. That is, the first standardized model corresponding to the data source that has an advantage in the first field is set to a weight greater than the weight threshold in the weight corresponding to the first field. When selecting a source for the first field, the first standardized model will be preferentially selected. The standard value of the first standard field corresponding to the first field in the first standardized model is used as the target value, and then the target value is assigned to the integration field value of the first field to complete the source selection for the first field. In addition, when the optimal data source cannot be clearly identified, that is, when the weight parameters of each standardized model corresponding to the first field are not greater than the weight threshold, the source selection can be assisted by the actual performance of the data. The assisted source selection will be explained in the following embodiments.

[0123] For example, in the integrated model, the source selection configuration for the d field is weighted source selection. If data analysis or application experience determines that data source D has an advantage in the d field, the weight of data source D in the d field is set to 1, and the weight of other data sources in the d field is set to 0.1. The preset weight threshold is set to 0.8. At this time, the weight of data source D in the d field is greater than the weight threshold. Then, when assigning a value to the d field in the integrated model, the field value of the d field in the standardized model D corresponding to data source D is selected as the target value according to the weight. Finally, the target value is assigned to the d field in the integrated model, thus completing the source selection for the d field in the integrated model.

[0124] Thus, compared to data integration methods that rely on a single data source, this application can dynamically select sources at the field level based on the data application preferences of each data source, thereby fully exploring and utilizing the advantages of each data source.

[0125] Furthermore, based on the first and second embodiments of the multi-source external data integration method of this application described above, a third embodiment of the multi-source external data integration method of this application is proposed.

[0126] In the third embodiment of the multi-source external data integration method of this application, the pre-agreed model further includes: auxiliary source selection, which includes: first-in, first-out;

[0127] In step S50 above, the step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0128] Step B10: If there is a second field in each of the integration fields that is configured as a first-in, first-out source selection field, then the second standardization model is determined according to the last update time of each of the standardization models.

[0129] Step B20: Take the standard value of the second standard field corresponding to the second field in the second standardization model as the integrated field value of the second field.

[0130] In this embodiment, if the integration field of the integration model contains a second field with the source selection configured as first-in-first-out, it indicates that the second field places more emphasis on timeliness. The integration field value of the second field is based on the latest data source in the data source, and the standard value of the standard field corresponding to the second field in the standardized model corresponding to the latest data source is used as the target value, and the target value is assigned to the second field.

[0131] For example, if the source selection configuration for field e in the integration field is first-in, and data source D is the latest updated data source, then the standard value of field e in the standardized model D corresponding to data source D is selected as the target value, and then the target value is assigned to field e in the integration field to complete the source selection of field e in the integration model.

[0132] Thus, compared to data integration methods that rely on a single data source, this application can dynamically select sources at the field level based on the timeliness of each data source, thereby ensuring the timeliness of the integrated data.

[0133] Furthermore, based on the first, second, and third embodiments of the multi-source external data integration method of this application described above, a fourth embodiment of the multi-source external data integration method of this application is proposed.

[0134] In the fourth embodiment of the multi-source external data integration method of this application, the auxiliary source selection includes: the first non-empty source;

[0135] In step S50 above, the step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0136] Step C10: If there is a third field in each of the integrated fields that is configured as the first non-empty source, then determine the third standard field corresponding to the third field in each of the standardized models.

[0137] Step C20: Determine each non-empty standardized model in which the standard value of the third standard field in each of the standardized models is not empty;

[0138] Step C30: Determine the third normalization model based on the last update time of each of the aforementioned non-empty normalization models;

[0139] Step C40: Use the standard value of the third standard field in the third standardization model as the integrated field value of the third field.

[0140] In this embodiment, if the integration field of the integration model contains a third field configured as the first non-empty source, it indicates that the third field pays more attention to the integrity of the field. When selecting a source for the third field, the latest source in the data source sorted by update time, i.e. the first non-empty source, is selected. The standard value of the third standard field corresponding to the third field in the first non-empty source is used as the target value and assigned to the third field.

[0141] For example, if the source selection configuration of field f in the integration field is the first non-empty source, and data source E is the latest data source, but the field value of field f in the normalized model E corresponding to data source E is empty, then data source E is excluded. After exclusion, data source F is the latest data source, and the field value of field f in the normalized model F is not empty. Then, the field value of field f in the normalized model F is selected as the target value, and the target value is then assigned to field f in the integration field to complete the source selection of field f in the integration model.

[0142] Alternatively, when the source selection configuration for field f in the integration field is set to the first non-empty source, first exclude the standardized models in each standardized model whose field f is empty, then select the field value of field f in the latest standardized model F after exclusion as the target value, and then assign the target value to field f in the integration field to complete the source selection for field f in the integration model.

[0143] Thus, compared to data integration methods that rely on a single data source, this application can dynamically select the first non-empty data source among the various data sources at the field level, thereby ensuring the integrity of the integrated data.

[0144] Furthermore, based on the first to fourth embodiments of the multi-source external data integration method of this application described above, a fifth embodiment of the multi-source external data integration method of this application is proposed.

[0145] In the fifth embodiment of the multi-source external data integration method of this application, the auxiliary source selection includes: specifying a data source;

[0146] In step S50 above, the step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes:

[0147] Step D10: If there is a fourth field in each of the integration fields that is configured as a specified data source, then determine the fourth standardized model corresponding to the specified data source according to the source configuration.

[0148] Step D20: Take the standard value of the fourth standard field corresponding to the fourth field in the fourth standardization model as the integrated field value of the fourth field.

[0149] In this embodiment, if the source of a certain field in the integrated data is only a single data source, the data source can be directly specified. If there is a fourth field in the integrated field of the integrated model that is configured as a specified data source, the standard value of the standard field corresponding to the fourth field in the standardized model corresponding to the specified data source is selected as the target value and assigned to the fourth field.

[0150] For example, if only data source G includes field g among all data sources, then the integrated field value of the integrated field g can be directly specified as the source field value of field g, thereby reducing the computational load of the method.

[0151] Furthermore, based on the first to fifth embodiments of the multi-source external data integration method of this application described above, a sixth embodiment of the multi-source external data integration method of this application is proposed.

[0152] In the sixth embodiment of the multi-source external data integration method of this application, the source selection configuration further includes: a post-event monitoring model, wherein the post-event monitoring model includes: voting for source selection;

[0153] Following step S50 above, where the method selects the respective integrated field value of each integrated field from the standard values, the method further includes:

[0154] Step S60: When integrating three or more data sources, if there is a fifth field in each of the integration fields that is configured as a voting source selection field, and the integration field value of the fifth field has not been updated within a preset time threshold, then the fifth standard field corresponding to the fifth field in each of the standardization models is determined.

[0155] Step S70: Determine the mode among the standard values ​​of each of the fifth standard fields, and update the integrated field value of the fifth field based on the mode.

[0156] In this embodiment, after the source selection is agreed upon in advance, if a certain field of a record of a certain entity from multiple sources has not been updated for a long time, the field value of the integration model is re-determined through a voting process, with the minority obeying the majority principle. The post-monitoring model is only applicable to the integration of three or more data sources.

[0157] For example, if the h field in the integration field has not been updated for more than a week, the standard values ​​of the h field in each standardized model are detected. If the standard values ​​of the h field in standardized model H, standardized model I, and standardized model J are detected to be 1, 2, and 2 respectively, the integration field value of the h field in the integration field is updated to 2.

[0158] Thus, compared to data integration methods that rely on a single data source, this application can further update the integrated data through a voting-based source selection process, thereby improving the accuracy of the integrated data.

[0159] Furthermore, based on the first to sixth embodiments of the multi-source external data integration method of this application described above, a seventh embodiment of the multi-source external data integration method of this application is proposed.

[0160] In the seventh embodiment of the multi-source external data integration method of this application, after step S40 above, which generates the integration model and the unique identifier of each of the standardized models based on the common identifier fields in each of the source field sets, the method further includes:

[0161] The mapping relationship between the integrated model and each of the standardized models is determined based on the unique identifier.

[0162] In this embodiment, ONEID mapping is used, which involves identifying a common identifying field across multiple sources and generating an ID value based on that field. This ID can serve as a unique identifier for an entity record in both the standardized model and the integrated model, thus forming a many-to-one mapping relationship between the multi-source standardized model and the integrated model.

[0163] Specifically, the structure of the final integrated model based on this mapping relationship is as follows: Figure 5 As shown, the standardized model obtained after standardizing each data source is based on ONEID mapping to a corresponding ONEID entity record. After selecting the source configuration, a multi-data source integration model can be obtained.

[0164] The aforementioned technical methods can be used to identify the many-to-one data relationship between each standardized model and the integrated model.

[0165] In addition, please refer to Figure 6 , Figure 6This is a functional block diagram of the multi-source external data integration device of this application. This application also provides a multi-source external data integration device, which includes:

[0166] The source data preparation module 10 is used to obtain the source field set of each data source and the source field value of each source field in the source field set in real time.

[0167] Field standardization module 20 is used to determine the standard field set of the standardization model corresponding to each of the source field sets, and the integrated field set of the integrated model, based on the union of the source field sets.

[0168] The field value standardization module 30 is used to standardize the values ​​of each of the source fields to obtain the standard values ​​of each standard field in the standard field set.

[0169] The unique identifier module 40 is used to generate unique identifiers for the integrated model and each of the standardized models based on the common identifier fields in each of the source field sets;

[0170] The source selection and integration module 50 is used to select the integration field value of each integration field from the standard values ​​based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, so as to integrate each data source into the integration model.

[0171] Optionally, the source selection and integration module includes:

[0172] The weighted source selection unit is used to obtain the weight parameters of each of the standardized models corresponding to the first field if there is a first field configured as a weighted source selection field among the integrated fields; if there is a target weight among the weight parameters that is greater than a preset weight threshold, then determine the first standardized model corresponding to the target weight; and take the standard value of the first standard field corresponding to the first field in the first standardized model as the integrated field value of the first field.

[0173] Optionally, the source selection and integration module also includes:

[0174] The first-in-first-out unit is used to determine the second standardized model based on the last update time of each standardized model if there is a second field in each of the integrated fields that is configured as first-in-first-out source selection; and to use the standard value of the second standard field corresponding to the second field in the second standardized model as the integrated field value of the second field.

[0175] Optionally, the source selection and integration module also includes:

[0176] The first non-empty source unit is used to determine the third standard field corresponding to the third field in each of the standardized models if there is a third field in each of the integrated fields that is configured as the first non-empty source; determine each non-empty standardized model in each of the standardized models whose standard value of the third standard field is not empty; determine the third standardized model according to the last update time of each of the non-empty standardized models; and use the standard value of the third standard field in the third standardized model as the integrated field value of the third field.

[0177] Optionally, the source selection and integration module also includes:

[0178] A designated data source unit is used to determine the fourth standardized model corresponding to the designated data source based on the source selection configuration if there is a fourth field in each of the integration fields that is configured as a designated data source; and to use the standard value of the fourth standard field corresponding to the fourth field in the fourth standardized model as the integration field value of the fourth field.

[0179] Optionally, the multi-source external data integration device also includes:

[0180] The post-event monitoring module is used to determine the fifth standard field corresponding to the fifth field in each of the standardized models when integrating three or more data sources. If there is a fifth field in each of the integrated fields that is configured as a voting source selection field, and the integrated field value of the fifth field has not been updated within a preset time threshold, then the module determines the mode of the standard values ​​of each of the fifth standard fields and updates the integrated field value of the fifth field based on the mode.

[0181] Optionally, the multi-source external data integration device also includes:

[0182] A mapping unit is used to determine the mapping relationship between the integrated model and each of the standardized models based on the unique identifier.

[0183] The specific implementation of the multi-source external data integration device in this application is basically the same as the embodiments of the multi-source external data integration method described above, and will not be repeated here.

[0184] Furthermore, this application also proposes a storage medium storing a program for integrating multi-source external data, which, when executed by a processor, implements the steps of the multi-source external data integration method of this application as described above.

[0185] The specific embodiments of the computer storage medium in this application are basically the same as the embodiments of the multi-source external data integration method described above, and will not be repeated here.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0187] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0189] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for integrating multi-source external data, characterized in that, The multi-source external data integration method includes: Real-time acquisition of the source field sets of each data source and the source field values ​​of each source field in the source field sets; The standard field set of the standardized model corresponding to each of the source field sets is determined based on the union of the source field sets, and the integrated field set of the integrated model is determined based on the union of the source field sets. The values ​​of each source field are standardized to obtain the standard values ​​of each standard field in each set of standard fields. The integrated model and each standardized model are generated based on the common identifying fields in each of the source field sets; Based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, the integration field value of each integration field is selected from each of the standard values ​​to integrate each of the data sources into the integration model; The source selection configuration includes: a pre-defined model, which includes: weighted source selection; the step of selecting the integration field value of each integration field from each of the standard values ​​includes: If any of the integration fields contains a first field configured as a weighted source selection field, then obtain the weight parameters of each of the standardized models corresponding to the first field. If any of the weight parameters contains a target weight greater than a preset weight threshold, then the first standardized model corresponding to the target weight is determined. The standard value of the first standard field corresponding to the first field in the first standardized model is used as the integrated field value of the first field.

2. The multi-source external data integration method according to claim 1, characterized in that, The pre-agreed model also includes: auxiliary source selection, which includes: first-come-first-served; The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes: If any of the integration fields has a second field configured as first-in-first-out for source selection, then the second standardized model is determined based on the last update time of each standardized model. The standard value of the second standard field corresponding to the second field in the second standardized model is used as the integrated field value of the second field.

3. The multi-source external data integration method according to claim 2, characterized in that, The auxiliary source selection includes: the first non-empty source; The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes: If any of the integration fields contains a third field configured as the first non-empty source, then the third standard field corresponding to the third field in each of the standardization models is determined. Identify each non-empty standardized model in which the standard value of the third standard field in each of the standardized models is not empty; The third normalization model is determined based on the last update time of each of the aforementioned non-empty normalization models; The standard value of the third standard field in the third standardization model is used as the integrated field value of the third field.

4. The multi-source external data integration method according to claim 2, characterized in that, The auxiliary source selection includes: specifying a data source; The step of selecting the respective integration field value of each integration field from the respective standard values ​​further includes: If any of the integration fields contains a fourth field configured as a specified data source, then the fourth standardized model corresponding to the specified data source is determined according to the specified data source configuration. The standard value of the fourth standard field corresponding to the fourth field in the fourth standardization model is used as the integrated field value of the fourth field.

5. The multi-source external data integration method according to claim 1, characterized in that, The source selection configuration also includes: a post-event monitoring model, which includes: a voting source selection; After the step of selecting the respective integrated field value of each of the said standard values, the method further includes: When integrating three or more data sources, if there is a fifth field in each of the integration fields that is configured as a voting selection field, and the integration field value of the fifth field has not been updated within a preset time threshold, then the fifth field is determined to be the fifth standard field corresponding to each of the standardization models. Determine the mode among the standard values ​​of each of the fifth standard fields, and update the integrated field value of the fifth field based on the mode.

6. The method for integrating multi-source external data according to any one of claims 1-5, characterized in that, After the step of generating unique identifiers for the integrated model and each of the standardized models based on the common identifying fields in each of the source field sets, the method further includes: The mapping relationship between the integrated model and each of the standardized models is determined based on the unique identifier.

7. A multi-source external data integration device, characterized in that, The multi-source external data integration device includes: The source data preparation module is used to obtain the source field set of each data source and the source field value of each source field in the source field set in real time; The field standardization module is used to determine the standard field set of the standardization model corresponding to each of the source field sets, and the integrated field set of the integrated model, based on the union of the source field sets. The field value standardization module is used to standardize the values ​​of each of the source fields to obtain the standard values ​​of each standard field in the standard field set. The unique identifier module is used to generate unique identifiers for the integrated model and each of the standardized models based on the common identifier fields in each of the source field sets; The source selection and integration module is used to select the integration field value of each integration field from the standard values ​​based on the unique identifier and the preset source selection configuration of each integration field in the integration field set, so as to integrate the data sources into the integration model. The source selection configuration includes: a pre-agreed model, which includes: weighted source selection; the source selection integration module includes: The weighted source selection unit is used to obtain the weight parameters of each of the standardized models corresponding to the first field if there is a first field configured as a weighted source selection field among the integrated fields; if there is a target weight among the weight parameters that is greater than a preset weight threshold, then determine the first standardized model corresponding to the target weight; and take the standard value of the first standard field corresponding to the first field in the first standardized model as the integrated field value of the first field.

8. A multi-source external data integration device, characterized in that, The multi-source external data integration device includes: a memory and a processor, wherein the memory stores a multi-source external data integration program, and when the multi-source external data integration program is executed by the processor, it implements the steps of the multi-source external data integration method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a multi-source external data integration program, which, when executed by a processor, implements the steps of the multi-source external data integration method as described in any one of claims 1 to 6.

Images