Heterogeneous data integration method, computer device and medium for off-site auditing

Abstract

The present application relates to the technical field of audit data integration, and specifically relates to a heterogeneous data integration method for off-site audit, a computer device and a medium. For different data groups in a single data type, the method identifies data groups with missing collection and combines the data groups with missing collection, and identifies data groups with repeated collection and combines the data groups with repeated collection, to obtain merged data corresponding to each data type. Based on the merged data, the method performs natural language processing and one-hot encoding on data of different data types, forms column vector combinations between different data categories of each data type, analyzes the matching degree between any two column vectors in the column vector combinations corresponding to different data types, obtains matching column vectors of the column vectors, and completes the integration between the same data categories in different data types. When performing off-site audit on the integrated data, data classification analysis can be more convenient, and the audit efficiency and accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of auditing data integration, in particular to a heterogeneous data integration method for off-site auditing, a computer device and a medium. BACKGROUND

[0002] Insurance companies are a high-risk industry and also a highly electronic industry. With the development of the industry, the types of business are increasing, the complexity of business processes is increasing, and transaction information and management information are expanding, which may lead to errors or fraud information and operating efficiency that may be hidden in various data information.

[0003] Traditional auditing methods mainly based on on-site and manual operations have been unable to face the huge amount of data in the insurance industry, and have been unable to find hidden audit clues in massive data, with the disadvantages of insufficient coverage, time lag, etc., and massive data information is prone to data duplication, data omission, etc. Off-site auditing refers to the use of computers as a means and database technology as a basis to continuously collect, organize data and information of business operation and management of the audited object, use queries and filters, record and analyze the data found to be abnormal or worthy of attention, and timely find problems, assess the risk status of the audited object, and monitor the operating status and development trend of the audited object. Therefore, the use of off-site auditing methods for auditing insurance industry data has good development prospects. SUMMARY

[0004] In order to solve the above technical problems, the purpose of the present application is to provide a heterogeneous data integration method for off-site auditing, a computer device and a medium.

[0005] According to the first aspect of the embodiment of the present application, a heterogeneous data integration method for off-site auditing is provided, and the technical solution adopted is as follows:

[0006] Data is collected based on multiple data sources to obtain data types corresponding to each data source, wherein each data type includes multiple data groups;

[0007] For different data groups in a single data type, identify data groups with missing collection and combine the data groups with missing collection, and identify data groups with repeated collection and combine the data groups with repeated collection, to obtain merged data corresponding to each data type;

[0008] Based on the merged data, natural language processing and one-hot encoding are performed on data of different data types to form a column vector combination between different data categories of each data type, and the matching degree between any two column vectors in the column vector combination corresponding to different data types is analyzed to obtain a matching column vector of the column vector, and the integration between the same data categories in different data types is completed.

[0009] In some embodiments of the present application, the method for identifying the existence of a data set with missing data includes:

[0010] Determining the number of complete data of the data set;

[0011] When the number of data collected in the data set is less than the number of complete data, marking the data set to obtain all incomplete data sets.

[0012] In some embodiments of the present application, the method for merging data sets with missing data includes:

[0013] Determining the number of the same data in any two different incomplete data sets;

[0014] Analyzing the relationship between the number of the same data and the number of complete data to obtain the data similarity between any two different incomplete data sets;

[0015] According to the data similarity, the data sets with missing data are merged.

[0016] In some embodiments of the present application, according to the data similarity, the data sets with missing data are merged, including:

[0017] Setting a data similarity threshold;

[0018] Determining whether the data similarity is greater than the data similarity threshold;

[0019] If yes, analyzing the distribution of different data in the two incomplete data sets, and if the distribution of different data in the two incomplete data sets has no intersection, merging the two incomplete data sets.

[0020] In some embodiments of the present application, identifying the data set with repeated collection and merging the data set with repeated collection includes:

[0021] Determining the importance of data in each data set, the importance including unimportant data, and obtaining the number relationship of the unimportant data;

[0022] analyzing the similarity of two different data groups in a single data type, combining the importance of the data and the quantity relationship of the unimportant data, and obtaining the mergability of the data in the two different data groups;

[0023] combining the repeatedly collected data according to the mergability.

[0024] In some embodiments of the present application, combining the repeatedly collected data according to the mergability comprises:

[0025] setting a mergability threshold;

[0026] determining whether the mergability is greater than the mergability threshold;

[0027] if yes, merging the data in the data group with the data of the corresponding data category in another data group.

[0028] In some embodiments of the present application, based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to form a column vector combination between different data categories of each data type, the matching degree between any two column vectors in the column vector combinations corresponding to different data types is analyzed, a matching column vector of the column vectors is obtained, and the integration between the same data categories in different data types is completed, comprising:

[0029] based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to convert the data of different data types into column vectors;

[0030] combining the column vectors corresponding to different data categories of each data type to form a column vector combination between different data categories of each data type;

[0031] analyzing the cosine similarity between each column vector in the column vector combination corresponding to a data source and any column vector in the column vector combination corresponding to another data source, obtaining the matching degree between any two column vectors in the column vector combinations corresponding to different data types;

[0032] according to the matching degree, completing the integration between the same data categories in different data types.

[0033] In some embodiments of the present application, according to the matching degree, the integration between the same data categories in different data types is completed, comprising:

[0034] setting a matching degree threshold;

[0035] determining whether the matching degree is greater than the matching degree threshold;

[0036] If yes, mark two column vectors as possible similar column vectors, and determine the column vector with the largest matching degree among all possible similar column vectors corresponding to each column vector as the matching column vector of the column vector;

[0037] Matching the column vector with the matching column vector corresponding thereto, the integration between the same data categories in different data types is completed.

[0038] According to a second aspect of the embodiment of the present application, a heterogeneous data integration computer device for off-site auditing is provided, and the computer device comprises:

[0039] a memory and a processor, wherein:

[0040] the memory is configured to store program code;

[0041] the processor is configured to read the program code stored in the memory and execute the method described above.

[0042] According to a third aspect of the embodiment of the present application, a computer readable storage medium having a computer program stored thereon is provided; the computer program is executed by a processor to implement the heterogeneous data integration method for off-site auditing according to the first aspect of the embodiment of the present application.

[0043] Compared with the prior art, the heterogeneous data integration method for off-site auditing, the computer device and the medium provided by the present application have the following beneficial effects:

[0044] The present application collects data based on multiple data sources to obtain data types corresponding to each data source, wherein each data type includes multiple data groups, thereby ensuring the integrity and accuracy of the original data. For different data groups in a single data type, the data groups with missing collection are identified and combined, and the data groups with repeated collection are identified and combined, thereby obtaining merged data corresponding to each data type, correcting the missing and repeated collection data, reducing the data volume, and making the collection result more accurate and facilitating subsequent calculation. Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to form column vector combinations between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combinations corresponding to different data types, obtain matching column vectors of the column vectors, complete the integration between the same data categories in different data types, thereby facilitating data classification according to data types, realizing data resource integration, facilitating subsequent off-site auditing analysis, and improving auditing efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, below will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A basic flowchart of a heterogeneous data integration method for off-site auditing provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined purpose, the following describes the specific embodiments, structure, features and effects of a heterogeneous data integration method for off-site auditing, computer equipment and medium according to the present application, in combination with the drawings and preferred embodiments. Different "one embodiment" or "another embodiment" in the following description do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms such as "comprise", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the circuit structure, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such article or device. Without more limitation, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the article or device comprising the element.

[0049] In the traditional auditing process, a large amount of data needs to be analyzed. For the staff, not only error data needs to be found, but also all data needs to be analyzed to find audit clues, which is difficult. The present application processes each data group of each data type of all data sources in the auditing process, and completes the merging of different data groups in each data type. Then, the cosine similarity of the data types on each data source is analyzed, and the integration of similar data types on different data sources is completed. The integrated data is used as the initial data for off-site auditing, which facilitates classification analysis and completes off-site auditing.

[0050] The specific scheme of the heterogeneous data integration method for off-site auditing provided by the present application will be described in detail below in combination with the drawings.

[0051] Please refer to Figure 1Figure 1 shows a basic flow of a heterogeneous data integration method for off-site audit provided by one embodiment of the present application.

[0052] As shown in Figure 1, one embodiment of the present application provides a heterogeneous data integration method for off-site audit, which specifically includes the following steps. Figure 1

[0053] S100: Data collection based on multiple data sources to obtain data types corresponding to each data source, wherein each data type includes multiple data groups.

[0054] Firstly, the data sources required by off-site audit need to be determined according to the analysis target. During the collection process, the integrity and accuracy of the data need to be ensured to avoid data loss and errors. For example, the main analysis target of the present embodiment is the overall business situation of the business department and the data characteristics of the customers, so the data at the business level and the data at the customer level need to be used for analysis. The data analysis at the business level is mainly through the analysis of business trends and profit models, etc., which is used to grasp the overall business situation of the business department and assist in determining the audit scheme and the audit focus. The data analysis at the customer level is mainly used to find audit clues and obtain the data characteristics of the customers for screening and classification of the customers. Therefore, firstly, the data at the business level and the data at the customer level corresponding to multiple different data sources are determined. Then, data collection is performed based on multiple data sources to obtain data types corresponding to each data source, wherein each data type includes multiple data groups.

[0055] Each data type corresponding to each data source may be different, so the sequence composed of the data types corresponding to all data sources is:

[0056] P I ={P1,P2…P i …P I}

[0057]

[0058] In the formula, P I represents a sequence composed of I data types corresponding to I data sources; P i represents a data vector of the i-th data type (the i-th data source), wherein i∈[1,I], I represents the total number of data types (data sources); x i,n represents the n-th data category in the i-th data type, wherein n∈[1,N i ], N i represents the number of data categories in the i-th data type.

[0059] ​It should be noted that the I data types corresponding to the I data sources in the above sequence can be completely the same data types, but no merging is performed here, and the most original data is retained. The data type obtained by a data source can be a (name, ID number, gender, age) sequence, so there are four data categories in the data type, namely name, ID number, gender, and age.

[0060] At this point, the data collection in all data types of different data sources is completed.

[0061] Non-site audit involves a large amount of data, not all of which are necessarily normal data, and there is a possibility that some data will be missed or repeatedly collected. Since the data in all data types in each data source is consistent, the data in the same data source is analyzed in detail to identify data with abnormal problems, and the data is processed under the corresponding conditions, so that the data in each data source is more accurate; further, by analyzing the similarity of the data types in different data sources, the integration of data between similar data types in different data sources is completed.

[0062] Therefore, the process of data integration for all data sources includes steps S200 and S300.

[0063] S200: For different data groups in a single data type, identify data groups that are missing and merge the data groups that are missing, and identify data groups that are repeatedly collected and merge the data groups that are repeatedly collected, to obtain the merged data corresponding to each data type.

[0064] The heterogeneous data sources involved in non-site audit can come from multiple different systems or platforms, and these systems can each store information about customers. When this information is integrated into the non-site audit system, if there is a mechanism for data sharing or data synchronization between different systems, it can happen that the same customer information is repeatedly obtained. In addition, in the process of heterogeneous data integration, data integration is a relatively key step, but due to differences in data format, storage method, and update frequency between different systems, there can be problems of repeated information or inconsistency in the data synchronization and integration process. Therefore, these repeated data need to be screened, and at the same time, the data format is made more regular, facilitating subsequent ETL format conversion.

[0065] In the data collection process, due to the large amount of data information collected, there can be errors such as missing data collection and repeated data collection, which need to be corrected to appropriately reduce the amount of data and also make the collection results more accurate, facilitating subsequent calculations.

[0066] Based on the above analysis, the embodiment identifies the data groups with missing data and merges the data groups with missing data, and identifies the data groups with repeated data and merges the data groups with repeated data, to obtain the merged data corresponding to each data type.

[0067] Further, the method of identifying the data groups with missing data includes determining the number of complete data of the data groups, and marking the data groups when the number of collected data in the data groups is less than the number of complete data, to obtain all the incomplete data groups.

[0068] The method of merging the data groups with missing data includes determining the number of same data in any two different incomplete data groups, analyzing the relationship between the number of same data and the number of complete data to obtain the data similarity between any two different incomplete data groups, and merging the data groups with missing data according to the data similarity. Further, the method of merging the data groups with missing data according to the data similarity includes setting a data similarity threshold, judging whether the data similarity is greater than the data similarity threshold, analyzing the distribution of different data in the two incomplete data groups if the data similarity is greater than the data similarity threshold, and merging the two incomplete data groups if the distribution of different data in the two incomplete data groups has no intersection, and otherwise, indicating that the possibility of similarity between the two incomplete data groups is low, and not further analyzing the two incomplete data groups.

[0069] The identification of the data groups with missing data and the merging of the data groups with missing data are specifically illustrated as follows: first, analyzing the data in a data type corresponding to a single data source, and for the data in the data source, the data is present in a database, and the number of labels on the database table header can represent the complete data amount. For example, the data type obtained by the data source is (name, ID number, gender, age) sequence, and the corresponding complete data amount is 4, i.e. the complete data amount is equal to the total number of data categories in the data type, thereby determining the complete data amount N i in the ith data type. The number of actually collected data n i,k of the kth data group P i,k in the ith data type is analyzed; when n i,k =N i , it indicates that the data group collects complete data; when n i,k <N i , it indicates that the data group collects incomplete data, and is marked to identify all the data groups with missing data and obtain all the incomplete data groups.

[0070] Then, the obtained incomplete data sets are analyzed to determine whether there is a similar relationship between the incomplete data sets, and the incomplete data sets are integrated through the similar relationship. The specific operation is as follows: it is determined that the number of same data in the vth incomplete data set and the hth incomplete data set is N v ,h The data similarity calculation formula between the vth incomplete data set and the hth incomplete data is constructed as follows:

[0071]

[0072] In the formula, γ v,n represents the data similarity between the vth incomplete data set and the hth incomplete data; N v ,h represents the number of same data in the vth incomplete data set and the hth incomplete data set; N i represents the number of complete data of the kth data set P i,k in the ith data type.

[0073] The data similarity threshold is set to 0.7. When γ v,n <0.7, it indicates that the possibility of similarity between the two incomplete data sets is low, and no further analysis is performed. When γ v,n ≥0.7, it indicates that the two incomplete data sets may be similar, and further consideration is needed. When γ v,n ≥0.7, the distribution of different data between the two incomplete data sets is analyzed. Due to the missing collection, there is no specific data under some data categories. When there is no intersection between the data categories corresponding to the remaining different data of the two incomplete data sets, the two incomplete data sets are combined into one data set. For example, the first data set: name A, age 3, gender no, birth date 2021.3; the second data set: name A, age 3, gender male, birth date no. For the first and second data sets, excluding the same data, the data categories corresponding to the different data are gender and birth date, and there is no intersection, so the first data set and the second data set can be combined to form a merged data set: name A, age 3, gender male, birth date 2021.3.

[0074] The above steps are repeated to complete the integration of all incomplete data sets.

[0075] ​​Furthermore, identifying and merging repeatedly collected data groups includes: determining the importance of data in each data group, where importance includes unimportant data, and obtaining the quantitative relationship of the unimportant data; analyzing the similarity between two different data groups within a single data type, and combining the importance of the data with the quantitative relationship of the unimportant data to obtain the merging feasibility of data in the two different data groups; and merging the repeatedly collected data groups based on the merging feasibility. Even further, merging the repeatedly collected data groups based on the merging feasibility includes: setting a merging feasibility threshold; determining whether the merging feasibility is greater than the merging feasibility threshold; and if so, merging the data in one data group with data of the corresponding data category in another data group.

[0076] A specific example illustrates this: Further analysis is performed based on the completion of incomplete data combination and merging. For each data point, its importance varies, and the nth data point in the i-th data type is assigned a value x. i,n The importance of The assignment of importance should be considered in conjunction with realistic factors. For example, for customer information, gender and age are generally more important than place of origin and education level. When all data in the i-th data type are equally important, Mark this data as important; when data of varying importance exist, assign values ​​to these data based on real-world factors. Mark this data as unimportant; This data is marked as very important. Within the i-th data type, there are some data with lower importance. When these two data points are similar and other data is identical, they can be merged.

[0077] Construct the k-th data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The formula for calculating the mergeability of the corresponding nth data is:

[0078]

[0079] In the formula, MP i,k,n P represents the k-th data group in the i-th data type. i,k The nth data and the lth data group P in the i-th data type i,+ The mergibility of the corresponding nth data; N i x represents the total number of data categories in the i-th data type; i,k,n P represents the k-th data group in the i-th data type. i,k The value of the nth data; x i,+,nThe value of the nth data of the kth data group P i,+ in the ith data type. The importance of the nth data of the kth data group P i,k in the ith data type. i,4 The number of data in the kth data group P i,k in the ith data type whose importance is less than Add 0.1 to prevent the denominator from being 0.

[0080] .x i,k,n -x i,+,n The absolute value of the difference between the value of the nth data of the kth data group P i,k in the ith data type and the value of the nth data of the lth data group P i,+ in the ith data type, the smaller the value, the smaller the difference between the corresponding data in the two data groups, and the greater the possibility of merging the data in the two data groups. i,k The importance of the nth data of the kth data group P in the ith data type, the greater the value, the more important the data, and the smaller the possibility of merging the data with other data. The data of all data categories in the ith data type is traversed, and the average value is calculated. The ratio of unimportant data to the total number of data in the kth data group P i,k in the ith data type, i.e., the number relationship of unimportant data, the greater the value, the more unimportant data in the data group, and the greater the possibility of merging the data in the data group.

[0081] Set the mergability threshold to 0.98, if MP i,k,n > 0.98, it means that the nth data of the kth data group P i,k in the ith data type can be merged with the corresponding nth data of the lth data group P i,+ in the ith data type; if MP i,k,n ≤ 0.98, it means that the nth data of the kth data group P i,k in the ith data type cannot be merged with the corresponding nth data of the lth data group P i,+ in the ith data type.

[0082] All data in any data group in all data types are traversed to complete the merging and integration of all duplicate data, and finally the merged data corresponding to each data type is obtained. Generally, the merging method is addition or coverage, and the specific method can be allocated according to the actual meaning of the data type.

[0083] S300: Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to form a column vector combination between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combination corresponding to different data types, obtain the matching column vector of the column vector, and complete the integration between the same data categories in different data types.

[0084] The data after merging in each data type is obtained through step S200. For heterogeneous data of off-site audit, by analyzing the specific content of the data, the same type of data can be integrated, and then the format is converted through ETL. The auditors can quickly determine the audit target by using the integrated data, and it is convenient to help analyze the possible problems of different customers from various feature directions.

[0085] For different data types, there may be the same data category, for example, one data type is (name, age, date of birth), and another data type is (name, place of origin, ID number). Two data types are different, but they have the same data category name. Therefore, the two different data categories obtained from two different data sources can be matched with each other, thereby facilitating data classification according to data types.

[0086] Based on the above analysis, the embodiment is based on the merged data, and natural language processing and one-hot encoding are performed on data of different data types to form column vector combinations between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combinations corresponding to different data types, obtain matching column vectors of the column vectors, and complete the integration between the same data categories in different data types. Further, including: based on the merged data, natural language processing and one-hot encoding are performed on data of different data types to convert data of different data types into column vectors; the column vectors corresponding to different data categories of each data type are combined to form column vector combinations between different data categories of each data type; analyze the cosine similarity between each column vector in the column vector combination corresponding to one data source and any column vector in the column vector combination corresponding to another data source, obtain the matching degree between any two column vectors in the column vector combinations corresponding to different data types; according to the matching degree, complete the integration between the same data categories in different data types. Further, a matching degree threshold is set; it is judged whether the matching degree is greater than the matching degree threshold; if yes, mark two column vectors as possible similar column vectors, determine the column vector with the maximum matching degree among all possible similar column vectors corresponding to each column vector as the matching column vector of the column vector; match the column vector with the matching column vector corresponding thereto to complete the integration between the same data categories in different data types.

[0087] Specific examples are: based on the merged data, analyze the data collected by different data sources, and these data all carry specific text information; convert the text information into column vectors a i by natural language processing and one-hot encoding; combine the column vectors corresponding to different data categories in the same data type to form column vector combinations between different data categories of each data type, and obtain the customer information matrix of different data sources as:

[0088]

[0089] In the matrix, represents the column vector of the data of the N i th data category in the i i th data type; N i represents the number of data categories in the i

[0090] Calculate the cosine similarity between column vector and each column vector in the customer information matrix of the remaining data types:

[0091]

[0092] In the formula, denotes a column vector the cosine similarity of column vector ; COS denotes the cosine function; denotes a column vector the angle between column vector .

[0093] In the vector space, the smaller the angle between two vectors, the closer their directions, and the closer the cosine value is to 1, indicating that the two vectors are more similar; on the contrary, the larger the angle, the smaller the cosine value (even negative, indicating opposite directions), indicating that the two vectors are less similar; the smaller the angle cosine value between column vector and , the higher the similarity of the two column vectors, and the greater the matching degree of the corresponding two column vectors.

[0094] Set the matching degree threshold to 0.7, when , mark that the two column vectors may be similar, but in reality, only one data category in the two data types will successfully match one data category, so select in as a match, that is, column vector and are most likely to be the same category of data. Obtain the best matching result of column vector in data type , and similarly obtain the best matching result of other column vectors.

[0095] According to the above method, the best matching result of column vectors in all data types is obtained, and the mutual matching between data categories in the data types corresponding to each data source is completed, so as to facilitate data classification according to data types, realize data resource integration, and facilitate subsequent non-site audit analysis.

[0096] At this point, we have completed the data integration on all data sources.

[0097] Through the above process, data integration on all data sources is completed, and through specific analysis of the data of different data types after integration, audit clues are helped to be found, so as to realize detailed analysis and individual analysis of the target customer.

[0098] Specifically as follows:

[0099] Preliminary analysis is performed on the data after integration of all data sources, the overall business situation of the business department is grasped, and the audit scheme and audit focus are determined; at the same time, the data characteristics of the customer are obtained, and the customer is screened and classified;

[0100] According to the data analysis result of the customer level, the isolated data characteristics different from other normal customers are found, the possible problems are judged, and the target customer of the audit is determined;

[0101] All information related to the target customer is found, and the nature of the account and potential problems are comprehensively judged;

[0102] According to the individual problems of a single audited unit, some data analysis is flexibly carried out to find audit clues, locate audit targets, or find the essence of the problem by comprehensively analyzing the relationship between multiple audit targets.

[0103] Based on the same inventive concept as the above method, the embodiment also provides a heterogeneous data integration computer device for off-site audit, which comprises:

[0104] a memory and a processor, wherein:

[0105] the memory is used to store program code;

[0106] the processor is used to read the program code stored in the memory, and perform data collection based on multiple data sources to obtain data types corresponding to each data source, wherein each data type includes multiple data groups; for different data groups in a single data type, identify the data groups with missing collection and combine the data groups with missing collection, and identify the data groups with repeated collection and combine the data groups with repeated collection to obtain merged data corresponding to each data type; based on the merged data, perform natural language processing and one-hot encoding on the data of different data types to form column vector combinations between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combinations corresponding to different data types, obtain matching column vectors of the column vectors, and complete the integration between the same data categories in different data types.

[0107] Based on the same inventive concept as the above method, the embodiment also provides a computer readable storage medium, which stores a computer program; the computer program is executed by a processor to implement data collection based on multiple data sources, to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups; for different data groups in a single data type, a data group with missing collection is identified and combined, and a data group with repeated collection is identified and combined, to obtain combined data corresponding to each data type; based on the combined data, data of different data types is subjected to natural language processing and one-hot encoding, to form a column vector combination between different data categories of each data type, to analyze a matching degree between any two column vectors in the column vector combination corresponding to different data types, to obtain a matching column vector of the column vector, and to complete integration between the same data categories in different data types.

[0108] It should be noted that the above-mentioned sequence of the embodiments of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0109] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

Claims

1. A method for heterogeneous data integration for off-site auditing, comprising: The method comprises: data collection based on multiple data sources to obtain data types corresponding to each data source, wherein each data type comprises multiple data groups; for different data groups in a single data type, identifying data groups with missing collection and merging the data groups with missing collection, which comprises determining the number of same data in any two different incomplete data groups, analyzing the relationship between the number of same data and the number of complete data to obtain data similarity, and merging the two incomplete data groups when the data similarity is greater than a data similarity threshold and the distribution of different data in the two incomplete data groups has no intersection; identifying data groups with repeated collection and merging the data groups with repeated collection to obtain merged data corresponding to each data type, which comprises determining the importance of data in each data group, combining the similarity of two different data groups and the number relationship of unimportant data to obtain the mergability of data in the two different data groups, and merging the data groups with repeated collection according to the mergability; based on the merged data, performing natural language processing and one-hot encoding on data of different data types to convert the data of different data types into column vectors, combining column vectors corresponding to different data categories of each data type to form column vector combinations between different data categories of each data type, analyzing the cosine similarity between each column vector in the column vector combination corresponding to one data source and any column vector in the column vector combination corresponding to another data source to obtain the matching degree between any two column vectors in the column vector combination corresponding to different data types, marking two column vectors as possible similar column vectors when the matching degree is greater than a matching degree threshold, determining the column vector with the largest matching degree in all possible similar column vectors corresponding to each column vector as the matching column vector of the column vector, and matching the column vector with its corresponding matching column vector to complete the integration between the same data categories in different data types; Wherein, the first data type is constructed The first data of the first data group in the first data type is compared with the corresponding first data of the first data group in the second data type, and the formula for calculating the mergability is as follows: ； In the formula, represents the total number of data types; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; represents the total number of data groups in the first data type; 2. The heterogeneous data integration method for off-site auditing of claim 1, wherein, the method for identifying data groups with missing collection comprises: determining the number of complete data of a data group; when the number of collected data in the data group is less than the number of complete data, marking the data group to obtain all incomplete data groups.

3. The heterogeneous data integration method for off-site auditing of claim 1, wherein, merging the data groups with repeated collection according to the mergability comprises: setting a mergability threshold; determining whether the mergability is greater than the mergability threshold; if yes, merging the data in the data group with the data of the corresponding data category in another data group.

4. A heterogeneous data integration computer device for off-site auditing, comprising: The computer device comprises a memory and a processor, wherein: the memory is used to store program code; the processor is used to read the program code stored in the memory and execute the method according to any one of claims 1 to 3.

5. A computer readable storage medium, characterized in that, A computer program is stored thereon; the computer program is executed to implement the heterogeneous data integration method for non-site auditing according to any one of claims 1 to 3.

Images