Heterogeneous data integration method for off-site auditing, computer equipment and medium

Abstract

The invention relates to the technical field of audit data integration, in particular to a heterogeneous data integration method for off-site audit, computer equipment and a medium, and aims at different data groups in a single data type to identify data groups with missed collection and combine the data with missed collection to obtain a data combination; identifying data groups which are collected repeatedly and combining the data groups which are collected repeatedly to obtain combined data corresponding to each data type; based on the merged data, performing natural language processing and one-hot coding on the data of different data types to form a column vector combination between different data types of each data type, analyzing the matching degree between any two column vectors in the column vector combination corresponding to different data types to obtain a matched column vector of the column vectors, and performing one-hot coding on the matched column vector; and completing integration of the same data type in different data types. And when the integrated data is subjected to off-site auditing, data classification analysis can be more conveniently carried out, and the auditing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audit data integration, and in particular to a heterogeneous data integration method, computer equipment and medium for off-site audit. Background Art

[0002] Insurance companies are a high-risk industry and a highly electronic industry. With the development of the industry, the types of business continue to increase, the complexity of business processes continues to increase, and transaction information and management information continue to expand, resulting in erroneous or fraudulent information and operating efficiency conditions being hidden in various types of data information.

[0003] Traditional audit methods that mainly rely on on-site and manual operations are no longer able to cope with the huge amount of data in the insurance industry, let alone find hidden audit clues in the massive data. They have shortcomings such as insufficient coverage and delayed timeliness, and massive data information easily leads to problems such as data duplication and data omission. Off-site audit refers to a supervision method in which auditors use computers as a means and database technology as a basis to continuously collect and organize data and information on the business operations and management of the audited object, use queries and screening to record and analyze the abnormal or noteworthy data found, promptly discover existing problems, assess the risk status of the audited object, and monitor the operating status and development trends of the audited object. Therefore, the use of off-site audit methods to audit insurance industry data has a good development prospect. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a heterogeneous data integration method, computer equipment and medium for off-site auditing.

[0005] According to a first aspect of an embodiment of the present invention, a heterogeneous data integration method for off-site audit is provided, and the technical solution adopted is specifically as follows:

[0006] Collect data based on multiple data sources to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups;

[0007] For different data groups in a single data type, identify the data groups with missed collection and merge the data groups with missed collection, and identify the data groups with repeated collection and merge the data groups with repeated collection to obtain merged data corresponding to each data type;

[0008] Based on the merged data, natural language processing and one-hot encoding are performed on data of different data types to form column vector combinations between different data categories of each data type. The matching degree between any two column vectors in the column vector combinations corresponding to different data types is analyzed to obtain matching column vectors of the column vectors, thereby completing the integration between the same data category in different data types.

[0009] In some embodiments of the present invention, a method for identifying a data group that has missed collection includes:

[0010] Determine the number of complete data sets;

[0011] When the amount of data collected in a data group is less than the amount of complete data, the data group is marked to obtain all incomplete data groups.

[0012] In some embodiments of the present invention, a method for merging data groups with missed collection includes:

[0013] Determining the number of identical data in any two different incomplete data groups;

[0014] Analyzing the relationship between the amount of the identical data and the amount of the complete data to obtain the data similarity between any two different incomplete data groups;

[0015] According to the data similarity, the data groups that have been missed are merged.

[0016] In some embodiments of the present invention, merging the data groups with missed collection according to the data similarity includes:

[0017] Set data similarity threshold;

[0018] Determining whether the data similarity is greater than the data similarity threshold;

[0019] If yes, the distribution of different data in the two incomplete data groups is analyzed, and if there is no intersection in the distribution of different data in the two incomplete data groups, the two incomplete data groups are merged.

[0020] In some embodiments of the present invention, identifying repeatedly collected data groups and merging the repeatedly collected data groups comprises:

[0021] Determining the importance of data in each of the data groups, wherein the importance includes unimportant data, and obtaining a quantitative relationship of the unimportant data;

[0022] Analyze the similarity of two different data groups in a single data type, and combine the importance of the data and the quantity relationship of the unimportant data to obtain the mergibility of the data in the two different data groups;

[0023] According to the merging ability, the repeatedly collected data sets are merged.

[0024] In some embodiments of the present invention, merging the repeatedly collected data groups according to the mergibility includes:

[0025] Set mergeability thresholds;

[0026] Determining whether the mergeability is greater than the mergeability threshold;

[0027] If so, merge the data in the data group with the data of the corresponding data category in another data group.

[0028] In some embodiments of the present invention, based on the merged data, natural language processing and one-hot encoding are performed on data of different data types to form column vector combinations between different data categories of each data type, and the matching degree between any two column vectors in the column vector combinations corresponding to different data types is analyzed to obtain matching column vectors of the column vectors, thereby completing the integration between the same data category in different data types, including:

[0029] Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to convert the data of different data types into column vectors;

[0030] Combining the column vectors corresponding to different data categories of each data type to form a column vector combination between different data categories of each data type;

[0031] Analyze the cosine similarity between each column vector in the column vector combination corresponding to one data source and any column vector in the column vector combination corresponding to another data source, and obtain the matching degree between any two column vectors in the column vector combination corresponding to different data types;

[0032] According to the matching degree, integration between the same data category in different data types is completed.

[0033] In some embodiments of the present invention, the integration of the same data category in different data types is completed according to the matching degree, including:

[0034] Set the matching degree threshold;

[0035] Determining whether the matching degree is greater than the matching degree threshold;

[0036] If yes, mark the two column vectors as possible similar column vectors, and determine the column vector with the greatest matching degree among all the possible similar column vectors corresponding to each column vector as the matching column vector of the column vector;

[0037] The column vector is matched with its corresponding matching column vector to complete the integration of the same data category in different data types.

[0038] According to a second aspect of an embodiment of the present invention, a heterogeneous data integration computer device for off-site audit is provided, the computer device comprising:

[0039] A memory and a processor, wherein:

[0040] The memory is used to store program code;

[0041] The processor is used to read the program code stored in the memory and execute the above method.

[0042] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement a heterogeneous data integration method for off-site auditing as described in the first aspect of an embodiment of the present invention.

[0043] Compared with the prior art, the heterogeneous data integration method, computer device and medium for off-site audit provided by the present invention have the following beneficial effects:

[0044] The present invention collects data based on multiple data sources to obtain the data type corresponding to each data source, wherein each data type includes multiple data groups, thereby ensuring the integrity and accuracy of the original data. For different data groups in a single data type, the data groups with missed collection are identified and the data groups with missed collection are merged, and the data groups with repeated collection are identified and the data groups with repeated collection are merged to obtain the merged data corresponding to each data type, thereby realizing the correction of missed collection and repeated collection data, reducing the amount of data, and also making the collection results more accurate, which is convenient for subsequent calculations. Based on the merged data, natural language processing and unique hot encoding are performed on the data of different data types to form a column vector combination between different data categories of each data type, and the matching degree between any two column vectors in the column vector combination corresponding to different data types is analyzed to obtain the matching column vector of the column vector, and complete the integration between the same data categories in different data types, thereby facilitating data classification according to data types, realizing data resource integration, facilitating subsequent off-site audit analysis, and improving audit efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A basic flow chart of a heterogeneous data integration method for off-site auditing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a heterogeneous data integration method, computer equipment and medium for off-site auditing proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. Terms such as "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the article or device including the element.

[0049] In the traditional audit process, a large amount of data needs to be analyzed. For the staff, not only do they need to find erroneous data, but they also need to analyze all the data to find audit clues, which is quite difficult. The present invention processes the data groups of each data type of all data sources in the audit process to complete the merging of different data groups in each data type; and then analyzes the cosine similarity of the data types on each data source to complete the integration of similar data types on different data sources. The integrated data is used as the initial data for off-site audit, which is convenient for classification analysis and completion of off-site audit.

[0050] A specific scheme of a heterogeneous data integration method for off-site auditing provided by the present invention is described in detail below with reference to the accompanying drawings.

[0051] See also Figure 1, which shows the basic process of a heterogeneous data integration method for off-site auditing provided by an embodiment of the present invention.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a heterogeneous data integration method for off-site audit, which specifically includes:

[0053] S100: Collect data based on multiple data sources to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups.

[0054] First, it is necessary to clarify the data sources required for off-site audits based on the analysis objectives. During the collection process, it is necessary to ensure the integrity and accuracy of the data to avoid data loss and errors. For example, the main analysis objectives of this embodiment are the overall operating conditions of the business department and the data characteristics of the customers. Therefore, it is necessary to use the data at the business level and the data at the customer level for analysis; the data analysis at the business level is mainly used to grasp the overall operating conditions of the business department through the analysis of business trends and profit models, etc., to assist in determining the audit plan and audit focus; the data analysis at the customer level is mainly used to find audit clues, obtain the data characteristics of the customers, and screen and classify the customers. Therefore, first determine the multiple different data sources corresponding to the data at the business level and the data at the customer level. Then, data collection is performed based on multiple data sources to obtain the data type corresponding to each data source, where each data type includes multiple data groups.

[0055] Each data source corresponds to a data type. Since the data types obtained by each data source may be different, the data types corresponding to all data sources are grouped into a sequence as follows:

[0056] P I = {P1,P2…P i …P I}

[0057]

[0058] Where P I represents a sequence of data of I data types corresponding to all I data sources; i represents the data vector of the i-th data type (i-th data source), where i∈[1,I], I represents the total number of data types (data sources); x i,n Represents the nth data category in the i-th data type, where n∈[1,N i ],N i Indicates the number of data categories in the i-th data type.

[0059] It should be noted that the data types corresponding to the I data sources in the above sequence may have exactly the same data type, but no merging is done here to retain the original data. The data type obtained by a data source may be a sequence of (name, ID number, gender, age), then there are 4 corresponding data categories in this data type, namely name, ID number, gender, and age.

[0060] At this point, the collection of data in all data types from different data sources has been completed.

[0061] Off-site audits involve a lot of data, and not all data are necessarily normal data. There is a possibility that some data may be missed or repeatedly collected. Since the data in all data types in each data source are consistent, based on this, specific analysis is conducted on the data in the same data source, data with abnormal problems is identified, and these data are processed according to corresponding conditions, so that the data in each data source is more accurate; further, by analyzing the similarity of data types in different data sources, the integration of data between similar data types in different data sources is completed.

[0062] Therefore, the process of integrating data in all data sources includes step S200 and step S300.

[0063] S200: For different data groups in a single data type, identify the data groups with missed collection and merge the data groups with missed collection, and identify the data groups with repeated collection and merge the data groups with repeated collection to obtain merged data corresponding to each data type.

[0064] The heterogeneous data sources involved in off-site audits may come from multiple different systems or platforms. These systems may store information about customers. When this information is integrated into the off-site audit system, if there is a data sharing or data synchronization mechanism between different systems, the same customer information may be obtained repeatedly. In addition, in the process of heterogeneous data integration, data integration is a key step. However, due to differences in data format, storage method and update frequency between different systems, information duplication or inconsistency may occur during data synchronization and integration. Therefore, it is necessary to filter these duplicate data, and at the same time make the data format more regular, which is convenient for subsequent ETL format conversion.

[0065] During the data collection process, due to the excessive amount of data information collected, some data may be missed or collected repeatedly, and this situation needs to be corrected to appropriately reduce the amount of data. At the same time, it also makes the collection results more accurate and facilitates subsequent calculations.

[0066] Based on the above analysis, this embodiment identifies the data groups that have been missed and merges the data groups that have been missed, and identifies the data groups that have been repeatedly collected and merges the data groups that have been repeatedly collected, for different data groups in a single data type, to obtain merged data corresponding to each data type.

[0067] Furthermore, the method for identifying data groups that have been missed includes: determining the number of complete data in the data group; when the number of collected data in the data group is less than the number of complete data, marking the data group to obtain all incomplete data groups.

[0068] The method for merging data groups with missing data includes: determining the number of identical data in any two different incomplete data groups; analyzing the relationship between the number of identical data and the number of complete data to obtain the data similarity between any two different incomplete data groups; merging the data groups with missing data according to the data similarity. Furthermore, merging the data groups with missing data according to the data similarity includes: setting a data similarity threshold; determining whether the data similarity is greater than the data similarity threshold; if yes, analyzing the distribution of different data in the two incomplete data groups, and merging the two incomplete data groups if there is no intersection in the distribution of different data in the two incomplete data groups; if not, it indicates that the possibility that the two incomplete data groups are similar is low, and no further analysis is performed on them.

[0069] Identify the data groups with missed collection and merge the data groups with missed collection. The specific examples are as follows: First, analyze the data in a data type corresponding to a single data source. For the data in the data source, it exists in a database. The number of labels on the database header can represent the complete data volume. For example, the data type obtained by the data source is a sequence of (name, ID number, gender, age), then the corresponding complete data volume is 4, that is, the complete data volume is equal to the total number of data categories in the data type. Therefore, the number of complete data in the i-th data type is determined to be N. i . Analyze the kth data group P in the i-th data type i,k The actual number of data collected n i,k ; When n i,k =N i When n i,k <N i , indicating that the data collection of this data group is incomplete, it is marked to identify all data groups with missed data collection and obtain all incomplete data groups.

[0070] Then, the obtained incomplete data sets are analyzed to see whether there are similar relationships between these incomplete data sets, and these incomplete data sets are integrated through similar relationships. The specific operation is as follows: Statistically determine the number of identical data in the vth incomplete data set and the hth incomplete data set as N v ' ,h ; The data similarity calculation formula between the vth incomplete data group and the hth incomplete data is constructed as follows:

[0071]

[0072] In the formula, γ v,n represents the data similarity between the vth incomplete data set and the hth incomplete data; N v ' ,h N represents the number of identical data between the vth incomplete data set and the hth incomplete data set; i Represents the kth data group P in the i-th data type i,k The number of complete data.

[0073] Set the data similarity threshold to 0.7, when γ v,n <0.7, indicating that the two incomplete data sets are less likely to be similar and will not be further analyzed; v,n When ≥0.7, it means that the two incomplete data sets may be similar and need further consideration. v,n When ≥0.7, the distribution of different data in the two incomplete data groups is analyzed. Due to missed collection, there will be no specific data under some data categories. When the data categories corresponding to the remaining different data of the two incomplete data groups do not have an intersection, the two incomplete data groups are combined into one data group. For example: the first data group: name A, age 3, gender none, date of birth 2021.3; the second data group: name A, age 3, gender male, date of birth none. For the first and second data groups, excluding their identical data, the data categories corresponding to the different data are gender and date of birth, which have no intersection. Therefore, the first data group and the second data can be combined to form a merged data group of name A, age 3, gender male, date of birth 2021.3.

[0074] Repeat the above steps to complete the integration of all incomplete data sets.

[0075] Further, identifying repeatedly collected data groups and merging the repeatedly collected data groups includes: determining the importance of the data in each of the data groups, the importance includes unimportant data, and obtaining the quantitative relationship of the unimportant data; analyzing the similarity of two different data groups in a single data type, combining the importance of the data and the quantitative relationship of the unimportant data, to obtain the mergeability of the data in the two different data groups; merging the repeatedly collected data groups according to the mergeability. Further, merging the repeatedly collected data groups according to the mergeability includes: setting a mergeability threshold; determining whether the mergeability is greater than the mergeability threshold; if so, merging the data in the data group with the data of the corresponding data category in another data group.

[0076] A specific example is: further analysis based on the completion of the incomplete data group merger. For each data, the corresponding data importance is different, and the nth data x in the i-th data type is assigned i,n The importance of The importance should be given in combination with practical factors. For example, for customer information, gender and age are generally more important than place of origin and education. Mark this data as important data; when there are different levels of importance of data, assign values ​​to these data in combination with realistic factors. Mark this data as unimportant; Mark this data as very important data. In the i-th data type, there are some data with lower importance. When their data are close and other data are the same, the two data can be merged.

[0077] Construct the kth data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The calculation formula for the mergeability of the corresponding nth data is:

[0078]

[0079] In the formula, MP i,k,n Represents the kth data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The corresponding n-th data can be merged; N i represents the total number of data categories in the i-th data type; x i,k,n Represents the kth data group P in the i-th data type i,k The value of the nth data; x i,+,nRepresents the lth data group P in the i-th data type i,+ The value of the nth data; Represents the kth data group P in the i-th data type i,k The importance of the nth data; N i,4 Represents the kth data group P of the i-th data type i,k Less important than The 0.1 is added to prevent the denominator from being 0.

[0080] .x i,k,n -x i,+,n . indicates the kth data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The smaller the value is, the smaller the data difference between the corresponding data categories in the two data groups is, and the greater the possibility that the data can be merged; the kth data group P in the i-th data type is i,k The importance of the nth data The larger the value, the more important the data is, and the smaller the possibility of combining it with other data is. Indicates traversing the data of all data categories in the i-th data type. Find the mean; Represents the kth data group P in the i-th data type i,k The ratio of unimportant data to the total amount of data, that is, the quantitative relationship of unimportant data. The larger the value, the more unimportant data there is in the data group. The more unimportant data there is in the data group, the greater the possibility that the data can be merged.

[0081] Set the mergeability threshold to 0.98. If MP i,k,n >0.98, indicating that the kth data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The corresponding nth data can be merged; if MP i,k,n ≤0.98, indicating that the kth data group P in the i-th data type i,k The nth data and the lth data group P in the i-th data type i,+ The corresponding nth data cannot be merged.

[0082] Traverse all data in any data group of all data types, complete the merging and integration of all duplicate data, and finally obtain the merged data corresponding to each data type. The merging method is generally addition or overwriting, and the specific method can be allocated according to the actual meaning of this type of data.

[0083] S300: Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to form a column vector combination between different data categories of each data type, and the matching degree between any two column vectors in the column vector combination corresponding to the different data types is analyzed to obtain the matching column vector of the column vector, thereby completing the integration between the same data category in different data types.

[0084] Through step S200, the merged data of each data type is obtained. For heterogeneous data of off-site audit, the same type of data can be integrated by analyzing the specific content of the data, and then after the ETL conversion format, the auditors can use the integrated data to quickly determine the audit objectives, and it is convenient to help analyze the possible problems of different customers from various feature directions.

[0085] For different data types, the same data category may exist. For example, one data type is (name, age, date of birth) and the other data type is (name, place of origin, ID number). The two data types are different, but they have the same data category name. Therefore, two different data categories obtained from two different data sources can also match each other, which facilitates data classification according to data type.

[0086] Based on the above analysis, this embodiment performs natural language processing and one-hot encoding on data of different data types based on the merged data to form a column vector combination between different data categories of each data type, analyzes the matching degree between any two column vectors in the column vector combination corresponding to different data types, obtains the matching column vector of the column vector, and completes the integration between the same data categories in different data types. Further, it includes: based on the merged data, performs natural language processing and one-hot encoding on data of different data types to convert data of different data types into column vectors; combines the column vectors corresponding to different data categories of each data type to form a column vector combination between different data categories of each data type; analyzes the cosine similarity between each column vector in the column vector combination corresponding to one data source and any column vector in the column vector combination corresponding to another data source to obtain the matching degree between any two column vectors in the column vector combination corresponding to different data types; according to the matching degree, completes the integration between the same data categories in different data types. Furthermore, a matching degree threshold is set; it is determined whether the matching degree is greater than the matching degree threshold; if so, the two column vectors are marked as possible similar column vectors, and the column vector with the greatest matching degree among all the possible similar column vectors corresponding to each column vector is determined as the matching column vector of the column vector; the column vector is matched with its corresponding matching column vector to complete the integration of the same data category in different data types.

[0087] A specific example is: based on the merged data, the data collected from different data sources are analyzed, and these data carry specific text information, and these text information are converted into column vectors α through natural language processing and one-hot encoding. i . Combine the column vectors corresponding to different data categories in the same data type to form a column vector combination between different data categories of each data type, and obtain the customer information matrix of different data sources:

[0088]

[0089] In the matrix, Indicates the Nth value in the i-th data type i Column vector of data categories; N i Indicates the number of data categories in the i-th data type.

[0090] Compute column vector The cosine similarity with each column vector of the other data types in the customer information matrix:

[0091]

[0092] In the formula, Represents a column vector and the column vector in the customer information matrix The cosine similarity of ; COS represents the cosine function; Represents a column vector With column vector The angle between.

[0093] In vector space, the smaller the angle between two vectors, the closer their directions are, so the closer the cosine value is to 1, the more similar the two vectors are. On the contrary, the larger the angle, the smaller the cosine value (even negative, indicating opposite directions), indicating that the two vectors are less similar. and The smaller the cosine value of the angle between them, the higher the similarity between the two column vectors, and the greater the matching degree between the corresponding two column vectors.

[0094] Set the matching threshold to 0.7. , the two column vectors may be similar, but in reality, only one data category will successfully match one data category in the two data types, so we choose In It is recorded as a successful match, that is, the column vector and Most likely the data is of the same category. Get the column vector In data type The best matching result of is obtained by analogy, and the best matching results of other column vectors are obtained by analogy.

[0095] According to the above method, the best matching result of column vectors in all data types is obtained, and the mutual matching between data categories in the data types corresponding to each data source is completed, so as to facilitate data classification according to data type, realize data resource integration, and facilitate subsequent off-site audit analysis.

[0096] At this point, we have completed data integration on all data sources.

[0097] The above process completes the data integration on all data sources. By specifically analyzing the data of different data types after integration, it helps to discover audit clues, thereby achieving detailed analysis and personality analysis of target customers.

[0098] The details are as follows:

[0099] Conduct preliminary analysis on the integrated data from all data sources to understand the overall operating conditions of the sales department and assist in determining the audit plan and audit focus; at the same time, obtain the data characteristics of the customers to screen and classify them;

[0100] Based on the data analysis results at the customer level, find isolated data features that are different from other normal customers, identify possible problems, and thus determine the target customers for the audit;

[0101] Find all information related to the target customer and comprehensively determine the nature of the account and potential problems;

[0102] According to the individual problems of each audited unit, some data analysis is carried out flexibly and individually to find audit clues, locate audit targets, or discover the essence of the problem through comprehensive analysis of the relationship between multiple audit targets.

[0103] Based on the same inventive concept as the above method, this embodiment also provides a heterogeneous data integration computer device for off-site auditing, the computer device comprising:

[0104] A memory and a processor, wherein:

[0105] The memory is used to store program code;

[0106] The processor is used to read the program code stored in the memory, and execute data collection based on multiple data sources to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups; for different data groups in a single data type, identify the data groups with missed collection and merge the data groups with missed collection, and identify the data groups with repeated collection and merge the data groups with repeated collection to obtain merged data corresponding to each data type; based on the merged data, perform natural language processing and one-hot encoding on data of different data types to form a column vector combination between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combination corresponding to different data types, obtain the matching column vector of the column vector, and complete the integration between the same data category in different data types.

[0107] Based on the same inventive concept as the above method, this embodiment also provides a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor to implement data collection based on multiple data sources to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups; for different data groups in a single data type, identify the data groups with missed collection and merge the data groups with missed collection, and identify the data groups with repeated collection and merge the data groups with repeated collection to obtain merged data corresponding to each data type; based on the merged data, perform natural language processing and one-hot encoding on data of different data types to form a column vector combination between different data categories of each data type, analyze the matching degree between any two column vectors in the column vector combination corresponding to different data types, obtain the matching column vector of the column vector, and complete the integration between the same data category in different data types.

[0108] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A heterogeneous data integration method for off-site audit, characterized in that: The method comprises: Collect data based on multiple data sources to obtain a data type corresponding to each data source, wherein each data type includes multiple data groups; For different data groups in a single data type, identify the data groups with missed collection and merge the data groups with missed collection, and identify the data groups with repeated collection and merge the data groups with repeated collection to obtain merged data corresponding to each data type; Based on the merged data, natural language processing and one-hot encoding are performed on data of different data types to form column vector combinations between different data categories of each data type. The matching degree between any two column vectors in the column vector combinations corresponding to different data types is analyzed to obtain matching column vectors of the column vectors, thereby completing the integration between the same data category in different data types.

2. The heterogeneous data integration method for off-site audit according to claim 1 is characterized in that: Methods for identifying data groups that have been missed include: Determine the number of complete data sets; When the amount of data collected in a data group is less than the amount of complete data, the data group is marked to obtain all incomplete data groups.

3. The heterogeneous data integration method for off-site audit according to claim 2 is characterized in that: Methods for merging data sets that have been missed include: Determining the number of identical data in any two different incomplete data groups; Analyzing the relationship between the amount of the identical data and the amount of the complete data to obtain the data similarity between any two different incomplete data groups; According to the data similarity, the data groups that have been missed are merged.

4. The heterogeneous data integration method for off-site audit according to claim 3 is characterized in that: According to the data similarity, the data groups with missed collection are merged, including: Set data similarity threshold; Determining whether the data similarity is greater than the data similarity threshold; If yes, the distribution of different data in the two incomplete data groups is analyzed, and if there is no intersection in the distribution of different data in the two incomplete data groups, the two incomplete data groups are merged.

5. The heterogeneous data integration method for off-site audit according to claim 1 is characterized in that: Identify and merge duplicate data sets, including: Determining the importance of data in each of the data groups, wherein the importance includes unimportant data, and obtaining a quantitative relationship of the unimportant data; Analyze the similarity of two different data groups in a single data type, and combine the importance of the data and the quantity relationship of the unimportant data to obtain the mergibility of the data in the two different data groups; According to the merging ability, the repeatedly collected data sets are merged.

6. The heterogeneous data integration method for off-site audit according to claim 5 is characterized in that: Based on the mergeability, the repeatedly collected data sets are merged, including: Set mergeability thresholds; Determining whether the mergeability is greater than the mergeability threshold; If so, merge the data in the data group with the data of the corresponding data category in another data group.

7. The heterogeneous data integration method for off-site audit according to claim 1 is characterized in that: Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to form a column vector combination between different data categories of each data type, and the matching degree between any two column vectors in the column vector combination corresponding to different data types is analyzed to obtain a matching column vector of the column vector, and the integration between the same data category in different data types is completed, including: Based on the merged data, natural language processing and one-hot encoding are performed on the data of different data types to convert the data of different data types into column vectors; Combining the column vectors corresponding to different data categories of each data type to form a column vector combination between different data categories of each data type; Analyze the cosine similarity between each column vector in the column vector combination corresponding to one data source and any column vector in the column vector combination corresponding to another data source, and obtain the matching degree between any two column vectors in the column vector combination corresponding to different data types; According to the matching degree, integration between the same data category in different data types is completed.

8. The heterogeneous data integration method for off-site audit according to claim 1 is characterized in that: According to the matching degree, the integration of the same data category in different data types is completed, including: Set the matching degree threshold; Determining whether the matching degree is greater than the matching degree threshold; If yes, mark the two column vectors as possible similar column vectors, and determine the column vector with the greatest matching degree among all the possible similar column vectors corresponding to each column vector as the matching column vector of the column vector; The column vector is matched with its corresponding matching column vector to complete the integration of the same data category in different data types.

9. A heterogeneous data integration computer device for off-site audit, characterized in that: The computer device comprises: a memory and a processor, wherein: The memory is used to store program code; The processor is configured to read the program code stored in the memory and execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed to implement the heterogeneous data integration method for off-site audit as described in any one of claims 1 to 8.

Images