Enterprise risk data fusion method and system based on multi-source fusion

Abstract

The invention discloses an enterprise risk data fusion method and system based on multi-source fusion, and relates to the technical field of enterprise risk data fusion, and the method comprises the implementation steps: data preprocessing: achieving data integrity, data uniqueness and data authority; dynamic rule constraint; performing case number standardization processing; state validity change and data layering are carried out, data with complicated sources are layered, a unified data structure is defined, and through data layering, an intermediate state is added so as to increase the reusability of the data; data de-duplication: de-duplication is carried out by using the state of Flink, Redis cache is utilized, and a unique data set is stored by using set; and data fusion: performing data fusion according to a dynamic loading rule based on a similarity improvement optimization algorithm of priorities. According to the method, the data can be simply and quickly clustered, human intervention is not needed in the implementation process, automatic calculation can be performed after parameters are configured, and the speed is extremely high.

Description

Technical Field

[0001] The present invention relates to the technical field of enterprise risk data fusion, and in particular to an enterprise risk data fusion method and system based on multi-source fusion. Background Art

[0002] In the current complex and ever-changing market environment, enterprise risk management has become a key link to ensure the stable operation of enterprises and enhance their competitiveness. From the perspective of business development, this information has great application value and research value. Different industries can grasp relevant information in a timely manner, understand the dynamics of partners, and take the initiative in advance, thereby avoiding or reducing direct or indirect losses caused by negative risk information of partners.

[0003] Since such data sources are numerous, data forms are diverse, business complexity is high, and sensitive information of legal entities is involved, the fusion process requires highly accurate and reliable program support. The technology of multi-source fusion is to better solve the problem of data heterogeneity in reality and to display the actual situation of data in a more comprehensive, accurate and timely manner. The main problems currently exist include data preprocessing, fusion field mapping, data conflict handling, data standardization, etc. Traditional similarity calculation methods may be inefficient when processing large-scale data sets, and in some cases it is difficult to accurately reflect the actual similarity between data. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and to provide an enterprise risk data fusion method and system based on multi-source fusion, which can cluster data simply and quickly. No human intervention is required during the implementation process. After the parameters are configured, the data can be automatically calculated at an extremely fast speed.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for fusion of enterprise risk data based on multi-source fusion, the implementation steps of the method include:

[0007] S1. Data preprocessing to achieve data integrity, data uniqueness and data authority;

[0008] S2, dynamic rule constraints;

[0009] S3, case number standardization;

[0010] S4, status validity change and data stratification, stratify data from complex sources to make the data more orderly, define a unified data structure so that all users can clearly understand the meaning of metadata, and through data stratification, add intermediate states to increase data reusability and reduce a lot of repeated calculations;

[0011] S5: Data deduplication. Use Flink's built-in state to deduplicate data. Use the mainstream Redis cache method and use set to save unique data sets. This method can be used to quickly determine whether an element exists in a set, and can also quickly calculate the number of elements in a set.

[0012] S6. Data fusion: Improve the optimization algorithm based on the similarity of priority, initialize the feature value, aggregate the same data into different groups, process the data of each group in a loop, and fuse the data according to the dynamically loaded rules.

[0013] This method proposes an improved priority similarity algorithm, which can flexibly set the priority in combination with the rule dynamic engine, and then calculate the similarity between multiple sources and merge related data according to field mapping.

[0014] Furthermore, in step S1,

[0015] The data integrity is to complete the data that is incomplete or more or less missing values ​​or key information in the original data;

[0016] The data is unique and duplicate data is removed;

[0017] The data authority is determined by reviewing and determining the priority of the data source in order to control the data quality, and the qualified data sources are divided.

[0018] Furthermore, in step S2,

[0019] A configurable constraint method based on dynamic rules is adopted, and the distributed configuration center Nacos is used to realize the immediate effectiveness and refresh of the configuration; the process includes starting, data source input, rule engine loading, data cleaning and storage.

[0020] Furthermore, the case number represents the specific information of the current record and includes the region and agency information of the punishment.

[0021] Furthermore, for case numbers in different formats, missing fields or symbols are first detected, and then the regular expression method is used to match the relevant groups, and standard symbols are used for replacement processing.

[0022] Furthermore, in step 5,

[0023] Define the cache expiration time; when the time is exceeded, the cache will be deleted immediately. Because it is a memory-based operation, the cached content can be created and updated quickly.

[0024] Furthermore, the specific implementation steps of step 6 are as follows:

[0025] S601: Input observation data X = (x1, x2, x3 ... x n ), and select the attribute value x that can be uniquely identified according to the scenario of the specific data dimension i , x j ;

[0026] S602: Initialize x i , use wei to process the coefficients; initialize x j , use the regularization method reg to deal with the noise; initialize the eigenvalue EI = Hash (wei (x i )+reg(x j ));

[0027] S603: Aggregate the data with the same EI into different groups through SQL;

[0028] S604: looping through the data of each group and fusing the data according to the dynamically loaded rules;

[0029] S605: Output the data of each group, which has only one piece of data, and record the original value;

[0030] Where wei represents the priority coefficient of the calculation participation, and reg represents the rule parameters of the manual initialization configuration; through the operation of the similarity algorithm, the HashCode value of the code is calculated to effectively distinguish the relationship between multiple data sources that need to be fused.

[0031] The present invention also claims protection for an enterprise risk data fusion system based on multi-source fusion, comprising: a data preprocessing module, a dynamic rule constraint module, a case number standardization processing module, a status validity change and data stratification module, a data deduplication module and a data fusion module.

[0032] The system realizes multi-source enterprise risk data fusion through the above-mentioned enterprise risk data fusion method based on multi-source fusion.

[0033] The present invention also claims protection for an enterprise risk data fusion device based on multi-source fusion, comprising: at least one memory and at least one processor;

[0034] The at least one memory is used to store a machine-readable program;

[0035] The at least one processor is used to call the machine-readable program to implement the above method.

[0036] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0037] Compared with the prior art, the enterprise risk data fusion method and system based on multi-source fusion of the present invention has the following beneficial effects:

[0038] The improved priority similarity algorithm proposed in the present invention can flexibly set the priority in combination with the rule dynamic engine, thereby calculating the similarity between multiple sources and merging related data according to field mapping. When data fusion is performed through this algorithm, data can be clustered simply and quickly, without human intervention in the implementation process, and can be automatically calculated after the parameters are configured, with extremely high speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of an implementation method of an enterprise risk data fusion method based on multi-source fusion provided by an embodiment of the present invention;

[0040] Figure 2 It is a flow chart of implementing the data fusion algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0042] The embodiment of the present invention provides an enterprise risk data fusion method based on multi-source fusion, such as Figure 1 As shown, the implementation steps of the method include:

[0043] S1, data preprocessing;

[0044] S2, dynamic rule constraints;

[0045] S3, case number standardization;

[0046] S4, status validity changes and data stratification;

[0047] S5. Data deduplication;

[0048] S6. Data fusion.

[0049] The specific implementation is as follows:

[0050] 1. Data preprocessing to achieve data integrity, data uniqueness and data authority;

[0051] Data integrity: fill in the incomplete data or data with missing values ​​or key information;

[0052] Data uniqueness, deduplication of data;

[0053] Data authority: In order to control data quality, data sources are reviewed and prioritized to identify qualified data sources.

[0054] 2. Dynamic rule constraints:

[0055] Adopt a configurable constraint method based on dynamic rules, and use the distributed configuration center Nacos to achieve the immediate effect and refresh of the configuration; the process includes start, data source input, rule engine loading, data cleaning and warehousing.

[0056] 3. Case number standardization processing:

[0057] In enterprise risk data, the case number is the most critical information, representing the specific information of the current record, and also containing information about the region and authority of the penalty. Therefore, it is necessary to standardize the case number to regulate its impact on data fusion, because incorrect processing may have a huge impact on the calculation of similarity.

[0058] For example, there are two case numbers, "Shanghai Securities Regulatory Commission Penalty Letter (2014) No. 10" and "Shanghai Securities Regulatory Commission Penalty (2014) No. 10", both representing the penalties imposed by the Shanghai Securities Regulatory Commission. The date is the 10th penalty decision letter in 2014, but the differences in format symbols and standard fonts lead to non-standard situations. To handle these two types of differences, first, it is necessary to detect the missing fields or symbols, such as "Penalty", "Letter", "(", etc., and then use the regular expression method to match the relevant groups and perform replacement processing with standard symbols.

[0059] 4. Status validity change and data stratification:

[0060] Stratify the data with complex sources to make the data more orderly, define a unified data structure so that all users can clearly understand the meaning of the metadata, and through data stratification, intermediate states can be added to increase data reusability and reduce a large amount of repeated calculations.

[0061] 5. Data deduplication:

[0062] Use the state provided by Flink itself for deduplication, utilize the mainstream method Redis cache, use set to save the unique data set, through which it can quickly determine whether a certain element exists in the set, and can also quickly calculate the number of elements in a certain set. Generally, the cache expiration time can be defined as several hours. When this time is exceeded, the cache will be immediately deleted. Because it is an in-memory operation, the content of the cache can be quickly created and updated.

[0063] 6. Data fusion:

[0064] In the data fusion method, the steps of the similarity improvement and optimization algorithm based on priority are as follows:

[0065] S601: Input the observed data X = (x1, x2, x3…x n), and select the attribute value x that can be uniquely identified according to the scenario of the specific data dimension i , x j ;

[0066] S602: Initialize x i , use wei to process the coefficients; initialize x j , use the regularization method reg to deal with the noise; initialize the eigenvalue EI = Hash (wei (x i )+reg(x j ));

[0067] S603: Aggregate the data with the same EI into different groups through SQL;

[0068] S604: looping through the data of each group and fusing the data according to the dynamically loaded rules;

[0069] S605: Output the data of each group, which has only one piece of data, and record the original value;

[0070] Where wei represents the priority coefficient of the calculation participation, and reg represents the rule parameters of the manual initialization configuration; through the operation of the similarity algorithm, the HashCode value of the code is calculated to effectively distinguish the relationship between multiple data sources that need to be fused.

[0071] This method proposes an improved priority similarity algorithm, which can flexibly set priorities in combination with the rule dynamic engine, and then calculate the similarity between multiple sources, and merge related data according to field mapping. When performing data fusion through this algorithm, data can be clustered simply and quickly, without human intervention in the implementation process, and can be automatically calculated after the parameters are configured, with extremely fast speed.

[0072] The embodiment of the present invention also provides an enterprise risk data fusion system based on multi-source fusion, which realizes multi-source enterprise risk data fusion through the enterprise risk data fusion method based on multi-source fusion described in the above embodiment. The system includes:

[0073] 1. Data preprocessing module to achieve data integrity, data uniqueness and data authority;

[0074] Data integrity: fill in the incomplete data or data with missing values ​​or key information;

[0075] Data uniqueness, deduplication of data;

[0076] Data authority: In order to control data quality, data sources are reviewed and prioritized to identify qualified data sources.

[0077] 2. Dynamic rule constraint module adopts a configurable constraint method based on dynamic rules and uses the distributed configuration center Nacos to realize the immediate effectiveness and refresh of the configuration; the process includes starting, data source input, rule engine loading, data cleaning and storage.

[0078] 3. Case number standardization processing module, standardizes case numbers to regulate their impact on data fusion. For example, there are two case numbers "Shanghai Securities Regulatory Commission Penalty (2014) No. 10" and "Shanghai Securities Regulatory Commission Penalty (2014) No. 10", both of which represent penalties made by the Shanghai Securities Regulatory Commission, and the date is the Penalty Decision No. 10 of 2014, but the difference in format symbols and standard fonts leads to non-standard situations. In order to handle these two types of differences, it is necessary to first detect missing fields or symbols, such as "penalty", "character", "(", etc., and then use the regular expression method to match the relevant groups and use standard symbols for replacement processing.

[0079] 4. The state validity change and data stratification module stratifies data from various sources to make the data more orderly and defines a unified data structure so that all users can clearly understand the meaning of metadata. Through data stratification, intermediate states can be added to increase data reusability and reduce a large amount of repeated calculations.

[0080] 5. Data deduplication module uses Flink's built-in state for deduplication, uses the mainstream Redis cache method, and uses set to save unique data sets. It can quickly determine whether an element exists in a set, and can also quickly calculate the number of elements in a set. Generally, the cache expiration time can be defined as several hours. When this time is exceeded, the cache will be deleted immediately. Because it is a memory-based operation, the cache content can be quickly created and updated.

[0081] 6. Data fusion module, based on the priority similarity improvement optimization algorithm, the steps are as follows:

[0082] S601: Input observation data X = (x1, x2, x3 ... x n ), and select the attribute value x that can be uniquely identified according to the scenario of the specific data dimension i , x j ;

[0083] S602: Initialize x i , use wei to process the coefficients; initialize x j , use the regularization method reg to deal with the noise; initialize the eigenvalue EI = Hash (wei (x i )+reg(x j ));

[0084] S603: Aggregate the data with the same EI into different groups through SQL;

[0085] S604: looping through the data of each group and fusing the data according to the dynamically loaded rules;

[0086] S605: Output the data of each group, which has only one piece of data, and record the original value;

[0087] Where wei represents the priority coefficient of the calculation participation, and reg represents the rule parameters of the manual initialization configuration; through the operation of the similarity algorithm, the HashCode value of the code is calculated to effectively distinguish the relationship between multiple data sources that need to be fused.

[0088] An embodiment of the present invention also provides an enterprise risk data fusion device based on multi-source fusion, comprising: at least one memory and at least one processor;

[0089] The at least one memory is used to store a machine-readable program;

[0090] The at least one processor is used to call the machine-readable program to implement the enterprise risk data fusion method based on multi-source fusion described in the above embodiment.

[0091] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the processor executes the enterprise risk data fusion method based on multi-source fusion described in the above embodiment. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes implementing the functions of any of the above embodiments are stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0092] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0093] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0094] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0095] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0096] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for fusion of enterprise risk data based on multi-source fusion, characterized in that: The implementation steps of this method include: S1. Data preprocessing to achieve data integrity, data uniqueness and data authority; S2, dynamic rule constraints; S3, case number standardization; S4, status validity change and data stratification, stratify data from various sources, define a unified data structure, and add intermediate states through data stratification to increase data reusability; S5: Data deduplication: Use Flink's built-in state to deduplicate data, use Redis cache, and use set to save unique data sets. S6. Data fusion: Improve the optimization algorithm based on the similarity of priority, initialize the feature value, aggregate the same data into different groups, process the data of each group in a loop, and fuse the data according to the dynamically loaded rules.

2. The enterprise risk data fusion method based on multi-source fusion according to claim 1 is characterized in that: The step S1, The data integrity is to complete the original data that is incomplete, missing values ​​or key information; The data is unique and duplicate data is removed; The authority of the data is determined by reviewing and determining the priority of the data source, and the qualified data sources are divided.

3. The enterprise risk data fusion method based on multi-source fusion according to claim 1 is characterized in that: The step S2, A configurable constraint method based on dynamic rules is adopted, and the distributed configuration center Nacos is used to realize the immediate effectiveness and refresh of the configuration; the process includes starting, data source input, rule engine loading, data cleaning and storage.

4. The enterprise risk data fusion method based on multi-source fusion according to claim 1 is characterized in that: The case number represents the specific information of the current record and includes the region and agency information of the punishment.

5. The enterprise risk data fusion method based on multi-source fusion according to claim 4 is characterized in that: For case numbers in different formats, we first detect missing fields or symbols, then use regular expressions to match related groups, and use standard symbols for replacement.

6. The enterprise risk data fusion method based on multi-source fusion according to claim 1 is characterized in that: Step 5, Defines the cache expiration time; when this time is exceeded, the cache will be deleted immediately.

7. The enterprise risk data fusion method based on multi-source fusion according to claim 1 is characterized in that: The specific implementation steps of step 6 are as follows: S601: Input observation data X = (x1, x2, x3 ... x n ), and select the attribute value x that can be uniquely identified according to the scenario of the specific data dimension i , x j ; S602: Initialize x i , use wei to process the coefficients; initialize x j , use the regularization method reg to deal with the noise; initialize the eigenvalue EI = Hash (wei (x i )+reg(x j )); S603: Aggregate the data with the same EI into different groups through SQL; S604: looping through the data of each group and fusing the data according to the dynamically loaded rules; S605: Output the data of each group, which has only one piece of data, and record the original value; Where wei represents the priority coefficient of the calculation participation, and reg represents the rule parameters of the manual initialization configuration; through the operation of the similarity algorithm, the HashCode value of the code is calculated to effectively distinguish the relationship between multiple data sources that need to be fused.

8. An enterprise risk data fusion system based on multi-source fusion, characterized in that: include: Data preprocessing module, dynamic rule constraint module, case number standardization processing module, status validity change and data stratification module, data deduplication module and data fusion module, The system realizes multi-source enterprise risk data fusion through the enterprise risk data fusion method based on multi-source fusion described in any one of claims 1 to 7.

9. An enterprise risk data fusion device based on multi-source fusion, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images