Multi-source college financial data aggregation method based on text filtering

Abstract

The invention belongs to the technical field of data aggregation, and particularly relates to a multi-source college financial data aggregation method based on text filtering. According to the method, firstly, multi-source college financial text data are crawled and preprocessed, and bad texts in the multi-source college financial data are filtered by utilizing a naive Bayes classifier and combining a category threshold design text filtering technology. And then, the filtered texts are aggregated together through a Jaro-Winkler similarity matching algorithm, so that intelligent aggregation of the multi-source college financial data is realized. According to the multi-source college financial data aggregation method based on text filtering, through crawling, preprocessing, bad text filtering and similarity matching aggregation of multi-source financial text data, the efficiency and accuracy of college financial data integration are effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of data aggregation, and in particular relates to a multi-source university financial data aggregation method based on text filtering. Background Art

[0002] With the rapid development of information technology and the continuous innovation of university management models, the sources of university financial data are becoming increasingly diversified, including financial statements, audit reports, online payment records, scientific research funding management systems, etc. These multi-source financial data have put forward higher requirements for aggregation methods.

[0003] The intelligent aggregation of multi-source university financial data aims to integrate financial data from different systems, formats, and sources through automated and intelligent technical means to form a unified, standardized, and easy-to-analyze data set. This can not only improve the efficiency and accuracy of data integration and reduce the cost of manual processing, but also provide more comprehensive and in-depth financial data analysis reports for university management, helping universities make more scientific and reasonable decisions. Therefore, studying the intelligent aggregation method of multi-source university financial data is of great significance for improving the efficiency of university financial management, optimizing resource allocation, and promoting the sustainable development of universities.

[0004] At present, more and more scholars have conducted relevant research on data aggregation methods and achieved a series of results. Literature 1: Research on aggregation and identification methods of digital resources in multi-value chain collaborative data space of manufacturing enterprises based on semantics. According to the spatial semantic network node discovery method of multi-source data, data aggregation and identification are realized, which has a good aggregation effect. However, the complexity of data semantic annotation is relatively high, which leads to certain limitations in the aggregation efficiency of this method in practical applications. Literature 2: A review of privacy-preserving data aggregation research based on the Internet of Vehicles. Aiming at the problem of privacy data aggregation in the Internet of Vehicles, a data aggregation scheme based on encryption algorithm is designed. However, the key exchange of data in the Internet of Vehicles often requires additional communication, which leads to a large actual communication overhead of this aggregation method. Literature 3: Research on cross-organizational medical and health big data aggregation and case knowledge reasoning methods. In the face of the secure sharing of large-scale multi-source heterogeneous medical and health data, a deep aggregation method based on federated learning mechanism is designed. However, this method has certain privacy leakage risks and poor practical application effect. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-source university financial data aggregation method based on text filtering, aiming to solve the technical problem that the sources of university financial data are wide and large, which affects the utilization efficiency of financial data.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] A method for aggregating multi-source university financial data based on text filtering includes the following steps:

[0008] S1. Crawling and preprocessing multi-source university financial text data: Taking university financial text data as the object, we conduct in-depth research on the intelligent aggregation of multi-source data; first, we introduce a deduplication algorithm based on a hash table, that is, after crawling new text data, we first convert the data into a hash value through a hash function, as shown in the following formula:

[0009] Z(x)=f MD5 (x)(1)

[0010] Where Z(x) is the hash value of the university financial data x; f MD5 is the MD5 hash function; through the above crawling and preprocessing, the present invention obtains high-quality multi-source university financial text data, providing a data basis for subsequent intelligent aggregation;

[0011] S2. Filter bad texts from multi-source university financial data: First, a naive Bayes classifier is constructed to classify multi-source university financial data. This is a text classification method based on Bayes theorem and feature conditional independence assumption. The key is to calculate the probability of the text data to be classified into each category. The calculation formula is shown below:

[0012]

[0013] In the formula, G(B n |x) is the given university financial text data x belongs to category B n The probability of x belonging to category B is greater than the value of n The greater the possibility; G(x|B n ) is in category B n Under this condition, the probability of university financial text data x appearing; G(B n ) is Category B n The probability of occurrence; G(x) is the probability of occurrence of university financial text data x. Assuming that the bad text category threshold is γ, if the probability G(B n |x)≥γ, then the category belongs to the positive category and is retained; if G(B n |x)<γ, then the category belongs to the counterexample category, that is, bad text, and it is removed;

[0014] S3. Aggregation of multi-source university financial text data: After filtering out bad texts from multi-source university financial data according to the above content, the remaining normal texts can be aggregated; Based on the particularity of multi-source university financial text data, the present invention uses the Jaro-Winkler similarity algorithm to match university financial text data from different sources, which is a similarity measurement method based on string comparison, combining the advantages of the Jaro algorithm and the Winkler algorithm, thereby improving the accuracy of similarity calculation. First, the Jaro algorithm needs to be used to calculate the JaroDistance similarity between each pair of texts, and the expression is as follows:

[0015]

[0016] Where D1 is the Jaro Distance similarity between the university financial text data from two different sources; M is the number of characters to be matched in the university financial text data; |L1| and |L2| are the lengths of the strings to be matched in the university financial text data from two different sources; is the number of characters that undergo transposition in the string to be matched; F0 is the number of times the two strings to be matched undergo transposition at the same position. Based on Jaro Distance, this paper uses the Winkler algorithm to make corrections and obtain the Jaro-Winkler similarity value, which is expressed as follows:

[0017] D2=D1+(ak(1-D1))(4)

[0018] Where D2 is the Jaro-Winkler similarity between the university financial text data from two different sources; a is the prefix length of the two strings to be matched; k is a constant, generally set to 0.1.

[0019] The multi-source university financial data aggregation method based on text filtering described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0020] The present invention proposes a multi-source university financial data aggregation method based on text filtering, which effectively improves the efficiency and accuracy of university financial data integration by crawling, preprocessing, bad text filtering and similarity matching aggregation of multi-source financial text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the method steps of an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of normalized mutual information index in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0024] like Figure 1 As shown, a method for aggregating multi-source university financial data based on text filtering includes the following steps:

[0025] S1. Crawling and preprocessing multi-source university financial text data: The present invention takes university financial text data as the object and deeply studies the intelligent aggregation of multi-source data; first, a deduplication algorithm based on a hash table is introduced, that is, after crawling new text data, the data is first converted into a hash value through a hash function, and the expression is shown in the following formula:

[0026] Z(x)=f MD5 (x)(1)

[0027] Where Z(x) is the hash value of the university financial data x; f MD5 is the MD5 hash function; through the above crawling and preprocessing, the present invention obtains high-quality multi-source university financial text data, providing a data basis for subsequent intelligent aggregation;

[0028] S2. Filter bad texts from multi-source university financial data: First, a naive Bayes classifier is constructed to classify multi-source university financial data. This is a text classification method based on Bayes theorem and feature conditional independence assumption. The key is to calculate the probability of the text data to be classified into each category. The calculation formula is shown below:

[0029]

[0030] In the formula, G(B n |x) is the given university financial text data x belongs to category B n The probability of the data x belonging to category Bn is greater. G(x|Bn) is the probability of the college financial text data x appearing under category Bn. G(Bn) is the probability of the data x belonging to category Bn. n The probability of occurrence; G(x) is the probability of occurrence of university financial text data x; assuming that the bad text category threshold is γ, if the belonging probability G(Bn|x)≥γ of the university financial text data x required by formula (3), then the category belongs to the positive example category and is retained; if G(Bn|x)<γ, then the category belongs to the negative example category, that is, bad text, and is eliminated; therefore, the present invention can effectively filter out bad texts in the original multi-source university financial data through the naive Bayes classifier combined with the category threshold.

[0031] S3. Aggregation of multi-source university financial text data: After filtering out bad texts from multi-source university financial data according to the above content, the remaining normal texts can be aggregated; Based on the particularity of multi-source university financial text data, the present invention uses the Jaro-Winkler similarity algorithm to match university financial text data from different sources, which is a similarity measurement method based on string comparison, combining the advantages of the Jaro algorithm and the Winkler algorithm, thereby improving the accuracy of similarity calculation; First, the Jaro algorithm needs to be used to calculate the JaroDistance similarity between each pair of texts, and the expression is as follows:

[0032]

[0033] Where D1 is the Jaro Distance similarity between the university financial text data from two different sources; M is the number of characters to be matched in the university financial text data; |L1| and |L2| are the lengths of the strings to be matched in the university financial text data from two different sources; is the number of characters that undergo transposition operation in the string to be matched; F0 is the number of times the two strings to be matched undergo transposition operation at the same position. Based on JaroDistance, the present invention uses Winkler algorithm for correction to obtain Jaro-Winkler similarity value, which is expressed as follows:

[0034] D2=D1+(ak(1-D1))(4)

[0035] Where D2 is the Jaro-Winkler similarity between the university financial text data from two different sources; a is the prefix length of the two strings to be matched; k is a constant, generally set to 0.1.

[0036] The present invention is based on the Jaro-Winkler similarity matching algorithm and effectively realizes the intelligent aggregation of multi-source university financial data after bad text filtering.

[0037] like Figure 2 As shown in the figure, a multi-source university financial data intelligent aggregation method based on deep learning and a multi-source university financial data intelligent aggregation method based on machine learning will be introduced, and a simulation comparative experiment will be carried out to explore the performance of the design method based on the experimental results.

[0038] First of all, following the conventional experimental environment settings and combining the actual needs of intelligent aggregation of multi-source university financial data, the key experimental environment configurations in this simulation experiment are Ubuntu 16.04LTS operating system, Intel Xeon E5-2650 CPU, TeslaP100-Pcie-8GB graphics card, and MySQL database.

[0039] Then, based on the existing data aggregation research, the present invention performs performance tests of the experimental group method and the control group method on 6 public university financial data sets. The distribution of each data set is shown in Table 1.

[0040] Table 1 Experimental dataset distribution table

[0041]

[0042] The purpose of this simulation comparison experiment is to prove the aggregation performance of the multi-source university financial data aggregation method designed by the present invention based on text filtering technology. Therefore, after obtaining the data aggregation results under the three methods, the present invention follows the research results of predecessors and adopts normalized mutual information (NMI) as the evaluation index. The value range is between [0,1]. The larger the value, the higher the correlation between the two random variables. The calculation formula is shown as follows:

[0043] NMI=H(x,y) / (S(x)+S(y)) 12 (5)

[0044] Where H(x,y) is the mutual information between the actual result x and the expected result y of the intelligent aggregation of multi-source university financial data; S(x) and S(y) are the entropies of the actual result x and the expected result y, respectively.

[0045] The normalized mutual information index of the intelligent aggregation results of the six data sets after applying different methods was calculated and statistically analyzed. The results are as follows: Figure 2 As shown. The NMI index of the experimental group and the control group methods in the university financial data set is more than 0.70, which proves their effectiveness in multi-source data aggregation. The present invention performs better in aggregation effect, with an average NMI value of 0.918, which is significantly improved compared with the control group method. Therefore, the multi-source university financial data aggregation method based on text filtering proposed in the present invention is feasible and reliable.

[0046] The sources of university financial data are wide and huge, which affects the utilization efficiency of financial data. To solve this problem, the present invention proposes a method for aggregating multi-source university financial data based on text filtering technology. First, the multi-source university financial text data is crawled and preprocessed, and the naive Bayes classifier is used to design text filtering technology in combination with the category threshold to filter out the bad text in the multi-source university financial data. Then, the filtered texts are aggregated together through the Jaro-Winkler similarity matching algorithm to realize the intelligent aggregation of multi-source university financial data. The experimental results show that after applying this method, the normalized mutual information of the intelligent aggregation results of multi-source university financial data is as high as 0.918, and the aggregation effect is excellent.

[0047] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for aggregating multi-source university financial data based on text filtering, characterized in that: The following steps are involved: S1. Crawling and preprocessing university financial text data from multiple sources: Taking university financial text data as the object, a deduplication algorithm based on a hash table is introduced. That is, after crawling new text data, the data is converted into a hash value through a hash function. The expression is shown in the following formula: Z(x)=f MD5 (x)(1) Where Z(x) is the hash value of the university financial data x; f MD5 is the MD5 hash function; through crawling and preprocessing, high-quality multi-source university financial text data is obtained; S2. Filter bad texts from multi-source university financial data: Construct a naive Bayes classifier to classify multi-source university financial data. Based on the Bayesian theorem and the text classification method of the feature conditional independence assumption, calculate the probability of the text data to be classified into each category. The calculation formula is shown as follows: In the formula, G(B n |x) is the given university financial text data x belongs to category B n The probability of x belonging to category B is greater than the value of n The greater the possibility; G(x|B n ) is in category B n Under this condition, the probability of university financial text data x appearing; G(B n ) is Category B n The probability of occurrence; G(x) is the probability of occurrence of university financial text data x; Assuming that the threshold of bad text category is γ, if the probability G(B) of university financial text data x in formula (3) is n |x)≥γ, the category belongs to the positive category and is retained; if G(B n |x)<γ, the category belongs to the counterexample category, that is, bad text, and it is removed; S3. Aggregation of multi-source university financial text data: Based on the particularity of multi-source university financial text data, the Jaro-Winkler similarity algorithm is used to match university financial text data from different sources. The similarity measurement method based on string comparison combines the Jaro algorithm and the Winkler algorithm. First, the Jaro algorithm needs to be used to calculate the Jaro Distance similarity between each pair of texts. The expression is as follows: Where D1 is the Jaro Distance similarity between the university financial text data from two different sources; M is the number of characters to be matched in the university financial text data; |L1| and |L2| are the lengths of the strings to be matched in the university financial text data from two different sources; is the number of characters that undergo transposition in the string to be matched; F0 is the number of times the two strings to be matched undergo transposition at the same position; based on Jaro Distance, the Winkler algorithm is used for correction to obtain the Jaro-Winkler similarity value, which is expressed as follows: D2=D1+(ak(1-D1))(4) Where D2 is the Jaro-Winkler similarity between the university financial text data from two different sources; a is the prefix length of the two strings to be matched; k is a constant, set to 0.1.

Images