Multi-feature enterprise data fusion method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of data fusion, and provides a multi-feature enterprise data fusion method and device, electronic equipment and a storage medium, which comprises the following steps: acquiring enterprise data in a target format, determining a first to-be-fused data set; performing data matching on the first to-be-fused data set based on a first preset enterprise data feature, determining a second to-be-fused data set according to the enterprise data that fails to match; and performing data fusion on the second to-be-fused data set based on at least one of a preset naming feature matching algorithm and a preset integrated learning algorithm. According to the application, the enterprise data that fails to match is fused through the preset naming feature matching algorithm and the preset integrated learning algorithm, the recognition range coverage is improved, and the possibility of losing marketing opportunities is reduced.

Description

Technical Field

[0001] This application relates to the field of data fusion technology, and in particular to a method, apparatus, electronic device and storage medium for multi-feature enterprise data fusion. Background Technology

[0002] With the continuous development of the internet economy, almost all traditional industries, including advertising, manufacturing, news, communications, logistics, healthcare, education, tourism, and catering, are migrating to internet platforms. Faced with enterprise data from different data sources, in different formats, and of different types, relying solely on business registration information is no longer sufficient to fully unlock the data value of government and enterprise clients. Furthermore, discrepancies between registered and actual business addresses, shell companies, and fictitious enterprises may arise, impacting the capture of market application opportunities in various scenarios. Therefore, integrating, transforming, processing, and analyzing multi-source heterogeneous data to obtain more comprehensive, accurate, reliable, and usable fused data is the foundation and key to data mining and big data analysis. It also provides more data support and decision-making basis for marketing and expanding important clients across different fields and industries.

[0003] Current market methods for fusing multi-source heterogeneous data focus on the precise fusion of basic data features in business registration information. This involves using the unified social credit code and company name as two feature data points to perform a database-breaking match on multi-source heterogeneous data. However, no further processing is provided for a large amount of heterogeneous data for which the above features are not available or do not match. Although the accuracy is relatively high, the coverage of the identification range is low, increasing the possibility of lost marketing opportunities. Summary of the Invention

[0004] This application provides a multi-feature enterprise data fusion method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that do not provide further processing for large amounts of heterogeneous data that lack the aforementioned features or whose features do not match. Although the accuracy is high, the coverage of the identification range is low, increasing the possibility of lost marketing opportunities.

[0005] In a first aspect, embodiments of this application provide a multi-feature enterprise data fusion method, including:

[0006] Acquire enterprise data in the target format to determine the first dataset to be merged;

[0007] Based on the first preset enterprise data characteristics, the first dataset to be fused is matched with data, and the second dataset to be fused is determined based on the enterprise data that failed to match.

[0008] The second dataset to be fused is fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0009] In one embodiment, data fusion is performed on the second dataset to be fused based on a preset ensemble learning algorithm, including:

[0010] Based on the second preset enterprise data characteristics, a search is performed in a preset multi-source enterprise database to obtain the recall list data;

[0011] Based on the third preset enterprise data features and the second dataset to be fused, the recall list data is subjected to feature derivation processing to obtain target derived feature data;

[0012] Based on the target derived feature data and the preset ensemble learning algorithm, the fusion similarity score corresponding to the recall list data is determined;

[0013] The recall list data with a fusion similarity score higher than the threshold are fused with the second dataset to be fused.

[0014] In one embodiment, before determining the fusion similarity score corresponding to the recall list data based on the target derived feature data and the preset ensemble learning algorithm, the method further includes:

[0015] The recall list data is filtered based on preset data features.

[0016] In one embodiment, data fusion is performed on the second dataset to be fused based on a preset named feature matching algorithm, including:

[0017] The second dataset to be fused is filtered based on a preset type to obtain the target dataset to be fused;

[0018] Based on the pre-trained Hidden Markov Model, the target dataset to be fused is segmented and identified using named features to obtain segmented data of the target dataset to be fused.

[0019] Based on the text similarity algorithm, the fusion similarity score between the target data segmented into words is determined, and the target data segmented into words with the fusion similarity score higher than the threshold are fused.

[0020] In one embodiment, the named feature word segmentation and recognition of the target dataset to be fused based on a pre-trained Hidden Markov Model to obtain segmented target data to be fused includes:

[0021] Based on the pre-trained Hidden Markov Model, determine the probability matrix corresponding to the target dataset to be fused;

[0022] Based on the dynamic programming algorithm and the probability matrix, determine the position state sequence corresponding to the target dataset to be fused;

[0023] Based on the positional state sequence, the target data to be fused is determined as segmented word data.

[0024] In one embodiment, acquiring enterprise data in a target format and determining the first dataset to be merged includes:

[0025] Obtain the enterprise data in the target format;

[0026] The enterprise data is parsed to determine the first dataset to be merged.

[0027] In one embodiment, data parsing of the enterprise data includes:

[0028] Heterogeneous data processing is performed on the enterprise data to obtain the target enterprise data;

[0029] The target company's data is cleaned of abnormal data to complete the data parsing.

[0030] Secondly, embodiments of this application also provide a multi-feature enterprise data fusion device, including:

[0031] The first determination module is used to acquire enterprise data in the target format and determine the first dataset to be merged.

[0032] The second determining module is used to perform data matching on the first dataset to be fused based on the first preset enterprise data characteristics, and to determine the second dataset to be fused based on the enterprise data that failed to match.

[0033] The data fusion module is used to perform data fusion on the second dataset to be fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0034] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-feature enterprise data fusion method as described above.

[0035] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-feature enterprise data fusion method as described above.

[0036] The multi-feature enterprise data fusion method, apparatus, electronic device, and storage medium provided in this application embodiment acquire enterprise data in a target format and determine a first dataset to be fused; perform data matching on the first dataset to be fused based on a first preset enterprise data feature; determine a second dataset to be fused based on the enterprise data that failed to match; and perform data fusion on the second dataset based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm. This application embodiment improves the coverage of the identification range and reduces the possibility of missing marketing opportunities by fusing data on enterprise data that failed to match through a preset named feature matching algorithm and a preset ensemble learning algorithm. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts illustrating the multi-feature enterprise data fusion method provided in the embodiments of this application;

[0039] Figure 2 This is provided by the embodiments of this application. Figure 1 A flowchart illustrating step S110;

[0040] Figure 3 This is provided by the embodiments of this application. Figure 1 One of the flowcharts for step S130;

[0041] Figure 4 This is provided by the embodiments of this application. Figure 3 A flowchart illustrating step S320;

[0042] Figure 5 This is provided by the embodiments of this application. Figure 1 The second flowchart of step S130;

[0043] Figure 6 This is a schematic diagram of an enterprise data fusion process based on a preset ensemble learning algorithm provided in an embodiment of this application;

[0044] Figure 7 This is the second flowchart illustrating the multi-feature enterprise data fusion method provided in the embodiments of this application;

[0045] Figure 8 This is one of the principle block diagrams of the multi-feature enterprise data fusion device provided in the embodiments of this application;

[0046] Figure 9 This is the second principle block diagram of the multi-feature enterprise data fusion device provided in the embodiments of this application;

[0047] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Figure 1 This is one of the flowcharts illustrating the multi-feature enterprise data fusion method provided in this application embodiment. Figure 7 This is the second flowchart illustrating the multi-feature enterprise data fusion method provided in this application embodiment, as follows: Figure 1 and Figure 7 As shown in the embodiments of this application, a multi-feature enterprise data fusion method is provided, including:

[0050] S110, Obtain enterprise data in the target format and determine the first dataset to be merged;

[0051] S120, perform data matching on the first dataset to be fused based on the first preset enterprise data features, and determine the second dataset to be fused based on the enterprise data that failed to match.

[0052] S130, perform data fusion on the second dataset to be fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0053] Optionally, in step S110, the target format enterprise data may include enterprise data in different data formats such as images, text, and audio obtained through monitoring of information sources and internal channel data feedback, etc. The enterprise data is processed, and the processed enterprise data is combined to obtain the first dataset to be fused.

[0054] Obtaining enterprise data in the target format may include the following steps:

[0055] Identify data sources to be integrated: Based on the preset business scenarios, identify the data sources to be integrated, including industrial and commercial enterprise data, Internet channel data, and internal multi-channel data (including outbound calls, hotlines, investigations, etc.) to monitor data sources from multiple dimensions.

[0056] Design a data acquisition solution: For data sources from different channels and in different formats, design a suitable data acquisition solution, including acquisition methods, frequency, time range, and storage media. For example, acquire enterprise data from internet channels through web crawlers, and acquire enterprise data from internal channels through scheduled tasks.

[0057] Optionally, step S120 is to use a precise matching algorithm to achieve precise matching and fusion of the first dataset to be fused. The first preset enterprise data features include enterprise name and social unified credit code. Based on credible enterprise data, N enterprise data with the same enterprise name or social unified credit code are fused into 1 enterprise data, where N is an integer. The set of enterprise data that cannot be fused by the precise matching algorithm is used as the second dataset to be fused.

[0058] Optionally, in step S130, either a preset named feature matching algorithm or a preset ensemble learning algorithm can be selected to perform data fusion on the second dataset to be fused, or both can be selected simultaneously. For example, the preset named feature matching algorithm can be used to fuse the enterprise data in the second dataset to be fused first. Then, a third dataset to be fused can be constructed based on the enterprise data that the preset named feature matching algorithm cannot fuse. The preset ensemble learning algorithm can then be used to fuse the enterprise data in the third dataset to be fused. The steps for fusing the enterprise data in the third dataset to be fused are the same as those for fusing the enterprise data in the second dataset to be fused.

[0059] Optionally, the purpose of the preset named feature matching algorithm is to segment the enterprise data in the second dataset to be fused according to the named feature segmentation rules, and then calculate the fusion similarity score of the segmented enterprise data based on the text similarity algorithm. Enterprise data with a fusion similarity score higher than a certain threshold standard are fused, and unfused enterprise data are matched and fused based on the preset ensemble learning algorithm.

[0060] Optionally, the purpose of the preset ensemble learning algorithm is to use the LightGBM ensemble learning algorithm to calculate basic and derived features of industrial and commercial enterprise data that have not been identified by the named feature matching algorithm, and select the most similar industrial and commercial enterprise names based on the prediction results to achieve enterprise data fusion. Here, the LightGBM ensemble learning algorithm refers to an ensemble learning algorithm based on Gradient Boosting Decision Tree (GBDT).

[0061] It is understood that the embodiments of this application use a preset named feature matching algorithm and a preset ensemble learning algorithm to fuse data of enterprises that failed to match, thereby improving the coverage of the identification range and reducing the possibility of missing marketing opportunities.

[0062] Figure 2 This is provided by the embodiments of this application. Figure 1 A flowchart of step S110 is shown below. Figure 2 As shown, in one embodiment, obtaining enterprise data in a target format and determining the first dataset to be fused includes:

[0063] S210, Obtain the enterprise data in the target format;

[0064] S220, perform data parsing on the enterprise data to determine the first dataset to be merged.

[0065] Optionally, the enterprise data may be parsed, including:

[0066] S221, perform heterogeneous data processing on the enterprise data to obtain target enterprise data;

[0067] S222, Perform abnormal data cleaning on the target enterprise data to complete data parsing.

[0068] Specifically, heterogeneous data processing of the enterprise data includes:

[0069] Text data processing: For text data, NLP natural processing methods are used to process enterprise text data obtained by web crawlers. Mining and analysis algorithms such as word segmentation, stop word removal, part-of-speech tagging, and keyword extraction are used to obtain basic characteristic data of enterprises.

[0070] Image data processing: For image data, such as business licenses and storefront photos, OCR (Optical Character Recognition) text recognition technology is used to recognize and analyze the text in the image to extract the basic feature data of the enterprise, and typos in the recognized enterprise name, address, etc. are manually corrected.

[0071] Audio data processing: For audio data, the librosa library in Python is used to preprocess the audio, and then a recurrent neural network (RNN) deep learning model is used to convert the audio into text to identify and extract basic enterprise feature data. The typos in the identified enterprise names, addresses, etc. are manually corrected.

[0072] Specifically, the abnormal data cleaning of the target enterprise data includes:

[0073] The text data after heterogeneous data processing is cleaned and processed to remove duplicate data from the same source, erroneous data such as garbled characters during parsing, and invalid data without company names. Basic feature data such as the unified social credit code, company name, province, city, district / county, address, latitude and longitude, and contact information are extracted.

[0074] Q i ={fno f name f prov f city f country f addr f long f lat f tel};

[0075] Among them, Q i This represents the data for the i-th company.

[0076] From the parsed multi-source data, select the enterprise dataset Q = {Q1, Q2, Q3...} that contains enterprise names and unified social credit codes.

[0077] It is understood that the embodiments of this application provide a method for preprocessing enterprise data, which can improve the accuracy of enterprise data fusion by preprocessing enterprise data.

[0078] In one embodiment, data matching of the first dataset to be fused based on first preset enterprise data characteristics includes: using enterprise data Q from a certain information source on the Internet channel. base ={f no f name Based on the principle of}, data fusion is achieved using precise matching methods, such as regular expressions, specifically including the following sub-steps:

[0079] If Q base With Q i If the unified social credit code is the same, the data of N identical enterprises will be merged into one enterprise information, including basic characteristic data such as enterprise name, unified social credit code, registered address, business address, latitude and longitude, and contact information.

[0080] If Q base With Q i If companies have the same name, the data of N identical companies will be merged into one company information, including basic characteristic data such as company name, unified social credit code, registered address, business address, latitude and longitude, and contact information.

[0081] Based on the above precisely matched fused data, the fused similarity score is calculated. i =100 unintegrated enterprise data constitute the second dataset to be integrated.

[0082] It is understood that the embodiments of this application provide a solution for enterprise data fusion based on a precise matching algorithm, which can effectively ensure the correctness of enterprise data fusion.

[0083] The following example illustrates how to perform data fusion on a second dataset by selecting a preset named feature matching algorithm.

[0084] Figure 3 This is provided by the embodiments of this application. Figure 1 One of the flowcharts for step S130 is shown below. Figure 3 As shown, in one embodiment, data fusion is performed on the second dataset to be fused based on a preset named feature matching algorithm, including:

[0085] S310, the second dataset to be merged is filtered based on a preset type to obtain the target dataset to be merged; for example, the preset type can be an industry type, that is, enterprise data with the industry type of "company" is filtered out from the second dataset to be merged. The company type can be enterprise data that is marked as limited liability company, limited company, joint-stock company, or joint-stock company.

[0086] S320, Based on the pre-trained Hidden Markov Model, the target dataset to be fused is segmented and identified using named features to obtain segmented data of the target dataset to be fused; the named features can be administrative region + company name + industry + nature of the company. For example, text segmentation technology is used to segment the company name, such as the company name is Zhongshan City Sailian Trading Co., Ltd., where the administrative region is Zhongshan City, the company name is Sailian, the industry is trade, and the nature of the company is a limited company.

[0087] S330, Based on the text similarity algorithm, determine the fusion similarity score between the target data segmented into words to be fused, and perform data fusion on the target data segmented into words to be fused whose fusion similarity score is higher than the threshold.

[0088] Specifically, Q is the enterprise characteristic data derived from the enterprise name of a certain information source on the Internet. base ={f area f code f field f type Based on}, through the enterprise's administrative region f area Company name f code Two derived feature data sets are precisely matched and filtered to obtain candidate datasets Q = {Q1, Q2, Q3...}. Then, a text similarity algorithm is used to calculate the industry f of the enterprises in the candidate dataset Q. field Similarity i A threshold is set based on a certain value (e.g., T = 95%), and the data with the highest similarity >= threshold T is selected for fusion to obtain enterprise fused data, where the fusion similarity score is... i =Similar i *100.

[0089] Figure 4This is provided by the embodiments of this application. Figure 3 A flowchart of step S320 is shown below. Figure 4 As shown, in one embodiment, the named feature word segmentation and recognition of the target dataset to be fused based on the pre-trained Hidden Markov Model to obtain the target data to be fused word segmentation data includes:

[0090] S410, Based on the pre-trained Hidden Markov Model, determine the probability matrix corresponding to the target dataset to be fused;

[0091] This application employs a Hidden Markov Model (HMM) Chinese word segmentation method to segment and recognize the naming features of company and enterprise categories. An HMM model consists of five elements: a StatusSet, an ObservedSet, a TransProbMatrix, an EmitRobMatrix, and an InitialStatus. A common method for applying HMM to Chinese word segmentation is to use the Viterbi algorithm to solve for the StatusSet state value sequence given the ObservedSet, TransProbMatrix, EmitRobMatrix, and InitStatus. The StatusSet state values ​​are typically labeled BMES, representing the position of the character within a word: B (begin) indicates the start of a word, M (middle) indicates the middle position, E indicates the end position, and S indicates the character stands alone as a word. Adding spaces or other identifiers after S and E completes the word segmentation task.

[0092] Hidden Markov Models (HMMs) are trained using existing manually segmented corpora. Statistical analysis of the frequency of word occurrences at various positional states is used to calculate three probability matrices: initial probability, transition probability, and emission probability. The probability of the {B, E, M, S} state at the word's prefix position is calculated to obtain the initial state probability distribution. The probability of transitioning from one positional state to another within a word is calculated to obtain the transition probability matrix. The emission probability matrix is ​​obtained by calculating the corresponding character and its frequency at each positional state and using additive smoothing.

[0093] S420, Based on the dynamic programming algorithm and the probability matrix, determine the position state sequence corresponding to the target dataset to be fused;

[0094] Specifically, the dynamic programming algorithm is the Viterbi algorithm. The core idea of the Viterbi algorithm is to implement the shortest path through dynamic programming, that is, to obtain the position state sequence of each character in the text to be segmented at the maximum probability. First, the initial probability of the first character of the text to be segmented is calculated, and the corresponding probability value of each character in the aforementioned emission probability dictionary is found to calculate its emission probability; then, under 4 position states, the position state with the maximum probability of each character in the text to be segmented is solved, and the maximum probability position state sequence is obtained; finally, according to the position state of the last character of the text to be segmented, the one with the maximum probability is selected from the state sequences, and the function will return the maximum probability value and the corresponding position state sequence.

[0095] S430: Determine the target word segmentation data of the to-be-fused data according to the position state sequence.

[0096] After completing the model training and the definition of the Viterbi algorithm, the position state sequence with the maximum probability can be obtained after inputting the text to be segmented. Finally, the position state of each character in the text to be segmented is judged, the text is segmented and the result is output. For example, in the word segmentation task, the Chinese sentence "Zhongshan Sailiang Trading Co., Ltd." to be segmented is regarded as the observed sequence, and its word segmentation label sequence is regarded as the unknown state sequence. The label sequence corresponding to the word segmentation mode of "Zhongshan / Sailiang / Trading / Co., Ltd." is BMEBMEBEBMME.

[0097] It can be understood that the embodiments of the present application provide an enterprise data fusion method based on a preset named feature matching algorithm, which can be combined with an accurate matching algorithm to improve the coverage of the identification range and reduce the possibility of missing marketing opportunities.

[0098] In the following, an example is given to illustrate the data fusion of the second to-be-fused data set by separately selecting the preset ensemble learning algorithm.

[0099] Figure 5 is the second schematic flowchart of step S130 provided in the embodiments of the present application Figure 1 , Figure 6 is the schematic flowchart of enterprise data fusion based on the preset ensemble learning algorithm provided in the embodiments of the present application. As shown in Figure 5 and Figure 6 , in one embodiment, performing data fusion on the second to-be-fused data set based on the preset ensemble learning algorithm includes:

[0100] S510: Perform retrieval in a preset multi-source enterprise database based on the second preset enterprise data features to obtain recall list data. The second preset enterprise data features include enterprise name, enterprise address and contact information.

[0101] Specifically, the enterprise data Q from a certain information source through Internet channels can be base={f no f name f addr f long f lat f tel Based on the data from multiple sources, the system uses three data features—name, address, and contact information—to perform recall, resulting in a recall set Q = Q′. name ∪Q′ addr UQ′ tel .

[0102] Name Recall: A keyword list is generated for the company name using the Hidden Markov Model (HMM) method based on the correlation between the names and keywords. A subset Q′ of the recall results is then obtained by searching these keywords in a pre-defined multi-source company database. name For example: Longting Hotel, keyword list = [Longting, Hotel, Longting Hotel].

[0103] Address Recall: Obtain a subset of recall results Q′ by searching for addresses in a pre-set multi-source enterprise database. addr .

[0104] Contact information retrieval: A subset of retrieval results Q′ is obtained by searching contact information in a pre-set multi-source enterprise database. tel .

[0105] S520, based on the third preset enterprise data features and the second dataset to be fused, the recall list data is subjected to feature derivation processing to obtain target derived feature data; the third preset enterprise data features include name, address, latitude and longitude, and contact information.

[0106] Specifically, the recall list data is used to derive features from four basic features: name, address, latitude and longitude, and contact information. These features include 15 derived features such as: name maximum common subsequence, address maximum common subsequence, name + address maximum common subsequence, subsequence length ratio, intersection, telephone consistency identifier, and latitude and longitude distance. The specific sub-steps are as follows:

[0107] 1) Longest Common Subsequence (LCS): This algorithm uses dynamic programming to find the similarity between two strings in text processing. For example: Longting Restaurant in Huoshan County and Longting Restaurant (Huoshan Avenue Branch).

[0108] a. Define two arrays dp[i][j], where dp[i][j] represents the length of the longest common subsequence of the first i characters and the first j characters of Longting Hotel and Longting Hotel (Huoshan Avenue Branch) in Huoshan County.

[0109] b. For each position i and j, if the characters they correspond to are the same, then dp[i][j] = dp[i-1][j-1] + 1; otherwise dp[i][j] = max(dp[i-1][j], dp[i][j-1]).

[0110] c. The length of the longest common subsequence is dp[m][n] = 3, where m and n are the lengths of Huoshan County Longting Hotel and Longting Hotel (Huoshan Avenue Branch) respectively, the common subsequence is Longting Hotel, and the subsequence length ratio = the length of the longest common subsequence of a certain feature / the total length of the feature field of the benchmark enterprise. For example, the subsequence length ratio of Huoshan County Longting Hotel = 3 / 7.

[0111] 2) Intersection: This is a new set consisting of elements common to two sets. However, unlike LCS, the intersection is not affected by the order of the two text elements, so it can be used as a supplement to LCS. For example, the intersection of Longting Restaurant and Longting Restaurant (Huoshan Avenue Branch) in Huoshan County is [Ting, Shan, Huo, Fan, Dian]. The intersection ratio = the length of the intersection of a certain feature / the total length of the feature fields of the benchmark enterprise.

[0112] 3) Latitude and longitude distance: The latitude and longitude of the enterprise in the recall list Q will be compared with the coordinates of Q... base The distance difference is calculated by taking the difference between the latitude and longitude of the enterprise.

[0113] 4) Telephone consistency identifier: Determines whether the numbers are consistent (0 or 1) and quantifies the contact information feature vector.

[0114] S530, based on the target derived feature data and the preset ensemble learning algorithm, determine the fusion similarity score corresponding to the recall list data; specifically, based on the calculation results of derived features such as latitude and longitude distance calculation, address (name) maximum common subsequence, subsequence ratio, and name (address) intersection, perform recall ranking probability prediction Similar using the LGB ensemble learning method. i The formula for calculating the fusion similarity score is: Score i =Similar i *100.

[0115] S540, the recall list data with a fusion similarity score higher than the threshold is fused with the second dataset to be fused.

[0116] Specifically, a threshold is set based on a certain value (e.g., T = 70%), and the data with the highest probability prediction >= threshold T is selected for fusion to obtain enterprise fusion data.

[0117] In one embodiment, before determining the fusion similarity score corresponding to the recall list data based on the target derived feature data and the preset ensemble learning algorithm, the method further includes:

[0118] The recall list data is filtered based on preset data features.

[0119] Specifically, the preset data features include province f prou City f city District / County f country and latitude and longitude difference f dis Four data features. By utilizing provincial f prou City f city District / County f country and latitude and longitude difference f dis Four data features are used to filter and eliminate the recall list result set Q, resulting in a coarse-ranked recall result set Q′. Specifically, the following sub-steps are included: removing recall data that are inconsistent in terms of province, city, and district / county, and removing recall data with a latitude and longitude distance greater than 10 kilometers.

[0120] It is understood that the embodiments of this application provide a technical solution for enterprise data fusion based on a preset integrated learning algorithm, which can be combined with a precise matching algorithm to improve the coverage of the recognition range and reduce the possibility of missing marketing opportunities.

[0121] The specific steps for selecting a preset named feature matching algorithm and a preset ensemble learning algorithm to perform data fusion on the second dataset to be fused can be found in the above embodiment.

[0122] In summary, this application proposes a multi-feature enterprise data fusion method based on short text matching. It focuses on the naming features of industrial and commercial enterprise names, constructing an enterprise name matching algorithm using text segmentation and text similarity calculation. Based on the inherent connections between multiple data features and multi-source enterprise data, it builds an integrated learning algorithm based on multiple features such as enterprise name, latitude and longitude, contact information, and address. By integrating the above two short text similarity algorithms with existing multi-level algorithms for precise matching, the characteristics of enterprise fusion data are expanded, solving the problem of inaccurate identification of data features and correlations between multi-source enterprise data due to single data features. By introducing a similarity algorithm on top of precise matching, it solves the problem of low coverage caused by the inability to fuse a large amount of enterprise data due to precise matching. Through dynamic adjustment of the similarity algorithm threshold, it reduces the possibility of missing government and enterprise marketing opportunities caused by single-feature (100% matching) fusion data scenarios, while providing a fusion similarity score to quantify matching accuracy. The multi-feature enterprise data fusion method based on short text matching provided in this application fully utilizes multi-feature factor analysis to focus on the inherent connections between multi-source heterogeneous data, improving the coverage of enterprise data fusion and exhibiting good generalization ability.

[0123] The multi-feature enterprise data fusion apparatus provided in this application is described below. The multi-feature enterprise data fusion apparatus described below and the multi-feature enterprise data fusion method described above can be referred to in correspondence.

[0124] Figure 8 This is one of the principle block diagrams of the multi-feature enterprise data fusion device provided in this application, such as... Figure 8 As shown in the illustration, this application also provides a multi-feature enterprise data fusion device, including:

[0125] The first determining module 810 is used to acquire enterprise data in the target format and determine the first dataset to be fused.

[0126] The second determining module 820 is used to perform data matching on the first dataset to be fused based on the first preset enterprise data features, and to determine the second dataset to be fused based on the enterprise data that failed to match.

[0127] The data fusion module 830 is used to perform data fusion on the second dataset to be fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0128] In one embodiment, the data fusion module 830 is further configured to:

[0129] Based on the second preset enterprise data characteristics, a search is performed in a preset multi-source enterprise database to obtain the recall list data;

[0130] Based on the third preset enterprise data features and the second dataset to be fused, the recall list data is subjected to feature derivation processing to obtain target derived feature data;

[0131] Based on the target derived feature data and the preset ensemble learning algorithm, the fusion similarity score corresponding to the recall list data is determined;

[0132] The recall list data with a fusion similarity score higher than the threshold are fused with the second dataset to be fused.

[0133] In one embodiment, the data fusion module 830 is further configured to:

[0134] The recall list data is filtered based on preset data features.

[0135] In one embodiment, the data fusion module 830 is further configured to:

[0136] The second dataset to be fused is filtered based on a preset type to obtain the target dataset to be fused;

[0137] Based on the pre-trained Hidden Markov Model, the target dataset to be fused is segmented and identified using named features to obtain segmented data of the target dataset to be fused.

[0138] Based on the text similarity algorithm, the fusion similarity score between the target data segmented into words is determined, and the target data segmented into words with the fusion similarity score higher than the threshold are fused.

[0139] In one embodiment, the data fusion module 830 is further configured to:

[0140] Based on the pre-trained Hidden Markov Model, determine the probability matrix corresponding to the target dataset to be fused;

[0141] Based on the dynamic programming algorithm and the probability matrix, determine the position state sequence corresponding to the target dataset to be fused;

[0142] Based on the positional state sequence, the target data to be fused is determined as segmented word data.

[0143] In one embodiment, the first determining module 810 is further configured to:

[0144] Obtain the enterprise data in the target format;

[0145] The enterprise data is parsed to determine the first dataset to be merged.

[0146] In one embodiment, the first determining module 810 is further configured to:

[0147] Heterogeneous data processing is performed on the enterprise data to obtain the target enterprise data;

[0148] The target company's data is cleaned of abnormal data to complete the data parsing.

[0149] Figure 8 This is the second principle block diagram of the multi-feature enterprise data fusion device provided in the embodiments of this application. The embodiments of this application propose a multi-feature enterprise data fusion device based on short text matching, including a data acquisition module, a data parsing module, a precise matching algorithm module, a named feature matching algorithm module, and an ensemble learning algorithm module.

[0150] The data acquisition module acquires enterprise data in different formats, such as images, text, and audio, through methods such as monitoring information source channels and internal channel data feedback.

[0151] The data parsing module parses basic feature data from multi-source data using unified rules, mainly including company name, address, latitude and longitude, and contact number.

[0152] The precise matching algorithm module performs single-feature matching based on the precise matching of basic features and business registration data to obtain enterprise integrated data.

[0153] The named feature matching algorithm module performs multi-feature similarity calculations on industrial and commercial enterprise data that cannot be accurately matched, and obtains enterprise fusion data.

[0154] The ensemble learning algorithm module uses the LGB ensemble learning algorithm to calculate basic and derived features of industrial and commercial enterprise data that have not been identified by the named feature matching algorithm. Based on the prediction results, it selects the most similar industrial and commercial enterprise names to achieve enterprise data fusion.

[0155] The multi-feature enterprise data fusion device based on short text matching proposed in this application uses a variety of algorithms, such as precise matching, named feature matching, and ensemble learning algorithm based on multi-dimensional features, to construct a hierarchical heterogeneous data fusion algorithm to replace the fusion method based on precise matching in the prior art. For company data with identifiable named features, the named feature matching algorithm is used for similarity matching. For unidentified enterprise data, the prediction results of the ensemble learning algorithm are used for association matching, thereby realizing the fusion of heterogeneous industrial and commercial enterprise data from multiple channels and the quantification of fusion similarity scores.

[0156] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a multi-feature enterprise data fusion method, which includes:

[0157] Acquire enterprise data in the target format to determine the first dataset to be merged;

[0158] Based on the first preset enterprise data characteristics, the first dataset to be fused is matched with data, and the second dataset to be fused is determined based on the enterprise data that failed to match.

[0159] The second dataset to be fused is fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0160] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-feature enterprise data fusion method provided by the above methods, the method including:

[0162] Acquire enterprise data in the target format to determine the first dataset to be merged;

[0163] Based on the first preset enterprise data characteristics, the first dataset to be fused is matched with data, and the second dataset to be fused is determined based on the enterprise data that failed to match.

[0164] The second dataset to be fused is fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0165] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the multi-feature enterprise data fusion method provided by the methods described above, the method comprising:

[0166] Acquire enterprise data in the target format to determine the first dataset to be merged;

[0167] Based on the first preset enterprise data characteristics, the first dataset to be fused is matched with data, and the second dataset to be fused is determined based on the enterprise data that failed to match.

[0168] The second dataset to be fused is fused based on at least one of a preset named feature matching algorithm and a preset ensemble learning algorithm.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for fusing multi-feature enterprise data, characterized in that, include: Acquire enterprise data in the target format to determine the first dataset to be merged; Based on the first preset enterprise data characteristics, the first dataset to be fused is matched with data, and the second dataset to be fused is determined based on the enterprise data that failed to match. The second dataset to be fused is fused based on a preset named feature matching algorithm and a preset ensemble learning algorithm. Data fusion is performed on the second dataset to be fused based on a preset ensemble learning algorithm, including: Based on the second preset enterprise data characteristics, a search is performed in a preset multi-source enterprise database to obtain the recall list data; Based on the third preset enterprise data features and the second dataset to be fused, the recall list data is subjected to feature derivation processing to obtain target derived feature data; Based on the target derived feature data and the preset ensemble learning algorithm, the fusion similarity score corresponding to the recall list data is determined; The recall list data with a fusion similarity score higher than the threshold are fused with the second dataset to be fused. Data fusion is performed on the second dataset to be fused based on a preset named feature matching algorithm, including: The second dataset to be fused is filtered based on a preset type to obtain the target dataset to be fused; Based on the pre-trained Hidden Markov Model, the target dataset to be fused is segmented and identified using named features to obtain the segmented data of the target dataset to be fused. Based on the text similarity algorithm, the fusion similarity score between the segmented data of the target data to be fused is determined, and the segmented data of the target data to be fused with the fusion similarity score higher than the threshold are fused. The named feature word segmentation and recognition of the target dataset to be fused, based on a pre-trained Hidden Markov Model, yields the word segmentation data of the target dataset to be fused, including: Based on the pre-trained Hidden Markov Model, determine the probability matrix corresponding to the target dataset to be fused; Based on the dynamic programming algorithm and the probability matrix, determine the positional state sequence corresponding to the target dataset to be fused; Based on the positional state sequence, the word segmentation data of the target data to be fused is determined.

2. The multi-feature enterprise data fusion method according to claim 1, characterized in that, Before determining the fusion similarity score corresponding to the recall list data based on the target derived feature data and the preset ensemble learning algorithm, the method further includes: The recall list data is filtered based on preset data features.

3. The multi-feature enterprise data fusion method according to claim 1, characterized in that, The step of acquiring enterprise data in the target format and determining the first dataset to be merged includes: Obtain the enterprise data in the target format; The enterprise data is parsed to determine the first dataset to be merged.

4. The multi-feature enterprise data fusion method according to claim 3, characterized in that, Data parsing of the enterprise data includes: Heterogeneous data processing is performed on the enterprise data to obtain the target enterprise data; The target company's data is cleaned of abnormal data to complete the data parsing.

5. A multi-feature enterprise data fusion device, characterized in that, include: The first determination module is used to acquire enterprise data in the target format and determine the first dataset to be merged. The second determining module is used to perform data matching on the first dataset to be fused based on the first preset enterprise data characteristics, and to determine the second dataset to be fused based on the enterprise data that failed to match. The data fusion module is used to perform data fusion on the second dataset to be fused based on a preset named feature matching algorithm and a preset ensemble learning algorithm; Data fusion is performed on the second dataset to be fused based on a preset ensemble learning algorithm, including: Based on the second preset enterprise data characteristics, a search is performed in a preset multi-source enterprise database to obtain the recall list data; Based on the third preset enterprise data features and the second dataset to be fused, the recall list data is subjected to feature derivation processing to obtain target derived feature data; Based on the target derived feature data and the preset ensemble learning algorithm, the fusion similarity score corresponding to the recall list data is determined; The recall list data with a fusion similarity score higher than the threshold are fused with the second dataset to be fused. Data fusion is performed on the second dataset to be fused based on a preset named feature matching algorithm, including: The second dataset to be fused is filtered based on a preset type to obtain the target dataset to be fused; Based on the pre-trained Hidden Markov Model, the target dataset to be fused is segmented and identified using named features to obtain the segmented data of the target dataset to be fused. Based on the text similarity algorithm, the fusion similarity score between the segmented data of the target data to be fused is determined, and the segmented data of the target data to be fused with the fusion similarity score higher than the threshold are fused. The named feature word segmentation and recognition of the target dataset to be fused, based on a pre-trained Hidden Markov Model, yields the word segmentation data of the target dataset to be fused, including: Based on the pre-trained Hidden Markov Model, determine the probability matrix corresponding to the target dataset to be fused; Based on the dynamic programming algorithm and the probability matrix, determine the positional state sequence corresponding to the target dataset to be fused; Based on the positional state sequence, the word segmentation data of the target data to be fused is determined.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-feature enterprise data fusion method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-feature enterprise data fusion method as described in any one of claims 1 to 4.

Images