Bidirectional data matching analysis method and system based on machine learning

Abstract

The invention discloses a two-way data matching analysis method and system based on machine learning, and relates to the technical field of machine learning, and the method comprises the steps: analyzing all matched data, searching data group types with two-way matching, and extracting the matching features of the two-way matching data group types in a working environment; screening out a data group type with a low matching degree as a low-configuration data group, and searching transition data of each low-configuration data group; data sets with large deviation are removed, and a matching database is constructed; training a matching model through machine learning by utilizing the constructed matching database, judging whether the real-time data needs bidirectional matching by utilizing a data group in the matching database, and inputting the data acquired in real time into the matching model for matching; and after bidirectional matching is performed on the real-time data by utilizing the matching model, calculating the matching degree of each matching result, judging the performance of the matching model by utilizing the matching degree, and performing real-time optimization and updating on the matching model.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a two-way data matching analysis method and system based on machine learning. Background Art

[0002] In life, data matching exists everywhere, including one-way matching, multi-way matching and two-way matching. With the development of the times and the evolution of society, the types of data generated in life are more diverse, and the accompanying data that needs two-way matching also increases, such as: labor market matching, university admissions and other practical problems. With the rapid development of computer algorithms, methods for solving two-way matching problems have also sprung up one after another. With the popularization of the Internet, recommendation systems have become an important application of two-way matching problems. Collaborative filtering algorithms improve the accuracy of personalized recommendations by matching users and products. However, there are still some problems in the problem of two-way matching. Due to the diversity of data, some data need two-way matching, but they do not have the characteristics of two-way matching or have fewer characteristics. This causes the system to match such data incorrectly, and how to discover such data and solve it is crucial to the matching efficiency of the system. When the matching model of the system uses machine learning, the unfairness and bias of the training data can easily lead to errors in system matching or matching errors for some low-proportion matching data. Summary of the invention

[0003] The purpose of the present invention is to provide a two-way data matching analysis method and system based on machine learning to solve the problems raised in the prior art.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A two-way data matching analysis method based on machine learning, the method comprising the following steps:

[0006] S100, collecting all the data that have been matched in the historical working environment, analyzing all the matched data, finding the types of data groups that have two-way matching, and extracting the matching features of the types of two-way matching data groups in the working environment;

[0007] Furthermore, the specific steps of extracting matching features of the two-way matching data group types in the working environment are:

[0008] S101, collect all the matching data in the historical working environment as {S1, S2, S3, ..., S n}, S1, S2, S3, ..., S nIndicates collecting the 1st, 2nd, 3rd, ..., nth data for matching in the historical working environment, where n is a positive integer; the matching object for each data in the extracted history when matching is {Sd1, Sd2, Sd3, ..., Sd m}, Sd1, Sd2, Sd3,..., Sd m Indicates the 1st, 2nd, 3rd, ..., mth matching object when matching each type of data in the collected history, where m is a positive integer;

[0009] S102, sequentially extract each type of data to be matched in the history as the initial data Sz, search for the initial data Sz in the matching object data set of the remaining n-1 data extracted, and when the initial data Sz is found in the matching object data set, use the data corresponding to the matching object data set as the matching source Sy of the initial data; suppose that the matching source of the initial data obtained after searching in the matching object data set of n-1 data is {Sy1, Sy2, Sy3, ..., Sy j},Sy1,Sy2,Sy3,...,Sy j Indicates the 1st, 2nd, 3rd, ..., jth matching source of the initial data after searching, where j is a positive integer; n types of data in the history are searched to obtain n matching source data sets;

[0010] S103, after obtaining the matching source data set of each type of initial data in the history, the matching source data set of each type of initial data and the matching object data set are compared. When the two data sets have the same data, it is determined that the same data and the initial data have a two-way match, and the same data is determined as the paired data Sp of the initial data; each type of initial data in the history is judged, and finally the data group with a two-way match in the working environment is obtained as {(Sz, Sp)1, (Sz, Sp)2, (Sz, Sp)3, ..., (Sz, Sp) g}, (Sz, Sp)1, (Sz, Sp)2, (Sz, Sp)3, ..., (Sz, Sp) g Indicates that there are 1st, 2nd, 3rd, ..., gth data groups with bidirectional matching in the search history, where g is a positive integer;

[0011] By using the matching data in the history, according to the matching object of each data, the data with two-way matching can be accurately found, the type of data with two-way matching can be determined, and a data group can be constructed to match the data with two-way matching.

[0012] S104, after finding a data group with a two-way match in the working environment, extract all data features in the data group as {T1, T2, T3, ..., T k}, T1, T2, T3, ..., T krepresents the 1st, 2nd, 3rd, ..., kth data feature in each data group, where k is a positive integer; for each feature, the feature is transformed into a distribution with a mean of 0 and a standard deviation of 1, and the data group and the extracted data features are standardized. The formula is: In the formula, S' represents the standardized data set, S represents all the data with two-way matching in the collected history, Tp represents the average value of all data features, and Tst represents the standard deviation of all data features; the covariance matrix of the data features is calculated using the standardized data set, and the formula is: In the formula, C represents the covariance matrix of the data features, r represents the number of each data group in the collected history, and S' T represents the transposed matrix of the standardized data matrix, and S' represents the standardized data matrix; then the covariance matrix is ​​used to calculate the eigenvalue of each data feature, and the formula is: det(C-λB)=0, where det represents the determinant of the covariance matrix, λ represents the eigenvalue of the data feature, and B represents the unit matrix with the same dimension as the matrix C;

[0013] Before extracting matching features from the data set, all features are standardized to remove feature deviations caused by data variance. During standardization, all data and features are in the same dimension, making it easier to match features in advance.

[0014] S105. Calculate the characteristic value of each data feature for each data group using the method described in S104, divide the characteristic values ​​into three categories from large to small, and extract the data feature corresponding to the characteristic value of the largest category among the three categories as the matching feature of the data group. The formula is: In the formula, A represents the matching feature set in each data group, λ min represents the minimum value of the calculated eigenvalue, λ max Indicates calculating the maximum value of the feature value; finally, the matching feature set of each data group is calculated.

[0015] The eigenvalue of the covariance matrix indicates the variance of the data in a specific direction, and the corresponding eigenvector indicates the corresponding direction. The larger the eigenvalue of a certain feature is, the larger its proportion in the data is, the more information it contains, and the better it can reflect the characteristics of the data.

[0016] S200, calculating the matching degree of the data groups in the two-way matching history, screening out the types of data groups with low matching degree as low-matching data groups, analyzing each type of low-matching data group, and searching for transition data of each low-matching data group;

[0017] Furthermore, the specific steps for finding the transition data of each low-profile data group are:

[0018] S201. Calculate the matching degree of the data groups of the two-way matching in the history, the formula is: Mat = ta × E, where Mat represents the matching degree of each data group, ta represents the matching time of each data group in the history, and E represents the number of matching errors of each data group in the history; calculate the matching degree of g data groups; analyze the g matching degrees, calculate the standard deviation of the g matching degrees as MST and the average value MP; calculate the difference between each matching degree and the average value, the formula is: Mc = Mat-Mp, where Mc represents the difference between each matching degree and the average value, when Mc>MST, judge that the data group corresponding to the difference Mc is a low-matching data group, and screen the g data groups to obtain x low-matching data groups;

[0019] S202. After screening out the low-matching data group in the two-way matching, extract the matching object set of the two data in the matching data group, search for matching objects that exist in the matching object sets of the two data in the low-matching data group as common objects, and use the matching degree calculation formula in S201 to calculate the matching degree of each common object, and select the common object with the smallest matching degree as the transition data tr of the low-matching data group.

[0020] When performing bidirectional matching on data, there are some bidirectional matching data that have a low matching degree when only matching by themselves, and cannot achieve accurate and fast matching, which seriously affects the matching efficiency of the system. However, by searching for transition data and quickly matching the transition data, the low-matching data group can achieve fast matching, which greatly increases the matching efficiency of the system.

[0021] S300, performing deviation analysis on the calculated data groups and the corresponding matching features, eliminating the data groups with large deviations, and using the matching features of the eliminated data groups and the low-matching data groups to build a matching database;

[0022] Furthermore, the specific steps of constructing a matching database by combining the matching features of the eliminated data group with the low-matching data group are as follows:

[0023] S301, collect the usage frequency of each matching feature in each data group in the history when matching {f1, f2, f3, ..., f e}, f1, f2, f3, ..., f e Indicates the usage frequency of the 1st, 2nd, 3rd, ..., eth matching features in each data group in the collected history, where e is a positive integer; calculates the standard deviation fst and the average value fp of the e usage frequencies, and screens out the deviation features among the e matching features, the formula is: f<fp-fst; when the usage frequency is less than the value of the average value minus the standard deviation, the corresponding matching feature is judged to be a deviation feature, and the deviation feature is eliminated;

[0024] S302, screening the matching features in each data group, finding the deviation features, eliminating the deviation features, and obtaining a screening matching feature data set; combining the low-matching data group, transitional data and the screening matching feature data set to construct a matching database.

[0025] S400, using the constructed matching database to train the matching model through machine learning, collecting data generated in real-time work, using the data group in the matching database to determine whether the real-time data requires two-way matching, and inputting the real-time collected data into the matching model for matching;

[0026] Furthermore, the specific steps of inputting the real-time collected data into the matching model for matching are:

[0027] S401. After screening the matching features in the searched data group, a vector tower of two data (Sz, Sp) in the data group is constructed using a neural network algorithm. The vector tower corresponding to Sz in the two data in the data group is set as tower α, and the vector tower corresponding to Sp is set as tower β. The matching features corresponding to the two data in the data group in the matching database are respectively input into the corresponding vector towers, and an embedding vector is generated through the neural network algorithm; the embedding vector of the Sz data output by tower α is set as u, and the embedding vector of the Sp data output by tower β is set as v; the cosine formula is used to calculate the similarity of the embedding vectors of the two data, and the formula is: In the formula, cos(u, v) represents the cosine similarity of two data, and u·v represents the dot product of embedding vectors u and v. The matching model is purchased using the double tower structure and cosine similarity.

[0028] A double-tower structure is constructed using a neural network, and cosine similarity is calculated. The two data are matched using cosine similarity. After the matching model is constructed, the matching database is input into the matching model for training using machine learning. The matching threshold is obtained to achieve automated data matching.

[0029] S402, inputting the data group and the corresponding matching features in the matching database into the matching model for training, and obtaining the matching threshold in the history as mat_y after training;

[0030] S403, collect the data generated in the real-time work, use the data group in the matching database to determine whether the real-time data needs two-way matching, and when it is determined that the data type collected in real time belongs to the data group in the matching database, input the two types of data collected in real time into the double towers of the matching model, output the embedding vectors respectively, and calculate the real-time cosine similarity mat_s; when mat_s≥mat_y, the matching is successful; when mat_s<mat_y, the matching fails;

[0031] When it is determined that the data collected in real time belongs to the low-matching data group in the matching database, the transition data tr of the low-matching data group in real-time work is extracted, and the transition data groups are constructed as (Sz, tr) and (tr, Sp). The cosine similarity of the two transition data groups is calculated using the matching model as mat_s1 and mat_s2; when mat_s1≥mat_y and mat_s2≥mat_y, the match is judged to be successful; otherwise, the match fails.

[0032] S500: After using the matching model to perform bidirectional matching on the real-time data, calculate the matching degree of each matching result, use the matching degree to judge the performance of the matching model, and optimize and update the matching model in real time.

[0033] Furthermore, the specific steps for real-time optimization and updating of the matching model are as follows:

[0034] S601. When using the matching model to perform real-time matching on the data in the working environment, collect the matching results in history, count the number of correct matches and the total number of matches, and calculate the proportion of correct matches. The formula is: In the formula, per represents the percentage of correct matches, nuz represents the number of correct matches, and num represents the total number of matches;

[0035] S602. Calculate the proportion of correct model matching times during normal operation in history as pery. When per<pery, determine that the model performance has degraded, collect a new round of data from the matching database, use the new matching database to train the matching model, and optimize and update the model.

[0036] After matching the data using the matching model, the matching results are used to evaluate the model performance. When the performance degrades, the matching database is updated and a new database is used again for a new round of machine learning to train the matching model and optimize the updated model; this increases the service life and breadth of the matching model and makes it more suitable for the working environment.

[0037] A two-way data matching and analysis system based on machine learning, the two-way data matching and analysis system comprising a data collection module, a data group search module, a database construction module, a model building module, a model training module, a matching module and a model optimization module;

[0038] The data collection module is used to collect matching data in the historical working environment;

[0039] The data group search module is used to search for data types with bidirectional matching according to the matching objects of each data in the history, construct data groups, and extract matching features;

[0040] The database construction module is used to search for low-matching data groups in the data groups, determine transition data of the low-matching data groups, and construct a matching database in combination with the data groups and corresponding matching features;

[0041] The module building module is used to construct a double tower structure through a neural network, output an embedding vector of a data set, and calculate cosine similarity to construct a matching model;

[0042] The model training module is used to train the matching model using the data group in the matching database to obtain a matching threshold;

[0043] The matching module is used to collect the data generated by the work in real time, determine whether matching is required, and use the matching model to match the real-time data;

[0044] The model optimization module is used to analyze and calculate the matching results of the matching model, judge the model performance, and complete the model optimization and update by updating the matching database and performing a new round of training on the matching model.

[0045] The data group search module includes a bidirectional matching search unit and a feature extraction unit;

[0046] The two-way matching search unit is used to analyze the data with matching in the history, find the matching object of each data, and build a data group by searching the data with two-way matching through the matching object;

[0047] The feature extraction unit is used to calculate the eigenvalue of each feature using the covariance matrix, and to determine the matching features of the data in each data group through the eigenvalue.

[0048] The database construction module includes a low-profile data group search unit and a transition data search unit;

[0049] The low-matching data group search unit is used to calculate the matching degree of the bidirectionally matched data in the history, search for the data group with low matching degree, and determine the low-matching data group;

[0050] The transition data search unit is used to determine the low-profile data group and then use the matching object set of two data in the low-profile data group to search for transition data of the low-profile data group.

[0051] The matching module includes a normal matching unit and a transition matching unit;

[0052] The normal matching unit is used to input the real-time collected data into the matching model, calculate the cosine similarity, and perform matching;

[0053] The transition matching unit is used to construct a transition data group using transition data when the real-time data is data in the low-profile data group, and input the transition data group into the matching model to perform data matching.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. The present invention searches for low-profile data groups, extracts transition data from the low-profile data groups, and uses the transition data to perform bidirectional matching on two types of data from the low-profile data groups, thereby greatly increasing the working efficiency of the system and reducing the matching time and error rate of the low-profile data groups.

[0056] 2. Before training the constructed model, the present invention performs deviation analysis on the data in the matching database and removes the data with large deviation, thereby avoiding the problem of matching deviation and unfair matching caused by the deviation of the training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a module distribution diagram of a two-way data matching analysis system based on machine learning in the present invention;

[0058] Figure 2 The present invention is a schematic diagram of the steps of a two-way data matching analysis method based on machine learning. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution.

[0061] A two-way data matching analysis method based on machine learning, the method comprising the following steps:

[0062] S100, collecting all the data that have been matched in the historical working environment, analyzing all the matched data, finding the types of data groups that have two-way matching, and extracting the matching features of the types of two-way matching data groups in the working environment;

[0063] The specific steps for extracting matching features of bidirectional matching data group types in the working environment are:

[0064] S101, collect all the matching data in the historical working environment as {S1, S2, S3, ..., S n}, S1, S2, S3, ..., S nIndicates collecting the 1st, 2nd, 3rd, ..., nth data for matching in the historical working environment, where n is a positive integer; the matching object for each data in the extracted history when matching is {Sd1, Sd2, Sd3, ..., Sd m}, Sd1, Sd2, Sd3,..., Sd m Indicates the 1st, 2nd, 3rd, ..., mth matching object when matching each type of data in the collected history, where m is a positive integer;

[0065] S102, sequentially extract each type of data to be matched in the history as the initial data Sz, search for the initial data Sz in the matching object data set of the remaining n-1 data extracted, and when the initial data Sz is found in the matching object data set, use the data corresponding to the matching object data set as the matching source Sy of the initial data; suppose that the matching source of the initial data obtained after searching in the matching object data set of n-1 data is {Sy1, Sy2, Sy3, ..., Sy j},Sy1,Sy2,Sy3,...,Sy j Indicates the 1st, 2nd, 3rd, ..., jth matching source of the initial data after searching, where j is a positive integer; n types of data in the history are searched to obtain n matching source data sets;

[0066] S103, after obtaining the matching source data set of each type of initial data in the history, the matching source data set of each type of initial data and the matching object data set are compared. When the two data sets have the same data, it is determined that the same data and the initial data have a two-way match, and the same data is determined as the paired data Sp of the initial data; each type of initial data in the history is judged, and finally the data group with a two-way match in the working environment is obtained as {(Sz, Sp)1, (Sz, Sp)2, (Sz, Sp)3, ..., (Sz, Sp) g}, (Sz, Sp)1, (Sz, Sp)2, (Sz, Sp)3, ..., (Sz, Sp) g Indicates that there are 1st, 2nd, 3rd, ..., gth data groups with bidirectional matching in the search history, where g is a positive integer;

[0067] By using the matching data in the history, according to the matching object of each data, the data with two-way matching can be accurately found, the type of data with two-way matching can be determined, and a data group can be constructed to match the data with two-way matching.

[0068] For example, when searching for two-way matching data on a shopping platform, the two-way matching data groups found are users and products;

[0069] S104, after finding a data group with a two-way match in the working environment, extract all data features in the data group as {T1, T2, T3, ..., T k}, T1, T2, T3, ..., T k represents the 1st, 2nd, 3rd, ..., kth data feature in each data group, where k is a positive integer; for each feature, the feature is transformed into a distribution with a mean of 0 and a standard deviation of 1, and the data group and the extracted data features are standardized. The formula is: In the formula, S' represents the standardized data set, S represents all the data with two-way matching in the collected history, Tp represents the average value of all data features, and Tst represents the standard deviation of all data features; the covariance matrix of the data features is calculated using the standardized data set, and the formula is: In the formula, C represents the covariance matrix of the data features, r represents the number of each data group in the collected history, and S' T represents the transposed matrix of the standardized data matrix, and S' represents the standardized data matrix; then the covariance matrix is ​​used to calculate the eigenvalue of each data feature, and the formula is: det(C-λB)=0, where det represents the determinant of the covariance matrix, λ represents the eigenvalue of the data feature, and B represents the unit matrix with the same dimension as the matrix C;

[0070] Before extracting matching features from the data set, all features are standardized to remove feature deviations caused by data variance. During standardization, all data and features are in the same dimension, making it easier to match features in advance.

[0071] S105. Calculate the characteristic value of each data feature for each data group using the method described in S104, divide the characteristic values ​​into three categories from large to small, and extract the data feature corresponding to the characteristic value of the largest category among the three categories as the matching feature of the data group. The formula is: In the formula, A represents the matching feature set in each data group, λ min represents the minimum value of the calculated eigenvalue, λ max Indicates calculating the maximum value of the feature value; finally, the matching feature set of each data group is calculated.

[0072] The eigenvalue of the covariance matrix indicates the variance of the data in a specific direction, and the corresponding eigenvector indicates the corresponding direction. The larger the eigenvalue of a certain feature is, the larger its proportion in the data is, the more information it contains, and the better it can reflect the characteristics of the data.

[0073] Extract matching features between users and products on shopping platforms, such as item type, brand, and price;

[0074] S200, calculating the matching degree of the data groups in the two-way matching history, screening out the types of data groups with low matching degree as low-matching data groups, analyzing each type of low-matching data group, and searching for transition data of each low-matching data group;

[0075] The specific steps to find the transition data for each low-profile data group are:

[0076] S201. Calculate the matching degree of the data groups of the two-way matching in the history, the formula is: Mat = ta × E, where Mat represents the matching degree of each data group, ta represents the matching time of each data group in the history, and E represents the number of matching errors of each data group in the history; calculate the matching degree of g data groups; analyze the g matching degrees, calculate the standard deviation of the g matching degrees as MST and the average value MP; calculate the difference between each matching degree and the average value, the formula is: Mc = Mat-Mp, where Mc represents the difference between each matching degree and the average value, when Mc>MST, judge that the data group corresponding to the difference Mc is a low-matching data group, and screen the g data groups to obtain x low-matching data groups;

[0077] S202. After screening out the low-matching data group in the two-way matching, extract the matching object set of the two data in the matching data group, search for matching objects that exist in the matching object sets of the two data in the low-matching data group as common objects, and use the matching degree calculation formula in S201 to calculate the matching degree of each common object, and select the common object with the smallest matching degree as the transition data tr of the low-matching data group.

[0078] When performing bidirectional matching on data, there are some bidirectional matching data that have a low matching degree when only matching by themselves, and cannot achieve accurate and fast matching, which seriously affects the matching efficiency of the system. However, by searching for transition data and quickly matching the transition data, the low-matching data group can achieve fast matching, which greatly increases the matching efficiency of the system.

[0079] By analyzing the shopping platform, it is determined that the low-matching data group is manufacturers and users; the transitional data extracted is retailers;

[0080] S300, performing deviation analysis on the calculated data groups and the corresponding matching features, eliminating the data groups with large deviations, and using the matching features of the eliminated data groups and the low-matching data groups to build a matching database;

[0081] The specific steps of building a matching database by combining the matching features of the eliminated data group with the low-matching data group are as follows:

[0082] S301, collect the usage frequency of each matching feature in each data group in the history when matching {f1, f2, f3, ..., f e}, f1, f2, f3, ..., f e Indicates the usage frequency of the 1st, 2nd, 3rd, ..., eth matching features in each data group in the collected history, where e is a positive integer; calculates the standard deviation fst and the average value fp of the e usage frequencies, and screens out the deviation features among the e matching features, the formula is: f<fp-fst; when the usage frequency is less than the value of the average value minus the standard deviation, the corresponding matching feature is judged to be a deviation feature, and the deviation feature is eliminated;

[0083] S302, screening the matching features in each data group, finding the deviation features, eliminating the deviation features, and obtaining a screening matching feature data set; combining the low-matching data group, transitional data and the screening matching feature data set to construct a matching database.

[0084] S400, using the constructed matching database to train the matching model through machine learning, collecting data generated in real-time work, using the data group in the matching database to determine whether the real-time data requires two-way matching, and inputting the real-time collected data into the matching model for matching;

[0085] The specific steps of inputting the real-time collected data into the matching model for matching are:

[0086] S401. After screening the matching features in the searched data group, a vector tower of two data (Sz, Sp) in the data group is constructed using a neural network algorithm. The vector tower corresponding to Sz in the two data in the data group is set as tower α, and the vector tower corresponding to Sp is set as tower β. The matching features corresponding to the two data in the data group in the matching database are respectively input into the corresponding vector towers, and an embedding vector is generated through the neural network algorithm; the embedding vector of the Sz data output by tower α is set as u, and the embedding vector of the Sp data output by tower β is set as v; the cosine formula is used to calculate the similarity of the embedding vectors of the two data, and the formula is: In the formula, cos(u, v) represents the cosine similarity of two data, and u·v represents the dot product of embedding vectors u and v. The matching model is purchased using the double tower structure and cosine similarity.

[0087] The double tower structure is constructed using a neural network, and the cosine similarity is calculated. The two data are matched using the cosine similarity. After the matching model is constructed, the matching database is input into the matching model for training using machine learning; the matching threshold is obtained to achieve automatic data matching. The value of cosine similarity is between -1 and 1. When the value is 1, it means that the two vectors are exactly the same (same direction), the value is 0, which means that the two vectors are perpendicular to each other and have no similarity. The value of -1 means that the two vectors are in completely opposite directions.

[0088] S402, inputting the data group and the corresponding matching features in the matching database into the matching model for training, and obtaining the matching threshold in the history as mat_y after training; and obtaining the matching threshold as 0.9 after training the matching model in the shopping platform;

[0089] S403, collect the data generated in the real-time work, use the data group in the matching database to determine whether the real-time data needs two-way matching, and when it is determined that the data type collected in real time belongs to the data group in the matching database, input the two types of data collected in real time into the double towers of the matching model, output the embedding vectors respectively, and calculate the real-time cosine similarity mat_s; when mat_s≥mat_y, the matching is successful; when mat_s<mat_y, the matching fails;

[0090] When it is determined that the real-time collected data belongs to the low-profile data group in the matching database, the transition data tr of the low-profile data group in real-time work is extracted, and the transition data groups are constructed as (Sz, tr) and (tr, Sp). The cosine similarity of the two transition data groups is calculated using the matching model as mat_s1 and mat_s2; when mat_s1≥mat_y and mat_s2≥mat_y, the match is judged to be successful; otherwise, the match fails. By implementing the collected data, the cosine similarity of the manufacturer and the retailer is set to 1.4, and the cosine similarity of the retailer and the user is set to 0.6; it is judged that the real-time matching of the manufacturer and the user fails, and they are not the customers needed by both parties;

[0091] S500: After using the matching model to perform bidirectional matching on the real-time data, calculate the matching degree of each matching result, use the matching degree to judge the performance of the matching model, and optimize and update the matching model in real time.

[0092] The specific steps for real-time optimization and updating of the matching model are as follows:

[0093] S601. When using the matching model to perform real-time matching on the data in the working environment, collect the matching results in history, count the number of correct matches and the total number of matches, and calculate the proportion of correct matches. The formula is: In the formula, per represents the percentage of correct matches, nuz represents the number of correct matches, and num represents the total number of matches;

[0094] S602. Calculate the proportion of correct model matching times during normal operation in history as pery. When per<pery, determine that the model performance has degraded, collect a new round of data from the matching database, use the new matching database to train the matching model, and optimize and update the model.

[0095] After matching the data using the matching model, the matching results are used to evaluate the model performance. When the performance degrades, the matching database is updated and a new database is used again for a new round of machine learning to train the matching model and optimize the updated model; this increases the service life and breadth of the matching model and makes it more suitable for the working environment.

[0096] A two-way data matching and analysis system based on machine learning, the two-way data matching and analysis system comprising a data collection module, a data group search module, a database construction module, a model building module, a model training module, a matching module and a model optimization module;

[0097] The data collection module is used to collect matching data in the historical working environment;

[0098] The data group search module is used to search for data types with bidirectional matching according to the matching objects of each data in the history, construct data groups, and extract matching features;

[0099] The database construction module is used to search for low-matching data groups in the data groups, determine transition data of the low-matching data groups, and construct a matching database in combination with the data groups and corresponding matching features;

[0100] The module building module is used to construct a double tower structure through a neural network, output an embedding vector of a data set, and calculate cosine similarity to construct a matching model;

[0101] The model training module is used to train the matching model using the data group in the matching database to obtain a matching threshold;

[0102] The matching module is used to collect the data generated by the work in real time, determine whether matching is required, and use the matching model to match the real-time data;

[0103] The model optimization module is used to analyze and calculate the matching results of the matching model, judge the model performance, and complete the model optimization and update by updating the matching database and performing a new round of training on the matching model.

[0104] The data group search module includes a bidirectional matching search unit and a feature extraction unit;

[0105] The two-way matching search unit is used to analyze the data with matching in the history, find the matching object of each data, and build a data group by searching the data with two-way matching through the matching object;

[0106] The feature extraction unit is used to calculate the eigenvalue of each feature using the covariance matrix, and to determine the matching features of the data in each data group through the eigenvalue.

[0107] The database construction module includes a low-profile data group search unit and a transition data search unit;

[0108] The low-matching data group search unit is used to calculate the matching degree of the bidirectionally matched data in the history, search for the data group with low matching degree, and determine the low-matching data group;

[0109] The transition data search unit is used to determine the low-profile data group and then use the matching object set of two data in the low-profile data group to search for transition data of the low-profile data group.

[0110] The matching module includes a normal matching unit and a transition matching unit;

[0111] The normal matching unit is used to input the real-time collected data into the matching model, calculate the cosine similarity, and perform matching;

[0112] The transition matching unit is used to construct a transition data group using transition data when the real-time data is data in the low-profile data group, and input the transition data group into the matching model to perform data matching.

[0113] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A two-way data matching analysis method based on machine learning, Features: The method comprises the following steps: S100, collecting all the data that have been matched in the historical working environment, analyzing all the matched data, finding the types of data groups that have two-way matching, and extracting the matching features of the types of two-way matching data groups in the working environment; S200, calculating the matching degree of the data groups in the two-way matching history, screening out the types of data groups with low matching degree as low-matching data groups, analyzing each type of low-matching data group, and searching for transition data of each low-matching data group; S300, performing deviation analysis on the calculated data groups and the corresponding matching features, eliminating the data groups with large deviations, and using the matching features of the eliminated data groups and the low-matching data groups to build a matching database; S400, using the constructed matching database to train the matching model through machine learning, collecting data generated in real-time work, using the data group in the matching database to determine whether the real-time data requires two-way matching, and inputting the real-time collected data into the matching model for matching; S500: After using the matching model to perform bidirectional matching on the real-time data, calculate the matching degree of each matching result, use the matching degree to judge the performance of the matching model, and optimize and update the matching model in real time.

2. According to the two-way data matching analysis method based on machine learning according to claim 1, Features: The specific steps of extracting the matching features of the two-way matching data group types in the working environment in S100 are: S101. Collect all the data for matching in the historical working environment as {S 1 , S 2 , S 3 ,...,S n }, S 1 , S 2 , S 3 ,...,S n Indicates collecting the 1st, 2nd, 3rd, ..., nth data for matching in the historical working environment, where n is a positive integer; the matching object for each data in the extracted history when matching is {Sd 1 , Sd 2 , Sd 3 , ..., Sd m }, Sd 1 , Sd 2 , Sd 3 , ..., Sd m Indicates the 1st, 2nd, 3rd, ..., mth matching object when matching each type of data in the collected history, where m is a positive integer; S102, sequentially extracting each type of data to be matched in the history as initial data Sz, searching for the initial data Sz in the matching object data set of the remaining n-1 data extracted, and when the initial data Sz is found in the matching object data set, using the data corresponding to the matching object data set as the matching source Sy of the initial data; Suppose the matching source of the initial data obtained after searching in the matching object data set of n-1 data is {Sy 1 , Sy 2 , Sy 3 ,...,Sy j }, Sy 1 , Sy 2 , Sy 3 ,...,Sy j Indicates the 1st, 2nd, 3rd, ..., jth matching source of the initial data after searching, where j is a positive integer; n types of data in the history are searched to obtain n matching source data sets; S103, after obtaining the matching source data set of each type of initial data in the history, the matching source data set of each type of initial data is compared with the matching object data set. When the two data sets have the same data, it is determined that the same data and the initial data have a two-way match, and the same data is determined as the paired data Sp of the initial data; each type of initial data in the history is judged, and finally the data set with a two-way match in the working environment is obtained as {(Sz, Sp) 1 、(Sz,Sp) 2 、(Sz,Sp) 3 , ..., (Sz, Sp) g }, (Sz, Sp) 1 、(Sz,Sp) 2 、(Sz,Sp) 3 , ..., (Sz, Sp) g Indicates that there are 1st, 2nd, 3rd, ..., gth data groups with bidirectional matching in the search history, where g is a positive integer; S104, after finding a data group with a two-way match in the working environment, extract all data features in the data group as {T 1 , T 2 , T 3 , ..., T k }, T 1 , T 2 , T 3 , ..., T k represents the 1st, 2nd, 3rd, ..., kth data feature in each data group, where k is a positive integer; for each feature, the feature is transformed into a distribution with a mean of 0 and a standard deviation of 1, and the data group and the extracted data features are standardized. The formula is: In the formula, S' represents the standardized data set, S represents all the data with two-way matching in the collected history, Tp represents the average value of all data features, and Tst represents the standard deviation of all data features; the covariance matrix of the data features is calculated using the standardized data set, and the formula is: In the formula, C represents the covariance matrix of the data features, r represents the number of each data group in the collected history, and S' T represents the transposed matrix of the standardized data matrix, and S' represents the standardized data matrix; then the covariance matrix is ​​used to calculate the eigenvalue of each data feature, and the formula is: det(C-λB)=0, where det represents the determinant of the covariance matrix, λ represents the eigenvalue of the data feature, and B represents the unit matrix with the same dimension as the matrix C; S105. Calculate the characteristic value of each data feature for each data group using the method described in S104, divide the characteristic values ​​into three categories from large to small, and extract the data feature corresponding to the characteristic value of the largest category among the three categories as the matching feature of the data group. The formula is: In the formula, A represents the matching feature set in each data group, λ min represents the minimum value of the calculated eigenvalue, λ max Indicates calculating the maximum value of the feature value; finally, the matching feature set of each data group is calculated.

3. According to the two-way data matching analysis method based on machine learning according to claim 2, Features: The specific steps of searching for transition data of each low-profile data group in S200 are: S201. Calculate the matching degree of the data groups of the two-way matching in the history, the formula is: Mat = ta × E, where Mat represents the matching degree of each data group, ta represents the matching time of each data group in the history, and E represents the number of matching errors of each data group in the history; calculate the matching degree of g data groups; analyze the g matching degrees, calculate the standard deviation of the g matching degrees as MST and the average value MP; calculate the difference between each matching degree and the average value, the formula is: Mc = Mat-Mp, where Mc represents the difference between each matching degree and the average value, when Mc>MST, judge that the data group corresponding to the difference Mc is a low-matching data group, and screen the g data groups to obtain x low-matching data groups; S202. After screening out the low-matching data group in the two-way matching, extract the matching object set of the two data in the matching data group, search for matching objects that exist in the matching object sets of the two data in the low-matching data group as common objects, and use the matching degree calculation formula in S201 to calculate the matching degree of each common object, and select the common object with the smallest matching degree as the transition data tr of the low-matching data group.

4. According to the two-way data matching analysis method based on machine learning according to claim 3, Features: The specific steps of constructing a matching database by combining the matching features of the eliminated data group and the low-matching data group in S300 are: S301, collect the usage frequency of each matching feature in each data group in the history when performing matching {f 1 、f 2 、f 3 , ..., f e }, f 1 、f 2 、f 3 , ..., f e Indicates the usage frequency of the 1st, 2nd, 3rd, ..., eth matching features in each data group in the collected history, where e is a positive integer; calculates the standard deviation fst and the average value fp of the e usage frequencies, and screens out the deviation features among the e matching features, the formula is: f<fp-fst; when the usage frequency is less than the value of the average value minus the standard deviation, the corresponding matching feature is judged to be a deviation feature, and the deviation feature is eliminated; S302, screening the matching features in each data group, finding the deviation features, eliminating the deviation features, and obtaining a screening matching feature data set; combining the low-matching data group, transitional data and the screening matching feature data set to construct a matching database.

5. According to the two-way data matching analysis method based on machine learning according to claim 4, Features: The specific steps of inputting the real-time collected data into the matching model for matching in S400 are: S401. After screening the matching features in the searched data group, a vector tower of two data (Sz, Sp) in the data group is constructed using a neural network algorithm. The vector tower corresponding to Sz in the two data in the data group is set as tower α, and the vector tower corresponding to Sp is set as tower β. The matching features corresponding to the two data in the data group in the matching database are respectively input into the corresponding vector towers, and an embedding vector is generated through the neural network algorithm; the embedding vector of the Sz data output by tower α is set as u, and the embedding vector of the Sp data output by tower β is set as v; the cosine formula is used to calculate the similarity of the embedding vectors of the two data, and the formula is: In the formula, cos(u, v) represents the cosine similarity of two data, and u·v represents the dot product of embedding vectors u and v. The matching model is purchased using the double tower structure and cosine similarity. S402, inputting the data group and the corresponding matching features in the matching database into the matching model for training, and obtaining the matching threshold in the history as mat_y after training; S403, collect the data generated in the real-time work, use the data group in the matching database to determine whether the real-time data needs two-way matching, and when it is determined that the data type collected in real time belongs to the data group in the matching database, input the two types of data collected in real time into the double towers of the matching model, output the embedding vectors respectively, and calculate the real-time cosine similarity mat_s; when mat_s≥mat_y, the matching is successful; when mat_s<mat_y, the matching fails; When it is determined that the data collected in real time belongs to the low-matching data group in the matching database, the transition data tr of the low-matching data group in real-time work is extracted, and the transition data groups are constructed as (Sz, tr) and (tr, Sp). The cosine similarity of the two transition data groups is calculated using the matching model as mat_s1 and mat_s2; when mat_s1≥mat_y and mat_s2≥mat_y, the match is judged to be successful; otherwise, the match fails.

6. According to the two-way data matching analysis method based on machine learning according to claim 5, Features: The specific steps of performing real-time optimization and updating of the matching model in S600 are: S601. When using the matching model to perform real-time matching on the data in the working environment, collect the matching results in history, count the number of correct matches and the total number of matches, and calculate the proportion of correct matches. The formula is: In the formula, per represents the percentage of correct matches, nuz represents the number of correct matches, and num represents the total number of matches; S602. Calculate the proportion of correct model matching times during normal operation in history as pery. When per<pery, determine that the model performance has degraded, collect a new round of data from the matching database, use the new matching database to train the matching model, and optimize and update the model.

7. A two-way data matching analysis system based on machine learning, Features: The two-way data matching and analysis system includes a data collection module, a data set search module, a database construction module, a model building module, a model training module, a matching module and a model optimization module; The data collection module is used to collect matching data in the historical working environment; The data group search module is used to search for data types with bidirectional matching according to the matching objects of each data in the history, construct data groups, and extract matching features; The database construction module is used to search for low-matching data groups in the data groups, determine transition data of the low-matching data groups, and construct a matching database in combination with the data groups and corresponding matching features; The module building module is used to construct a double tower structure through a neural network, output an embedding vector of a data set, and calculate cosine similarity to construct a matching model; The model training module is used to train the matching model using the data group in the matching database to obtain a matching threshold; The matching module is used to collect data generated by work in real time, determine whether matching is required, and use the matching model to match the real-time data; The model optimization module is used to analyze and calculate the matching results of the matching model, judge the model performance, and complete the model optimization and update by updating the matching database and performing a new round of training on the matching model.

8. According to the two-way data matching analysis system based on machine learning according to claim 7, Features: The data group search module includes a two-way matching search unit and a feature extraction unit; The two-way matching search unit is used to analyze the data with matching in the history, find the matching object of each data, and find the data with two-way matching through the matching object to construct a data group; The feature extraction unit is used to calculate the eigenvalue of each feature using the covariance matrix, and to determine the matching features of the data in each data group through the eigenvalue.

9. The two-way data matching analysis system based on machine learning according to claim 7, Features: The database construction module includes a low-profile data group search unit and a transition data search unit; The low-matching data group search unit is used to calculate the matching degree of the bidirectionally matched data in the history, search for the data group with low matching degree, and determine the low-matching data group; The transition data search unit is used to determine the low-profile data group and then use the matching object set of two data in the low-profile data group to search for transition data of the low-profile data group.

10. The two-way data matching analysis system based on machine learning according to claim 7, Features: The matching module includes a normal matching unit and a transition matching unit; The normal matching unit is used to input the real-time collected data into the matching model, calculate the cosine similarity, and perform matching; The transition matching unit is used to construct a transition data group using transition data when the real-time data is data in the low-matching data group, and input the transition data group into the matching model to perform data matching.

Images