An intelligent data mining method and system based on big data

Abstract

The application provides an intelligent data mining method and system based on big data, wherein a theme keyword is selected in an existing large database, the keyword is arranged and combined to construct a keyword screening channel, big data is screened and separated through the keyword screening channel, a mining database meeting the keyword channel is acquired, data preprocessing is performed on the mining database, mining data with concentrated density is acquired, linear regression calculation is performed on the mining data with concentrated density through keywords and other high-frequency words, and a data mining result is acquired, so as to solve the problem that in a big data environment, too much data causes the accuracy and efficiency of data mining to be reduced, and solve the problem that data mining is limited by themes and fields.

Description

TECHNICAL FIELD

[0001] The application provides an intelligent data mining method and system based on big data, and belongs to the technical field of intelligent data mining. BACKGROUND

[0002] At present, by mining various data behaviors in big data, some intention data in the corresponding field can be mined, and by deep analysis of the data, the development trend of the data and the influence of other data factors on the development trend are obtained. However, as the amount of data generated by big data is increasing, the influence of irrelevant data on intelligent data mining is increasing, which reduces the accuracy and efficiency of intelligent data mining. In addition, the more data, the more obvious the influence of data mining, and the direction of data mining also has limitations. SUMMARY

[0003] The application provides an intelligent data mining method and system based on big data, which solves the problem of reducing the accuracy and efficiency of data mining caused by excessive data in a big data environment, and solves the problem of limiting data mining by theme and field.

[0004] An intelligent data mining method based on big data, the method comprising:

[0005] S1, selecting theme keywords in an existing large database, arranging and combining the keywords to construct a keyword screening channel;

[0006] S2, screening and separating the big data through the keyword screening channel to obtain a mining database conforming to the keyword channel;

[0007] S3, pre-processing the mining database to obtain mining data concentrated in density; S4, performing linear regression calculation on the mining data concentrated in density through keywords and other high-frequency words to obtain a data mining result.

[0008] Further, the selecting theme keywords in an existing large database, arranging and combining the keywords to construct a keyword screening channel comprises:

[0009] According to the direction of data mining, set the mining direction to set the keywords, arrange and combine the keywords according to the number of keywords, the content of arrangement and combination is increased from one keyword to all keywords, arrange and combine different numbers of keywords to obtain various arrangement and combination schemes, formulate data channel screening rules according to the arrangement and combination schemes to obtain data screening channels corresponding to the arrangement and combination schemes.

[0010] Further, the big data is screened and separated through the keyword screening channel to obtain a mining database meeting the keyword channel, including: the big data is screened through the data screening channel, and the data meeting the keyword channel screening rule is retained, and the screening data not containing all the keywords continues to be screened, the screening times are set to 3-5 times, or the screening is performed until the screened data result does not change, and the mining database is obtained.

[0011] Further, the mining database is preprocessed to obtain mining data with concentrated density, including:

[0012] The mining database data is subjected to logical regression processing to obtain a linear regression equation, the density diameter is set according to the distribution density of the mining database data near the regression equation, the data with concentrated density is divided, the data with high density near the regression equation is collected to obtain high-density data near the regression equation, and the data deviating from the regression equation and showing concentrated distribution is collected to obtain high-density data far from the regression equation.

[0013] For the data with irregular discrete distribution, clustering processing is performed, a point with relatively close distance to the data with discrete distribution is set as a discrete data distribution center, if the discrete data distribution center falls into the high-density data, the discrete data is considered as an expansion item of the high-density data, if there is still discrete data that cannot fall into the high-density data range through the distribution center, the discrete data is subjected to retention processing and is not subjected to data mining.

[0014] The repeated numerical records in the high-density data are deleted, the data with missing data of more than 2% is deleted, and the data with missing data of less than 2% is retained, and the missing part is filled with an average value.

[0015] Further, the mining data with concentrated density is subjected to linear regression calculation through keywords and other high-frequency words to obtain a data mining result, including:

[0016] In the mining database, the appearance frequency proportion of the keywords in the high-density data is counted, the high-density data of the keywords is subjected to linear regression calculation, the internal relationship between the keywords is constructed, the prediction direction of the keyword data is obtained, and keyword data mining is completed.

[0017] In the mining database, the appearance frequency proportion of the keywords in the high-density data is counted, other words with appearance frequency close to or exceeding the keywords are counted, the logical relationship between the other words and the associated words is established, the keyword extension data association is obtained, and keyword extension data mining is completed.

[0018] In the database mining, the high-frequency same words except the keywords are counted, the proportion of the high-frequency same words in the total data set is obtained, other high-frequency same words with a similar proportion are retrieved, logical connections are established between the same words with the same proportion, the relationship between the number of words and variables is constructed, and the hidden data mining results other than the keywords are obtained.

[0019] Further, the high-frequency same words with a proportion of 0.2% are added as keywords, the keywords are put into the database for re-mining, new mining results and other high-frequency same words are obtained, a machine model is established, and the mining is repeatedly performed to increase the keywords until no keywords can be obtained. Linear regression calculation is performed on the keywords, the internal relationship between the keywords is constructed, and the data mining results are obtained.

[0020] An intelligent data mining system based on big data, the system comprises:

[0021] A data screening module is configured to set data mining keywords, form a keyword screening channel, screen the database, and obtain a data mining database.

[0022] A data processing module is configured to perform data preprocessing on the mining database to obtain mining data with concentrated density.

[0023] A data mining module is configured to perform linear regression calculation on the mining data with concentrated density by using keywords and other high-frequency words to obtain data mining results.

[0024] Further, the data screening module comprises:

[0025] A keyword combination module is configured to sequentially add one keyword to all keywords, arrange and combine different numbers of keywords, and obtain various arrangement and combination schemes.

[0026] A channel screening module is configured to formulate data channel screening rules according to the arrangement and combination schemes, and obtain data screening channels corresponding to the arrangement and combination schemes.

[0027] A channel iteration module is configured to continue screening the screening data of the screening channel that does not contain all the keywords to obtain a data mining database.

[0028] Further, the data processing module comprises:

[0029] A logistic regression processing module is configured to perform logistic regression processing on the mining database to obtain a linear regression equation and understand the distribution of data.

[0030] A density data acquisition module is configured to acquire density data near and deviating from the regression equation.

[0031] The discrete data clustering module is used for clustering processing of discrete distributed data, and arranging the discrete data into density data.

[0032] The repeated missing processing module is used for processing the repeated data and the missing data, and obtaining the preprocessed mining database.

[0033] Further, the data mining module comprises:

[0034] The prediction mining module is used for calculating the appearance frequency proportion of the keywords in the high-density data in the mining database, performing linear regression calculation on the high-density keyword data, constructing the internal relationship between the keywords, and obtaining the prediction direction of the keyword data.

[0035] The extension mining module is used for calculating the appearance frequency proportion of the keywords in the high-density data in the mining database, calculating other words similar to or exceeding the appearance frequency of the keywords, establishing the logical relationship between the other words and the associated words, and obtaining the extension data association of the keywords.

[0036] The hidden mining module is used for calculating the high-frequency same words other than the keywords in the mining database, obtaining the proportion of the high-frequency same words in the total data set, additionally calling other high-frequency same words close to the proportion of the high-frequency same words, establishing the logical connection between the same words, constructing the relationship between the number and the variable of the words, and obtaining the hidden data mining result other than the keywords.

[0037] The iteration mining module is used for increasing the high-frequency same words with a proportion of 0.2% to the keywords, putting the keywords into the database for re-mining, obtaining new data mining results and other high-frequency same words, establishing a machine model, repeatedly mining and increasing the keywords until no keywords can be obtained, performing linear regression calculation on the keywords, constructing the internal relationship between the keywords, and obtaining the data mining result.

[0038] The present application has the following beneficial effects:

[0039] By arranging and combining the keywords in the preliminary input mining direction in the big data, a multi-channel data screening channel is formed, which can accelerate the screening speed of the data, and the multi-channel composite screening mode can accurately mine the database while ensuring the screening speed, so as to make the data mining result more accurate and ensure the accuracy of the data mining; meanwhile, by processing the data in the mined database, the overall data is subjected to logistic regression to obtain the preliminary direction of data mining, the mining data is divided in density near the regression curve, and the mining data is collected in different density areas, so that the mining data can be subdivided in the general direction of data mining, and the result of data mining is more detailed; meanwhile, the processed data is mined, the trend association between keywords, the trend association between keywords and extended multi-frequency words, and the trend association of the multi-frequency words are obtained, and multi-angle data mining is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 a flow chart of the method of the present application;

[0041] Figure 2 a system block diagram of the system of the present application. DETAILED DESCRIPTION

[0042] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0043] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. The described embodiments are merely some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application.

[0045] One embodiment of the present application is an intelligent data mining method based on big data, which comprises:

[0046] S1, selecting a direction keyword in an existing large database, and arranging and combining the keyword to obtain a keyword screening channel;

[0047] S2, screen and separate the big data through the keyword screening channel to obtain a mining database conforming to the keyword channel;

[0048] S3, data preprocessing is performed on the mining database to obtain mining data concentrated in density; S4, linear regression calculation is performed on the mining data concentrated in density through keywords and other high-frequency words to obtain a data mining result.

[0049] The working principle of the above technical solution is:

[0050] Firstly, keywords related to the theme or target to be mined are selected from the existing large database, the keywords are arranged and combined, and the screening rules of the big data are set based on the keyword arrangement and combination;

[0051] Then, according to the set keyword screening channel, the system screens and separates the big data, the data not matching the keyword channel is excluded, and the data matching the keyword channel is retained to form a relatively small but more concentrated mining database;

[0052] After that, the data in the mining database is preprocessed, the preprocessing includes data cleaning, format conversion, missing value processing and outlier detection, and more standardized, accurate and concentrated mining data is obtained;

[0053] Finally, in-depth feature analysis is performed on the preprocessed mining database to identify patterns, trends and correlations in the data, based on the results of the feature analysis, corresponding mining rules are set, the data is deeply mined, the results of data mining are obtained using statistical methods and machine learning algorithms, and valuable insights about the big data are provided.

[0054] The effect of the above technical solution is:

[0055] By selecting keywords to arrange and combine to set data screening rules, a screening standard or framework can be constructed to screen data matching the keywords from the big data, the screening database obtained by setting the screening rules can obtain a relatively small but more concentrated mining database, which can reduce the data reference range of the mining database and increase the time and accuracy of data mining; by preprocessing the data in the mining database, the quality and consistency of the data can be improved, the interference of abnormal data and missing value data can be reduced, and the data can be more standardized and accurate; by performing feature analysis on the preprocessed mining database, patterns, trends and correlations in the data can be identified, corresponding mining rules can be set, valuable information or knowledge can be quickly and accurately extracted from the data through the set mining rules, and efficient data mining results can be provided to users in the form of reports and charts.

[0056] One embodiment of the present application, the intelligent data mining method based on big data, characterized in that, in the existing large database, the direction keyword is selected, the keyword is arranged and combined to obtain the keyword screening channel, comprising:

[0057] According to the direction of data mining, set the mining direction setting keyword, arrange and combine the keyword, the content of arrangement and combination is increased from one keyword to all keywords, arrange and combine different number of keywords, obtain various arrangement and combination schemes, and formulate data channel screening rules according to the arrangement and combination scheme, and obtain the data screening channel corresponding to the arrangement and combination scheme.

[0058] The working principle of the above technical scheme is:

[0059] Firstly, according to the direction of data mining, the related keywords are selected, the keywords represent different mining data characteristics or attributes, and the keywords are arranged according to different combination modes, from single keyword to all keyword combination, and the combination mode is:

[0060] Let the number of keywords be n, and the keywords be a1, a2, a3, a4,...., an;

[0061] For single keyword combination mode, there are n choices;

[0062] For two keyword combination mode, there are n*(n-1) choices;

[0063] For three keyword combination mode, there are n*(n-1)(n-2) choices (the first keyword has n choices, the second keyword has n-1 choices, and the third keyword has n-2 choices);

[0064] In this way, until the combination mode of all keywords, that is, n*(n-1)*(n-2)...2*1;

[0065] Therefore, arrange and combine n numbers in order from single keyword to all keywords, the number of selected keywords is x, x≤n-1,

[0066] The total number of combination modes is N: N=

[0067] Secondly, based on the content of arrangement and combination, formulate the corresponding data channel screening rules for each combination scheme, use all number arrangement and combination screening channels according to the field direction familiarity, or use specific keyword number arrangement and combination screening channel, or gradually increase the number of keywords for step-by-step screening for unfamiliar field direction;

[0068] Finally, each keyword permutation corresponds to a data filtering channel, creating multiple data channels, each corresponding to a different keyword combination; when the depth of data mining is selected according to the number of keywords, the more the number of keywords, the closer the data relationship between the keywords, and the narrower the data range; the fewer the number of keywords, the wider the data range, and the most content containing other hidden information.

[0069] The effect of the above technical solution is:

[0070] Through the permutation and combination of keywords, different aspects of data can be fully explored, from a single keyword to the combination of all keywords, ensuring the depth and breadth of data mining; based on the permutation and combination scheme, the data channel filtering rules are formulated, making the data filtering process more standardized and systematic, which helps to eliminate subjectivity and randomness, improve the accuracy and reliability of data filtering; at the same time, the permutation and combination method greatly reduces the amount of data that needs to be filtered, improving the efficiency of data processing. At the same time, through the preset keyword channel, valuable data can be quickly located; at the same time, based on the condition of big data, the scheme allows flexible setting of keywords according to different mining directions, and adjusts the combination method of keywords according to actual needs, making the scheme have strong adaptability and application range; at the same time, through multi-dimensional permutation and combination analysis of data, hidden patterns, associations and trends in data can be deeply mined, which helps to provide deeper data insights and provide more valuable data mining results; at the same time, through the preset keyword channel and filtering rules, the complexity of data filtering and processing is greatly simplified, so that non-professionals can easily perform data mining and analysis; in addition, the scheme can be expanded as the amount of data increases, has good scalability, and through continuous optimization of the permutation and combination of keywords, the efficiency and accuracy of data mining can be improved.

[0071] In one embodiment of the present application, the intelligent data mining method based on big data, characterized by filtering big data through a keyword channel to obtain a mining database that meets the keyword channel, comprises: filtering big data through a data filtering channel, data that meets the keyword channel filtering rule is retained, and filtering data that does not contain all keywords continues to be filtered, the number of filtering is set to 3-5 times, or continues until the filtered data result does not change, and a mining database is obtained.

[0072] The working principle of the above technical solution is:

[0073] First, according to the screening rule of the keyword channel, the big data is preliminarily screened, the data meeting the screening rule of the keyword channel is reserved, and the data not meeting the screening rule is saved separately, then the keyword channel screening is continuously carried out until the screened data amount does not change, and the data subset related to the keyword is quickly located from the large amount of data; then, for the screening result not containing all the keywords, iterative screening is continuously carried out, and the data not meeting the keyword channel is continuously eliminated through multiple repeated screening processes until the screening result does not change or the preset screening times are reached, generally 3-5 times of screening, the data subset is gradually optimized by using the iterative screening mode, so that the data subset more accurately meets the screening rule of the keyword channel; finally, through the data screening and iterative screening processes, the data subset with the highest matching degree with the keyword channel is obtained, and the mining database meeting the requirement of the keyword channel is obtained.

[0074] The effect of the above technical solution is:

[0075] The data screening through the keyword channel can quickly locate the data related to the keyword, improve the efficiency of data screening, and significantly reduce the data processing time compared with the traditional piece-by-piece screening method; through the iterative screening process, the data not meeting the keyword channel is continuously eliminated, so that the finally obtained mining database is highly matched with the keyword channel, and the quality and reliability of the data are improved; since the scheme is based on the keyword channel for screening, it can focus on the data related to the keyword, so as to deeply mine the mode, association and trend of the data, which helps to provide deeper data insight and more valuable insights; in addition, with the increase of the data amount, the scheme can process larger scale data by expanding the keyword channel and the number of iterative screening. At the same time, by continuously optimizing the screening rule of the keyword channel, the efficiency and accuracy of data mining can be improved, the subjectivity and randomness can be eliminated, and the accuracy of data screening and processing can be ensured, which helps to avoid wrong decisions and conclusions.

[0076] In an embodiment of the present application, the intelligent data mining method based on big data is characterized in that the mining database is subjected to data preprocessing to obtain mining data with concentrated density, comprising:

[0077] The mining database is subjected to logistic regression processing to obtain a linear regression equation, and data with high density concentration near the regression equation is collected according to the distribution density of the data in the mining database near the regression equation to obtain a near-regression high-density data set;

[0078] Data deviating from the regression equation and showing concentrated distribution is collected to obtain a far-regression high-density data set;

[0079] For irregularly distributed data, clustering processing is performed, a point with relatively close distance to the irregularly distributed data is set as a distribution center point of the irregular data, if the distribution center point of the irregular data falls in the high-density data, the irregular data is considered as an expansion item of the high-density data, if there is still irregular data that cannot fall in the high-density data range through the distribution center point, the irregular data is reserved and is not subjected to data mining;

[0080] The repeated numerical records in the high-density data are deleted, the data with missing data of more than 2% is deleted, the data with missing data of less than 2% is reserved, the missing part is filled with an average value, and pre-processing mining data is obtained.

[0081] The working principle of the above technical solution is:

[0082] Firstly, the data of the mining database is subjected to logical regression processing, a linear regression equation is obtained, the linear regression equation is used to describe the relationship between the data, and the mode and trend in the data are found;

[0083] Then, according to the distribution density of the data near the regression equation, the data with high density is collected to form a near-regression high-density data set, and the data deviating from the regression equation and showing centralized distribution is collected to form a far-regression high-density data set, and a representative data set is obtained;

[0084] After that, for irregularly distributed data, clustering processing and distribution center point setting are performed to determine whether the distribution center point of the irregular data falls in the high-density data range, if it falls, it is considered as an expansion item of the high-density data, if it does not fall, the irregular data is reserved, the data is reserved for future data increase, and the irregular data is densely distributed to form a new high-density data range, and the processing of abnormal values and outliers is completed;

[0085] Finally, in the mining database, the repeated numerical values are deleted after recording the number of times, and the missing data is processed, the data with missing data of more than 2% is deleted, the data with missing data of less than 2% is reserved and filled with an average value, and pre-processing mining data is obtained.

[0086] The effect of the above technical solution is:

[0087] Through logical regression processing and data distribution analysis, it is helpful to focus on representative data set points, reduce the workload of data processing, and discover patterns and trends in the data, thereby improving the quality and reliability of the data, avoiding subjectivity and randomness, and ensuring the accuracy of data processing; through clustering processing of the discrete data around the regression equation, by setting the distribution center point for calculation, abnormal values and outliers can be identified and processed, which helps to eliminate noise and anomalies in the data, further improving the accuracy and reliability of data processing; through processing of repeated values and missing values, the integrity and consistency of the data can be ensured, which helps to avoid data redundancy and information omission, and provides more complete and accurate data sets for subsequent data analysis and mining; by obtaining near-regression high-density data sets and far-regression high-density data sets, and processing repeated values and filling missing data, the technical scheme can effectively utilize high-density data sets for subsequent data analysis and mining, and provide more valuable insights.

[0088] In one embodiment of the present application, the intelligent data mining method based on big data is characterized in that, for the mining data concentrated in the density set, linear regression calculation is performed on the keywords and other high-frequency words to obtain the data mining result, including:

[0089] In the mining database, the frequency ratio of the keywords in the high-density data is counted, linear regression calculation is performed on the high-density data of the keywords, the internal relationship between the keywords is constructed, the prediction direction of the keyword data is obtained, and the keyword data mining is completed.

[0090] In the mining database, the frequency ratio of the keywords in the high-density data is counted, other words with similar or higher frequency than the keywords are counted, the logical relationship between the other words and the associated words is established, the keyword extension data association is obtained, and the keyword extension data mining is completed.

[0091] In the mining database, the high-frequency same words other than the keywords are counted, the proportion of the high-frequency same words in the total data set is obtained, other high-frequency same words with similar proportion are retrieved, logical connections are established between the same proportion words, the relationship between the number of words and variables is constructed, and the hidden data mining result other than the keywords is obtained.

[0092] The working principle of the above technical scheme is as follows:

[0093] First, in the database mining, the proportion of the occurrence frequency of the keywords is analyzed to determine which keywords are dominant in the data set, which may be the focus of data mining. Linear regression is calculated for high-density data of keywords to build an internal relationship model between keywords, helping to understand how keywords are related to each other and how they affect the data mining results. Through the linear regression model, the direction of keyword data prediction is obtained;

[0094] Then, in addition to directly focusing on keywords, other words with similar or higher occurrence frequency than keywords are also counted. These words may have a logical connection with keywords, so they are also an important part of data mining. The logical relationship between these other words and associated words is established to further extend the depth and breadth of data mining and discover hidden information and patterns in the extended keywords.

[0095] Finally, in addition to keywords and their extended words, other high-frequency words in the data set are also focused on. Although these words are not keywords, they may occupy a certain proportion in the data set and can be included in the keywords, so they also contain valuable information. The high-frequency same words are counted and their proportion in the total data set is calculated to identify which non-keyword words are significant in the data set. Other high-frequency same words with similar proportions are retrieved, and logical connections are established between these same words with similar proportions to reveal the relationship between the number and variables of the words, thereby obtaining hidden data mining results beyond keywords.

[0096] The effect of the above technical solution is:

[0097] Through feature analysis and rule setting, and in-depth processing of high-density data sets, the technical solution can efficiently mine keyword data, keyword extension data and hidden data, which not only improves the efficiency of data mining, but also ensures the accuracy of the results; by linear regression calculation, the internal relationship between keywords is constructed, and the development direction of keyword data is obtained, which helps to predict future trends and patterns, and provides valuable predictive insights for decision makers; in addition to focusing on keywords themselves, the technical solution can also discover other words related to keywords and establish logical relationships between them, which helps to reveal more associations of keyword extensions, thereby enriching the results of data mining; by counting high-frequency same words other than keywords and retrieving other high-frequency same words with a similar proportion of high-frequency same words, the technical solution can obtain hidden data mining results other than keywords, which helps to discover important information that has been overlooked and provides a more comprehensive data perspective; at the same time, the technical solution can flexibly adjust the feature analysis, mining rules and data processing methods according to different needs and scenarios. This makes the solution have good adaptability and scalability, and can deal with different types of data and problems; through the preset feature analysis, mining rules and data processing flow, the technical solution can realize automatic and intelligent data processing and analysis, which helps to reduce manual intervention and error rate, and improve the efficiency and accuracy of data mining and analysis.

[0098] In one embodiment of the present application, the intelligent data mining method based on big data is characterized in that high-frequency same words with a proportion of 0.2% are added as keywords, the keywords are put into the database for re-mining, new mining results and other high-frequency same words are obtained, a machine model is established, and the mining is repeated to increase the keywords until no keywords can be obtained. Linear regression calculation is performed on the keywords to construct the internal relationship between the keywords and obtain the data mining results.

[0099] The working principle of the above technical solution is as follows:

[0100] First, the initial keywords are used to filter and mine the database, and the data containing these keywords in the database is searched, or more complex text mining techniques are used to extract information related to the keywords. New information is collected from the filtered results, including other high-frequency words or related words. These new words may be co-occurring words with the initial keywords, or other words that frequently appear in the mining results. Words with a frequency of more than 0.2% are considered to have the same importance as keywords, and are added as keywords. The same meaning is replaced or added as a keyword, the keywords are gradually adjusted, and the appropriate mining range is obtained.

[0101] Then, using the keywords and the mined information, a machine learning model is trained, which uses the updated keyword list to repeatedly mine the keyword and newly discovered high-frequency vocabulary database, and through continuous data iteration, the keyword set is continuously expanded until no new high-frequency or related vocabulary can be found, so as to capture more information related to the theme field;

[0102] After that, linear regression analysis is performed on the collected keywords to quantify the relationship between the dependent variable and one or more independent variables, to explain the association between different keywords and their common influence on a certain result or output through linear regression, and to build an internal relationship model between keywords through the results calculated by linear regression, to reveal which keywords are strongly related and which may be weakly related, and how they interact with each other through the model;

[0103] Finally, according to the results of all the above steps, the final data mining conclusion is obtained, which includes the key trends, patterns, and association rules of the theme field.

[0104] The effects of the above technical solutions are:

[0105] By using the initial keywords, the technical solution can effectively filter and mine data related to these keywords from the database; at the same time, by using text mining technology, information related to the keywords can be further extracted, thereby ensuring the accuracy and completeness of the data; in addition, during the filtering and mining process, the technical solution will continuously adjust the keywords according to newly discovered information, thereby gradually optimizing the scope of data mining, which helps to more comprehensively capture information related to the theme field and avoid missing due to improper selection of initial keywords; at the same time, by using the updated keyword list and related information, the technical solution can train a machine learning model, which can not only help to further mine the database, but also expand the keyword set through data iteration, thereby more deeply understanding the theme field; in addition, through linear regression analysis, the technical solution can quantify the relationship between keywords and explain how they jointly affect a certain result or output, which helps to reveal the internal relationship and interaction between keywords, making the data mining results more persuasive; at the same time, after all the technical steps, a comprehensive and in-depth data mining conclusion about the theme field can be obtained, which includes key trends, patterns, and association rules, which has important guiding significance for understanding the theme field, predicting future trends, and formulating related strategies.

[0106] One embodiment of the present application is an intelligent data mining system based on big data, which comprises:

[0107] The data screening module is configured to set data mining keywords, form a keyword screening channel, screen the database, and obtain a data mining database.

[0108] The data processing module is configured to pre-process the mining database to obtain mining data concentrated in density.

[0109] The data mining module is configured to perform linear regression calculation on the mining data concentrated in density through keywords and other high-frequency words to obtain a data mining result.

[0110] The working principle of the above technical solution is as follows:

[0111] First, the data screening module sets data mining keywords and forms a keyword screening channel according to the keywords, screens the database through the keyword screening channel, obtains data related to the keywords, and forms a data mining database; then, the data processing module pre-processes the mining database obtained from the data screening module, the pre-processing content includes missing value processing, duplicate value processing and aggregation of discrete data, so that the data is concentrated in density and meets the requirements of further analysis; finally, the data mining module performs feature analysis and pattern recognition on the pre-processed data, it first analyzes the characteristics of the data, and then performs in-depth analysis on the data according to the characteristics and preset mining rules, this process includes classification, clustering, association rule mining and other operations, valuable information and knowledge are extracted from a large amount of data, and finally the data mining result is obtained; the running process of the whole system is an iterative process, there is data transmission and feedback between each module, so that the whole system can be continuously optimized and improved.

[0112] The effect of the above technical solution is as follows:

[0113] The data screening module sets data mining keywords and forms a keyword screening channel, the system can quickly and accurately screen out data related to the keywords, greatly improving the efficiency and accuracy of data screening; the data processing module can automatically pre-process the mining database, thereby improving the quality and consistency of the data and reducing the need for manual intervention; the data mining module can perform in-depth feature analysis and pattern recognition on the pre-processed data, and mine the internal relationship and rules of the data, providing valuable information and knowledge for enterprise or personal decision-making; the system composed of the above modules can obtain intelligent data mining results from big data, provide data support for decision-making, and improve the scientificity and accuracy of decision-making; the system has good flexibility and scalability, can be customized and optimized according to the needs of users and the characteristics of data, and meets the data mining needs in different scenarios; the system can automatically iterate and update the data mining process, continuously optimize and improve the data mining result, and improve the efficiency and accuracy of data mining.

[0114] In one embodiment of the present application, the data screening module comprises:

[0115] The keyword combination module is used to arrange and combine different numbers of keywords in sequence from one keyword to all keywords to obtain various arrangement and combination schemes.

[0116] The channel screening module is used to formulate data channel screening rules according to the arrangement and combination schemes to obtain data screening channels corresponding to the arrangement and combination schemes.

[0117] The channel iteration module is used to continue screening the screening data of the screening channel not containing all the keywords to obtain the data mining database.

[0118] The working principle of the above technical solution is as follows:

[0119] The keyword combination module is used to arrange and combine the existing keywords, starting from one keyword and then increasing to all keywords to form different keyword combinations, and various keyword combination schemes can be obtained. The channel screening module uses the arrangement and combination schemes generated by the keyword combination module to formulate corresponding data channel screening rules. The screening rules are based on the matching degree and importance of the keywords and are used to screen the data matching the keyword combination scheme from the database. Since one data channel cannot pass all the data, the channel iteration module is used to iteratively screen the screening channel not containing all the keywords. For the data not completely matching the keywords, the screening will be performed again until all the keywords are satisfied, thereby obtaining the data mining database.

[0120] The effect of the above technical solution is as follows:

[0121] The keyword combination module allows one keyword to be sequentially added to all keywords to generate different keyword combination schemes, and the combination flexibility enables the technical scheme to adapt to different data mining requirements and perform data screening according to different keyword combinations; through the keyword combination module and the channel screening module, the technical scheme can generate various permutation and combination schemes, and formulate data channel screening rules according to the schemes, thereby ensuring the accuracy of data screening, improving the efficiency of data processing, selecting data completely matched with the keywords to improve the accuracy and reliability of data mining; through the cooperative work of the keyword combination module and the channel screening module, the technical scheme can quickly generate data channel screening rules and perform efficient data screening on the database, which greatly makes the data mining process faster and more efficient; the channel iteration module can automatically perform iterative screening on the screening channels that do not contain all the keywords until a data-rich mining database is obtained, thereby reducing the need for manual intervention, improving the efficiency and accuracy of data screening.

[0122] In an embodiment of the present application, the data processing module comprises:

[0123] A logistic regression processing module is configured to perform logistic regression processing on the mining database to obtain a linear regression equation and understand the distribution of the data.

[0124] A density data acquisition module is configured to acquire density data near and deviating from the regression equation.

[0125] A discrete data clustering module is configured to perform clustering processing on the discrete distributed data and arrange the discrete data into the density data.

[0126] A repeated missing data processing module is configured to process the repeated data and the missing data to obtain a preprocessed mining database.

[0127] The working principle of the above technical scheme is as follows:

[0128] First, the logistic regression processing module is used to perform logistic regression processing on the data of the mining database to obtain a linear regression equation, which is used to describe the relationship between the data and find the patterns and trends in the data.

[0129] Then, according to the distribution density of the data near the regression equation, the density data acquisition module is used to acquire the data with high density near the regression equation to form a near-regression high-density data set, and the data deviating from the regression equation and showing a centralized distribution is acquired to form a far-regression high-density data set, thereby obtaining a representative data set.

[0130] After that, for irregularly discrete distributed data, the discrete data is clustered and the distribution center point is set by using the discrete data clustering module, whether the distribution center point of the discrete data falls into the high-density data range is judged, if it falls into, it is considered to be the spread item of the high-density data, if it does not fall into, the discrete data is reserved, and the processing of the abnormal value and the outlier is completed;

[0131] Finally, the repeated values and the missing values in the high-density data set are processed by using the repeated missing processing module, the repeated values are recorded in number and then deleted, and the missing data is processed, for the data missing more than 2%, the deletion processing is performed, for the data missing within 2%, the average value is used for filling, and the pre-processing mining data is obtained.

[0132] The effect of the above technical solution is:

[0133] Through the logical regression processing module and the data distribution analysis, the representative data set points can be focused, the workload of data processing is reduced, the patterns and trends in the data can be found, the quality and reliability of the data are improved, the subjectivity and randomness can be avoided, and the accuracy of data processing is ensured, through the density data acquisition module, the near regression high-density data set and the far regression high-density data set are obtained, and the repeated values are processed and the missing data is filled, the technical solution can effectively utilize the high-density data set for subsequent data analysis and mining, and more valuable insights are provided, through the discrete data clustering module, the discrete data around the regression equation is clustered, the distribution center point is set for calculation, the abnormal value and the outlier can be identified and processed, the noise and the abnormality in the data can be eliminated, and the accuracy and the reliability of data processing are further improved;

[0134] Through the repeated missing processing module, the repeated values and the missing values are processed, the integrity and the consistency of the data can be ensured, the data redundancy and the information omission can be avoided, more complete and accurate data sets are provided for subsequent data analysis and mining, in general, through the logical regression processing, the density data acquisition, the discrete data clustering and the repeated missing processing, the comprehensive processing of the mining database is realized, the quality and the consistency of the data are improved, and more accurate and reliable data basis is provided for subsequent data mining and analysis.

[0135] In an embodiment of the application, the data mining module comprises:

[0136] The prediction mining module is used for calculating the appearance frequency proportion of the keywords in the high-density data in the mining database, performing linear regression calculation on the high-density data of the keywords, constructing the internal relationship between the keywords, and obtaining the prediction direction of the keyword data.

[0137] The extension mining module is used for mining the database to count the proportion of the occurrence frequency of the high-density data keyword, count other words similar to or exceeding the occurrence frequency of the keyword, establish a logical relationship between the other words and the keyword, and obtain the keyword extension data association.

[0138] The hidden mining module is used for mining the database to count high-frequency same words other than the keyword, count the proportion of the high-frequency same words in the total data set, additionally call other high-frequency same words close to the proportion of the high-frequency same words, establish a logical connection between the same words, build a relationship between the number and variable of the words, and obtain the hidden data mining result other than the keyword.

[0139] The iteration mining module is used for increasing the high-frequency same words with a proportion of 0.2% to the keyword, putting the keyword into the database to re-mine, obtaining new mining results and other high-frequency same words, establishing a machine model, repeatedly mining and increasing the keyword until no keyword can be obtained, performing linear regression calculation on the keyword, building an internal relationship between the keywords, and obtaining the data mining result.

[0140] The working principle of the above technical solution is as follows:

[0141] The prediction mining module counts the occurrence frequency of the keyword in the high-density data, performs linear regression calculation on the keyword, builds an internal relationship between the keywords, and helps to predict the development direction of the keyword data through the establishment of the internal relationship, thereby providing a basis for decision-making and prediction. The extension mining module finds other words similar to or exceeding the keyword in the mining database, and establishes a logical relationship between these words and the keyword, thereby obtaining the data mining result of the keyword extension association through the establishment of the logical relationship. The hidden mining module mainly focuses on counting high-frequency same words other than the keyword in the mining database, and can obtain the proportion of the high-frequency same words in the total data set through the statistics of the high-frequency same words. In addition, a logical connection is established between the same words, which helps to find the relationship between the number and variable of the words, thereby obtaining the hidden data mining result other than the keyword. The iteration mining module increases the high-frequency same words with a proportion of 0.2% to the keyword, and puts the keyword into the database for screening and mining. Through repeated screening and increasing the keyword, a more comprehensive and in-depth mining database can be obtained. On this basis, linear regression calculation is performed to build an internal relationship between the keywords, and finally a comprehensive data mining result is obtained. In summary, the data mining module in the above technical solution deeply mines the internal relationship and rule of the data from the high-density data set, and provides valuable information and knowledge for decision-making and prediction.

[0142] The effect of the above technical solution is that:

[0143] The prediction mining module counts the high density data of the keywords, performs linear regression calculation, constructs the internal relationship between the keywords, and thus obtains the prediction direction of the keyword data. The acquisition of the prediction direction helps to make scientific prediction of future trends and changes, and provides basis for decision-making. The extension mining module focuses on discovering other words similar to or exceeding the keywords, and establishes the logical relationship between these words and the associated words. Through the establishment of the logical relationship, the extension data association of the keywords can be obtained, the value of the data is further expanded, the internal contact and regularity of the data are helped to be understood in depth, and the depth and breadth of data mining are improved. The hidden mining module counts the same high frequency words except the keywords, discovers the same words with a proportion of 0.2%, and establishes the logical connection between the words. The establishment of the connection helps to discover the relationship between the quantity and the variable between the words, and thus obtains the hidden data mining result outside the keywords, can reveal more potential contact and regularity, and improves the comprehensiveness and accuracy of data mining. The iterative mining module obtains new mining results and other same words by repeatedly increasing the keywords and re-mining. Through the establishment of a machine model, intelligent repeated screening can be performed, the keywords are increased until no new keywords can be obtained, the result of data mining can be continuously optimized and improved, and the efficiency and accuracy of data mining are improved. In summary, the data mining module in the above technical solution realizes comprehensive and in-depth mining of the high density data set through various ways and means, improves the efficiency and accuracy of data mining, and the application scene of the technical solution is wide, which can provide strong support for decision-making and prediction in various subject fields.

[0144] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A big data based intelligent data mining method, characterized in that, The method comprises: S1, selecting a theme keyword in an existing large database, and arranging and combining the keyword to construct a keyword screening channel; S2, screening and separating the large data through the keyword screening channel to obtain a mining database conforming to the keyword channel; S3, pre-processing the mining database data to obtain mining data with concentrated density; S4, performing linear regression calculation on the mining data with concentrated density through keywords and other high-frequency words to obtain a data mining result. In the mining database, the appearance frequency proportion of the high-density data keywords is statistically analyzed to obtain a keyword data mining result, a keyword extension data mining result, and a hidden data mining result.

2. The big data based intelligent data mining method as claimed in claim 1, wherein In the existing large database, a theme keyword is selected, and the keyword is arranged and combined to construct a keyword screening channel, which comprises: According to the data mining direction, a mining direction is set, a keyword is set, the keywords are arranged and combined according to the number of keywords, the arrangement and combination content is sequentially increased from one keyword to all keywords, different numbers of keywords are arranged and combined, various arrangement and combination schemes are obtained, data channel screening rules are formulated according to the arrangement and combination schemes, and a data screening channel corresponding to the arrangement and combination scheme is obtained.

3. The big data based intelligent data mining method as claimed in claim 2, wherein The large data is screened and separated through the keyword screening channel to obtain a mining database conforming to the keyword channel, which comprises:

4. The intelligent data mining method based on big data according to claim 2, characterized in that The large data is screened through the data screening channel, and the data conforming to the keyword channel screening rule is retained. The mining database data is processed by a logistic regression to obtain a linear regression equation, a density diameter is set according to the distribution density of the mining database data near the regression equation, the data with concentrated density is divided, the data with high density near the regression equation is collected to obtain near-regression high-density data, and the data deviating from the regression equation and showing concentrated distribution is collected to obtain far-regression high-density data. For the irregularly distributed discrete data, a clustering processing is performed, a point with a relatively close distance to the discrete data is set as a discrete data distribution center, if the discrete data distribution center falls into the high-density data, the discrete data is considered as an expansion item of the high-density data, if there is still discrete data that cannot fall into the high-density data range through the distribution center, the discrete data is retained, The repeated numerical records in the high-density data are deleted, the data with a missing rate of more than 2% is deleted, the data with a missing rate of less than 2% is retained, and the missing part is filled with an average value.

5. The big data based intelligent data mining method according to claim 4, characterized in that ,The high-frequency same words with a proportion of 0.2% are added as keywords, the keywords are put into the database for re-mining, new mining results and other high-frequency same words are obtained, a machine model is established, and mining is repeatedly performed to increase the keywords until no keywords can be obtained, linear regression calculation is performed on the keywords, internal relationships between the keywords are constructed, and data mining results are obtained. The system comprises: A data screening module is configured to set data mining keywords, form a keyword screening channel, screen a database, and obtain a data mining database. A data processing module is configured to pre-process the mining database, and obtain mining data with concentrated density.

6. The intelligent data mining method based on big data according to claim 5, characterized in that The data in the mining database is processed by logistic regression, the density diameter is set, the data with concentrated density is divided, near-regression high-density data and far-regression high-density data are obtained, and the data with irregular discrete distribution is processed by clustering, deletion, and filling.

7. A big data based intelligent data mining system characterized by, A data mining module is configured to perform linear regression calculation on the mining data with concentrated density through keywords and other high-frequency words, and obtain data mining results. In the mining database, the proportion of the occurrence frequency of the keywords in the high-density data is statistically analyzed to obtain the results of keyword data mining, keyword extension data mining, and hidden data mining. The data screening module comprises: A keyword combination module is configured to sequentially add one keyword to all keywords, arrange and combine different numbers of keywords, and obtain various arrangement and combination schemes. A channel screening module is configured to formulate data channel screening rules according to the arrangement and combination schemes, and obtain data screening channels corresponding to the arrangement and combination schemes. A channel iteration module is configured to continue screening the screening data in the screening channel that does not contain all the keywords, and obtain a data mining database.

8. The big data based intelligent data mining system as claimed in claim 7, wherein, The data processing module comprises: A logistic regression processing module is configured to perform logistic regression processing on the mining database, obtain a linear regression equation, and understand the distribution of the data. A density data acquisition module is configured to acquire density data near and deviating from the regression equation. ​ 9. The big data based intelligent data mining system as claimed in claim 7, wherein, ​ ​ ​ The discrete data clustering module is used for clustering processing of discrete distributed data, and arranging the discrete data into density data. The repeated missing data processing module is used for processing repeated data and missing data to obtain a preprocessed mining database.

10. The big data based intelligent data mining system as claimed in claim 7, wherein, The data mining module comprises: The prediction mining module is used for counting the proportion of the occurrence frequency of keywords in high-density data in the mining database, performing linear regression calculation on the high-density keyword data, constructing the internal relationship between keywords, and obtaining the prediction direction of keyword data. The extension mining module is used for counting the proportion of the occurrence frequency of keywords in high-density data in the mining database, counting other words similar to or exceeding the occurrence frequency of keywords, establishing the logical relationship between the other words and the associated words, and obtaining the extension data association of keywords. The hidden mining module is used for counting high-frequency same words other than keywords in the mining database, obtaining the proportion of the high-frequency same words in the total data set, additionally calling other high-frequency same words close to the proportion of the high-frequency same words, establishing logical connections between the same words, constructing the relationship between the number of words and variables, and obtaining the hidden data mining result other than the keywords. The iteration mining module is used for increasing the high-frequency same words with a proportion of 0.2% to keywords, putting the keywords into the database for re-mining, obtaining new data mining results and other high-frequency same words, establishing a machine model, repeatedly mining and increasing keywords until no keywords can be obtained, performing linear regression calculation on the keywords, constructing the internal relationship between the keywords, and obtaining the data mining result.

Images