Integrated AI-driven data mining tool training method and device

Abstract

The present application provides an integrated AI-driven data mining tool training method and device, which relates to the field of data mining technology, including: reading source data types, performing hierarchical decomposition of data, determining the topological network of data mining, determining mining driving rules based on mining targets of network nodes, executing multi-logic block training and network integrated deployment, determining initialization mining tools, introducing proxy functions, performing cross-validation and multi-objective optimization, determining optimization strategies, optimizing and independently allocating domains to the initialization mining tools based on the optimization strategies, and determining data mining tools. The present application can solve the technical problem in the prior art that the accuracy of mining results is low due to the low efficiency of traditional data mining tools in processing large-scale and complex data. Through integrated AI drive and combined with a variety of artificial intelligence technologies, the automation and intelligence of the data mining process are realized, thereby improving the accuracy and efficiency of data mining.

Description

Technical Field

[0001] The present application relates to the field of data mining technology, and in particular to an integrated AI-driven data mining tool training method and device. Background Art

[0002] Data mining is an important branch of artificial intelligence, which aims to extract valuable information and patterns from large amounts of data. With the advent of the big data era, data mining technology is increasingly being used in various industries, such as finance, healthcare, and e-commerce. However, existing data mining tools face some technical challenges when processing large-scale and complex data. Existing data mining tools are often inefficient when processing large-scale data sets, resulting in a time-consuming mining process. Some data mining tools do not have a high enough accuracy in their models when processing complex data, resulting in deviations and errors in the mining results.

[0003] In summary, there is a technical problem in the prior art that the accuracy of mining results is low due to the low efficiency of traditional data mining tools in processing large-scale and complex data. Summary of the invention

[0004] The purpose of this application is to provide an integrated AI-driven data mining tool training method and device to solve the technical problem in the prior art that traditional data mining tools have low processing efficiency for large-scale and complex data, resulting in low accuracy of mining results.

[0005] In view of the above problems, the present application provides an integrated AI-driven data mining tool training method and device.

[0006] In a first aspect, the present application provides an integrated AI-driven data mining tool training method, which is implemented by an integrated AI-driven data mining tool training device, wherein the integrated AI-driven data mining tool training method includes: reading the source data type, performing hierarchical decomposition of the data, and determining the topological network of data mining, wherein the decomposition layer at least includes a domain-scenario-multi-level refinement layer; based on the topological network, determining the mining driving rules based on the mining target of the network node, and the mining driving rules include brain-like processing rules, rigid logic rules and long short-term memory rules; based on the mining driving rules and the topological network, performing multi-logic block training and network integrated deployment to determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented to the mining target; introducing a proxy function, cross-validating and multi-objective optimization of the initialization mining tool, and determining the optimization strategy, wherein the proxy function is an approximate gradient relationship for a nonlinear optimization target; based on the optimization strategy, optimizing and independently allocating the initialization mining tool to the independent calculation domain to determine the data mining tool.

[0007] In a second aspect, the present application further provides an integrated AI-driven data mining tool training device for executing the integrated AI-driven data mining tool training method as described in the first aspect, wherein the integrated AI-driven data mining tool training device comprises: a data reading module, the data reading module is used to read the source data type, perform data hierarchical decomposition, and determine the topological network of data mining, wherein the decomposition layer at least includes a domain-scenario-multi-level refinement layer; a mining drive rule determination module, the mining drive rule determination module is used to determine the mining drive rules based on the topological network and the mining target based on the network node, and the mining drive rules include brain-like processing rules, rigid logic Rules and long short-term memory rules; a mining tool determination module, the mining tool determination module is used to perform multi-logic block training and network integration deployment based on the mining driving rules and the topological network, and determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented to the mining target; an optimization module, the optimization module is used to introduce a proxy function, cross-validate and perform multi-objective optimization on the initialization mining tool, and determine the optimization strategy, wherein the proxy function is an approximate gradient relationship for a nonlinear optimization target; a calculation domain allocation module, the calculation domain allocation module is used to optimize and independently allocate the initialization mining tool to the calculation domain based on the optimization strategy, and determine the data mining tool.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] By reading the data type of the source end, the data is hierarchically decomposed to determine the topological network of data mining, wherein the decomposition layer at least includes the domain-scenario-multi-level refinement layer; based on the topological network, the mining driving rules are determined based on the mining target of the network node, and the mining driving rules include brain-like processing rules, rigid logic rules and long short-term memory rules; based on the mining driving rules and the topological network, multi-logic block training and network integration deployment are performed to determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented to the mining target; the agent function is introduced to cross-validate and multi-objective optimization of the initialization mining tool to determine the optimization strategy, wherein the agent function is an approximate gradient relationship for nonlinear optimization targets; based on the optimization strategy, the initialization mining tool is optimized and the independent calculation domain is allocated to determine the data mining tool. In other words, through integrated AI drive, combined with a variety of artificial intelligence technologies, the automation and intelligence of the data mining process is realized, and the accuracy and efficiency of data mining are improved.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0012] Figure 1 A flowchart of the integrated AI-driven data mining tool training method of this application;

[0013] Figure 2 This is a schematic diagram of the structure of the integrated AI-driven data mining tool training device of this application.

[0014] Explanation of the reference numerals: data reading module 11 , mining driving rule determination module 12 , mining tool determination module 13 , optimization module 14 , computational domain allocation module 15 . DETAILED DESCRIPTION

[0015] This application solves the technical problem in the prior art that the accuracy of mining results is low due to the low efficiency of traditional data mining tools in processing large-scale and complex data by providing an integrated AI-driven data mining tool training method and device. Through integrated AI drive and combining multiple artificial intelligence technologies, the data mining process is automated and intelligent, improving the accuracy and efficiency of data mining.

[0016] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0017] For example, please refer to the attached Figure 1The present application provides an integrated AI-driven data mining tool training method, wherein the integrated AI-driven data mining tool training method is applied to an integrated AI-driven data mining tool training device, and the integrated AI-driven data mining tool training method specifically includes the following steps:

[0018] Step 1: Read the data type at the source end, perform hierarchical decomposition of the data, and determine the topological network for data mining, where the decomposition layer at least includes the domain-scenario-multi-level refinement layer.

[0019] Specifically, data is loaded from a database, file, or data stream, and the type of each column of data is identified, such as integer, floating point number, string, date, etc. The raw data is classified and organized according to specific dimensions to form a hierarchical data structure, including a domain layer, a scenario layer, and a multi-level refinement layer. The domain refers to the vertical industry or application field to which the data belongs, such as finance, medical care, education, etc. After determining the domain of the data, the scenario layer further refines the classification of the data. The scenario refers to the specific context or goal of the data mining application. For example, in the financial field, the scenario may include stock trading, credit risk assessment, etc. The multi-level refinement layer is a further refinement of the scenario layer, including multiple sub-levels, each of which focuses on different aspects or features of the data. Based on the above hierarchical decomposition, a topological network for data mining is constructed, which defines the connection and interaction between different levels and components in the data mining process. The result of the hierarchical decomposition of data is converted into a topological network, where each node represents a data level (domain, scenario, multi-level refinement layer), and the edge represents the association relationship between nodes. For example, a domain can have multiple scenarios, and a scenario can contain multiple refinement layers. Through hierarchical decomposition, data mining tools can more accurately locate and analyze data in specific fields and scenarios, thereby improving the pertinence and effectiveness of analysis.

[0020] Step 2: Based on the topological network, with mining targets based on network nodes, determine mining driving rules, wherein the mining driving rules include brain-like processing rules, rigid logic rules, and long-short-term memory rules.

[0021] Specifically, the topological network is determined in the previous data hierarchical decomposition step, which defines the connection and interaction between different levels and components in the data mining process. The network nodes represent specific tasks or goals in the data mining process, such as specific data sets, data features, or analysis goals. According to the structure and node characteristics of the topological network, the driving rules of data mining are determined to guide the data mining tools on how to process and analyze data to achieve the predetermined mining goals. Mining driving rules include brain-like processing rules, rigid logic rules, and long and short-term memory rules, which respectively simulate the processing methods of the human brain, rules based on clear logic and facts, and rules based on the time characteristics and dynamic changes of data. By determining the mining driving rules based on the mining goals of the topological network and network nodes, the most appropriate mining rules are adopted according to different business needs and data characteristics, thereby improving the intelligence and automation level of the entire data mining process.

[0022] Step 3: Based on the mining driving rules and the topological network, perform multi-logic block training and network integration deployment, and determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented towards the mining target.

[0023] Specifically, mining driving rules define the rules that need to be followed in the data mining process, such as brain-like processing rules, rigid logic rules, and long-short-term memory rules. The topological network is determined in the data hierarchical decomposition step, which defines the connection and interaction between different levels and components in the data mining process. Multiple logical blocks are different processing modules or units in the data mining process, which can process different data sets or tasks in parallel. Use mining driving rules and topological networks to guide the training process of each logical block to achieve the goal of data mining. During the training process, adjust the parameters of the logical block to optimize its performance on specific mining tasks. Based on the training results of the logical blocks, network integration deployment is carried out, and different logical blocks are combined and deployed according to the structure and association relationship of the topological network to form an integrated AI-driven data mining tool. Through the training of multiple logical blocks and network integration deployment, the initialization mining tool is determined. The initialization mining tool refers to an AI model that has been preliminarily trained and deployed, which contains multiple logical blocks and can perform data mining tasks. The driving algorithm is an algorithm that guides the logical block on how to process and analyze data, guided by the mining goal. Each logic block is configured with a mining-oriented driving algorithm, and each logic block has its own specific data processing and analysis algorithm to achieve specific mining goals. Through multi-logic block training and network integrated deployment, the efficiency of data mining can be improved, and parallel processing and rapid learning can be achieved.

[0024] Step 4: Introduce a proxy function, perform cross-validation and multi-objective optimization on the initialization mining tool, and determine the optimization strategy, wherein the proxy function is an approximate gradient relationship for the nonlinear optimization target.

[0025] Specifically, a surrogate function is a function used to approximate the objective function, which is very useful when the objective function is too complex or difficult to calculate the gradient directly. In nonlinear optimization problems, the surrogate function is designed to provide approximate gradient information of the nonlinear optimization objective function to help the optimization algorithm find the optimal solution. By using surrogate functions, the system can evaluate and compare the performance of different models more quickly, thereby selecting the optimal model. Cross-validation is a method for evaluating model performance. By dividing the data set into a training set and a validation set, the model is trained and tested multiple times to evaluate the generalization ability of the model. Use surrogate functions to evaluate multiple performance indicators of the model, such as accuracy, recall, F1 score, etc. Through multi-objective optimization, a balance point is found so that the model can achieve good performance in multiple performance indicators. Multi-objective optimization refers to optimizing the model by considering multiple performance indicators at the same time during the data mining process. According to the results of cross-validation and multi-objective optimization, the optimization strategy of the data mining tool is determined, which guides how to adjust model parameters, select algorithms, and adjust the training process to achieve the best mining effect. Surrogate functions are used to approximate complex nonlinear optimization objectives, and the best optimization strategy is determined through cross-validation and multi-objective optimization, thereby improving the performance and generalization ability of the model.

[0026] Step 5: Based on the optimization strategy, the initialization mining tool is optimized and an independent calculation domain is allocated to determine the data mining tool.

[0027] Specifically, the optimization strategy obtained by cross-validation and multi-objective optimization is a specific method or step for improving model performance. According to the determined optimization strategy, the initialization mining tool is adjusted and improved, including adjusting model parameters, improving algorithms, changing the training process, etc., to improve the performance of the model. The computational domain refers to the computing resources or computational domain used to execute the data mining algorithm. Independent computational domain allocation refers to the allocation of independent computing resources to different parts of the data mining tool for parallel processing or optimized resource utilization. According to the needs of the logical block, such as computational complexity, data processing volume, etc., dynamic adjustments are made to allocate different computing resources to different logical blocks to ensure that each logical block can perform its tasks efficiently. The final data mining tool is determined by optimizing the initialization mining tool and allocating independent computational domains based on the optimization strategy. By optimizing the initialization mining tool and allocating independent computational domains based on the optimization strategy, it is ensured that the data mining process can be carried out efficiently and accurately, the efficiency and accuracy of data mining are improved, and the changing data analysis needs are adapted.

[0028] Furthermore, step 2 of this application includes:

[0029] The topological network is traversed to segment and determine a plurality of topological links, wherein each topological link contains mining targets of multi-level refined link nodes; based on the plurality of topological links, a homology search based on big data is performed to determine a plurality of homology data sets; the plurality of homology data sets are traversed to analyze and determine rules and perform personalized link compensation correction to determine the mining driving rules.

[0030] Specifically, the topological network is traversed to check each node and connection in the network to understand the overall structure of the data and the mining target. During the traversal process, the topological network is divided into multiple topological links according to the characteristics of the data and the mining requirements. Each link represents a specific path or subtask in the data mining process, connects a series of nodes, and represents one or more specific data mining tasks. Each topological link contains multiple nodes, which represent different levels of data refinement. Each node has its own specific mining target, and the mining is refined recursively layer by layer. Big data technology is used to perform homology retrieval on each topological link to find data sets with the same source or correlation as the data nodes in the link. Through homology retrieval, multiple homology data sets related to each topological link are determined. Homologous data sets are data sets with the same source or correlation as the original data set, including historical data, real-time data, social media data and other types of data.

[0031] Traverse the homologous data sets, identify the patterns, relationships and rules in the data, perform confidence checks, and determine the confidence data sets. In the process of traversing the homologous data sets, analyze and determine the rules that need to be followed in the data mining process, including rigid logic rules, brain-like processing rules and long-short-term memory rules, which are mined based on the objective knowledge system, subjective thinking links and subjective and objective learning and updating rules. Rigid logic rules are rules based on objective facts and logical reasoning, brain-like processing rules are rules that imitate the human brain to process information, and long-short-term memory rules are rules based on time series data.

[0032] Mining is generally universal and can be applied to a variety of different data sets and scenarios. Each specific data mining task or application scenario has its own unique needs and characteristics, and general data mining tools may not be able to fully meet the personalized needs of specific scenarios. In order to resolve the difference between universal and personalized needs, personalized link compensation correction is required. In the context of data mining, link compensation correction refers to the analysis and adjustment of data sets to solve the problem of inconsistent or missing data. Through personalized link compensation correction, the logical fit between data mining tools and specific application scenarios can be improved to ensure that specific needs can be met more accurately. Mining-driven rules refer to a set of rules that are ultimately determined based on the analysis of homologous data sets and the results of link compensation correction to guide the data mining process, including data processing, feature selection, model parameter setting, etc. By analyzing homologous data sets and determining rules, data can be understood and analyzed more accurately. Personalized link compensation correction enables data mining tools to adapt to the characteristics and needs of different data sets and improve the adaptability of analysis.

[0033] Furthermore, the present application also includes the following steps:

[0034] The multiple homologous data sets are traversed, and confidence verification is performed to determine a confidence data set; the confidence data set is traversed, and the rigid logic rules are mined based on the objective knowledge system, the brain-like processing rules are mined based on the subjective thinking link, and the long-short-term memory rules are mined based on the subjective and objective learning and updating laws; the brain-like processing rules, rigid logic rules and long-short-term memory rules are traversed, and rule clustering is performed and the topological network is divided into link areas.

[0035] Specifically, the homologous data sets are traversed to check the content and structure of each data. Each homologous data set is confidence checked. Confidence checking is a process of evaluating the quality of a data set, which involves checking the accuracy, completeness and consistency of the data. Through confidence checking, data of low quality are identified and excluded to ensure that only reliable and accurate data are used in the subsequent data mining process. Confidence data sets refer to those data sets with high quality that have passed the verification and are considered to be accurate and reliable. These data sets are checked and analyzed one by one, and rigid logic rules are mined from the confidence data sets using objective knowledge systems such as mathematical formulas, scientific principles, and statistical laws. Rigid logic rules refer to rules based on clear logic and facts, which are usually fixed in data analysis and mining and do not change with time and environment. Subjective thinking links, such as pattern recognition, intuitive judgment, and rules of thumb, are used to analyze and mine brain-like processing rules in data. Brain-like processing rules refer to those rules that simulate the human brain's processing of information, usually involving fuzzy logic and pattern recognition, are non-deterministic, and have certain flexibility and adaptability.

[0036] The subjective and objective learning update rules, such as forgetting curve and memory reinforcement, are used to analyze and mine the long and short-term memory rules in the data. Long and short-term memory rules refer to rules based on the time characteristics and dynamic changes of data, which can capture the changing trend of data over time. The brain-like processing rules, rigid logic rules and long and short-term memory rules are traversed, and these rules are checked and analyzed one by one for subsequent clustering and link area division. In the process of traversal, the mined rules are clustered. Rule clustering is a data mining technology that can group similar or related rules. Specific algorithms and strategies are used to identify and cluster similar rules, including rule-based similarity metrics, clustering algorithms (such as K-means, hierarchical clustering, etc.). Through rule clustering, rules with similar logic or functions are grouped together, thereby reducing the redundancy of rules and improving the utilization efficiency of rules. Based on the results of rule clustering, the topological network is divided into link areas, and topological links with the same or similar logic rules are divided into the same area. In this way, each area contains a group of links with similar functions or goals, which helps to improve the efficiency and accuracy of data mining. By verifying confidence, mining three types of rules, and clustering rules, we ensure that high-quality data is used for data mining, and we can mine different types of rules, organize and plan the data mining process more effectively, and improve overall efficiency.

[0037] Further, step three of this application includes:

[0038] Based on the topological network, multi-level data elements are determined, wherein the data element levels are set based on the mining criticality; a mapping relationship between the data element levels and the learning rates is established; the confidence data set is traversed, and in combination with the mapping relationship, block mapping and supervised training are performed on the multiple logical blocks, and integrated deployment is performed based on the network association relationship, and the initialization mining tool is determined.

[0039] Specifically, according to the structure and node characteristics of the topological network, the multi-level data elements in the data mining process are determined. Data elements refer to the key data points or features that need to be paid attention to and analyzed in the data mining process. In the topological network, data elements are divided into different levels, which are set based on the criticality of the data in the mining process. Mining criticality refers to the importance and influence of data elements in the data mining process. According to mining criticality, data elements are divided into different levels so that different degrees of attention can be given in the data mining process. The learning rate is an important hyperparameter in machine learning, which determines the update speed of the model during the learning process. The learning rate affects the convergence speed and final performance of the model. Too high or too low learning rates may lead to poor model training results. A mapping relationship between the level of data elements and the learning rate is established, and the learning rate is adjusted according to the level of data elements. For example, for data elements with high criticality, a higher learning rate can be set to ensure that the model can quickly learn these important features. For data elements with low criticality, a lower learning rate can be set to avoid overfitting.

[0040] Traverse the confidence data set and check the content and structure of each data set. Combined with the previously established mapping relationship between data element levels and learning rates, perform block mapping and supervised training on multiple logical blocks. Multiple logical blocks refer to driving algorithms configured for specific mining targets, and block mapping refers to mapping data sets to different logical blocks for parallel processing. Supervised training refers to training models with labeled data to improve model accuracy and performance. Integrate and deploy logical blocks based on network associations, combine and deploy different logical blocks according to the structure and association of the network, and form an integrated AI-driven data mining tool. Determine the initialization mining tool through block mapping, supervised training, and integrated deployment. The initialization mining tool is an AI model formed based on the training and integrated deployment of multiple logical blocks. It contains multiple logical blocks and can perform data mining tasks. Starting from determining multi-level data elements based on the topological network, to establishing the mapping relationship between data element levels and learning rates, and finally performing block mapping, supervised training and integrated deployment, the mining tools are initialized to ensure that high-quality data can be used for data mining. Different logical blocks can also be effectively matched and trained to meet the ever-changing data analysis needs.

[0041] Further, step 4 of this application includes:

[0042] Determine the optimization target of the initialization mining tool and extract the nonlinear target; identify the first nonlinear target, perform big data mining, and determine the matching function set; based on the rule fitness and target fit, traverse the matching function set to perform adaptive analysis and proofreading, and select the first proxy function; traverse the nonlinear target and determine the Nth proxy function, and add the first proxy function to the Nth proxy function into the proxy function.

[0043] Specifically, determine the optimization target for initializing the mining tool. The optimization target refers to the performance standard that the data mining tool needs to achieve during training and operation, such as accuracy, recall rate, F1 score, etc. On the basis of determining the optimization target, extract the nonlinear target. Nonlinear targets refer to those targets that cannot be accurately described simply by linear models or methods, and usually require more complex models and algorithms to handle. Identify the first nonlinear target, use big data resources for mining, and use big data technology to process and analyze large amounts of data to discover the relationships and patterns between nonlinear targets. Through big data mining, determine the matching function set. The matching function set is a set of functions used to match and associate different data points or features, including nonlinear regression, support vector machines, neural networks, etc., which can handle complex nonlinear relationships and improve the accuracy and efficiency of data mining.

[0044] The suitability of the matching function set is evaluated based on the rule fitness and target fit. Rule fitness refers to the degree of matching between the matching function and the data mining rule, and target fit refers to the degree of matching between the matching function and the mining target. Based on the evaluation, the matching function set is traversed to check the performance and suitability of each matching function. Adaptability analysis and proofreading are performed to check the performance of the matching function on a specific data set and whether it can effectively match and associate data points or features. The first proxy function is a function selected from the matching function set to approximate nonlinear objectives and guide optimization. The characteristics and requirements of each nonlinear objective are checked to determine the Nth proxy function. The proxy function set refers to a set of functions used to process nonlinear objectives. The Nth proxy function is the best proxy function selected for the Nth nonlinear objective. The first proxy function to the Nth proxy function are added to the proxy function set, that is, all selected proxy functions are brought together for use in the data mining process. In the optimization process, for some nonlinear objectives, there is a problem that the optimization direction is difficult to determine. Using a proxy function (such as the gradient approximation method) to solve this problem, because it provides approximate information about the direction of the objective function's gradient, thereby guiding the optimization algorithm (such as gradient descent) to search for the optimal solution more effectively in a complex nonlinear space. By traversing the matching function set and performing adaptability analysis and proofreading, evaluating the performance of each matching function in the actual data mining task, and selecting the best matching function as the first proxy function, building a proxy function set can provide optimization directions from multiple perspectives, which helps to improve the performance of mining tools more comprehensively.

[0045] Furthermore, the present application also includes the following steps:

[0046] Traversing the multiple logic blocks, determining the distributed block convergence conditions, wherein the block convergence conditions are determined based on the refinement level and mining accuracy of the mining target; determining the first convergence degree based on the validation set of the same target sample, and determining the second convergence degree by cross-validation of different target training samples; setting the convergence interval of the multiple logic blocks, if the first convergence degree is greater than the upper limit of the convergence interval, and the second convergence degree is less than the lower limit of the convergence interval, then the initialization mining tool is trained to convergence, otherwise the optimization target is located.

[0047] Specifically, multiple logical blocks are traversed and appropriate convergence conditions are set for these logical blocks. Block convergence conditions refer to the criteria for stopping the training process of the logical block when specific conditions are met during the data mining process. It is usually determined based on the degree of refinement of the mining target and the mining accuracy. The degree of refinement of the mining target refers to the requirements of the data mining task for details and accuracy, and the mining accuracy refers to the degree of closeness between the data mining results and the actual data. Different data mining tasks have different requirements for the quality of the results, and the block convergence conditions need to be set according to the specific needs and the characteristics of the source data. For example, for mining tasks that require high accuracy, more stringent convergence conditions are set to ensure that the model can achieve the expected prediction accuracy.

[0048] The samples in the same block are divided into training set and validation set, which are used for training the model and evaluating the performance of the model respectively. Use the trained model to make predictions on the validation set to evaluate the performance of the model and ensure that the model has achieved a high accuracy on the training set as the first verification. Use the sample sets of different blocks to verify the model of the current block and evaluate the generalization ability of the model on different data sets as the second verification. The first verification must be high enough to ensure the accuracy and pertinence of the model on a specific block; the second verification must be low enough to ensure that the model will not make misjudgments when processing data from different blocks. If the second verification is high, it means that there is approximation or verification error between blocks. If the blocks are similar, fusion processing can be considered to improve the overall performance of the model. The first convergence degree refers to the degree of convergence of the model on a specific data set, that is, the degree to which the model performance reaches a stable or optimal state. The second convergence degree refers to the degree of convergence of the model on different target data, that is, the stability and accuracy of the model when processing different targets.

[0049] The convergence interval refers to a range set for the first convergence degree and the second convergence degree, and refers to the threshold range where the model performance reaches a stable or optimal state. If the first convergence degree is greater than the upper limit of the convergence interval, and the second convergence degree is less than the lower limit of the convergence interval, it is considered that the initialization mining tool has been trained to convergence. If the block does not reach convergence, it is necessary to review and adjust the optimization goal, which may be caused by problems such as model structure, parameter setting or data preprocessing. By cross-validating the initialization mining tool, it is ensured that the model has good performance on different data sets and improves the generalization ability of the model.

[0050] Further, step five of this application includes:

[0051] Traversing the multiple logical blocks, determining a first allocation weight based on the driving requirements of algorithms and rules, and determining a second allocation weight based on the single processing volume and the concurrent processing volume; fitting the first allocation weight and the second allocation weight to determine the logical block allocation ratio; and performing operator allocation on the initialized mining tool based on the logical block allocation ratio, wherein the data mining tool has logical block correlation and relative independence.

[0052] Specifically, multiple logic blocks are traversed and corresponding weights are assigned to each block. According to the algorithm and rule-driven requirements of each logic block, a first allocation weight is determined, which reflects the importance of different logic blocks in the data mining process. According to the single processing volume and concurrent processing volume, a second allocation weight is determined, which reflects the efficiency and resource requirements of different logic blocks when processing data. The first allocation weight and the second allocation weight are fitted. Fitting refers to adjusting the weight by a certain algorithm or method, including optimization algorithms such as gradient descent and genetic algorithms, to achieve a better allocation effect. By fitting the first allocation weight and the second allocation weight, the weight of each logic block is determined, thereby forming the allocation ratio of the logic block. The allocation ratio reflects the relative importance of different logic blocks in the data mining process and the proportion of processing resources allocated to each logic block. According to the logic block allocation ratio, an appropriate operator is assigned to each logic block. An operator refers to an algorithm or operation used for data mining, such as classification, regression, clustering, etc. There is a correlation between different logic blocks in the data mining tool, that is, the output of one logic block may be the input of another logic block. At the same time, there may be relative independence between logical blocks, that is, the output of one logical block may not affect another logical block. When allocating operators, the system needs to consider the correlation and relative independence between logical blocks to ensure that the data mining process can be carried out efficiently and accurately. Through independent domain allocation, operators are allocated according to the specific needs and computing resource requirements of each logical block, thereby achieving efficient data mining.

[0053] In summary, the integrated AI-driven data mining tool training method provided in this application has the following technical effects:

[0054] By reading the data type of the source end, the data is hierarchically decomposed to determine the topological network of data mining, wherein the decomposition layer at least includes the domain-scenario-multi-level refinement layer; based on the topological network, the mining driving rules are determined based on the mining target of the network node, and the mining driving rules include brain-like processing rules, rigid logic rules and long short-term memory rules; based on the mining driving rules and the topological network, multi-logic block training and network integration deployment are performed to determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented to the mining target; the agent function is introduced to cross-validate and multi-objective optimization of the initialization mining tool to determine the optimization strategy, wherein the agent function is an approximate gradient relationship for nonlinear optimization targets; based on the optimization strategy, the initialization mining tool is optimized and the independent calculation domain is allocated to determine the data mining tool. In other words, through integrated AI drive, combined with a variety of artificial intelligence technologies, the automation and intelligence of the data mining process is realized, and the accuracy and efficiency of data mining are improved.

[0055] Embodiment 2: Based on the same inventive concept as the integrated AI-driven data mining tool training method in the aforementioned embodiment, the present application also provides an integrated AI-driven data mining tool training device, see the attached Figure 2 , the integrated AI-driven data mining tool training device comprises:

[0056] The data reading module 11 is used to read the data type of the source end, perform hierarchical decomposition of the data, and determine the topological network of data mining, wherein the decomposition layer at least includes a domain-scenario-multi-level refinement layer.

[0057] The mining driving rule determination module 12 is used to determine the mining driving rules based on the topological network and the mining targets based on the network nodes. The mining driving rules include brain-like processing rules, rigid logic rules and long short-term memory rules.

[0058] The mining tool determination module 13 is used to perform multi-logic block training and network integration deployment based on the mining driving rules and the topological network, and determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented towards the mining target.

[0059] The optimization module 14 is used to introduce a proxy function, perform cross-validation and multi-objective optimization on the initialization mining tool, and determine an optimization strategy, wherein the proxy function is an approximate gradient relationship for a nonlinear optimization target.

[0060] The calculation domain allocation module 15 is used to optimize and independently allocate the calculation domain to the initialized mining tool based on the optimization strategy, and determine the data mining tool.

[0061] Furthermore, the mining driving rule determination module 12 in the integrated AI-driven data mining tool training device is also used for:

[0062] The topological network is traversed to segment and determine a plurality of topological links, wherein each topological link contains mining targets of multi-level refined link nodes; based on the plurality of topological links, a homology search based on big data is performed to determine a plurality of homology data sets; the plurality of homology data sets are traversed to analyze and determine rules and perform personalized link compensation correction to determine the mining driving rules.

[0063] Furthermore, the integrated AI-driven data mining tool training device further includes a determination rule analysis module for:

[0064] The multiple homologous data sets are traversed, and confidence verification is performed to determine a confidence data set; the confidence data set is traversed, and the rigid logic rules are mined based on the objective knowledge system, the brain-like processing rules are mined based on the subjective thinking link, and the long-short-term memory rules are mined based on the subjective and objective learning and updating laws; the brain-like processing rules, rigid logic rules and long-short-term memory rules are traversed, and rule clustering is performed and the topological network is divided into link areas.

[0065] Furthermore, the mining tool determination module 13 in the integrated AI-driven data mining tool training device is also used for:

[0066] Based on the topological network, multi-level data elements are determined, wherein the data element levels are set based on the mining criticality; a mapping relationship between the data element levels and the learning rates is established; the confidence data set is traversed, and in combination with the mapping relationship, block mapping and supervised training are performed on the multiple logical blocks, and integrated deployment is performed based on the network association relationship, and the initialization mining tool is determined.

[0067] Furthermore, the optimization module 14 in the integrated AI-driven data mining tool training device is also used for:

[0068] Determine the optimization target of the initialization mining tool and extract the nonlinear target; identify the first nonlinear target, perform big data mining, and determine the matching function set; based on the rule fitness and target fit, traverse the matching function set to perform adaptive analysis and proofreading, and select the first proxy function; traverse the nonlinear target and determine the Nth proxy function, and add the first proxy function to the Nth proxy function into the proxy function.

[0069] Furthermore, the optimization module 14 in the integrated AI-driven data mining tool training device is also used for:

[0070] Traversing the multiple logic blocks, determining the distributed block convergence conditions, wherein the block convergence conditions are determined based on the refinement level and mining accuracy of the mining target; determining the first convergence degree based on the validation set of the same target sample, and determining the second convergence degree by cross-validation of different target training samples; setting the convergence interval of the multiple logic blocks, if the first convergence degree is greater than the upper limit of the convergence interval, and the second convergence degree is less than the lower limit of the convergence interval, then the initialization mining tool is trained to convergence, otherwise the optimization target is located.

[0071] Furthermore, the computational domain allocation module 15 in the integrated AI-driven data mining tool training device is also used for:

[0072] Traversing the multiple logical blocks, determining a first allocation weight based on the driving requirements of algorithms and rules, and determining a second allocation weight based on the single processing volume and the concurrent processing volume; fitting the first allocation weight and the second allocation weight to determine the logical block allocation ratio; and performing operator allocation on the initialized mining tool based on the logical block allocation ratio, wherein the data mining tool has logical block correlation and relative independence.

[0073] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. Figure 1 The integrated AI-driven data mining tool training method and specific examples in Example 1 are also applicable to the integrated AI-driven data mining tool training device in this embodiment. Through the above detailed description of the integrated AI-driven data mining tool training method, those skilled in the art can clearly understand the integrated AI-driven data mining tool training device in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0074] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0075] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. An integrated AI-driven data mining tool training method, characterized in that: include: Read the data type of the source end, perform hierarchical decomposition of the data, and determine the topological network of data mining, where the decomposition layer at least includes the domain-scenario-multi-level refinement layer; Based on the topological network, determining mining driving rules based on mining targets based on network nodes, wherein the mining driving rules include brain-like processing rules, rigid logic rules, and long-short term memory rules; Based on the mining driving rules and the topological network, multi-logic block training and network integration deployment are performed to determine an initialization mining tool, wherein each logic block is configured with a driving algorithm oriented toward a mining target; Introducing a proxy function, performing cross-validation and multi-objective optimization on the initialization mining tool, and determining an optimization strategy, wherein the proxy function is an approximate gradient relationship for a nonlinear optimization objective; Based on the optimization strategy, the initialization mining tool is optimized and an independent calculation domain is allocated to determine the data mining tool; According to the structure and node characteristics of the topological network, the driving rules of data mining are determined to guide the data mining tool on how to process and analyze the data to achieve the predetermined mining goal; The determining of the mining driving rules includes: Traversing the topological network, segmenting and determining a plurality of topological links, wherein each topological link includes a mining target of a multi-level refined link node; Based on the multiple topological links, perform homology retrieval based on big data to determine multiple homology data sets; The plurality of homologous data sets are traversed, rules are analyzed and determined, and personalized link compensation correction is performed to determine the mining driving rules.

2. The integrated AI-driven data mining tool training method according to claim 1, characterized in that: Traversing the plurality of homologous data sets and analyzing and determining rules, including: Traversing the multiple homologous data sets, and performing confidence verification to determine a confidence data set; Traversing the confidence data set, mining the rigid logic rules based on the objective knowledge system, mining the brain-like processing rules based on the subjective thinking link, and mining the long-term and short-term memory rules based on the subjective and objective learning and updating laws; The brain-like processing rules, rigid logic rules and long short-term memory rules are traversed to perform rule clustering and divide the topological network into link areas.

3. The integrated AI-driven data mining tool training method according to claim 1, characterized in that: The introducing proxy function comprises: Determining the optimization target of the initialization mining tool and extracting nonlinear targets; Identify the first nonlinear target, conduct big data mining, and determine the matching function set; Based on the rule adaptability and the target fit, the matching function set is traversed to perform adaptability analysis and proofreading, and a first proxy function is selected; The nonlinear objective is traversed and an Nth proxy function is determined, and the first proxy function to the Nth proxy function are added into the proxy function.

4. The integrated AI-driven data mining tool training method according to claim 2, characterized in that: The step of determining and initializing the mining tool comprises: Based on the topological network, determining multiple levels of data elements, wherein the levels of the data elements are set based on mining criticality; Establish a mapping relationship between data element levels and learning rates; The confidence data set is traversed, and block mapping and supervised training are performed on the multiple logical blocks in combination with the mapping relationship. Integrated deployment is performed based on the network association relationship, and the initialization mining tool is determined.

5. The integrated AI-driven data mining tool training method according to claim 1, characterized in that: The cross-validation of the initialization mining tool comprises: Traversing the plurality of logical blocks, determining a distributed block convergence condition, wherein the block convergence condition is determined based on a refinement degree and a mining accuracy of a mining target; Determine the first convergence degree based on the validation set of the same target sample, and determine the second convergence degree by cross-validation of the training samples with different targets; The convergence interval of the multiple logic blocks is set. If the first convergence degree is greater than the upper limit of the convergence interval and the second convergence degree is less than the lower limit of the convergence interval, the initialization mining tool is trained to convergence, otherwise the optimization target is located.

6. The integrated AI-driven data mining tool training method according to claim 1, characterized in that: The initialization mining tool is assigned an independent computational domain, including: Traversing the multiple logic blocks, determining a first allocation weight based on the driving requirements of the algorithm and the rule, and determining a second allocation weight based on the single processing volume and the concurrent processing volume; Fitting the first allocation weight and the second allocation weight to determine a logic block allocation ratio; Based on the logic block allocation ratio, operators are allocated to the initialization mining tools, wherein the data mining tools have logic block association and relative independence.

7. An integrated AI-driven data mining tool training device, characterized in that: The steps for implementing the integrated AI-driven data mining tool training method according to any one of claims 1 to 6, wherein the integrated AI-driven data mining tool training device comprises: A data reading module, which is used to read the data type of the source end, perform hierarchical decomposition of the data, and determine the topological network of data mining, wherein the decomposition layer at least includes a domain-scenario-multi-level refinement layer; A mining driving rule determination module, wherein the mining driving rule determination module is used to determine mining driving rules based on the topological network and mining targets based on network nodes, wherein the mining driving rules include brain-like processing rules, rigid logic rules, and long-short-term memory rules; A mining tool determination module, the mining tool determination module is used to perform multi-logic block training and network integration deployment based on the mining driving rules and the topological network, and determine the initialization mining tool, wherein each logic block is configured with a driving algorithm oriented to the mining target; An optimization module, the optimization module is used to introduce a proxy function, perform cross-validation and multi-objective optimization on the initialization mining tool, and determine an optimization strategy, wherein the proxy function is an approximate gradient relationship for a nonlinear optimization target; A calculation domain allocation module is used to optimize and independently allocate calculation domains to the initialized mining tools based on the optimization strategy to determine the data mining tools.

Images