Data accurate matching analysis method based on machine learning

Abstract

The invention discloses an accurate data matching analysis method based on machine learning, and relates to the technical field of data matching, and the method comprises the following specific steps: dividing data to be processed into a plurality of sub-data sets, dividing the size of each sub-data set according to the performance index and data characteristic of a computational node, and obtaining a data set; according to the method, the to-be-processed data is divided into a plurality of sub-data sets through a distributed parallel processing technology, the sub-data sets are distributed to a plurality of computing nodes for parallel processing, each computing node can independently process the sub-data set distributed to the computing node, and therefore the parallelism of computing resources is fully utilized, and the computing efficiency is improved. Compared with the prior art, the method has the advantages that the data processing efficiency and speed are remarkably improved, meanwhile, an optimized synchronization mechanism is adopted, coordination work among the computing nodes is ensured, the processing bottleneck caused by waiting or communication delay is avoided, and the overall processing performance is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data matching, and in particular to a data accurate matching analysis method based on machine learning. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, data has become an indispensable core resource in corporate operations, scientific research and government decision-making. The scale, type and generation speed of data are growing at an unprecedented rate, which puts extremely high demands on data processing capabilities. The traditional centralized data processing model is limited by computing resources, storage capacity and network bandwidth, and it is difficult to cope with the real-time processing needs of massive data. Therefore, distributed computing architecture has gradually become the mainstream choice for processing big data.

[0003] At present, existing data matching analysis methods often use simple random segmentation and segmentation based on fixed rules in terms of data segmentation. They lack in-depth analysis of data characteristics, which can easily lead to data imbalance between sub-data sets and affect the efficiency and accuracy of subsequent processing. In terms of node allocation, existing technologies usually only consider a single performance indicator of the computing node, resulting in unreasonable allocation plans, insufficient integration of data parallelism and model parallelism, and synchronization problems and performance bottlenecks.

[0004] To sum up, the existing technology has obvious shortcomings in the precise matching analysis of large-scale data and cannot meet the requirements of high efficiency, accuracy and reliability of data processing in the big data era. Therefore, there is an urgent need for a data precise matching analysis method based on machine learning that can make full use of distributed parallel processing technology, overcome the shortcomings of the existing technology, and realize fast and accurate processing of large-scale data. Summary of the invention

[0005] The purpose of the present invention is to make up for the shortcomings of the prior art and to provide a data precision matching analysis method based on machine learning. It can better adapt to data of different types and structures through intelligent data segmentation algorithms and feature extraction methods. Whether it is text, numerical or image data, the system can perform effective preprocessing and feature engineering according to its characteristics, thereby improving the support capability for diversified data sources. In addition, the machine learning-based method allows the model to self-optimize with the continuous input of new data, which enables the system to continuously improve its performance to cope with the ever-changing data environment and business needs.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a data accurate matching analysis method based on machine learning, the specific steps of the method are: S100, dividing the data to be processed into multiple sub-data sets, and the size of each sub-data set is divided according to the performance index of the computing node and the data characteristics; The performance indicators of each computing node are monitored in real time, including CPU utilization. , memory usage , Network bandwidth and disk I / O speed , define the computing node performance index ,Right now ,in, are the weight coefficients of CPU utilization, memory usage, network bandwidth, and disk I / O speed, respectively, and , is the maximum value of the network bandwidth, The maximum value of the disk I / O speed; The data characteristics are as follows: For each sub-dataset, analyze its size , Data type complexity and difficulty in handling , define the distribution index ,Right now ,in, is a weight parameter used to balance the performance of computing nodes and subdataset size The impact on distribution, , the allocation is more likely to allocate sub-datasets to performance index On the compute nodes with high values, When , allocation pays more attention to the matching of the size of the sub-dataset and the performance of the computing node; S200, allocating each sub-data set to multiple computing nodes, and each computing node independently performs data preprocessing, feature extraction, and model training operations; S300, processing the allocated sub-dataset on each computing node, wherein: The preprocessing stage includes data cleaning and normalization operations; In the feature extraction stage, effective features are extracted according to the characteristics of the data and the goal of matching analysis; In the model training phase, the extracted features are used to train the machine learning model; S400, through a distributed synchronization mechanism, merges and summarizes the results on each computing node; S500, after merging and summarizing, the final data matching analysis result is obtained.

[0007] Furthermore, in the node allocation process of S200, for each sub-data set, all computing nodes are traversed to calculate the allocation index between them and each computing node. , select the allocation index The largest computing node is used as the distribution target for the sub-dataset.

[0008] Furthermore, the S200 performs real-time performance monitoring on each computing node, including CPU utilization. , memory usage , Network bandwidth and disk I / O speed Indicators, at the same time, analyze the size of each sub-dataset , Data type complexity and difficulty in handling The characteristics of the sub-dataset are formed, and the sub-dataset feature vector is combined with the computing node performance index To match, specifically: For sub-datasets that are difficult to process and have complex data types, performance indicator values ​​are assigned On a high computing node; For small-scale sub-datasets, assign performance indicator values The resource is low and idle on the computing nodes to make full use of computing resources.

[0009] Furthermore, in the process of model training, S300 adopts a distributed training algorithm, that is, a method combining data parallelism and model parallelism, wherein: The data parallelism is to divide the data into multiple small batches and distribute them to different computing nodes for training, and each computing node independently updates the model parameters; The model parallelism is to distribute different parts of the model to different computing nodes for parallel computing.

[0010] Furthermore, the data parallelism is for computing nodes, the total data set is , split the data set into small batches, each of which has a size of , in data parallelism, each computing node The model parameters are updated based on the small batch data assigned to it, that is, the model parameters are , the loss function is , then in the computing node The parameter update formula on is: ,in, Represents a compute node In the The model parameters at the iteration, is the learning rate, is the gradient of the loss function under the current parameters. After one round of iteration, the parameters of each computing node are synchronized, and the parameter averaging method is adopted, that is: , so that all computing nodes use the synchronized parameters for training in the new round of iterations.

[0011] Furthermore, the model parallelism can be divided into parts, respectively, On the computing nodes, for an input sample , the forward propagation process of the model is expressed as: ,in, Represents the model The calculation function of each part, in the back propagation process, each computing node Calculate the gradient of the part it is responsible for , pass the gradient to the previous computing node for further gradient calculation, and get the gradient of the entire model, that is, the loss function is , then the gradient of the model is calculated by the chain rule: , each computing node Responsible for calculation and ,in It is a computing node Responsible model parameters, when the parameters are updated, each computing node The gradient calculated by Update the parameters: .

[0012] Furthermore, during the model training process, the S300 combines data parallelism and model parallelism, divides the data set into multiple small batches, and distributes them to different computing nodes for data parallel training. At the same time, the model is divided into multiple parts and distributed to different computing nodes for model parallel computing. In each round of iteration, each computing node first performs data parallel parameter update, and then performs model parallel gradient calculation and parameter update. The parameters of each computing node are merged and updated through the parameter synchronization mechanism, that is, the model parameters are initialized. , divide the data set into small batches and distribute them to different computing nodes. For each round of iteration Perform data parallel update and model parallel gradient calculation, the data parallel update is performed on each computing node Perform data-parallel parameter updates based on the small batches of data assigned to it: , the model parallel gradient calculation for each computing node Perform model-parallel gradient computation: , the parameter synchronization and update synchronizes the parameters of each computing node: , each computing node is updated according to the synchronized parameters: .

[0013] Furthermore, the S400 synchronization mechanism includes a distributed message queue and a distributed file system to ensure data consistency and accurate results between various computing nodes, and to set up a synchronization monitoring and error handling mechanism to monitor the synchronization status of each computing node in real time, and to immediately reallocate tasks and start backup nodes when node failures and synchronization delays are detected. At the same time, the data in the synchronization process is checked and verified to ensure data integrity and accuracy.

[0014] Furthermore, the distributed message queue and distributed file system in the synchronization mechanism specifically include: The distributed message queue sends the intermediate results to the distributed message queue after each computing node completes part of the task. That is, the message queue adopts message middleware, and each computing node acts as a message producer and sends the results to the central node. At the same time, the central node is set as a message consumer to read the intermediate results of each computing node from the message queue; Distributed file system: At different stages of model training, model parameters and intermediate results are saved in the distributed file system. Each computing node reads and updates the model parameters from the distributed file system at any time. In the result merging stage, the parallel reading and writing functions of the distributed file system are used to merge the final results on each computing node into a unified file.

[0015] Compared with the existing technology, this data accurate matching analysis method based on machine learning has the following beneficial effects: 1. The present invention uses distributed parallel processing technology to divide the data to be processed into multiple sub-datasets, and distributes these sub-datasets to multiple computing nodes for parallel processing. Each computing node can independently process the sub-dataset assigned to it, thereby fully utilizing the parallelism of computing resources and significantly improving the efficiency and speed of data processing. At the same time, the present invention also adopts an optimized synchronization mechanism to ensure the coordinated work between the various computing nodes, avoiding processing bottlenecks caused by waiting or communication delays, and further improving the overall processing performance.

[0016] 2. The present invention adopts a distributed synchronization mechanism in the result merging stage, which summarizes and merges the processing results of each computing node through message middleware and other means. This synchronization mechanism can not only ensure the data consistency between each computing node, but also effectively avoid the problem of data loss or duplication, thereby ensuring the accuracy and reliability of the analysis results. In addition, the present invention also introduces a load balancing strategy to dynamically adjust the task allocation according to the actual performance of the computing node, ensure the load balance of each node, and further improve the reliability and stability of the system.

[0017] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 The figure is an operational flow chart of a data precision matching analysis method based on machine learning. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0021] This embodiment elaborates on the application of the data accurate matching analysis method based on machine learning in the personalized recommendation system of the e-commerce platform. Through this method, efficient processing and accurate analysis of user behavior data are achieved, providing users with personalized product recommendations.

[0022] In the specific implementation, first, the e-commerce platform collects the user's browsing history, purchase history, search keyword behavior data, as well as the attribute information and category information of the goods, cleans and preprocesses these data, removes noise and abnormal data, removes duplicate records and obviously erroneous data, performs data normalization operations, and maps the data to a unified numerical range to ensure data quality and consistency. According to the performance indicators and data characteristics of the computing nodes, the preprocessed data is divided into multiple sub-data sets, and the performance indicators of each computing node are monitored in real time. The monitored indicators include CPU utilization , memory usage , Network bandwidth and disk I / O speed , define the computing node performance index ,Right now ,in, are the weight coefficients of CPU utilization, memory usage, network bandwidth, and disk I / O speed, respectively, and , is the maximum value of the network bandwidth, is the maximum value of the disk I / O speed. For each sub-dataset, analyze its size , Data type complexity and difficulty in handling , define the distribution index ,Right now ,in, is a weight parameter used to balance the performance of computing nodes and subdataset size The impact on distribution, , the allocation is more likely to allocate sub-datasets to performance index On the compute nodes with high values, When the allocation is more focused on matching the size of the sub-dataset with the performance of the computing node, the allocation index , distribute the sub-datasets to appropriate computing nodes.

[0023] Then, the performance indicators of the computing nodes are monitored in real time to ensure accurate understanding of the status of each node. The characteristics of the sub-datasets, including size, data type complexity, and processing difficulty, are analyzed. For sub-datasets with high processing difficulty and complex data types, performance indicators are assigned. The performance index is assigned to the computing nodes with high values ​​to give full play to the advantages of high-performance nodes. For small-scale sub-data sets, On computing nodes with low values ​​and idle resources, to make full use of computing resources and avoid resource waste. On each computing node, the allocated sub-dataset is cleaned, duplicate data is removed, missing values ​​are filled, and data is normalized. The data is mapped to a unified numerical range to facilitate subsequent feature extraction and model training. According to the characteristics of e-commerce data and the goal of personalized recommendation, user behavior characteristics are extracted, such as browsing time, purchase frequency, and product categories of interest, as well as product characteristics such as price, brand, and sales volume. An effective feature selection algorithm is used to screen out features that have a greater impact on recommendation results, thereby improving the accuracy and efficiency of the model.

[0024] Then, the extracted features are used to train the machine learning model using a distributed training algorithm. computing nodes, the total data set is , split the data set into small batches, each of which has a size of , in data parallelism, each computing node The model parameters are updated based on the small batch data assigned to it, that is, the model parameters are , the loss function is , then in the computing node The parameter update formula on is: ,in, Represents a compute node In the The model parameters at the iteration, is the learning rate, is the gradient of the loss function under the current parameters. After one round of iteration, the parameters of each computing node are synchronized, and the parameter averaging method is adopted, that is: , so that all computing nodes use the synchronized parameters for training in the new round of iterations, and the model can be divided into parts, respectively, On the computing nodes, for an input sample , the forward propagation process of the model is expressed as: ,in, Represents the model The calculation function of each part, in the back propagation process, each computing node Calculate the gradient of the part it is responsible for , pass the gradient to the previous computing node for further gradient calculation, and get the gradient of the entire model, that is, the loss function is , then the gradient of the model is calculated by the chain rule: , each computing node Responsible for calculation and ,in It is a computing node Responsible model parameters, when the parameters are updated, each computing node The gradient calculated by Update the parameters: , divide the data set into multiple small batches and assign them to different computing nodes for data parallel training. At the same time, divide the model into multiple parts and assign them to different computing nodes for model parallel computing. In each round of iteration, each computing node first performs data parallel parameter update, and then performs model parallel gradient calculation and parameter update. The parameters of each computing node are merged and updated through the parameter synchronization mechanism, that is, the model parameters are initialized. , divide the data set into small batches and distribute them to different computing nodes. For each round of iteration Perform data parallel update and model parallel gradient calculation, the data parallel update is performed on each computing node Perform data-parallel parameter updates based on the small batches of data assigned to it: , the model parallel gradient calculation for each computing node Perform model-parallel gradient computation: , the parameter synchronization and update synchronizes the parameters of each computing node: , each computing node is updated according to the synchronized parameters: The results on each computing node are merged and summarized through distributed message queues and distributed file systems.

[0025] Among them, after each computing node completes part of the task, the distributed message queue sends the intermediate results to the distributed message queue. The message queue adopts highly reliable message middleware. Each computing node acts as a message producer and sends the results to the central node. At the same time, the central node is set as a message consumer to read the intermediate results of each computing node from the message queue. The distributed file system saves the model parameters and intermediate results to the distributed file system at different stages of model training. Each computing node reads and updates the model parameters from the distributed file system at any time. In the result merging stage, the parallel reading and writing functions of the distributed file system are used to merge the final results on each computing node into a unified file. A synchronization monitoring and error handling mechanism is set to monitor the synchronization status of each computing node in real time. Once a node failure or synchronization delay is found, the task is immediately reallocated and the backup node is started. At the same time, the data in the synchronization process is checked and verified to ensure the integrity and accuracy of the data.

[0026] Finally, based on the results of model training, a personalized product recommendation list is generated for users. When users visit the e-commerce platform, the system uses the trained model to perform real-time analysis based on the users' historical behavior data, predicts the products that the users may be interested in, and recommends these products to the users, thereby improving the users' shopping experience and the platform's sales.

[0027] To sum up, through the application of this embodiment in the personalized recommendation system of the e-commerce platform, the data precise matching analysis method based on machine learning can give full play to its advantages, realize efficient processing and precise analysis of large-scale data, and provide users with more personalized and accurate product recommendation services through reasonable data segmentation, node allocation and parallel processing, combined with an effective synchronization mechanism, thereby improving user satisfaction and the competitiveness of the e-commerce platform. At the same time, the distributed parallel processing and synchronization mechanism of the method ensure the stability and reliability of the system, and can cope with the processing needs of massive data on the e-commerce platform.

[0028] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data accurate matching analysis method based on machine learning, characterized in that: The specific steps of this method are: S100, dividing the data to be processed into multiple sub-data sets, and the size of each sub-data set is divided according to the performance index of the computing node and the data characteristics; The performance indicators of each computing node are monitored in real time, including CPU utilization. , memory usage , Network bandwidth and disk I / O speed , define the computing node performance index ,Right now ,in, are the weight coefficients of CPU utilization, memory usage, network bandwidth, and disk I / O speed, respectively, and , is the maximum value of the network bandwidth, The maximum value of the disk I / O speed; The data characteristics are as follows: For each sub-dataset, analyze its size , Data type complexity and difficulty in handling , define the distribution index ,Right now ,in, is a weight parameter used to balance the performance of computing nodes and subdataset size The impact on distribution, , the allocation is more likely to allocate sub-datasets to performance index On the compute nodes with high values, When , allocation pays more attention to the matching of the size of the sub-dataset and the performance of the computing node; S200, allocating each sub-data set to multiple computing nodes, and each computing node independently performs data preprocessing, feature extraction, and model training operations; S300, processing the allocated sub-dataset on each computing node, wherein: The preprocessing stage includes data cleaning and normalization operations; In the feature extraction stage, effective features are extracted according to the characteristics of the data and the goal of matching analysis; In the model training phase, the extracted features are used to train the machine learning model; S400, through a distributed synchronization mechanism, merges and summarizes the results on each computing node; S500, after merging and summarizing, the final data matching analysis result is obtained.

2. According to the method of accurate data matching analysis based on machine learning in claim 1, it is characterized in that: In the node allocation process of S200, for each sub-data set, all computing nodes are traversed to calculate the allocation index between them and each computing node. , select the allocation index The largest computing node is used as the distribution target for the sub-dataset.

3. According to the data accurate matching analysis method based on machine learning as described in claim 2, it is characterized in that: The S200 performs real-time performance monitoring on each computing node, including CPU utilization , memory usage , Network bandwidth and disk I / O speed Indicators, at the same time, analyze the size of each sub-dataset , Data type complexity and difficulty in handling The characteristics of the sub-dataset are formed, and the sub-dataset feature vector is combined with the computing node performance index To match, specifically: For sub-datasets that are difficult to process and have complex data types, performance indicator values ​​are assigned On a high computing node; For small-scale sub-datasets, assign performance indicator values The resource is low and idle on the computing nodes to make full use of computing resources.

4. According to the data accurate matching analysis method based on machine learning in claim 1, it is characterized in that: In the process of model training, S300 adopts a distributed training algorithm, that is, a method combining data parallelism and model parallelism, wherein: The data parallelism is to divide the data into multiple small batches and distribute them to different computing nodes for training, and each computing node independently updates the model parameters; The model parallelism is to distribute different parts of the model to different computing nodes for parallel computing.

5. According to claim 4, a data accurate matching analysis method based on machine learning is characterized in that: The data parallelism is computing nodes, the total data set is , split the data set into small batches, each of which has a size of , in data parallelism, each computing node The model parameters are updated based on the small batch data assigned to it, that is, the model parameters are , the loss function is , then in the computing node The parameter update formula on is: ,in, Represents a compute node In the The model parameters at the iteration, is the learning rate, is the gradient of the loss function under the current parameters. After one round of iteration, the parameters of each computing node are synchronized, and the parameter averaging method is adopted, that is: , so that all computing nodes use the synchronized parameters for training in the new round of iterations.

6. According to claim 4, a data accurate matching analysis method based on machine learning is characterized in that: The model parallelism can be divided into parts, respectively, On the computing nodes, for an input sample , the forward propagation process of the model is expressed as: ,in, Represents the model The calculation function of each part, in the back propagation process, each computing node Calculate the gradient of the part it is responsible for , pass the gradient to the previous computing node for further gradient calculation, and get the gradient of the entire model, that is, the loss function is , then the gradient of the model is calculated by the chain rule: , each computing node Responsible for calculation and ,in It is a computing node Responsible model parameters, when the parameters are updated, each computing node The gradient calculated by Update the parameters: .

7. The data accurate matching analysis method based on machine learning according to claim 6 is characterized in that: During the model training process, S300 combines data parallelism and model parallelism, divides the data set into multiple small batches, and distributes them to different computing nodes for data parallel training. At the same time, the model is divided into multiple parts and distributed to different computing nodes for model parallel calculation. In each round of iteration, each computing node first performs data parallel parameter update, and then performs model parallel gradient calculation and parameter update. The parameters of each computing node are merged and updated through the parameter synchronization mechanism, that is, the model parameters are initialized. , divide the data set into small batches and distribute them to different computing nodes. For each round of iteration Perform data parallel update and model parallel gradient calculation, the data parallel update is performed on each computing node Perform data-parallel parameter updates based on the mini-batch data assigned to it: , the model parallel gradient calculation for each computing node Perform model-parallel gradient computation: , the parameter synchronization and update synchronizes the parameters of each computing node: , each computing node is updated according to the synchronized parameters: .

8. The data accurate matching analysis method based on machine learning according to claim 1 is characterized in that: The S400 synchronization mechanism includes a distributed message queue and a distributed file system to ensure data consistency and accuracy among computing nodes, and to set up a synchronization monitoring and error handling mechanism to monitor the synchronization status of each computing node in real time. When node failures and synchronization delays are detected, tasks are immediately reallocated and backup nodes are started. At the same time, data during the synchronization process is checked and verified to ensure data integrity and accuracy.

9. The data accurate matching analysis method based on machine learning according to claim 8 is characterized in that: The distributed message queue and distributed file system in the synchronization mechanism specifically include: The distributed message queue sends the intermediate results to the distributed message queue after each computing node completes part of the task. That is, the message queue adopts message middleware, and each computing node acts as a message producer and sends the results to the central node. At the same time, the central node is set as a message consumer to read the intermediate results of each computing node from the message queue; Distributed file system: At different stages of model training, model parameters and intermediate results are saved in the distributed file system. Each computing node reads and updates the model parameters from the distributed file system at any time. In the result merging stage, the parallel reading and writing functions of the distributed file system are used to merge the final results on each computing node into a unified file.

Images