Large-scale data analysis and prediction system based on machine learning

Abstract

The invention discloses a large-scale data analysis and prediction system based on machine learning, and relates to the technical field of machine learning and big data. The system comprises a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, an analysis prediction output module and a visual display module. According to the method, the optimal BPNN architecture is selected according to the prediction precision returned by the test set, the optimal neural network architecture containing the optimal initialization weight and bias is acquired, the future state is predicted by using the machine learning model, and the guidance path is dynamically adjusted, so that accurate analysis of logistics information is realized, the logistics efficiency is improved, and the logistics operation cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine learning and big data technology, and particularly relates to a large-scale data analysis and prediction system based on machine learning, which is suitable for multiple scenarios such as express delivery, warehousing, cross-border logistics, etc. BACKGROUND

[0002] Traditional logistics management methods have been difficult to meet the efficient, accurate and intelligent needs of modern logistics industry. The rapid development of big data technology has provided strong support for the construction of logistics intelligent management platform. Big data technology can realize the collection, storage, processing and analysis of massive data, and mine valuable information behind the data. However, the current logistics industry lacks sufficient application of logistics goods data, and cannot modify the path in real time combined with weather factors and traffic conditions, and it is also difficult to mine key useful data information from massive logistics information.

[0003] In the invention patent with application number 202411687978.X, a big data prediction system based on machine learning is disclosed. Through the feature engineering module, specific algorithms are used, correlation is considered and a penalty factor is introduced, the relevant feature subset is focused to improve accuracy, and the hidden layer and parameters of the model module can be self-adaptively adjusted. The main use is the self-adaptive adjustment mechanism to make the model better adapt to the characteristics of the data to obtain higher prediction accuracy. However, the invention patent only considers the correlation and introduces the penalty factor, and the main direction is to focus on the relevant feature subset to improve the accuracy, and to realize the self-adaptive adjustment of the hidden layer and parameters of the model module.

[0004] The present application provides a large-scale data analysis and prediction system based on machine learning, which mainly starts from the optimal selection of deep neural network architecture, and realizes the improvement of logistics information analysis efficiency and resource allocation efficiency. SUMMARY

[0005] The present application aims to provide a large-scale data analysis and prediction system based on machine learning, which selects or constructs an effective feature subset as an input variable from preprocessed data, selects the optimal BPNN architecture by using the prediction accuracy returned by the test set, constructs a BPNN population according to the optimal network architecture, and obtains the best neural network architecture containing optimal initialization weights and biases, thereby solving the problems of low logistics efficiency, inadequate analysis of logistics information, unreasonable resource allocation and high operating cost.

[0006] To solve the above technical problems, the present application is realized by the following technical scheme:

[0007] The application discloses a large-scale data analysis and prediction system based on machine learning, which comprises a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, an analysis and prediction output module and a visual display module.

[0008] The data acquisition module is used for collecting a large data sample set of logistics goods, wherein the large data sample set covers historical data at different time nodes in the logistics process of the goods, and the historical data comprises a sending address, a sending contact number, a receiving address, a receiving contact number, a goods name, a quantity, a size, a weight, a material, a use, a transportation mode, a transportation time, a transportation cost and a passing city.

[0009] The data preprocessing module is used for cleaning and normalizing the collected data, eliminating abnormal values and noises, and setting a data quality monitoring module to monitor the quality of the preprocessed data.

[0010] The feature engineering module is used for screening or constructing an effective feature subset from the preprocessed data as input variables.

[0011] The machine learning prediction model module is based on an improved deep neural network architecture, wherein the deep neural network architecture comprises an input layer, a plurality of hidden layers and an output layer, and the deep neural network architecture needs to initialize the weights and biases of the hidden layers and the output layer before training, and the learning rate and the activation function are given.

[0012] The activation function is The number of units of the hidden layer is In the formula, g(z) represents the output value of the activation function, z represents the input value of the activation function, e represents a natural constant, w1 is the number of units of the hidden layer, w is the number of input units, m is the number of output units, and a is a constant between 1 and 10.

[0013] The deep neural network architecture firstly traverses the first hidden layer and the second hidden layer, selects the optimal BPNN architecture by the prediction accuracy returned by the test set, secondly constructs a BPNN population according to the optimal network architecture and optimizes the initialization parameters of the BPNN by using an improved PSO algorithm, and finally obtains the best neural network architecture containing the optimal initialization weights and biases; the application file finds the optimal neural network architecture from the first hidden layer and the second hidden layer neural network, and the neural network is sensitive to the initialized weights and biases, so as to easily fall into a local optimum.

[0014] The model training module is used for training the machine learning prediction model by using the processed feature subset.

[0015] The analysis prediction output module is configured to input the data to be predicted into the trained prediction model to obtain a data prediction analysis result.

[0016] The visual display module is configured to visually display the prediction analysis result of the data.

[0017] As a preferred technical solution, the data collection module acquires the latest big data sample set of the logistics goods in real time through a synchronization mechanism with a data source; the structured data is directly acquired from a business database in a database direct connection manner, that is, the business database such as MySQL / Oracle is connected through JDBC / ODBC; the newly added or changed data is identified based on a time stamp or a version number; the non-structured data is identified by using an OCR technology to identify key fields; if it is a language record, the customer's telephone recording is converted into text by using an ASR; and the identified fields are automatically converted by using a format conversion tool and then input into the data preprocessing module for processing.

[0018] As a preferred technical solution, the data quality monitoring module is configured to monitor the data blood relationship and record the source of each field and the transformation history; and the specific process of the data blood relationship tracking is as follows:

[0019] According to the data collected by the data collection module, a unified metadata center is built, and a goods waybill table and a goods transportation track table are generated;

[0020] Apache Calcite is used to parse the SQL job, and the JSON relationship of the goods waybill table and the goods transportation track table is extracted;

[0021] The OpenLineage SDK is embedded in the Spark pipeline, and the real-time field transformation in the goods waybill table and the goods transportation track table is dynamically marked;

[0022] The Kafka message header in the ELK log is collected, and the upstream and downstream transportation jobs are automatically associated;

[0023] An blood relationship map is created, and a blood relationship health index is constructed, so as to obtain the goods data flow path for visual processing; the data blood relationship is monitored and the source of each field and the transformation history are recorded, the life cycle of the logistics data is tracked, the data error can be identified, the compliance can be ensured, the data architecture can be optimized to reduce redundancy, the dependence relationship of the logistics data can be visually displayed, the data background can be quickly understood by an analyst, and the communication cost is reduced; by tracking the sensitive data flow, the problem source can be quickly located

[0024] When the bloodline map is created, three core nodes and two types of relationships are mainly included, namely, original nodes, cleaning nodes and job nodes, and EEDS relationship and PRODUCES relationship; the original node represents the original waybill data table in the MySQL system; the cleaning node represents the processed Hive data table; the job node records the SparkSTL task and the person in charge; the EEDS relationship describes that the original table is converted by "field mapping" to supply the ETL job; and the PRODUCES relationship represents that the job performs the "remove abnormal coordinates" logic to generate the cleaning table.

[0025] As a preferred technical solution, the data preprocessing module adopts a dynamic anomaly monitoring mechanism, and the specific steps are as follows:

[0026] Step one, the sliding window is used to statistically adjust the abnormal value judgment threshold, to avoid the misjudgment caused by the fixed threshold, and to distinguish the real abnormal data from the environment caused abnormal data in combination with the weather data;

[0027] Step two, a typical transportation mode portrait (such as the reasonable temperature zone of fresh products) is established for each type of goods, and the data deviating from the portrait triggers secondary verification;

[0028] Step three, the goods are clustered (according to weight / volume / category), and the normalization processing is independently performed in each cluster; different normalization coefficients can also be used for night transportation data than for daytime, to reflect the time period difference of logistics efficiency; the normalization parameters are dynamically updated through online learning, to adapt to the data distribution drift caused by business scale expansion.

[0029] As a preferred technical solution, the feature engineering module adopts the combination of XGBoost model and RFECV algorithm for screening; wherein, in the XGBoost model, the objective function is defined as the sum of training loss and regularization, and the specific expression is as follows:

[0030] obj(θ)=L(θ)+Ω(θ);

[0031] In the formula, L(θ) is a training loss term, which is used to measure the prediction accuracy of the model on the training data; and Ω(θ) is a regularization term, which is used to measure the complexity of the model; the objective function for learning is defined based on the objective function, and the specific expression is as follows:

[0032]

[0033] In the formula, I represents the loss function of the prediction result, t represents the number of times of lifting, represents the prediction result of the t-th time of lifting, f i represents the training function involved in the i-th time of lifting, and n represents the total number of times;

[0034] The RFECV algorithm finds the optimal feature set by obtaining the best cross-validation score, and obtains the best feature subset; the RFECV algorithm includes two stages of RFE (recursive feature elimination) and CV (cross-validation); wherein, the RFE stage updates the feature space by repeatedly constructing a model and sequentially deleting the feature with the lowest model evaluation importance in the feature space until the required number of features is obtained; the CV stage is used to evaluate the performance of the trained model on new data, which can reduce the possibility of overfitting to a certain extent.

[0035] As a preferred technical solution, the improved PSO algorithm is used to optimize the initialization weight and bias of the deep neural network, by setting the particle swarm size and search space dimension, setting the number of particle swarm size (such as 50-100), and each particle represents a candidate solution of a set of neural network weights and biases; the search space dimension is determined according to the neural network structure, and the dimension number is the sum of all weight and bias parameters; at the same time, the velocity range of the particle, the learning factor and the inertia weight can also be set; generally, the particle update speed is limited to ±0.5 to prevent parameter oscillation; the learning factor includes individual learning factor and social learning factor, and is usually set to 1.5-2.0; the initial value of the inertia weight is set to 0.9, and the linear decreasing value is 0.4 to balance global exploration and local development.

[0036] Before iteration, the particles need to be encoded and initialized, and the position vector of each particle is directly mapped to the weight matrix and bias vector of the neural network; the speed is initialized, and the initial speed vector is randomly generated, and the speed range is within the preset speed interval; a particle swarm composed of N particles is used to find the optimal solution in D-dimensional space through repeated iteration, and the specific implementation process is as follows:

[0037] Suppose each particle m has a speed of and a position at the tth generation, where t is the current iteration number, D is the dimension of the problem function, and m is a positive integer less than or equal to m; each particle m saves its historical best position P m =[p m1 ,…,p md ,…,p mD ], and the historical best position of the population is represented as B=[b1,…,b d ,…,b D ]; the speed of the mth particle at the t+1th generation is updated by the following formula:

[0038]

[0039] where ω is the inertia weight, c1 and c2 are constants of 2, and two values randomly generated in the range of [0, 1];

[0040] According to the velocity of the particles, the position of the mth particle in the t+1 generation is updated by the following formula:

[0041]

[0042] stop when the maximum number of iterations is reached or the fitness change is less than a threshold value; through the above scheme, PSO can provide the neural network with globally optimized initial parameters, significantly improving the model convergence speed and generalization performance.

[0043] As a preferred technical solution, before the model training module trains the machine learning prediction model using the processed feature subset, a multiple linear regression model is first established, the least squares method is used to analyze the influence of the goods transportation mode and the transportation environment on the logistics efficiency, and the operation speed of the deep neural network is improved by reducing the analysis dimension. The specific multiple linear regression model formula is as follows:

[0044] logistics i =α 12 +α 13 X 1i +α 14 X 2i +…+α 19 X 7i +∑ j β ij Z ij +ε i ;

[0045] In the formula, logistics i is the transportation efficiency of the ith logistics goods, X 1i to X 7i are the sending address, receiving address, quantity, size, weight, material, and transportation mode of the ith goods, respectively; Z ij is the jth control variable of the ith logistics goods, α 12 is the intercept term, α 13 to β ij are the regression coefficients of each variable, and ε i is the system error; the subsequent BP neural network analysis can be facilitated by reducing the number of input layer nodes, reducing the computational complexity, and speeding up the back propagation process; and the analysis dimension (i.e., eliminating insignificant factors in the control variables) is reduced to improve the operation speed of the BP neural network.

[0046] As a preferred technical solution, the specific process of obtaining the best neural network architecture containing the optimal initial weights and biases of the deep neural network architecture is as follows:

[0047] Step S1: read training samples and test samples, set the maximum number of iterations T;

[0048] Step S2: traverse the neural network of one hidden layer, select the network architecture with the highest test accuracy as layer1;

[0049] Step S3: traverse the neural network of two hidden layers, select the network architecture with the highest test accuracy as layer2;

[0050] Step S4: compare the optimal architectures in layer1 and layer2 as bestlayer;

[0051] Step S5: build N neural networks with the same network architecture as bestlayer;

[0052] Step S6: randomly initialize the weights and biases of the N BP neural networks;

[0053] Let the initialization iteration number h=1, the current individual y=1; it also needs to be determined whether h is less than or equal to T, if not, the optimal neural network architecture can be obtained;

[0054] If yes, it is determined whether y is less than or equal to N; if yes, the weights and biases of the yth neural network are updated according to the particle swarm algorithm; the yth neural network is trained and the global optimal network structure and individual historical optimal are obtained;

[0055] Step S7: obtain the optimal neural network architecture.

[0056] As a preferred technical solution, the analysis and prediction output module inputs the multi-source data to be predicted into the trained model, predicts the future state by using the machine learning model, dynamically adjusts the guidance path, uses time series to predict the future cargo volume of each region, directly inputs the prediction result into the path planning algorithm to pre-allocate resources, models the path problem as a Markov decision process, learns the optimal strategy through a reward function, and outputs a real-time dynamic path sequence.

[0057] As a preferred technical solution, the analysis and prediction output module breaks the "data island" by inputting the to-be-predicted multi-source data such as vehicle GPS position, real-time traffic flow, historical cargo volume, weather information and order distribution into the trained model to construct a unified data lake; the future state is predicted by using a machine learning model to dynamically adjust the guidance path; the future cargo volume of each region is predicted by using a time series prediction, the future cargo volume of each region is predicted based on a time series model, and the accuracy is improved based on external factors; the prediction result is directly input into a path planning algorithm to pre-allocate resources; the path problem is modeled as a Markov decision process, and the agent learns the optimal strategy through a reward function; the prediction result is directly input into a path planning algorithm to pre-allocate resources; meanwhile, the path problem is modeled as a Markov decision process, and the optimal strategy is learned through a reward function, and a real-time dynamic path sequence is output.

[0058] As a preferred technical solution, the visualization display module displays the prediction result to the user in the form of intuitive charts, including line charts, column charts, scatter charts, city cargo volume distribution heat maps and cargo logistics transportation guidance paths, and the user can select different chart types for viewing according to needs.

[0059] The present application has the following beneficial effects:

[0060] (1) The present application selects the optimal BPNN architecture according to the prediction accuracy returned by the test set, constructs a BPNN population according to the optimal network architecture, obtains the best neural network architecture containing the optimal initialization weight and bias, predicts the future state by using a machine learning model, dynamically adjusts the guidance path, realizes accurate analysis of logistics information, improves logistics efficiency, optimizes resource allocation, and reduces the operating cost of logistics;

[0061] (2) The present application preprocesses the big data sample set of logistics goods, monitors the data bloodline and records the source and transformation history of each field, traces the life cycle of logistics data, can identify data errors, ensure compliance, optimize the data architecture to reduce redundancy, and visualize the dependency relationship of logistics data to help analysts quickly understand the data background and reduce communication costs; by tracing the flow of sensitive data, the problem source can be quickly located;

[0062] (3) The present application selects or constructs effective features as input variables from the preprocessed data by using the combination of XGBoost model and RFECV algorithm, repeatedly constructs the model, and deletes the features with the lowest model evaluation importance in the feature space to update the feature space, so as to obtain the required number of features, reduce the overfitting degree, and improve the model training accuracy and training efficiency.

[0063] (4) the application breaks the "data island" by inputting the multi-source data to be predicted into the trained model, constructs a unified data lake, uses a machine learning model to predict the future state, dynamically adjusts the guidance path, uses a time series to predict the future cargo volume of each region, predicts the future cargo volume of each region based on a time series model, improves the accuracy in combination with external factors, directly inputs the prediction result into a path planning algorithm to pre-allocate resources, and improves the logistics efficiency.

[0064] Of course, implementing any product of the application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed for the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 A large-scale data analysis and prediction system structure diagram based on machine learning of the application;

[0067] Figure 2 A workflow diagram of a large-scale data analysis and prediction system based on machine learning of the application;

[0068] Figure 3 A specific flowchart for obtaining the best neural network architecture by the deep neural network architecture;

[0069] Figure 4 A specific flowchart for obtaining the best neural network architecture by the deep neural network architecture in the embodiment. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0071] In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0072] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will combine the drawings to further describe the present application. Figures 1-4and examples, the present application is further described in detail. It should be appreciated that the detailed example described herein are merely for explanation and do not limit the present application.

[0073] Please refer to Figures 1-2 The present application is a large-scale data analysis and prediction system based on machine learning, comprising a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, an analysis and prediction output module, and a visualization display module.

[0074] The data acquisition module is used to collect a large data sample set of logistics goods. The large data sample set covers historical data at different time nodes in the logistics process of the goods. The historical data includes the sender's address, the sender's contact information, the recipient's address, the recipient's contact information, the item name, the quantity, the size, the weight, the material, the purpose, the transportation method, the transportation time, the transportation cost, and the passing city.

[0075] The data preprocessing module is used to clean and normalize the collected data, eliminate outliers and noise, and set up a data quality monitoring module to monitor the quality of the preprocessed data.

[0076] The feature engineering module is used to select or construct an effective feature subset from the preprocessed data as an input variable.

[0077] The machine learning prediction model module is based on an improved deep neural network architecture. The deep neural network architecture includes an input layer, multiple hidden layers, and an output layer. Before training the deep neural network architecture, the weights and biases of the hidden layers and the output layer need to be initialized, and the learning rate and the activation function need to be given.

[0078] The activation function is The number of units in the hidden layer is where g(z) represents the output value of the activation function, z represents the input value of the activation function, e represents the natural constant, w1 is the number of units in the hidden layer, w is the number of input units, m is the number of output units, and a is a constant between 1 and 10.

[0079] The deep neural network architecture first traverses the first hidden layer and the second hidden layer, selects the optimal BPNN architecture by the prediction accuracy returned by the test set, then constructs a BPNN population according to the optimal network architecture, and optimizes the initialization parameters of the BPNN using the improved PSO algorithm. Finally, the best neural network architecture containing the optimal initialization weights and biases is obtained. This document will find the optimal neural network architecture from the first hidden layer and the second hidden layer neural network. Since the neural network has great sensitivity to the initialized weights and biases, it is easy to fall into local optimum.

[0080] The model training module is configured to train the machine learning prediction model by using the processed feature subset;

[0081] The analysis prediction output module is configured to input the data to be predicted into the trained prediction model to obtain a data prediction analysis result.

[0082] The visual display module is configured to visually display the prediction analysis result of the data.

[0083] The data acquisition module acquires the latest big data sample set of the logistics goods in real time through a synchronization mechanism with a data source; the structured data is directly acquired from a business database in a database direct connection manner, that is, the business database such as MySQL / Oracle is connected through JDBC / ODBC; the newly added or changed data is identified based on a timestamp or a version number; the non-structured data is identified by using an OCR technology to identify key fields; if it is a language record, the customer's telephone recording is converted into text by using an ASR; and the identified fields are automatically converted by using a format conversion tool and then input into the data preprocessing module for processing.

[0084] The field mapping specification is as shown in the following table:

[0085] Sender information Address standardization (converted to province-city-district three-level code) Recipient information Contact information desensitization processing (keep the first 3 and the last 4 digits) Timestamp Unified conversion to UTC time and record time zone information Material type Mapping to UNSPSC standard classification system Size and weight Unified conversion to international standard units (cm / kg) Special attributes Mark fragile / dangerous goods label (0 / 1 indicates)

[0086] The data quality monitoring module is configured to monitor data blood relationship and record the source of each field and the transformation history; and the specific process of data blood relationship tracking is as follows:

[0087] According to the data collected by the data acquisition module, a unified metadata center is built, and a goods waybill table and a goods transportation track table are generated;

[0088] Apache Calcite is used to parse SQL jobs, and JSON relationships of the goods waybill table and the goods transportation track table are extracted;

[0089] The OpenLineage SDK is embedded in the Spark pipeline to dynamically mark real-time field transformation in the goods waybill table and the goods transportation track table;

[0090] The Kafka message header in the ELK log is collected to automatically associate the upstream and downstream transportation jobs;

[0091] An ancestry map is created, and an ancestry health index is constructed to obtain a goods data flow path for visual processing; the data blood relationship is monitored and the source of each field and the transformation history are recorded, the life cycle of the logistics data is tracked, data errors can be identified, compliance can be ensured, and the data architecture can be optimized to reduce redundancy, and the dependency relationship of the logistics data can be visually displayed to help an analyst quickly understand the data background and reduce communication costs; by tracking the flow of sensitive data, the source of a problem can be quickly located

[0092] When creating the bloodline map, three core nodes and two types of relationships are mainly included; namely, the original node, the cleaning node and the job node, and the EEDS relationship and the PRODUCES relationship. The original node represents the original waybill data table in the MySQL system. The cleaning node represents the processed Hive data table. The job node records the Spark STL task and the person in charge. The EEDS relationship describes the conversion of the original table through "field mapping" to supply the ETL job. The PRODUCES relationship represents the execution of the "remove abnormal coordinates" logic to generate the cleaning table. The specific process of creating the bloodline map is as follows:

[0093] 1. Define the metadata model: clearly define the management objects (such as database tables, fields) and their attributes (name, type, source, etc.).

[0094] 2. Collect metadata: extract data from databases, data lakes, and other sources, and use tools to automate collection to improve efficiency.

[0095] 3. Establish a blood relationship model: define upstream and downstream relationships through graphical methods, such as using React+Canvas technology to implement interactive maps.

[0096] 4. Track data flow: parse SQL statements or dynamically generate blood links through tools to form a visual map; when the waybill status field is not updated for more than 2 hours, automatically trigger root cause analysis.

[0097] The data preprocessing module adopts a dynamic anomaly monitoring mechanism, and the specific steps are as follows:

[0098] Step one, use sliding window statistics (such as EWMA exponential weighted moving average) to dynamically adjust the threshold for judging abnormal values, avoid false positives caused by fixed thresholds, and distinguish between real abnormal data and environment-induced abnormal data by combining weather data;

[0099] Step two, establish a typical transportation mode portrait for each type of goods (such as the reasonable temperature zone for fresh products), and trigger secondary verification for data that deviates from the portrait;

[0100] Step three, cluster the goods (by weight / volume / category), and perform normalization processing independently within each cluster; different normalization coefficients can also be used for night transportation data than for daytime, reflecting the time period differences in logistics efficiency; online learning is used to dynamically update the normalization parameters to adapt to the data distribution drift caused by business expansion.

[0101] The feature engineering module uses a combination of XGBoost models and RFECV algorithms for screening; in the XGBoost model, the objective function is defined as the sum of the training loss and the regularization, and the specific expression is:

[0102] obj(θ) = L(θ) + Ω(θ);

[0103] In the formula, L(θ) is a training loss term for measuring the prediction accuracy of the model on the training data; Ω(θ) is a regularization term for measuring the complexity of the model; the objective function for learning is defined based on the objective function, and the specific expression is as follows:

[0104]

[0105] In the formula, I represents the loss function of the prediction result, t represents the number of times of lifting, represents the prediction result of the tth time of lifting, f i represents the training function involved in the ith time of lifting, and n represents the total number of times;

[0106] The RFECV algorithm finds the optimal feature set by obtaining the best cross-validation score, and obtains the best feature subset; the RFECV algorithm includes two stages of RFE (recursive feature elimination) and CV (cross-validation); wherein, the RFE stage repeatedly constructs a model, and deletes the feature with the lowest model evaluation importance in the feature space to update the feature space, until the required number of features is obtained; the CV stage is used to evaluate the performance of the trained model on new data, which can reduce the possibility of overfitting to a certain extent.

[0107] The improved PSO algorithm is used to optimize the initialization weights and biases of the deep neural network, by setting the particle swarm size and search space dimension, setting the number of particle swarm size (such as 50-100), and each particle represents a candidate solution of a set of neural network weights and biases; the search space dimension is determined according to the neural network structure, and the dimension number is the sum of all weight and bias parameters; at the same time, the velocity range of the particle, the learning factor and the inertia weight can also be set; generally, the particle update velocity is limited to ±0.5 to prevent parameter oscillation; the learning factor includes individual learning factor and social learning factor, and is usually set to 1.5-2.0; the initial value of the inertia weight is set to 0.9, and the linear decreasing value is 0.4 to balance global exploration and local development;

[0108] Before iteration, the particles need to be encoded and initialized, and the position vector of each particle is directly mapped to the weight matrix and bias vector of the neural network; the speed is initialized, and the initial speed vector is randomly generated, and the speed range is within the preset speed interval; using a particle swarm consisting of N particles, the optimal solution is found in D-dimensional space through repeated iteration, and the specific implementation process is as follows:

[0109] Let each particle m have a speed of and position In the formula, t is the current iteration number, D is the dimension of the problem function, m is a positive integer less than or equal to m; each particle m saves its historical best position P m = [p m1 ,…,p md ,…,p mD ,…,p d ,…,p D ] is the historical best position in the population, which is represented as B = [b1, …, b i ,…,b 12 ]; the velocity of the mth particle in the t+1 generation is updated by the following formula:

[0110]

[0111] In the formula, ω is the inertia weight, c1 and c2 are constants of 2, and are two values randomly generated in the range of [0, 1];

[0112] According to the velocity of the particle, the position of the mth particle in the t+1 generation is updated by the following formula:

[0113]

[0114] Stop when the maximum number of iterations is reached or the fitness change is less than the threshold value; through the above scheme, PSO can provide the initial parameters of global optimization for the neural network, significantly improving the model convergence speed and generalization performance.

[0115] Before the model training module trains the machine learning prediction model using the processed feature subset, a multiple linear regression model is established, the least squares method is used to analyze the influence of the cargo transportation mode and the transportation environment on the logistics efficiency, and the operation speed of the deep neural network is improved by reducing the analysis dimension. The specific multiple linear regression model formula is as follows:

[0116] logistics i = α 12 + α 13 X 1i + α 14 X 2i + … + α 19 X 7i + ∑ j β ij Z ij + ε i ;

[0117] In the formula, logistics i is the transportation efficiency of the ith logistics cargo, X 1i to X 7i are the sending address, receiving address, quantity, size, weight, material, and transportation mode of the ith cargo, respectively; Zij is the jth control variable of the ith logistics cargo, a 12 is the intercept term, a 13 to b ij are the regression coefficients of the respective variables, and e i is the system error; can provide convenience for subsequent BP neural network analysis, reduce the number of input layer nodes, reduce the calculation complexity, speed up the back propagation process; at the same time, reduce the analysis dimension (i.e. eliminate the insignificant factors in the control variable) to improve the operation speed of BP neural network.

[0118] Referring to Figure 3 As shown in the figure, the specific process of obtaining the best neural network architecture including the optimal initialization weight and bias of the deep neural network architecture is as follows:

[0119] Step S1: read the training samples and test samples, and set the maximum iteration number T;

[0120] Step S2: traverse the neural network of one hidden layer, select the network architecture with the highest test accuracy as layer1;

[0121] Step S3: traverse the neural network of two hidden layers, select the network architecture with the highest test accuracy as layer2;

[0122] Step S4: compare the optimal architectures in layer1 and layer2 as bestlayer;

[0123] Step S5: construct N neural networks with the same network architecture of bestlayer;

[0124] Step S6: randomly initialize the weights and biases of the N BP neural networks;

[0125] Referring to Figure 4 As shown in the figure, set the initialization iteration number h=1, and the current individual y=1; it also needs to be determined whether h is less than or equal to T, if not, the optimal neural network architecture can be obtained;

[0126] If yes, it is determined whether y is less than or equal to N; if yes, the weights and biases of the yth neural network are updated according to the particle swarm optimization algorithm; the yth neural network is trained, and the global optimal network structure and individual historical optimum are obtained.

[0127] Step S7: obtain the optimal neural network architecture.

[0128] The analysis and prediction output module breaks the "data island" by inputting the multi-source data to be predicted, such as vehicle GPS position, real-time traffic flow, historical cargo volume, weather information and order distribution, to the trained model, and building a unified data lake;

[0129] Detailed multi-source data features include:

[0130] spatiotemporal features: such as period congestion coefficient, grid area cargo density, road network topology, and spatial autocorrelation preserved through GeoHash encoding;

[0131] dynamic features: real-time traffic events (accidents or construction), weather anomalies (rainfall leading to speed reduction), and demand fluctuations (sudden orders), which need to be captured through sliding window statistics (such as 1-hour mean) to capture short-term trends;

[0132] Predict future states using machine learning models to dynamically adjust the guidance path; predict future cargo volumes in each region using time series prediction, based on time series models (such as LSTM) to predict future cargo volumes in each region, combined with external factors (such as holiday economic index) to improve accuracy; the prediction results are directly input into the path planning algorithm for pre-allocation of resources; model the path problem as a Markov decision process (MDP), and the agent learns the optimal strategy through a reward function (such as reducing delivery time and reducing fuel consumption); for example, combine graph neural networks (GNN) to encode node relationships and output dynamic path sequences.

[0133] Specifically, time series modeling is as follows:

[0134] Input data engineering: build a three-dimensional input tensor [sample number, time step, feature dimension], which specifically includes: historical cargo volume sequence, spatiotemporal features, and external factors; the historical cargo volume sequence is the regional cargo volume data for the past 7-30 days, which is generated through a sliding window; the spatiotemporal features are grid coordinates GeoHash encoding and period features (morning and evening peak indicators); the external factors are weather indexes (precipitation probability / wind force) and economic indicators (such as Jingdong 618 activity intensity); the network architecture uses a Seq2Seq structure, and both the encoder and the decoder contain: double-layer LSTM units (hidden layer 128 dimensions) to control the historical information decay rate through the forget gate; the attention mechanism dynamically weights the influence of different time steps to improve the ability to capture sudden demand; the output adaptation result is as follows: a predicted future 6-hour cargo volume distribution heat map with a grid accuracy of 500m x 500m and an error rate of <8%.

[0135] The prediction results are directly input into the path planning algorithm for pre-allocation of resources; at the same time, the path problem is modeled as a Markov decision process, and the optimal strategy is learned through a reward function, outputting real-time dynamic path sequences.

[0136] The visualization module displays the prediction results to the user in the form of intuitive charts, including line charts, bar charts, scatter plots, and city cargo volume distribution heat maps and cargo logistics transportation guidance paths, allowing users to select different chart types for viewing according to their needs.

[0137] Specifically, the Apache Spark is used to generate time series data (such as the goods volume per hour), spatial data (grid coordinates and goods volume values), and path data (longitude and latitude sequences) by real-time aggregation of prediction results, and the output is in a standardized JSON format; the front-end rendering is realized based on ECharts, and the chart types are adapted to different data types, such as a line chart (with a confidence interval) reflecting a time series trend, a stacked column chart (layered according to goods types) reflecting regional comparison, a heat map superimposed on a Gaode map API reflecting spatial distribution, and a Leaflet dynamic drawing of a weighted directed graph reflecting path planning. The city is divided into 1km*1km grids by dynamic rendering of the heat map, the Kriging interpolation algorithm is used to smooth the discrete goods volume data, and the HSL gradient (blue->yellow->red representing low->high goods volume) is used for color mapping.

[0138] Meanwhile, the current optimal path is highlighted on the map, and the semi-transparent display is used for other feasible paths; and the risk is marked on each path, which facilitates the background observation and real-time path adjustment.

[0139] It should be noted that the units included in the above system embodiments are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for easy differentiation, and do not limit the protection scope of the present application.

[0140] In addition, those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by programs instructing related hardware, and the corresponding programs can be stored in a computer-readable storage medium.

[0141] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details, nor limit the application to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited only by the claims and their entire scope and equivalents.

Claims

1. A large-scale data analysis and prediction system based on machine learning, comprising a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, an analysis and prediction output module, and a visualization module, characterized in that: The data acquisition module is used to collect a large data sample set of logistics goods; the large data sample set covers historical data at different time points in the logistics process of goods. The data preprocessing module is used to clean and normalize the collected data, eliminate outliers and noise, and set up a data quality monitoring module to monitor the quality of the preprocessed data. The feature engineering module is used to filter or construct an effective subset of features from the preprocessed data as input variables; The machine learning prediction model module is based on an improved deep neural network architecture; the deep neural network architecture includes an input layer, multiple hidden layers, and an output layer; before training the deep neural network architecture, the weights and biases of the hidden layers and the output layer need to be initialized, and the learning rate and activation function need to be given. The deep neural network architecture first traverses the first and second hidden layers and selects the optimal BPNN architecture based on the prediction accuracy returned by the test set; then, it constructs a BPNN population based on the optimal network architecture and optimizes the initialization parameters of the BPNN using an improved PSO algorithm; finally, it obtains the optimal neural network architecture containing the optimal initial weights and biases. The model training module is used to train the machine learning prediction model using the processed feature subset; The analysis and prediction output module is used to input the data to be predicted into the trained prediction model to obtain the data prediction and analysis results. The visualization module is used to visualize the predictive analysis results of the data.

2. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The data acquisition module acquires the latest big data sample set of logistics goods in real time through a synchronization mechanism with the data source; it directly obtains the business database for structured data by direct database connection and identifies new or changed data based on timestamps or version numbers; it uses OCR technology to identify key fields for unstructured data; and it automatically calls a format conversion tool to convert the identified fields before incorporating them into the data preprocessing module for further processing.

3. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The data quality monitoring module is used to monitor data lineage and record the source and transformation history of each field; the specific process of data lineage tracking is as follows: Based on the data collected by the data acquisition module, a unified metadata center is built and a cargo waybill table and a cargo transportation trajectory table are generated. Parse the SQL task to extract the JSON relationship between the waybill table and the cargo transportation trajectory table; Dynamically mark real-time field changes in the waybill table and cargo transportation trajectory table; Collect Kafka message headers from ELK logs and automatically associate them with upstream and downstream transportation jobs; Create a lineage map and construct a lineage health index to obtain cargo data flow paths for visualization processing.

4. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The data preprocessing module employs a dynamic anomaly monitoring mechanism, and the specific steps are as follows: Step 1: Use a sliding window to dynamically adjust the outlier threshold and combine it with weather data to distinguish between real outlier data and environmentally caused outlier data; Step 2: Create a typical transportation mode profile for each type of goods; data deviating from the profile will trigger secondary verification. Step 3: Cluster the goods and perform normalization independently within each cluster.

5. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The feature engineering module employs a combination of the XGBoost model and the RFECV algorithm for screening; wherein, the objective function of the XGBoost model is defined as the sum of the training loss and the regularization, specifically expressed as: obj(θ) = L(θ) + Ω(θ); In the formula, L(θ) is the training loss term, used to measure the prediction accuracy of the model on the training data; Ω(θ) is the regularization term, used to measure the complexity of the model; the objective function used for learning is defined based on the objective function, and its specific expression is as follows: In the formula, I represents the loss function of the prediction result, and t represents the number of boosts. f represents the prediction result of the t-th improvement. i Let represent the training function involved in the i-th boost, and n represent the total number of boosts; The RFECV algorithm finds the optimal feature set by obtaining the best cross-validation score, thus obtaining the best feature subset. The RFECV algorithm consists of two stages: RFE and CV. In the RFE stage, the model is repeatedly built, and the feature space is updated by successively deleting the features with the lowest evaluation importance in the feature space until the required number of features is obtained. The CV stage is used to evaluate the performance of the trained model on new data.

6. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The improved PSO algorithm is used to optimize the initial weights and biases of a deep neural network. By setting the particle swarm size and the search space dimension, a particle swarm consisting of N particles is used to find the optimal solution in the D-dimensional space through repeated iterations. The specific implementation process is as follows: Let each particle m have a velocity of in the t-th generation. and location In the formula, t is the current iteration number, D is the dimension of the problem function, and m is a positive integer less than or equal to m; each particle m stores its historical best position P. m =[p m1 ,…,p md ,…,p mD The historical best position in the population is represented as B = [b1, ..., b]. d ,…,b D The velocity of the m-th particle in generation t+1 is updated using the following formula: In the formula, ω is the inertia weight, and c1 and c2 are constants of 2. and Two values ​​are randomly generated within the range [0,1].

7. The large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, Before training the machine learning prediction model using the processed feature subset, the model training module first establishes a multiple linear regression model. It then uses the least squares method to analyze the impact of cargo transportation methods and the transportation environment on logistics efficiency. This reduces the analytical dimensions to improve the computational speed of the deep neural network. The specific formula for the multiple linear regression model is as follows: logistics i =a 12 +a 13 X 1i +a 14 X 2i +…+a 19 X 7i +∑ j b ij Z ij +e i ; In the formula, logistics i Let x be the transportation efficiency of the i-th item in the logistics. 1i To x 7i Z represents the sender's address, recipient's address, quantity, dimensions, weight, material, and mode of transport for the i-th item; ij Let α be the j-th control variable for the i-th item in the logistics shipment. 12 For the intercept term, α 13 To β ij These are the regression coefficients for each variable, ε. i This represents systematic error.

8. A large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The specific process for obtaining the optimal neural network architecture, which includes the optimal initial weights and biases, is as follows: Step S1: Read the training samples and test samples, and set the maximum number of iterations T; Step S2: Traverse the neural network with one hidden layer and select the network architecture with the highest test accuracy as layer 1; Step S3: Traverse the neural network with two hidden layers and select the network architecture with the highest test accuracy as layer 2; Step S4: Compare the optimal architecture between layer 1 and layer 2, and select the best layer; Step S5: Construct N neural networks with the same network architecture for the best layer; Step S6: Randomly initialize the weights and biases of N BP neural networks; Step S7: Obtain the optimal neural network architecture.

9. A large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The analysis and prediction output module inputs multi-source data to be predicted into the trained model, uses a machine learning model to predict future states, and dynamically adjusts the guidance path; it uses time series to predict the future cargo volume of each region, and directly inputs the prediction results into the path planning algorithm to pre-allocate resources; at the same time, it models the path problem as a Markov decision process, learns the optimal strategy through a reward function, and outputs a real-time dynamic path sequence.

10. A large-scale data analysis and prediction system based on machine learning according to claim 1, characterized in that, The visualization module presents the prediction results to users in intuitive chart formats, including line charts, bar charts, scatter plots, heat maps of urban freight volume distribution, and freight logistics transportation guidance routes. Users can choose different chart types to view according to their needs.

Images