Method, device and equipment for screening data items of railway freight data and medium

Abstract

The invention discloses a data item screening method, device and equipment for railway freight data and a medium, and aims to solve the problem that railway freight transportation data is numerous and redundant and optimize transportation management. The method comprises the following steps: acquiring original data of railway cargo transportation from a plurality of data sources and preprocessing the original data, then constructing an initial data set and calculating a value evaluation score of the initial data set; generating a new data set by dynamically adjusting data items in the current data set, and calculating a score; obtaining a more valuable data set by comparing the evaluation scores of the new and old data sets; and repeatedly generating a new data set and calculating the score to obtain a data set with the highest score, and the data set is the most valuable data set. According to the method, valuable freight data records can be effectively searched from mass data, so that the utilization rate of the valuable data is improved, the transportation cost is reduced, the overall efficiency of transportation management is improved, and the high requirements of the modern railway freight industry on big data processing and application are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of railway data processing, and in particular relates to a method, device, equipment and medium for screening data items of railway freight data. Background Art

[0002] With the growth of railway freight transportation demand, rapid technological progress and the improvement of informatization, the data generated in the field of railway freight has shown explosive growth, and its potential economic and social value has gradually become prominent. The railway digital economy is becoming an important engine to promote railway business innovation, improve railway transportation productivity, and optimize railway resource allocation. Railway freight data includes but is not limited to cargo type, transportation time, transportation route, station information, etc. Traditional railway freight transportation data is usually collected and summarized according to stations, railway bureaus or group companies, and then stored uniformly according to business categories and time tags. These data are usually large and complex, containing a large number of data items. Among these data items, some have significant analytical value, while others may have limited contribution to the overall railway freight transportation management. Therefore, how to effectively screen out data items with high value and generate and optimize data sets among these data items has become a key issue.

[0003] Current technical solutions usually include screening methods based on certain static rules or simple algorithms, but these methods are often difficult to adapt to the dynamic changes of data and may not fully explore the complex relationships between data items. A more advanced and flexible technology is needed for efficient data screening and evaluation to ensure the high value and applicability of the final data set. Summary of the invention

[0004] The present invention provides a data item screening method, device, equipment and medium for railway freight data, which can effectively search for valuable freight data records in massive data to improve the utilization rate of valuable data, thereby reducing transportation costs, improving the overall efficiency of transportation management, and meeting the high requirements of the modern railway freight industry for big data processing and application.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A method for screening data items of railway freight data, comprising:

[0007] Step 1, obtaining raw data of railway freight transportation from multiple data sources and performing data preprocessing; wherein the raw data includes M data items;

[0008] Step 2: randomly select N non-repeating data items from the M data items of the original data as the initial candidate freight data set X1; store the initial candidate freight data set X1 as the current value data set Y, and store X1 in the memory set U; where N is the target number of data items to be screened, 1 <N<M;

[0009] Step 3: Calculate the value assessment score f(X1) of the data set X1; where f(X1) i ) is the data set value assessment function;

[0010] Step 4: Use X i+1 =D(X i ) Generate a new dataset X i+1 , and the new dataset X i+1 Stored in the memory set U; where the function D(X i ) represents the data set X i The data items in the current data set X are adjusted. i Select n data items and replace them with new data items, and 1≤n <N,n<M-N;

[0011] Step 5: Calculate the data set X i+1 The value assessment score f(X i+1 );

[0012] Step 6, compare f(X i+1 ) and f(X i )Numerical size:

[0013] If f(X i+1 )>f(X i ), represents the data set X i+1 If the value is higher, go directly to step 7;

[0014] If f(X i+1 )≤f(X i ), indicating that the current modification operation is invalid, and then further determine whether the memory set U has stored the data set of all N data item combinations. If so, the screening is terminated and the current value data set Y is returned as the final screened data set. Otherwise, return to step 4 and perform the modification on the data set X. i to readjust;

[0015] Step 7: Update the current value dataset Y to dataset X i+1 ;

[0016] Step 8: If the termination condition is not met, update i=i+1 and go to step 4 to enter the next round; if the termination condition is met, return the current value data set Y as the final screened data set.

[0017] Furthermore, the type of the data item belongs to a customer information type, a waybill data type or a cargo invoice information type, and each type includes one or more data items.

[0018] Furthermore, the data preprocessing includes cleaning, conversion, normalization, missing value processing and outlier processing.

[0019] Furthermore, the dataset value evaluation function f(X i ), using a regression model based on historical data and expert scores to obtain an invisible representation; wherein the regression model is obtained based on historical data and expert scores, including the following steps:

[0020] (1) Data collection: Collect historical data item records and expert evaluation scores of the data set;

[0021] (2) Data cleaning: processing missing values ​​and outliers, and standardizing the data;

[0022] (3) Constructing training and test sets: Split the data set into training and test sets in proportion, define the data item set as the model input, and define the expert score value as the target variable of the corresponding model output;

[0023] (4) Model training and validation: Select a regression model and model evaluation indicators, use the training set data to train the model, optimize the model parameters, and evaluate the performance of the model on the test set.

[0024] Furthermore, in step 4, two methods, local search and dynamic step size adjustment, are used to search the dataset X i Adjust the data items in:

[0025] (1) Local search means limiting the data set X i The range of changes in the data items involved in the adjustment is specified to select data set X according to the order of data item numbers. i The data items involved in the adjustment;

[0026] (2) Dynamically adjusting the step size means dynamically adjusting the number of replacement data items according to the magnitude of the change in the data set score.

[0027] Furthermore, the method for dynamically adjusting the number n of data items involved in the adjustment is:

[0028] (1) Get the current dataset X i The score f(X i ) and the new dataset X i+1 The score f(X i+1 ), calculate the score change θ:

[0029]

[0030] (2) If the score change θ is greater than the preset replacement threshold upper limit, the number of adjustments is increased based on the current number of adjustments for the next iteration;

[0031] (3) If the score change θ is less than the preset replacement threshold, the number of adjustments is reduced based on the current number of adjustments for the next iteration;

[0032] (4) If the score change θ is within the preset replacement threshold, the current adjustment number will remain unchanged in the next iteration.

[0033] Furthermore, the termination condition of step 8 is one of the following:

[0034] (1) The memory set U already stores the data set of all N data item combinations, that is,

[0035] (2) The change rate of the value assessment scores of two consecutive data sets is within the preset termination threshold ε, that is:

[0036]

[0037] A data item screening device for railway freight data, comprising:

[0038] The data preprocessing module is used to: perform data preprocessing on the original data of railway freight transportation obtained from multiple data sources; wherein the original data includes M data items;

[0039] The initialization module is used to randomly extract N non-repeating data items from the M data items of the original data as the initial candidate freight data set X1; store the initial candidate freight data set X1 as the current value data set Y, and store X1 in the memory set U; where N is the target number of data items to be screened, 1 <N<M;

[0040] Value evaluation module, used to calculate any data set X i The value assessment score f(X i ), where f(X i ) is the data set value assessment function;

[0041] Dataset adjustment module, used to: i+1 =D(X i ) Generate a new dataset X i+1 , and the new dataset X i+1 Stored in the memory set U; where the function D(X i ) represents the data set X i The data items in the current data set X are adjusted. iSelect n data items and replace them with new data items, and 1≤n <N,n<M-N;

[0042] Numeric comparison module, used to compare f(X i+1 ) and f(X i )Numerical value;

[0043] The value data set update module is used to: determine f(X i+1 )>f(X i ), update the current value dataset Y to dataset X i+1 ;

[0044] The value data set output module is used to: when the termination condition is reached, output the current value data set Y as the final screened data set.

[0045] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor implements any one of the above-mentioned methods for screening data items of railway freight data.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-mentioned methods for screening data items of railway freight data.

[0047] The method, device, equipment and medium for selecting data items of railway freight data provided by the present invention have the following beneficial effects:

[0048] (1) Improve data utilization: By comprehensively collecting data from multiple data sources and screening out high-value data items, the present invention can extract the most informative parts from a large amount of data and optimize data utilization efficiency. This screening mechanism ensures the refinement and efficiency of the data set, making subsequent analysis and decision-making more targeted.

[0049] (2) Flexible iterative optimization mechanism: The present invention adopts an iterative optimization mechanism based on value assessment. By continuously adjusting and evaluating the data set, the value of the screening data set can be dynamically optimized. This mechanism can not only gradually improve the overall value of the data set, but also adapt to different business needs and actual situations, providing a more flexible and efficient optimization solution.

[0050] (3) Scientific regression model evaluation: By constructing a regression model to evaluate the value of a data set, the present invention provides a scientific and systematic evaluation method. Combining historical data and expert scoring, the value of each data set can be objectively measured, ensuring the accuracy and objectivity of the evaluation process, and providing a solid theoretical basis for data screening.

[0051] (4) Reduce transportation costs: By optimizing the data screening process, the present invention can accurately identify and evaluate high-value data sets, thereby providing a more accurate basis for transportation decision-making. By optimizing the data processing process, transportation costs can be effectively reduced, for example, by optimizing route planning, reducing unnecessary transportation links, and other means to achieve cost savings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the method described in the embodiment of the present application. DETAILED DESCRIPTION

[0053] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0054] The embodiment of the present invention provides a method for screening data items of railway freight data, referring to Figure 1 As shown, the following steps are included:

[0055] Step 1: Obtain the original data of railway freight transportation from multiple data sources and perform data preprocessing.

[0056] The data sources include China Railway 95306 website, freight transportation information integration platform, etc. The data items in the original data include:

[0057] (1) Customer information (such as customer ID, customer type, contact information);

[0058] (2) Waybill data (such as waybill number, place of shipment, destination, and mode of transportation);

[0059] (3) Cargo invoice information (such as cargo type, quantity, weight, and freight).

[0060] The data preprocessing includes operations such as cleaning, conversion, normalization, missing value processing, and outlier processing of the original data.

[0061] (1) Data cleaning: delete duplicate records.

[0062] (2) Data conversion: Convert data from different formats to a unified format, such as unifying the date format to YYYY-MM-DD.

[0063] (3) Data normalization: Scale numerical features (such as weight and shipping cost) to the range of 0 to 1 for comparison.

[0064] (4) Missing value processing: Use mean imputation or interpolation to fill missing values, or delete records with a large number of missing values.

[0065] (5) Outlier handling: Identify and handle outliers, for example, using the Z-score method or the IQR method.

[0066] Step 2: Randomly select N non-repeating data items from the M data items of the original data as the initial alternative freight dataset X1; store the initial alternative freight dataset X1 as the current value dataset Y, and store X1 in the memory set U.

[0067] Among them, N is the target number of data items to be screened, and 1 < N < M. The present invention assumes that N non-repeating and high-value data items are to be screened out from the railway freight data with a total number of M data items after preprocessing. According to the combination formula, there are different combinations of selecting N non-repeating data items from M data items. Number the dataset composed of N data items as X i , i = 1, 2, 3, … I, Initially set i = 1.

[0068] In addition, the memory set U is used to store the evaluated datasets to ensure that the datasets are not repeated in the update stage of Step 4.

[0069] For example: Screen out N = 5 non-repeating and high-value data items from the preprocessed railway freight data with a total number of M = 10 data items. The data items include the shipper name, commodity name, quantity, weight, shipping place, destination, waybill number, goods packaging, transportation cost, and transportation mode, which are represented by x1, x2, x3, …, x 10 respectively. The dataset composed of 5 data items is numbered as X i and i = 1, 2, 3, … I, Suppose the initial alternative freight dataset X1 = {x1, x2, x6, x8, x9}.

[0070] Step 3: Calculate the value evaluation score f(X1) of the dataset X1. This score is used to measure the overall value of the dataset.

[0071] Among them, f(X i ) is the dataset value evaluation function, which is implicitly represented by a machine learning model and obtained based on historical data and expert scoring. It includes the following steps:

[0072] (1) Data collection. Collect historical data item records of different freight operations, and collect the evaluation scores of experts for each dataset as the target variable of the regression task.

[0073] (2) Data cleaning. This includes filling or deleting missing data, identifying and handling outliers in the data, and normalizing or standardizing the features to ensure that all features are on the same scale.

[0074] (3) Construct training and test sets. First, define the target variable as the numerical value of the expert score, which is 0-100. Second, select and construct features related to the prediction target, which may include numerical values ​​and category features in historical records. Then, randomly split the data set into training and test sets according to a certain ratio. The common ratio is 80% training set and 20% test set.

[0075] (4) Model training and validation. Select random forest regression or support vector regression model and use the training set data to train the model, optimize the model parameters to minimize the evaluation indicators. The evaluation indicators are mean square error (MSE), root mean square error (RMSE), determination coefficient (R 2 ) and other common regression indicators. During model training, grid search or random search can be used to adjust the model's hyperparameters to find the best parameter combination.

[0076] For example, the score of the dataset X1 predicted by the pre-trained regression model is f(X1)=70.

[0077] Step 4: Use X i+1 =D(X i ) Generate a new dataset X i+1 , and the new dataset X i+1 Stored in memory set U.

[0078] Among them, the function D(X i ) represents the data set X i The data items in the current data set X are adjusted. i Select n data items and replace them with new data items, and 1≤n <N,n<M-N。

[0079] This embodiment uses two methods, local search and dynamic adjustment of the number of data items to be replaced in the next iteration, to adjust the data set, so as to optimize the iteration process and improve efficiency.

[0080] (1) Local search.

[0081] Local search limits the range of data items in each iteration, and stipulates that from the current data set X i The data items in the table are replaced starting with those with smaller subscript numbers.

[0082] (2) Dynamically adjust the number of data items to be replaced.

[0083] Dynamically adjust the step size to dynamically adjust the number of data items to be replaced in the next iteration according to the magnitude of the change in the score of the data set during the iteration. Dynamically adjust the number of data items to be replaced, which is used in the next iteration and will use the value evaluation score f(X i+1 ), therefore, the dynamic adjustment process can be performed after step 5.

[0084] The replacement threshold is set to θ in advance. change =5%~6%. In each iteration, record the current data set X i The score f(X i ). Get a new dataset X i+1 The score f(X i+1 ), and calculate the score change θ:

[0085]

[0086] Adjust the rules for the number of data item replacements n:

[0087] 1) The score changes greatly: when θ>θ change , indicating that the score of the new dataset has been greatly improved. At this time, you can consider increasing the number of data items to be replaced in the next iteration, for example, updating n = n + 1, to explore the dataset space faster.

[0088] 2) Small score change: When θ<θ change , indicating that the score improvement is small. In this case, the number of data items to be replaced in the next iteration should be reduced, for example, updating n = n-1 to search the current area more finely.

[0089] 3) Maintaining the step size: If θ is within the set range, the number of replacements for the current data item can be kept unchanged in the next iteration, that is, n=n.

[0090] This embodiment assumes that the number of initial data item replacements is n=2, then selects the two data items x1 and x2 with the smallest subscript numbers from the current data set X1={x1, x2, x6, x8, x9}, and replaces them with the newly selected data items x4 and x5, to obtain a new data set X2={x4, x5, x6, x8, x9}.

[0091] Due to the variation in scores According to the adjustment rules, 2+1=3 data items should be adjusted and replaced in X3=D(X2).

[0092] Step 5: Calculate the data set X i+1 The value assessment score f(X i+1 ).

[0093] Step 6, compare f(X i+1 ) and f(X i )Numerical size:

[0094] If f(X i+1 )>f(X i ), represents the data set X i+1 If the value is higher, go directly to step 7;

[0095] If f(Xi+1 )≤f(X i ), indicating that the current modification operation is invalid, and then further determine whether the memory set U has stored the data set of all N data item combinations. If so, the screening is terminated and the current value data set Y is returned as the final screened data set. Otherwise, return to step 4 to modify the data set X. i Make readjustments.

[0096] For example, the above calculation yields f(X2)=75, f(X1)=70, f(X2)>f(X1), so the new data set X2 is more valuable, and we go directly to step 7.

[0097] Step 7: Update the current value dataset Y to dataset X i+1 , that is, update Y to the current data set X with higher value i+1 , prepare for the next round of iteration.

[0098] Step 8: If the termination condition is not met, update i=i+1 and go to step 4 to enter the next round; if the termination condition is met, return the current value data set Y as the final screened data set.

[0099] The termination condition is one of the following:

[0100] (1) The memory set U already stores the data set of all N data item combinations, that is,

[0101] (2) The change rate of the value assessment scores of two consecutive data sets is within the preset termination threshold i, that is:

[0102]

[0103] In this embodiment, the termination threshold ε=2%. At this time, i=2, so At the same time, only one change rate calculation is currently performed, that is, Therefore, the termination condition is not met, and the process goes to step 4 to enter the next round.

[0104] The above are only exemplary embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for screening data items of railway freight data, characterized in that: include: Step 1, obtaining raw data of railway freight transportation from multiple data sources and performing data preprocessing; wherein the raw data includes M data items; Step 2: randomly select N non-repeating data items from the M data items of the original data as the initial candidate freight data set X1; store the initial candidate freight data set X1 as the current value data set Y, and store X1 in the memory set U; where N is the target number of data items to be screened, 1 <N<M; Step 3: Calculate the value assessment score f(X1) of the data set X1; where f(X1) i ) is the data set value assessment function; Step 4: Use X i+1 =D(X i ) Generate a new dataset X i+1 , and the new dataset X i+1 Stored in the memory set U; where the function D(X i ) represents the data set X i The data items in the current data set X are adjusted. i Select n data items and replace them with new data items, and 1≤n <N,n<M-N; Step 5: Calculate the data set X i+1 The value assessment score f(X i+1 ); Step 6, compare f(X i+1 ) and f(X i )Numerical size: If f(X i+1 )>f(X i ), represents the data set X i+1 If the value is higher, go directly to step 7; If f(X i+1 )≤f(X i ), indicating that the current modification operation is invalid, and then further determine whether the memory set U has stored the data set of all N data item combinations. If so, the screening is terminated and the current value data set Y is returned as the final screened data set. Otherwise, return to step 4 and perform the modification on the data set X. i to readjust; Step 7: Update the current value dataset Y to dataset X i+1 ; Step 8: If the termination condition is not met, update i=i+1 and go to step 4 to enter the next round; if the termination condition is met, return the current value data set Y as the final screened data set.

2. The method for selecting data items of railway freight data according to claim 1, characterized in that: The type of the data item belongs to the customer information type, the waybill data type or the cargo invoice information type, and each type includes one or more data items.

3. The method for selecting data items of railway freight data according to claim 1, characterized in that: The data preprocessing includes cleaning, conversion, normalization, missing value processing and outlier processing.

4. The method for selecting data items of railway freight data according to claim 1, characterized in that: Dataset value evaluation function f(X i ), using a regression model based on historical data and expert scores to obtain an invisible representation; wherein the regression model is obtained based on historical data and expert scores, including the following steps: (1) Data collection: Collect historical data item records and expert evaluation scores of the data set; (2) Data cleaning: processing missing values ​​and outliers, and standardizing the data; (3) Constructing training and test sets: Split the data set into training and test sets in proportion, define the data item set as the model input, and define the expert score value as the target variable of the corresponding model output; (4) Model training and validation: Select a regression model and model evaluation indicators, use the training set data to train the model, optimize the model parameters, and evaluate the performance of the model on the test set.

5. The method for selecting data items of railway freight data according to claim 1, characterized in that: In step 4, two methods, local search and dynamic step size adjustment, are used to search the data set X. i Adjust the data items in: (1) Local search means limiting the data set X i The range of changes in the data items involved in the adjustment is specified to select data set X according to the order of data item numbers. i The data items involved in the adjustment; (2) Dynamically adjusting the step size means dynamically adjusting the number of replacement data items according to the magnitude of the change in the data set score.

6. The method for selecting data items of railway freight data according to claim 5, characterized in that: The dynamic adjustment method for the number of data items n involved in the adjustment is: (1) Get the current dataset X i The score f(X i ) and a new dataset X i+1 The score f(X i+1 ), calculate the score change θ: (2) If the score change θ is greater than the preset replacement threshold upper limit, the number of adjustments is increased based on the current number of adjustments for the next iteration; (3) If the score change θ is less than the preset replacement threshold, the number of adjustments is reduced based on the current number of adjustments for the next iteration; (4) If the score change θ is within the preset replacement threshold, the current adjustment number will remain unchanged in the next iteration.

7. The method for selecting data items of railway freight data according to claim 1, characterized in that: The termination condition of step 8 is one of the following: (1) The memory set U already stores the data set of all N data item combinations, that is, (2) The change rate of the value assessment scores of two consecutive data sets is within the preset termination threshold i, that is:

8. A data item screening device for railway freight data, characterized in that: include: The data preprocessing module is used to: perform data preprocessing on the original data of railway freight transportation obtained from multiple data sources; wherein the original data includes M data items; The initialization module is used to randomly extract N non-repeating data items from the M data items of the original data as the initial candidate freight data set X1; store the initial candidate freight data set X1 as the current value data set Y, and store X1 in the memory set U; where N is the target number of data items to be screened, 1 <N<M; Value evaluation module, used to calculate any data set X i The value assessment score f(X i ); where f(X i ) is the data set value assessment function; Dataset adjustment module, used to: i+1 =D(X i ) Generate a new dataset X i+1 , and the new dataset X i+1 Stored in the memory set U; where the function D(X i ) represents the data set X i The data items in the current data set X are adjusted. i Select n data items and replace them with new data items, and 1≤n <N,n<M-N; Numeric comparison module, used to compare f(X i+1 ) and f(X i )Numerical value; The value data set update module is used to: determine f(X i+1 )>f(X i ), update the current value dataset Y to dataset X i+1 ; The value data set output module is used to: when the termination condition is reached, output the current value data set Y as the final screened data set.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images