Bulk commodity transportation loss assessment method and system based on big data

Abstract

The invention discloses a bulk commodity transportation loss assessment method and system based on big data, and particularly relates to the technical field of bulk commodity transportation. The method comprises the following steps: collecting and sorting time and geographical dimension data of bulk commodity transportation, preprocessing original data, extracting time balance features and coverage consistency features, and evaluating the overall deviation severity degree through a machine learning model in combination with the features, so that a data set is divided into a high-equal-deviation data set and a low-equal-deviation data set; and finally, a service decision is optimized according to the deviation degree, so that the representativeness and integrity of data analysis are improved, wrong decisions caused by data deviation are avoided, and transportation loss management is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of bulk commodity transportation, and in particular to a bulk commodity transportation loss assessment method and system based on big data. Background Art

[0002] The evaluation of bulk commodity transportation loss based on big data refers to the use of big data technology to analyze and evaluate the loss of bulk commodities during transportation. Bulk commodity transportation involves many factors, such as weather, transportation methods, route selection, loading and unloading links, etc., which will affect the degree of loss of commodities. Through big data technology, a large amount of transportation-related data can be collected and analyzed, including historical transportation records, environmental conditions, sensor data, etc., to accurately evaluate the loss of each link. The advantage of this evaluation method is that it can discover the potential patterns and trends of loss by analyzing multi-dimensional data, thereby improving the accuracy of loss prediction. Enterprises can use this technology to optimize transportation routes, improve logistics management processes, reduce transportation losses, and reduce costs. At the same time, big data can also help enterprises monitor abnormal situations in the transportation process in real time, take timely countermeasures, and ensure the safe transportation of bulk commodities.

[0003] The prior art has the following deficiencies:

[0004] The analysis results of big data depend largely on the representativeness and completeness of the data set used. If the collected data is biased (such as over-reliance on data from a certain time period or a certain geographical area), the loss pattern analyzed may not be applicable to other scenarios or time points. This deviation may cause decision makers to make wrong business decisions. For example, if the model draws a more optimistic assessment result based on a past low-loss period, the company may reduce its investment in preventive measures, but when the actual transportation environment changes (such as a transportation route suddenly faces a higher risk), the company will encounter unforeseen high losses. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for evaluating the transportation loss of bulk commodities based on big data, so as to solve the shortcomings in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a method for evaluating the transportation loss of bulk commodities based on big data, comprising the following steps:

[0007] S1: According to the time range and transportation route of bulk commodity transportation, collect and organize the original data of time span and geographical area, organize the data according to time and geographical dimensions, and generate the original data set of bulk commodity transportation loss in each region;

[0008] S2: preprocessing the raw data of the time span and the geographic area, extracting features from the preprocessed raw data of the time span and the geographic area, respectively extracting the time balance feature from the raw data of the time span and the coverage consistency feature from the raw data of the geographic area;

[0009] S3: Convert the extracted time balance features and coverage consistency features into comprehensive feature vectors and input them into the machine learning model for training, and determine the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model;

[0010] S4: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

[0011] Preferably, in S2, a time balance anomaly index is generated according to the time balance feature in the extracted time span original data, and the method for obtaining the time balance anomaly index is:

[0012] Construct a time series data set containing all time periods. The data volume of each time period includes the number of transportation events and the number of losses. Define the neighborhood of each time period, that is, the k value in the local anomaly factor. The k value indicates that each time period will be compared with its k nearest neighbors. For each time period, calculate the distance to its k nearest neighbors. The expression is:

[0013] k-distance(T i )=max(dist(T i ,T j )); where T i is the amount of data in the i-th time period, dist(T i ,T j ) represents the data volume difference between time period i and adjacent time period j, max represents the maximum distance in the selected neighborhood, and measures the time period T i The density difference to another time period in its neighborhood is used to calculate the reachable distance. The formula is: reachable distance (T i ,T j )=max(k-distance(T j ),dist(T i ,T j )); k-distance (T j ) is the k-distance of time period j, dist(T i ,T j ) is the distance between time period i and time period j, and the local reachability density measures the time period T i For the reachability of its neighbors, it means that all time periods in the neighborhood to T iThe inverse of the reachable distance is expressed as:

[0014] Locally accessible density Where k is the number of time periods in the neighborhood, and ∑ represents the number of time periods in the neighborhood from T i The reachable distances are summed, and the time balance anomaly index is calculated based on the calculated local reachable density. The expression is:

[0015] Where LK is the time balance anomaly index.

[0016] Preferably, in S2, a coverage consistency index is generated according to the coverage consistency features in the extracted original data of the geographic area, and the method for obtaining the coverage consistency index is:

[0017] Construct the original dataset of geographic regions, including the transportation data and the number of loss events in each region, sort the data volume of each geographic region from small to large, and calculate the total data volume of all geographic regions. The expression is: In the formula, QD is the total amount of data in all geographic regions, and N is the number of regions. The ratio of the amount of data in each geographic region to the total amount of data, that is, the data coverage ratio, is calculated as follows: In the formula, RT is the data volume coverage ratio. For each geographical area i, the weight coefficient is calculated

[0018] Multiply the weight coefficient of each area by the proportion of the data volume to obtain the coverage consistency index PL.

[0019] Preferably, in S3, the extracted time balance features and coverage consistency features are converted into comprehensive feature vectors and input into the machine learning model for training, and the overall deviation severity of the original data set of commodity transportation loss is determined according to the output results of the model, specifically:

[0020] The time balance anomaly index and coverage consistency index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as the input of the machine learning model. The machine learning model predicts the overall deviation severity value label of the original data set of commodity transportation loss with each group of comprehensive feature vectors as the prediction target, and minimizes the sum of prediction errors of the overall deviation severity value labels of all original data sets of commodity transportation loss as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The overall deviation severity value of the original data set of commodity transportation loss is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0021] Preferably, in S4, the original data set of commodity transportation loss is divided into a high-deviation data set and a low-deviation data set according to the overall deviation severity of the data set, and business decision optimization is performed based on the division result, specifically:

[0022] The overall deviation severity value of the obtained original data set of commodity transportation loss is compared with the preset reference threshold of the overall deviation severity value. If the overall deviation severity value of the original data set of commodity transportation loss is greater than or equal to the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is high. At this time, an early warning signal is generated, and the corresponding data set is divided into a high-deviation data set; if the overall deviation severity value of the original data set of commodity transportation loss is less than the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is low. At this time, no early warning signal is generated, and the corresponding data set is divided into a low-deviation data set.

[0023] Preferably, in S4, when the data set is classified as a high-deviation data set, it indicates that there is a significant deviation in time or geographical area, and optimization measures need to be taken, specifically including:

[0024] For geographical areas with uneven data coverage, add data collection and monitoring equipment to ensure adequate data coverage in key areas and time periods with high transportation loss rates. Where D 缺失区域 Indicates the amount of supplementation in areas with insufficient data coverage; for data sets with high deviation, optimize the transportation routes and time nodes, the expression is: Where R i is the risk factor of different transportation routes; for the part classified as low deviation data set, for the data set with low deviation, the transportation path and scheduling are further optimized, and the expression is optimized path = argmin (time + cost): Since the data is evenly distributed, the accuracy of the model prediction is high, and the accuracy is further improved by continuously updating and improving the prediction model, and the expression is: The improvement of model accuracy is based on the continuous optimization of time and geographical coverage balance.

[0025] The present invention also provides a bulk commodity transportation loss assessment system based on big data, including a data integration module, a feature extraction module, a deviation prediction module and a classification optimization module;

[0026] Data integration module: collects and organizes the original data of time span and geographical area according to the time range and transportation route of bulk commodity transportation, organizes the data according to time and geographical dimensions, and generates the original data set of bulk commodity transportation loss in each region;

[0027] Feature extraction module: preprocess the raw data of time span and geographic area, extract features from the preprocessed raw data of time span and geographic area, and respectively extract the time balance feature in the raw data of time span and the coverage consistency feature in the raw data of geographic area;

[0028] Deviation prediction module: converts the extracted time balance features and coverage consistency features into comprehensive feature vectors and inputs them into the machine learning model for training. It also determines the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model.

[0029] Classification optimization module: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

[0030] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0031] 1. The present invention organizes and preprocesses the raw data of time span and geographical area, extracts time balance features and coverage consistency features, and converts these features into comprehensive feature vectors and inputs them into the machine learning model to evaluate the severity of the overall deviation. According to the results of the model output, the data set can be divided into high-deviation and low-deviation data sets, and corresponding business decision optimization measures are taken based on this, which effectively solves the problem of inaccurate analysis results caused by data deviation in the prior art, especially in dealing with wrong decisions caused by uneven distribution of time or geographical data.

[0032] 2. The present invention not only improves the accuracy of transportation loss prediction, but also realizes intelligent optimization of business decision-making by dynamically adjusting data collection strategies, optimizing transportation routes and improving prediction models, especially in the processing of high-loss areas and key time periods. Overall, it reduces the potential risks caused by insufficient data representativeness, reduces the risk of errors in transportation loss assessment by enterprises, improves supply chain management efficiency, and provides a more reliable basis for decision makers. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0034] Figure 1 The figure is a flow chart of the method of the present invention.

[0035] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example 1, please refer to Figure 1 and Figure 2 As shown, the method for evaluating bulk commodity transportation loss based on big data described in this embodiment includes the following steps:

[0038] S1: According to the time range and transportation route of bulk commodity transportation, collect and organize the original data of time span and geographical area, organize the data according to time and geographical dimensions, and generate the original data set of bulk commodity transportation loss in each region;

[0039] S2: preprocessing the raw data of the time span and the geographic area, extracting features from the preprocessed raw data of the time span and the geographic area, respectively extracting the time balance feature from the raw data of the time span and the coverage consistency feature from the raw data of the geographic area;

[0040] S3: Convert the extracted time balance features and coverage consistency features into comprehensive feature vectors and input them into the machine learning model for training, and determine the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model;

[0041] S4: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

[0042] Among them, in S1, according to the time range and transportation route of bulk commodity transportation, the original data of time span and geographical area are collected and sorted, and the data are sorted according to time and geographical dimensions to generate the original data set of bulk commodity transportation loss in each region, specifically:

[0043] Clearly define the time range and transportation routes to better divide and collect data: Determine the time span of the data, including: Short-term data: divided by time units such as days, weeks, and months to capture short-term fluctuations and seasonal changes. Divided by time units such as quarters and years to capture long-term trends in commodity transportation, such as the impact of long-term climate change or market fluctuations. Including specific holidays, extreme weather events, seasonal peaks, port congestion periods, etc., these time points may be significantly associated with transportation losses.

[0044] Identify the main transport routes and geographical areas, including: Ports: such as the port of departure and the port of destination. Cross-border transport involves multiple countries and regions and can be divided by country. Divide the main transport routes according to different modes of transport (such as sea, land, rail, etc.).

[0045] Extract the transportation records of bulk commodities from the company's internal transportation management system (TMS), supply chain management system (SCM) or enterprise resource planning system (ERP), covering: Cargo information: type, quantity, weight, etc. of bulk commodities. Mode of transportation: choice of transportation means such as sea, land, and air. Transportation time: the time from loading to unloading at the destination. External data: Weather data: obtain weather conditions in various places during transportation, such as temperature, humidity, wind speed, precipitation, etc. from a third-party meteorological service platform or historical weather database. Port operation data: obtain key data such as port congestion, cargo throughput, loading and unloading time, etc. from the operation reports of major ports. Traffic data: For land transportation, obtain data such as traffic congestion and road conditions, which can come from government departments, navigation systems, or transportation monitoring equipment. Macroeconomic data: Combine external economic factors such as market fluctuations, changes in fuel prices, and adjustments to logistics policies to analyze their impact on transportation routes and time.

[0046] Organize the data according to different time granularities (such as day, week, month, quarter, year). The data in each time period should include: Transportation start and end time: the start and end time of each transportation task. Calculate the total transportation time and summarize it by day, week or month to analyze the relationship between time and loss. Record whether transportation loss occurred in each time period, and mark the type and amount of loss (such as damaged goods, lost goods, etc.).

[0047] Use date and time stamp as index to sort the transportation records in chronological order and generate transportation loss statistics by day or week.

[0048] Choose the appropriate time granularity based on different analysis objectives. For example, if you are concerned about seasonal factors, you may organize data by month or quarter; if you need to analyze the impact of a specific event (such as bad weather), you can summarize data by hour or day.

[0049] Classify data by geographic region (such as country, port, transportation route), generate transportation records and loss information for each region, including: Origin and destination: mark the starting point and end point of each transportation task, and clarify the specific area involved in each transportation task. Divide the main transportation routes by sea, land, etc., and record the main places or stations along the way. Summarize the transportation loss data of each region and generate the loss rate of each region by geographical location.

[0050] Mark each transport record with the geographical area to which it belongs (such as the port of departure, port of destination, countries or regions passed through) to ensure that each record corresponds to a specific geographical location. Generate a geographical dimension loss table, which summarizes the transportation events and loss events in each region by geographical location (such as port, country, route), indicating the transportation volume, number of losses and loss ratio. After the data is sorted, data cleaning is required to ensure the accuracy and completeness of the data: remove duplicate transportation records or loss events, and ensure that each transportation task is unique in the data set. For some missing data, use appropriate methods to fill in (such as filling based on the average transportation time and loss rate of similar routes). Use statistical methods to identify and eliminate obvious outliers, such as abnormally long transportation times or excessively low loss values.

[0051] Summarize the transport tasks according to the time dimension to generate a time dimension loss data set. Each row represents a time period (such as a month or a quarter), including: Time period: a specific year, month or quarter. The total transportation volume in the time period (such as the number of transport tasks, the number of tons of goods transported). Number of losses: the number of loss events that occurred in the time period. Loss rate: the loss rate obtained by dividing the number of losses by the total transportation volume.

[0052] Geographic dimension loss data set generation: Generate a geographic region loss data set according to the geographic dimension. Each row represents a region or route, including: Region name: country, port or transportation route name. Total transportation volume: total transportation volume in the region (such as tonnage, quantity of goods). Number of losses: number of loss events in the region. Loss rate: number of losses divided by total transportation volume in the region.

[0053] The loss datasets of time dimension and geographic dimension are integrated to generate a comprehensive loss dataset with complete information. Each record contains the transportation time, origin, destination, transportation volume, number of losses, loss rate, etc. This dataset is used for subsequent pattern analysis and loss prediction model training.

[0054] Ensure data integrity and consistency by cross-checking data to verify that transportation and loss conditions are accurate for each time period and geographic region. For missing or abnormal data, use expert judgment or supplement external data sources to make corrections.

[0055] S2: Preprocess the original data of the time span and the geographic area, and perform feature extraction on the preprocessed original data of the time span and the geographic area, respectively extracting the time balance feature in the original data of the time span and the coverage consistency feature in the original data of the geographic area.

[0056] Before feature extraction, the original data of time span and geographic area are preprocessed to ensure data quality and consistency. The time format is unified, such as converting the original timestamp into a unified time unit (such as day, week, month, quarter, year).

[0057] If there is no data record for some time periods, interpolation or filling methods can be used, such as using the average weekly or monthly transportation volume or loss rate to fill the missing values. Detect abnormal time period data (such as abnormally high loss rate or extremely low transportation volume) and improve the accuracy of the data through denoising or correction.

[0058] Standardize the geographic location information in data from different sources. For certain geographic areas where data is lacking, use the average data of adjacent areas or estimate based on historical data. Identify and correct abnormal transportation volume or loss rate that may exist in certain areas to ensure data consistency.

[0059] The time balance anomaly index is generated according to the time balance feature in the extracted time span original data. The method for obtaining the time balance anomaly index is:

[0060] Construct a time series dataset containing all time periods. The data volume of each time period includes the number of transportation events and the number of losses. Define the neighborhood of each time period, that is, the k value in the local anomaly factor. k is a parameter that controls the number of neighboring time periods. Generally, a suitable k value (such as 3 or 5) is selected to calculate the local density. The k value means that each time period will be compared with its k nearest neighbors (in time). Smaller k values ​​are used to detect local anomalies, and larger k values ​​are used to detect global anomalies. For each time period, calculate the distance to its k nearest neighbors (that is, the difference in the amount of data between it and the adjacent time period). The k-distance is used to measure the distance between a time period and its neighborhood. The expression is: k-distance(T i )=max(dist(T i ,T j )); where T i is the amount of data in the i-th time period, dist(T i ,T j ) represents the data volume difference between time period i and adjacent time period j, max represents the maximum distance in the selected neighborhood, and measures the time period T i The density difference to other time periods in its neighborhood is used to calculate the reachable distance, which is defined as the time period Tj k-distance and time period T i With T j The larger distance between them is:

[0061] Reachable distance (T i ,T j )=max(k-distance(T j ),dist(T i ,T j )); k-distance (T j ) is the k-distance of time period j, dist(T i ,T j ) is the distance between time period i and time period j (i.e., the difference in data volume); the local reachability density measures the time period T i Relative to the reachability of its neighbors, indicating the reachability of all time periods in the neighborhood to T i The reciprocal of the reachable distance. The larger the local reachable density, the higher the density in this time period, and the less likely it is an abnormal point. The expression is: Local reachable density Where k is the number of time periods in the neighborhood, and ∑ represents the number of time periods in the neighborhood from T i The reachable distances are summed, and the time balance anomaly index is calculated based on the calculated local reachable density. The expression is: Where LK is the time balance anomaly index.

[0062] The larger the time balance anomaly index is, the more it indicates that the local density of the original data set of commodity transportation loss in certain time periods is significantly lower than that in adjacent time periods. This means that the data distribution is uneven and there are obvious abnormal time points, indicating that the data is overly dependent on certain time periods. A high anomaly index reflects that the overall data set is more biased, especially in the time dimension, where the amount of data in certain time periods may be significantly less than that in other time periods, or the loss rate is significantly different. This situation will affect the accurate analysis of the loss pattern and cause the model to have poor prediction results for certain specific time periods.

[0063] On the contrary, the smaller the time-balanced anomaly index is, the more evenly distributed the data is, the smaller the density difference between time periods is, and the fewer anomalies are. A small anomaly index indicates that the data has good coverage in the time dimension, and the data volume and loss rate in each time period are close to the same, which helps to improve the representativeness and prediction accuracy of the model. Overall, a lower anomaly index means that the deviation of the data set is smaller, and the model can be evenly applied in all time periods, avoiding over-reliance on data at certain time points, thereby improving the accuracy of analysis and decision-making.

[0064] The coverage consistency index is generated according to the coverage consistency characteristics in the extracted original data of the geographic area. The method for obtaining the coverage consistency index is:

[0065] Construct the original dataset of geographic regions, including the transportation data and the number of loss events in each region, sort the data volume of each geographic region from small to large, and calculate the total data volume of all geographic regions. The expression is: In the formula, QD is the total amount of data in all geographic regions, and N is the number of regions. The ratio of the amount of data in each geographic region to the total amount of data is calculated, that is, the data coverage ratio is calculated, and the expression is: In the formula, RT is the data volume coverage ratio. For each geographical area i, the weight coefficient is calculated Multiply the weight coefficient of each area by the proportion of the data volume to obtain the coverage consistency index PL.

[0066] The larger the coverage consistency index, the more uneven the distribution of the original dataset of commodity transportation losses in geographical regions, and the more serious the deviation of the dataset. This shows that the data is concentrated in a few geographical regions, while the amount of data in other regions is small or even not covered. In this case, model analysis and prediction will be overly dependent on data from certain regions, resulting in insufficient representation of global or multi-regional transportation conditions, and the accuracy of the results may be greatly reduced. For example, if the coverage consistency index is high, it means that there are large differences in the distribution of data in different geographical regions, and it is impossible to effectively assess the loss risk of all regions.

[0067] On the contrary, the smaller the coverage consistency index, the more evenly the data set is distributed in the geographical area, and the overall degree of deviation is less. The distribution of data in different geographical areas is more balanced, indicating that each area has sufficient data to support analysis and modeling. In this case, the model can make full use of data from all regions, and the prediction results are more representative and reliable, and have wider applicability. A low coverage consistency index means that the data set can fully reflect the transportation loss situation in all geographical areas, with less deviation, and the analysis results are more helpful for global decision-making.

[0068] S3: The extracted time balance features and coverage consistency features are converted into comprehensive feature vectors and input into the machine learning model for training, and the overall deviation severity of the original data set of commodity transportation loss is determined based on the output results of the model.

[0069] The time balance anomaly index and coverage consistency index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as the input of the machine learning model. The machine learning model predicts the overall deviation severity value label of the original data set of commodity transportation loss with each group of comprehensive feature vectors as the prediction target, and minimizes the sum of prediction errors of the overall deviation severity value labels of all original data sets of commodity transportation loss as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The overall deviation severity value of the original data set of commodity transportation loss is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0070] The method for obtaining the overall deviation severity value of the original data set of commodity transportation losses is as follows: from the comprehensive feature vector training data of the trained machine learning model, the corresponding function expression is obtained: CR=F(LK, PL); where F is the output function of the model, LK is the time equilibrium anomaly index, PL is the coverage consistency index, and CR is the overall deviation severity value of the original data set of commodity transportation losses.

[0071] S4: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

[0072] The overall deviation severity value of the obtained original data set of commodity transportation loss is compared with the preset reference threshold of the overall deviation severity value. If the overall deviation severity value of the original data set of commodity transportation loss is greater than or equal to the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is high. At this time, an early warning signal is generated, and the corresponding data set is divided into a high-deviation data set; if the overall deviation severity value of the original data set of commodity transportation loss is less than the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is low. At this time, no early warning signal is generated, and the corresponding data set is divided into a low-deviation data set.

[0073] When a dataset is classified as a high-bias dataset, it indicates that there is significant deviation in time or geographic area, and business optimization measures need to be taken to ensure the representativeness of the data and reduce risks in loss forecasting.

[0074] Reallocate resources: Add data collection and monitoring equipment to geographical areas with uneven data coverage, especially in areas and time periods with high transportation loss rates, to ensure that data in these key areas and time periods are fully covered. Where D 缺失区域 Indicates the amount of supplementation in areas with insufficient data coverage;

[0075] Re-evaluate transportation plans: For data sets with high deviation, optimize transportation routes or time nodes. Areas with high loss or insufficient data can reduce deviations and risks by replanning routes, increasing transportation monitoring, or changing transportation methods. The expression is: Where R i For the risk factors of different transportation routes, choose the option that minimizes cost and risk.

[0076] For the part classified as low-bias data set, it indicates that the data is evenly distributed in time and geographic dimensions, so the optimization strategy should focus on cost control and efficiency improvement.

[0077] Continuously optimize transportation efficiency: For data sets with low deviation, transportation routes and scheduling can be further optimized to improve transportation efficiency and reduce transportation costs. The expression is optimized path = argmin (time + cost): optimize transportation by selecting the route with the shortest transportation time and the lowest cost.

[0078] Improve prediction accuracy: Since the data is evenly distributed, the model prediction accuracy is high. Therefore, the accuracy can be further improved by continuously updating and improving the prediction model, especially real-time analysis based on big data. The expression is: The improvement of model accuracy is based on the continuous optimization of time and geographical coverage balance.

[0079] For highly biased data sets, the system should issue early warning signals to remind decision makers that the data has a high degree of deviation, and conduct regular data evaluation and business adjustments based on the early warning signals. The feedback mechanism should include: regularly evaluating the overall deviation severity of the data set and observing the effects of optimization measures. Dynamically adjust the reference threshold of the overall deviation severity value to adapt to the changing needs of the business and the influence of the external environment.

[0080] In this embodiment, according to the time range and transportation route of bulk commodity transportation, the original data of time span and geographical area are collected and sorted to generate a complete bulk commodity transportation loss data set. Then, these data are preprocessed, and the time balance features of the time dimension and the coverage consistency features of the geographical dimension are extracted. Then, the extracted features are converted into comprehensive feature vectors, input into the machine learning model for training, and the overall deviation severity of the bulk commodity transportation loss data set is output. According to the results of the model, the data set is divided into high deviation and low deviation data sets, and corresponding business decision optimization strategies are adopted for different deviation situations.

[0081] Embodiment 2, the commodity transportation loss assessment system based on big data described in this embodiment includes a data integration module, a feature extraction module, a deviation prediction module and a classification optimization module;

[0082] Data integration module: collects and organizes the original data of time span and geographical area according to the time range and transportation route of bulk commodity transportation, organizes the data according to time and geographical dimensions, and generates the original data set of bulk commodity transportation loss in each region;

[0083] Feature extraction module: preprocess the raw data of time span and geographic area, extract features from the preprocessed raw data of time span and geographic area, and respectively extract the time balance feature in the raw data of time span and the coverage consistency feature in the raw data of geographic area;

[0084] Deviation prediction module: converts the extracted time balance features and coverage consistency features into comprehensive feature vectors and inputs them into the machine learning model for training. It also determines the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model.

[0085] Classification optimization module: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

[0086] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0087] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0088] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for evaluating bulk commodity transportation losses based on big data, characterized by: The steps include: S1: According to the time range and transportation route of bulk commodity transportation, collect and organize the original data of time span and geographical area, organize the data according to time and geographical dimensions, and generate the original data set of bulk commodity transportation loss in each region; S2: preprocessing the raw data of the time span and the geographic area, extracting features from the preprocessed raw data of the time span and the geographic area, respectively extracting the time balance feature from the raw data of the time span and the coverage consistency feature from the raw data of the geographic area; S3: Convert the extracted time balance features and coverage consistency features into comprehensive feature vectors and input them into the machine learning model for training, and determine the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model; S4: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

2. The method for evaluating bulk commodity transportation loss based on big data according to claim 1 is characterized by: In S2, a time balance anomaly index is generated according to the time balance feature in the extracted time span original data, and the method for obtaining the time balance anomaly index is: Construct a time series data set containing all time periods. The data volume of each time period includes the number of transportation events and the number of losses. Define the neighborhood of each time period, that is, the k value in the local anomaly factor. The k value indicates that each time period will be compared with its k nearest neighbors. For each time period, calculate the distance to its k nearest neighbors. The expression is: k-distance(T i )=max(dist(T i ,T j )); where T i is the amount of data in the i-th time period, dist(T i ,T j ) represents the data volume difference between time period i and adjacent time period j, max represents the maximum distance in the selected neighborhood, and measures the time period T i The density difference to another time period in its neighborhood is used to calculate the reachable distance. The formula is: reachable distance (T i ,T j )=max(k-distance(T j ),dist(T i ,T j )); k-distance (T j ) is the k-distance of time period j, dist(T i ,T j ) is the distance between time period i and time period j, and the local reachability density measures the time period T i For the reachability of its neighbors, it means that all time periods in the neighborhood to T i The inverse of the reachable distance is expressed as: local reachable density Where k is the number of time periods in the neighborhood, and ∑ represents the number of time periods in the neighborhood from T i The reachable distances are summed, and the time balance anomaly index is calculated based on the calculated local reachable density. The expression is: Where LK is the time balance anomaly index.

3. The method for evaluating bulk commodity transportation loss based on big data according to claim 2 is characterized by: In S2, a coverage consistency index is generated according to the coverage consistency features in the extracted original data of the geographic area. The method for obtaining the coverage consistency index is: Construct the original dataset of geographic regions, including the transportation data and the number of loss events in each region, sort the data volume of each geographic region from small to large, and calculate the total data volume of all geographic regions. The expression is: In the formula, QD is the total amount of data in all geographic regions, and N is the number of regions. The ratio of the amount of data in each geographic region to the total amount of data, that is, the data coverage ratio, is calculated as follows: In the formula, RT is the data volume coverage ratio. For each geographical area i, the weight coefficient is calculated Multiply the weight coefficient of each area by the proportion of the data volume to obtain the coverage consistency index PL.

4. The method for evaluating bulk commodity transportation loss based on big data according to claim 3 is characterized by: In S3, the extracted time balance features and coverage consistency features are converted into comprehensive feature vectors and input into the machine learning model for training. The overall deviation severity of the original data set of commodity transportation loss is determined based on the output results of the model, specifically: The time balance anomaly index and coverage consistency index are converted into comprehensive feature vectors, and the comprehensive feature vectors are used as the input of the machine learning model. The machine learning model predicts the overall deviation severity value label of the original data set of commodity transportation loss with each group of comprehensive feature vectors as the prediction target, and minimizes the sum of prediction errors of the overall deviation severity value labels of all original data sets of commodity transportation loss as the training target. The machine learning model is trained until the sum of prediction errors converges, and the model training is stopped. The overall deviation severity value of the original data set of commodity transportation loss is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

5. The method for evaluating bulk commodity transportation loss based on big data according to claim 4 is characterized by: In S4, according to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results, specifically: The overall deviation severity value of the obtained original data set of commodity transportation loss is compared with the preset reference threshold of the overall deviation severity value. If the overall deviation severity value of the original data set of commodity transportation loss is greater than or equal to the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is high. At this time, an early warning signal is generated, and the corresponding data set is divided into a high-deviation data set; if the overall deviation severity value of the original data set of commodity transportation loss is less than the preset reference threshold of the overall deviation severity value, it means that the overall deviation severity of the original data set of commodity transportation loss is low. At this time, no early warning signal is generated, and the corresponding data set is divided into a low-deviation data set.

6. The method for evaluating bulk commodity transportation loss based on big data according to claim 5 is characterized by: In S4, when a dataset is classified as a high-bias dataset, it indicates that there is a significant deviation in time or geographic area, and optimization measures need to be taken, including: For geographical areas with uneven data coverage, add data collection and monitoring equipment to ensure adequate data coverage in key areas and time periods with high transportation loss rates. Where D 缺失区域 Indicates the amount of supplementation in areas with insufficient data coverage; for data sets with high deviation, optimize the transportation routes and time nodes, the expression is: Where R i is the risk factor of different transportation routes; for the part classified as low deviation data set, for the data set with low deviation, the transportation path and scheduling are further optimized, and the expression is optimized path = argmin (time + cost): Since the data is evenly distributed, the accuracy of the model prediction is high, and the accuracy is further improved by continuously updating and improving the prediction model, and the expression is: The improvement of model accuracy is based on the continuous optimization of time and geographical coverage balance.

7. A bulk commodity transportation loss assessment system based on big data, used to implement the bulk commodity transportation loss assessment method based on big data as described in any one of claims 1 to 6, characterized in that: It includes data integration module, feature extraction module, deviation prediction module and classification optimization module; Data integration module: collects and organizes the original data of time span and geographical area according to the time range and transportation route of bulk commodity transportation, organizes the data according to time and geographical dimensions, and generates the original data set of bulk commodity transportation loss in each region; Feature extraction module: preprocess the raw data of time span and geographic area, extract features from the preprocessed raw data of time span and geographic area, and respectively extract the time balance feature in the raw data of time span and the coverage consistency feature in the raw data of geographic area; Deviation prediction module: converts the extracted time balance features and coverage consistency features into comprehensive feature vectors and inputs them into the machine learning model for training. It also determines the overall deviation severity of the original data set of commodity transportation loss based on the output results of the model. Classification optimization module: According to the overall deviation severity of the original data set of commodity transportation loss, it is divided into a high-deviation data set and a low-deviation data set, and business decision optimization is performed based on the division results.

Images