Multi-dimensional intelligent cleaning and charging method based on network freight data set

Abstract

The invention belongs to the technical field of big data, and particularly relates to a multi-dimensional intelligent cleaning and charging method based on a network freight data set, which comprises the following steps of: S1, data set acquisition: aggregating real service data into a cleaning data set pool; s2, data set cleaning rule configuration: combining static rules, dynamic rules and priority weight proportion configuration to form a comprehensive configuration system, and obtaining corresponding cleaning rules; s3, data set cleaning: configuring a cleaning data set pool according to a cleaning rule; s4, a data set report is obtained, wherein the data set report is a multi-dimensional visual report generated by analyzing and summarizing the cleaned data set; and S5, data set auditing: performing manual verification on an abnormal data list, and optimizing a data set configuration rule. Through multi-dimensional rule application and automatic and intelligent data cleaning, the cleaning efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically relating to a multi-dimensional intelligent cleaning and billing method based on network freight datasets. Background Technology

[0002] Existing traditional online freight data sources contain significant gaps, anomalies, or format errors in data information, such as missing cargo type, cargo weight and volume, vehicle model, and approved load capacity. These errors drastically reduce data credibility, and traditional data cleaning methods often fall short in handling the complexity of online freight scenarios. Secondly, they typically rely on manual operation, which is not only inefficient and costly but also ill-suited for processing massive amounts of data. Furthermore, fixed cleaning rules cannot effectively address the impact of dynamic market changes and unforeseen events, such as fuel price fluctuations, severe weather conditions, or traffic restrictions. More importantly, traditional methods often focus on single-dimensional data verification, neglecting the inherent correlations between different data dimensions, resulting in poor cleaning effects and failing to meet the stringent data quality requirements of advanced data analysis and artificial intelligence applications. Against this backdrop, there is an urgent need for a multi-dimensional intelligent cleaning and billing method based on online freight datasets. This method aims to overcome the limitations of traditional data cleaning through automation, intelligence, multi-dimensionality, and dynamic means, significantly improving data quality and processing efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-dimensional intelligent cleaning and billing method based on network freight datasets, in order to solve the technical problems of existing technologies that make it difficult to intelligently and dynamically adjust multi-dimensional data during the data cleaning process, and that cleaning rules cannot adapt to the dynamic changes in actual conditions.

[0004] The multi-dimensional intelligent cleaning and billing method based on network freight datasets includes the following steps: S1. Dataset Collection: Through real business data, collect waybill information, cargo information, vehicle information, and trajectory information from multiple dimensions of multiple relationships and waybills, and aggregate them into a cleaned dataset pool. S2. Dataset cleaning rule configuration: Combining static rules, dynamic rules, and priority weight ratio configurations to form a comprehensive configuration system, resulting in corresponding cleaning rules; S3. Dataset Cleaning: Configure a cleaned dataset pool according to the cleaning rules, perform in-depth verification and filtering of waybill data, including determining whether the waybill is qualified through transportation cost billing during the verification process. S4. Dataset Report: The dataset report is an analysis summary of the cleaned dataset, producing a multi-dimensional visual report; S5. Dataset Review: For abnormal data sheets, assign them to professional annotators for manual verification and annotation administrator review. If the verification is correct, repeat the cleaning steps and perform cleaning again; optimize the dataset configuration rules.

[0005] Preferably, step S2 includes: S2.1 Configure static rules: The photographer's initial screening rules are based on industry standards and business requirements. The initial screening rules are static rules. S2.2 Configuring Dynamic Rules: By introducing external data sources and setting corresponding trigger conditions and adjustment strategies, dynamic allocation is achieved, enabling adaptive adjustment of scenario-based weights. S2.3 Priority and weighting of cleaning rules: Priority allocation for static and dynamic rules; S2.4, Comprehensive configuration weight formula, used to balance the overall impact of static and dynamic rules.

[0006] Preferably, step S2.2 includes: S2.2.1 External data source introduction: When the data from the external data source meets the triggering conditions, the dynamic rules will be automatically adjusted. S2.2.2 Dynamic adjustment based on triggering conditions and adjustment strategies. The preset triggering conditions in this step will activate the corresponding adjustment strategies. Through a specific adjustment function, external factors are quantified as influence factors on static rule parameters or thresholds.

[0007] Preferably, step S2.3 includes: S2.3.1, the method for configuring the priority weight ratio of static rules: the initial formula for the total static rule weight is as follows: , In the formula, Indicates the first i The percentage coefficient of static rules; Indicates the first i The weight coefficients of the static rules, This represents the total weight of the static rules. n The number of static rules; the weighting coefficient measures the weight of a single rule, and the percentage coefficient is used from the bottom up to limit the relative influence of static and dynamic rules; Weighting coefficients are introduced into the information gain algorithm to measure the contribution of each static rule to anomaly detection. The formula for the information gain algorithm is: IG(Ri) = H(D) - H(D|Ri), where H(D) is the information entropy of dataset D, and H(D|Ri) is the conditional entropy when the static rule Ri is known; static weights The complete calculation formula is: , in, Initial weights set manually; The adaptive coefficient controls the dynamic adjustment range; IG(Max) is the maximum value of the information gain of all rules, used for normalization.

[0008] Preferably, in step S2.3.1, a real-time weight update mechanism is applied: IG(Ri) is recalculated and the weights are updated based on the trigger conditions set according to the data volume; a smoothing process is used during the update, and the calculation formula is as follows: , Where γ is the smoothing coefficient. t To update the round, The weighting coefficients from the previous round. To update the latest weights, The weighting coefficient for the current round of actual application after smoothing.

[0009] Preferably, step S2.3 includes: S2.3.2, adjusting the function. The dynamic rule weights are calculated based on real-time external factors, as shown in the following formula: , Where, k j Indicates the set number j The default dynamic weight of the dynamic rule; f j ( x ): No. j The adjustment function of the dynamic rules calculates the weight coefficients based on real-time input; m is the number of dynamic rules. f represents the total dynamic rule weight; j ( x The settings are based on the patterns of relevant dynamic rules, including: 2.3.2.1 Periodic Dynamic Rule: The periodic adjustment function is:

[0010] In the above formula, m represents the downward adjustment percentage of the off-season index; n represents the upward adjustment percentage of the hot-season index. x For date parameters based on periodic detection, T 1, T 2 represents the set of dates for the off-season and peak season of holidays, respectively; 2.3.2.2 Dynamic Rule for Mean Dispersion: The adjustment function for mean dispersion is:

[0011] Where: P avg ΔP represents the historical average. x- P avgΔP represents the difference between the current value and the historical average. x This represents the current value; k is the floating weight of the corresponding category value, and the "±" in the calculation is determined by the direction of the increase or decrease of the difference; 2.3.2.3 Adaptive Adjustment of Contextualized Weights: To calculate this type of adjustment function, firstly, a contextualized labeling system is established, and a fuzzy logic system is used to map the context to the weights. The calculation formula is as follows: , in, x Features of the current scene; β s Let be the scene sensitivity coefficient of the s-th scene rule, obtained by training with historical business data. p The number of scene rules; f s ( x ) for scene features and scene rules x The influence function.

[0012] Preferably, step S2.4 includes: S2.3.3, the weight formula for the comprehensive configuration is as follows: ,in, For the configured comprehensive weight, This represents the global priority coefficient of static rules, and the rule association matrix enables deep coupling between dynamic and static rules.

[0013] Preferably, step S3 includes: S3.1 Preprocessing: Basic logic verification for formatting source data and checking for missing source data information; S3.2 Redundancy handling: Use hash algorithms to calculate and remove completely duplicate or redundant waybill information to reduce data duplication. S3.3 Data Cleaning: Based on the configured cleaning rules, the dataset is cleaned according to static rules, dynamic rules, and the weight of each rule. This step involves in-depth verification and filtering of waybill data. The transportation cost calculation includes the following specific implementation steps: 1) Match the vehicle model code of the vehicle being transported on the waybill; 2) Match the mileage range to which the transportation mileage belongs; 3) Calculate the range of transportation costs based on the transportation distance; 4) Judgment rules: If the transportation cost is between the minimum and maximum transportation costs, it is considered a qualified waybill; otherwise, it is an abnormal waybill, marked and archived into the manual verification data pool for manual review and verification.

[0014] Preferably, step S4 includes: classifying the dataset report according to categories such as route, vehicle type code, and shipping month, comparing the data before and after cleaning, and generating a multi-dimensional cleaning report; classifying the route comparison according to the administrative codes of the pick-up city and the receiving city, forming a data comparison report for different transportation routes; comparing the vehicle type according to different vehicle type codes to statistically analyze the performance of each vehicle type on different routes; and comparing the shipping month according to the shipping time to analyze the data feature change trend of each month.

[0015] Preferably, step S5 includes: S5.1 Review and Classification: Statistically analyze different causes of anomalies and classify abnormal data; S5.2 Data Desensitization: Data is desensitized to different degrees depending on the cause of the anomaly; S5.3, Annotation Personnel Review: Data is assigned to annotators with specialized skills for manual review according to the same anomaly type; S5.4, Review by the administrator: A certain percentage of data of the same abnormal type is randomly selected and enters the administrator's review pool. The reviewer reviews whether the data passes the review. If the pass rate of this type of abnormal data is lower than the threshold, it is still marked as abnormal and the review steps of the reviewer are repeated. S5.5 Review Feedback: The annotation review administrator reviews the abnormal data and, based on expert analysis experience, adjusts the static, dynamic, and weight ratio cleaning configurations to confirm whether the entire dataset needs to be cleaned again.

[0016] The technical advantages of this invention are as follows: 1. Intelligent Dynamic Adjustment: Changes in the market environment and operating conditions directly impact data. For example, capacity constraints during holidays may lead to increased freight costs; severe weather or emergencies (such as road closures) can affect transportation timeliness and route selection. These dynamic factors make it difficult for current fixed data cleaning rules to maintain long-term effectiveness, lacking the ability to perceive market dynamics and emergencies. This method, through the application of multi-dimensional rules (static, dynamic, and weighted configuration) and automated and intelligent data cleaning, significantly reduces the workload of manual review and labeling. Compared to traditional manual-led cleaning methods, it is estimated that overall cleaning efficiency will be improved while ensuring or even enhancing data quality.

[0017] 2. Multi-dimensional data collection: Network freight data often comes from multiple systems and channels, including waybill systems, vehicle management systems, GPS tracking devices, external weather services, fuel price APIs, etc. Existing technologies often only collect waybill attributes for data cleaning, while this method can collect information such as waybill attributes, cargo attributes, and vehicle attributes from multiple dimensions in parallel and build a dataset, verify the correctness of the data from multiple dimensions, and improve the quantity and quality of the dataset sample pool.

[0018] 3. Enable the cleaning results to feed back into the rule base: By establishing quantitative feedback from manual review results to the adjustment of rule parameters (static, dynamic, weight ratio adjustment, etc.), we can summarize, analyze and optimize, forming a closed-loop system of "cleaning → review → optimization". Attached Figure Description

[0019] Figure 1 This is a flowchart of a multi-dimensional intelligent cleaning and billing method based on a network freight dataset according to the present invention. Detailed Implementation

[0020] The following detailed description of the embodiments, with reference to the accompanying drawings, will further illustrate the specific implementation of the present invention, in order to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0021] like Figure 1 As shown, the present invention provides a multi-dimensional intelligent cleaning and billing method based on network freight datasets, including the following steps.

[0022] S1. Dataset Collection: Collect real business data from the production environment that can be used for training. Through real business data, collect information from multiple dimensions related to network freight documents (i.e., waybills), such as waybill information (shipping and receiving time, pick-up and unloading locations, etc.), cargo information (cargo weight and volume, transportation costs, etc.), vehicle information (vehicle type, vehicle length, rated load, etc.), and trajectory information (GPS coordinates, speed, etc.), and aggregate them into a clean dataset pool.

[0023] In this step, the required waybill information is collected in the production environment of the online freight system. The collected waybill information must have been reported to the provincial online freight platform detection system and must be free of abnormal data to ensure the compliance of the source data as genuine business data.

[0024] The data collection scope includes: 1.1 Waybill Attributes: Including but not limited to waybill number, shipping time, arrival time, shipping city, shipping address, receiving city, receiving address, shipping coordinates, receiving coordinates, and shipping costs.

[0025] 1.2 Cargo Attributes: Key information such as cargo type, cargo weight, and cargo volume.

[0026] 1.3 Vehicle attributes: license plate number, vehicle type, vehicle length, vehicle rated load, vehicle gross weight, etc.

[0027] 1.4 Track attributes: transportation mileage, track information, vehicle speed, vehicle location point latitude and longitude information, etc.

[0028] S2. Dataset Cleaning Rule Configuration. This step employs a comprehensive configuration system that combines static rules, dynamic rules, and priority weighting to achieve a cleaning logic that is both stable and reliable, as well as flexible and adaptable. This step includes the following sub-steps.

[0029] S2.1 Configuring Static Rules: This step involves configuring static rules based on industry standards and strict business requirements. These initial screening rules are static and include basic configurations such as cargo weight greater than 0 kg and transportation costs greater than 0 yuan. This step also includes rules configured based on business expert experience, such as vehicle model mileage pricing rules and dynamic adjustment mechanisms, to balance the need for adjusting static and dynamic rules and better reflect real-world business rules.

[0030] Static rules form the basic defense line for data cleaning, ensuring the basic compliance and rationality of the data. These include: shipping time, arrival time, shipping city, receiving city, detailed shipping address, detailed receiving address, shipping coordinates, receiving coordinates, cargo type, cargo weight, cargo volume, transportation cost, license plate number, vehicle length, vehicle type, vehicle load capacity, transportation mileage, trajectory information, vehicle speed, configuration vehicle mileage unit price table, mileage range, unit price, starting price, highest price, lowest price, highest upward adjustment range, and lowest downward adjustment range. Among these, shipping time, arrival time, shipping city, receiving city, detailed shipping address, detailed receiving address, cargo type, cargo weight, cargo volume, transportation cost, license plate number, vehicle length, vehicle type, vehicle load capacity, transportation mileage, trajectory information, and vehicle speed are mandatory fields and cannot be empty.

[0031] Vehicle model mileage pricing table: Configuration is based on expert experience, including vehicle model codes, mileage ranges, unit prices (vehicle / yuan / km), starting price, maximum price, minimum price, maximum upward adjustment range, and minimum downward adjustment range. (Vehicle model codes are classified according to the "National Operating Vehicle Type Code" standard, such as: heavy-duty van (H12), medium-duty van (H22), light-duty flatbed van (H35), etc.) The configuration of vehicle mileage unit price table includes: configuration based on the vehicle code, mileage range, unit price (vehicle / yuan / km), starting price, maximum price, minimum price, maximum upward adjustment range, minimum downward adjustment range, etc. provided by expert experience (the vehicle code is classified according to the name classification of the "National Operating Vehicle Type Code" standard, such as: heavy-duty van (H12), medium-duty van (H22), light-duty flatbed van (H35), etc.).

[0032] S2.2 Configuring Dynamic Rules: Dynamic allocation is achieved by introducing external data sources and setting corresponding trigger conditions and adjustment strategies, enabling adaptive adjustment of weights based on specific scenarios. Trigger conditions include periodic changes in shipping and receiving times (holidays, peak and off-peak seasons, etc.) and fluctuations in cargo market prices (changes in cargo supply and demand, fuel prices, tariff impacts), etc. This step specifically includes the following sub-steps.

[0033] S2.2.1 External Data Source Introduction. The system will connect to various external data sources. When the data from an external data source meets the trigger conditions, dynamic rules will be automatically adjusted. Examples include national statutory holiday information, real-time weather forecasts, fuel price indices, commodity market price fluctuation data, and traffic control information.

[0034] S2.2.2 Dynamic adjustment based on trigger conditions and adjustment strategies. The preset trigger conditions in this step (such as specific holidays, severe weather warnings, and significant oil price fluctuations) will activate corresponding adjustment strategies. These strategies, through specific adjustment functions, quantify external factors as influence factors on static rule parameters or thresholds. Examples are as follows: Impact of Holidays: By importing the calendar and holiday schedule, the system automatically identifies the holiday periods and may adjust the reasonable fluctuation range of transportation costs. For example, the upper limit of the unit price for long-distance transportation may be appropriately increased during the Spring Festival; when the dynamic rules detect warnings of severe weather such as heavy rain or heavy snow, the "vehicle load" in the static rules is automatically reduced by 20%, and the mileage unit price fluctuation coefficient in the transportation cost calculation model is adjusted simultaneously (formula: adjusted unit price = original unit price × (1 - disaster impact coefficient × load safety weight)).

[0035] S2.3 Priority and weighting configuration of cleaning rules: Prioritize static and dynamic rules to ensure that cleaning rules meet both hard business requirements (static rules) and adapt to dynamic changes in the external environment (dynamic rules). Setting appropriate weighting in this step is crucial for accurate cleaning.

[0036] S2.3.1 The priority weight ratio of static rules is configured as follows.

[0037] The initial formula for the total static rule weights is as follows: , In the formula, Indicates the first i The proportion coefficient of static rules (0≤ ≤1, and each The sum is 1). Indicates the first i The weight coefficient of the static rule (default value is 1). This represents the total weight of the static rules. n This represents the number of static rules. Among them, the weight coefficient is more inclined to measure an individual rule. On this basis, the information gain algorithm index is further introduced to measure the contribution of each static rule to the identification of data anomalies. The proportion coefficient is a higher-level macro control, which is used from the bottom up to limit the relative influence of static and dynamic rules.

[0038] The formula for the information gain algorithm is: IG(Ri) = H(D) - H(D|Ri), where H(D) is the information entropy of dataset D, and H(D|Ri) is the conditional entropy of the known static rule Ri. A larger IG(Ri) value indicates that the rule is more important for anomaly detection. Thus, the static weights... The complete calculation formula is: , in, Initial weights set manually (e.g., shipping time, arrival time) = 0.10); The adaptive coefficient (0.20-0.50) controls the dynamic adjustment range; IG(Max) is the maximum value of the information gain of all rules, used for normalization.

[0039] After setting the initial weights for each static rule, historical data is imported for training. The input consists of cleaned data from the past 12 months (including 100,000 normal data entries and 30,000 abnormal data entries). For each rule Ri, its hit frequency and information gain IG (Ri) in the abnormal data are calculated. Example: If the IG value of the "cargo weight" rule is 0.20, which is higher than the IG value of "shipping address" (0.15), then the former's weight is automatically increased by 5%.

[0040] Real-time weight update mechanism: Trigger conditions are set based on data volume: for example, every time 5,000 new approved data entries are added, IG (Ri) is recalculated and the weight is updated.

[0041] The update process employs a smoothing technique, specifically using an exponential moving average (EMA) to prevent excessive weight fluctuations. The calculation formula is as follows: , Where γ is the smoothing coefficient (0.7-0.9). t To update the round, The weighting coefficients from the previous round. To update the latest weights, The weighting coefficient for the current round of actual application after smoothing.

[0042] S2.3.2 The weights of dynamic rules are not fixed, but are adjusted by a function. The calculation is based on real-time external factors (such as holidays and market prices). The formula for calculating the total dynamic rule weight is as follows: , Where, k j Indicates the set number j The default dynamic weight of the dynamic rule (default value is 1); f j ( x ): No. j The adjustment function of the dynamic rules calculates weight coefficients based on real-time input (such as holidays, off-peak seasons, and changes in the commodity market); m is the number of dynamic rules. This represents the total dynamic rule weight.

[0043] f j ( x The settings are based on the relevant dynamic rules, as shown in the following example: 2.3.2.1 Periodic Dynamic Rules: For example, by adding factors such as holidays, off-peak seasons, and market fluctuations in commodity prices, the corresponding periodic adjustment function for shipments would be:

[0044] In the above formula, m represents the downward adjustment percentage of the off-season index; n represents the upward adjustment percentage of the hot-season index. x For date parameters based on periodic detection, T 1, T 2 represents the set of dates for the off-season and peak season of holidays, respectively.

[0045] 2.3.2.2 Dynamic Rules for Mean Difference: For example, the market adjustment function for commodity prices is:

[0046] Where: P avg Represents the historical average price, ΔP= x- P avg ΔP represents the difference between the current price and the historical average price. x This represents the current price; k is the market fluctuation weight of the commodity price, and the "±" in the calculation is determined by the direction of the increase or decrease of the difference.

[0047] 2.3.2.3 Adaptive adjustment of scene-based weights: For example, the adjustment function of the four-dimensional scene label system.

[0048] To calculate this type of adjustment function, a four-dimensional scene label system is first established, as detailed in Table 1.

[0049] Table 1: Four-Dimensional Scene Labeling System

[0050] A fuzzy logic system is used to map scenes to weights, and the calculation formula is as follows: , in, x Features of the current scenario (e.g., "Goods Type = Fresh Produce"); β s Let be the scene sensitivity coefficient of the s-th scene rule, obtained by training with historical business data. p The number of scene rules; f s ( x ) for scene features and scene rules x The influence function.

[0051] For example, in the fresh produce transportation scenario, the weight adjustment mode is as follows: Triggering conditions: The cargo type label is "fresh produce" and the transportation distance is >2000km; The influence coefficient of fresh produce on the "goods type" rule is 0.40, that is... x For the "fresh produce" scenario, the s-th scenario rule is "goods type". f s ( x The result changed from the initial value to 0.40.

[0052] S2.4, The comprehensive configuration weight formula is used to balance the overall impact of static and dynamic rules. This step allows for macro-level adjustments based on business stability and external environmental volatility. The comprehensive configuration weight formula is as follows: ,in, For the configured comprehensive weight, This represents the global priority coefficient of static rules (0 < 0). ≤1) For example: A value of 0.8 indicates that static rules account for 80% of the weight.

[0053] This step also enables deep coupling between dynamic and static rules through a rule association matrix, a crucial collaborative mechanism. The triggering of dynamic rules (such as detecting a "rainstorm disaster") not only adjusts their own weights but also directly adjusts the parameters of related static rules through a predefined rule association matrix. It goes beyond simple rule stacking, introducing information gain for data-driven weight optimization, and combining scene awareness and rule-based linkage to make the data cleaning process more intelligent and adaptive. This not only improves the accuracy of cleaning but also significantly reduces the need for manual intervention, especially when processing large-scale, high-dimensional, and rapidly changing network freight data, where its advantages are particularly evident.

[0054] S3. Dataset Cleaning: Configure the dataset pool for cleaning according to the cleaning rules. This step includes the following sub-steps.

[0055] S3.1 Preprocessing: Formatting source data and performing basic logical checks for missing source data information. Formatting includes various types of time formatting, cargo weight unit formatting, transportation cost amount unit formatting, latitude and longitude formatting, and location coordinate speed formatting, etc.

[0056] The formatting involves waybill information, cargo information, vehicle information, and tracking information. Details are as follows: 1) Waybill Information: 1.1) Delivery time and arrival time: The time information shall be formatted in accordance with the ISO 8601 standard, and the formatting standard shall be "yyyy-MM-dd HH:mm:ss" (e.g., 2025-01-01 00:00:00).

[0057] 1.2) City of shipment and city of receipt: Import the latest administrative division codes released by the Ministry of Civil Affairs and match the administrative division codes of the city of shipment and city of receipt. For example: Jinghu District, Wuhu City, Anhui Province: 340202.

[0058] 1.3) Detailed address of the shipping location, detailed address of the receiving location, shipping coordinates, and receiving coordinates: Call the Baidu coordinate conversion API interface based on the detailed address information, and uniformly use the Baidu coordinate system (BD09) to represent the shipping coordinates and receiving coordinates; if the coordinates have been filled in, then match whether the latitude and longitude of the coordinate points use the Baidu coordinate system and are consistent.

[0059] 2) Cargo Information: 2.1) Cargo Weight and Volume: The weight and volume of the transported goods. A weight-to-volume ratio is used to classify goods as heavy or bulky. For example, a ratio greater than 3 indicates heavy cargo, otherwise it's bulky cargo. The unit for cargo weight is kilograms, and the unit for cargo volume is cubic meters. 2.2) Transportation costs: The actual costs incurred in transporting the goods, including fuel, tolls, and the actual amount paid, in yuan; 3) Vehicle Information: 3.1) Vehicle type: The unified vehicle type adopts the "National Operating Vehicle Type Code" standard, using the operating vehicle code to represent different vehicle types; 3.2) Approved load capacity: The maximum load capacity of the vehicle under specific road conditions and load standards, as filled in on the vehicle registration certificate, in kilograms; 3.3) Gross weight: The weight of a vehicle when it is fully equipped and loaded with passengers (including the driver) and cargo as required, in kilograms; 4) Track information: 4.1) Vehicle speed: The vehicle speed collected through positioning during the transportation process, in kilometers per hour; 4.2) Trajectory information: The latitude and longitude information of the vehicle's location point obtained by GPS, Beidou or base station positioning at certain time intervals (3min / 5min / 10min), in the format: "longitude, latitude"; 4.3) Transportation mileage: By passing the vehicle location trajectory information already obtained above to the Eagle Eye Track Service API interface of Baidu Maps, the standard mileage returned by the trajectory mileage interface is used as the transportation mileage, in kilometers; S3.2 Redundancy handling: A hash algorithm is used to calculate and remove completely duplicate or redundant waybill information to reduce the amount of data that needs to be cleaned repeatedly.

[0060] The system categorizes waybill data based on predefined key dimensions (such as pick-up city, delivery city administrative code, vehicle type code, shipping time, and delivery time). For each waybill within a category, its multi-dimensional information is combined into a unique string, and then the hash value of this string is calculated. By comparing hash values, identical waybill records can be quickly identified. For duplicate records, the system typically retains one (e.g., the earliest entered or most recently updated record), while the rest are marked or removed.

[0061] The advantages of hash algorithms lie in their determinism (the same input always yields the same output) and efficiency (fast computation speed), making them ideal for deduplication tasks on large datasets. This ensures the accuracy of data analysis and avoids statistical bias caused by duplicate data.

[0062] S3.3 Data Cleaning: Based on the configured cleaning rules, the dataset is cleaned according to static rules, dynamic rules, and the weight of each rule.

[0063] The system will comprehensively utilize pre-processed data, configured static rules, dynamic rules, and real-time calculated weights to perform in-depth verification and filtering of each waybill data. The verification process can also determine the eligibility of a waybill through transportation cost calculation. The specific implementation steps are as follows.

[0064] 1) Match the vehicle model code of the vehicle being transported on the waybill.

[0065] 2) Match the mileage range to which the transportation mileage belongs.

[0066] 3) Calculate the range of transportation costs based on the transportation distance. The calculation formula is as follows: Minimum shipping cost = Shipping distance unit price (1 - Minimum downward adjustment range) , Maximum transportation cost = Transportation distance unit price (1 + maximum upward adjustment range) , The unit price, the minimum downward adjustment range, and the maximum upward adjustment range are all derived from the vehicle mileage unit price table in the static rules. The comprehensive weighting combines static and dynamic rules (such as fuel price fluctuations and holiday impacts), including the results of scenario-based weight adjustments, making the range of transportation costs more dynamically adaptable.

[0067] 4) Judgment Rules: If the transportation cost is between the minimum and maximum transportation costs, the waybill is considered a qualified waybill; otherwise, it is considered an abnormal waybill. Qualified waybills are added to the training data set, while abnormal waybills are marked and archived into the manual verification data pool for manual review and verification. That is, when the minimum transportation cost ≤ the maximum transportation cost, the waybill is considered a qualified waybill.

[0068] S4. Dataset Report: The dataset report is an analysis and summary of the cleaned dataset, producing a multi-dimensional visualization report to provide decision support for subsequent fine-tuning of rules, strategies, weights, etc.

[0069] The dataset report is categorized by route (pickup city - delivery city), vehicle type code, and shipping month, and compares the data before and after cleaning for each route, vehicle type code, and shipping month, generating a multi-dimensional cleaning report. The cleaning report should include: total cleaned data volume, qualified data quantity (rate), abnormal data quantity (rate), specific reasons for abnormal data, dimensional mean (route, vehicle type code, month), and dimensional variance (route, vehicle type code, month), ultimately forming visual charts (such as bar charts, line charts, pie charts, etc.) to view the detailed dataset cleaning report for a specific route, vehicle type code, month, or other dimensions. The comparison method before and after multi-dimensional cleaning is as follows.

[0070] Route Comparison: Data comparison reports are generated for different transportation routes, categorized by administrative codes of the pick-up city and the delivery city. For example, key indicators such as average transportation costs, cargo weight, and vehicle type are compared across different routes. Vehicle type comparison: Based on different vehicle type codes (such as heavy-duty vans, medium-duty vans, etc.), the performance of each vehicle type on different routes is statistically analyzed. This includes metrics such as average speed and fuel costs.

[0071] Shipment Month Comparison: Classify by shipment time (formatted as "yyyy-MM") and analyze the changing trends of data characteristics for each month, such as the impact of holidays on transportation costs and changes in seasonal demand for goods.

[0072] S5. Dataset Review: For abnormal data sheets, archive and isolate them into anomaly analysis datasets, assign them to professional annotation personnel for manual verification and annotation administrator review. If the verification is correct, repeat the cleaning steps for a second cleaning; optimize dataset configuration rules, etc. The following are the steps for dataset review.

[0073] S5.1 Review and Classification: Statistically analyze different causes of anomalies and classify abnormal data. Refer to the anomaly cleaning classification in the preprocessing steps.

[0074] S5.2 Data Desensitization: Data is desensitized to different degrees depending on the cause of the anomaly.

[0075] S5.3, Annotation Personnel Review: Data is assigned to annotators with professional skills for manual review according to the same anomaly type.

[0076] S5.4, Administrator Review: 20% of the data of the same abnormal type (for the same abnormal data annotated by different annotators) are randomly selected and entered into the administrator review pool. The reviewer reviews whether the data passes the review. If the pass rate of this type of abnormal data is less than 95%, it will still be marked as abnormal and the annotator review steps will be repeated.

[0077] S5.5 Review Feedback: The annotation review administrator reviews the abnormal data and adjusts the static, dynamic and weight ratio cleaning configurations based on expert analysis experience to confirm whether the entire dataset (training dataset and abnormal dataset) needs to be cleaned again.

[0078] For abnormal data in the manually verified data pool, the data is first classified and anonymized, then verified by professional annotators, and a certain number of data are randomly checked by the annotation review administrator for verification. Any abnormalities are reported and the rules and weights are adjusted accordingly.

[0079] Each piece of data undergoes independent review by at least two annotators. If the two annotators agree, the data is considered acceptable; if there is a disagreement, further investigation is required until a consensus is reached. To reduce subjective bias, a blind-box review mechanism is introduced, meaning that reviewers are unaware of the other reviewers' results until all reviews are completed.

[0080] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A multi-dimensional intelligent cleaning and billing method based on a network freight dataset, characterized in that: Comprise the following steps: S1, data set collection: through the real business data, from multiple relationships to the waybill multiple dimensions to collect waybill information, cargo information, vehicle information, track information, aggregated into a cleaning data set pool; S2, data set cleaning rule configuration: combined with static rules, dynamic rules and priority weight proportion configuration form a comprehensive configuration system, get the corresponding cleaning rules; S3, data set cleaning: according to the cleaning rule configuration cleaning data set pool, the waybill data for deep checking and screening, including in the process of checking by transportation cost charging to determine whether the waybill is qualified; S4, data set report: data set report is the analysis summary of the data set which has been executed cleaning, output multi-dimensional visual report; S5, data set review: for abnormal data, assigned to professional labeling personnel for artificial verification and labeling administrator review, check no error repeat cleaning step again cleaning; Optimization of data set configuration rules.

2. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 1, characterized in that: Step S2 includes: S2.1, configuration of static rules: the preliminary screening rules based on industry standards and business hard requirements, the preliminary screening rules are static rules; S2.2, configuration of dynamic rules: by introducing external data source and setting corresponding trigger condition and adjustment strategy to realize dynamic allocation, realize scene weight self-adaptive adjustment; S2.3, priority weight proportion configuration of cleaning rules: priority allocation is made to static rules and dynamic rules; S2.4, comprehensive configuration weight formula, used to balance the overall influence of static rules and dynamic rules.

3. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 2, characterized in that: Step S2.2 includes: S2.2.1, external data source introduction, when the data transmitted by external data source reaches the trigger condition, the dynamic rule is triggered to adjust automatically; S2.2.2, dynamic adjustment based on trigger condition and adjustment strategy, the preset trigger condition of this step will activate the corresponding adjustment strategy, and the external factors are quantified as influence factor of static rule parameter or threshold value through specific adjustment function.

4. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 2, characterized in that: Step S2.3 includes: S2.3.1, the priority weight proportion configuration mode of static rules: the initial formula of total static rule weight is as follows: ， In the formula, represents the proportion coefficient of the static rule of the first i represents the weight coefficient of the static rule of the first i is the total weight of the static rule, n is the number of static rules; the weight coefficient measures a single rule, and the proportion coefficient is used from the bottom to limit the relative influence of static and dynamic rules.​​ Weighting coefficients are introduced into the information gain algorithm to measure the contribution of each static rule to anomaly detection. The formula for the information gain algorithm is: IG(Ri) = H(D) - H(D|Ri), where H(D) is the information entropy of dataset D, and H(D|Ri) is the conditional entropy when the static rule Ri is known; static weights The complete calculation formula is: ， wherein, initial weights set by human; is an adaptive coefficient, controlling the dynamic adjustment range; IG(Max) is the maximum value of all rule information gains, used for normalization.

5. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 4, characterized in that: In step S2.3.1, real-time weight updating mechanism is applied: based on data volume, trigger condition is set to recalculate IG(Ri) and update weight; When updating, smoothing processing is adopted, and the calculation formula is as follows: ， where γ is a smoothing coefficient, t is the update round, is the weight coefficient of the previous round, is the latest weight for updating, is the weight coefficient of the current round after smoothing.

6. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 2, characterized in that: Step S2.3 comprises: S2.3.2, adjusting the function The dynamic rule weight is calculated according to real-time external factors, and the calculation formula is as follows: ， where k j represents the default dynamic weight of the set j th dynamic rule; f j ( x ) is the adjustment function of the j th dynamic rule, which calculates the weight coefficient according to real-time input; m is the number of dynamic rules, represents the total dynamic rule weight; f j ( x ) is set according to the law of the related dynamic rule, including: 2.3.2.1, periodic dynamic rules: the periodic adjustment function is: In the above formula, m represents the floating proportion of the off-season index downward adjustment; n represents the floating proportion of the hot-season index upward adjustment, x a date parameter based on periodic detection, T 1, T 2 respectively represent the date set of the holiday off-season and the hot-season. 2.3.2.2, mean difference dynamic rules: the adjustment function of mean difference is: wherein: P avg represents the historical average value, ΔP x- P avg represents the difference between the current value and the historical average value, x represents the current value; k is the value floating weight of the corresponding category, and the "+" or "-" in the calculation is determined by the increasing or decreasing direction of the difference. 2.3.2.3, scene weight self-adaptive adjustment: to calculate this kind of adjustment function, first, a scene-based label system is established, and fuzzy logic system is used to realize the mapping from scene to weight, and the calculation formula is: ， wherein, x is the current scene feature; β s is the scene sensitive coefficient of the s-th scene rule, which is trained by historical business data, p is the number of scene rules; f s ( x ) is the influence function of the scene feature on the scene rule x .

7. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 2, characterized in that: Step S2.4 includes: S2.3.3, the comprehensive configured weight formula is as follows: Wherein, The comprehensive weight configured, The global priority coefficient of static rules, the deep coupling of dynamic rules and static rules is realized through the rule association matrix.

8. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 1, characterized in that: Step S3 includes: S3.1, preprocessing: format source data and basic logic verification of source data information missing; S3.2, redundancy processing: use hash algorithm to calculate and eliminate completely repeated redundant waybill information, reduce repeated cleaning data; S3.3, data cleaning: based on the completion of cleaning rule configuration, the data set is cleaned according to the static rule, the dynamic rule and the weight of each rule; this step deeply checks and filters the waybill data, and the transportation cost charging includes the following specific implementation steps: 1) matching the vehicle model code of the waybill carrying vehicle; 2) matching the mileage range to which the transportation mileage belongs; 3) calculating the transportation mileage and transportation cost range interval; 4) judging rule: if the transportation cost is between the minimum and maximum transportation cost, it is considered as a qualified waybill, otherwise it is an abnormal waybill, which is marked and archived into the artificial verification data pool for manual review and verification.

9. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 1, characterized in that: Step S4 includes: the data set report is classified according to the same line, vehicle model code, delivery month and other categories, and the comparison before and after cleaning is made to generate a multi-dimensional cleaning report; the line comparison is classified according to the administrative code of the pickup city-delivery city to form a data comparison report of different transportation lines; the vehicle model comparison is based on different vehicle model codes to count the performance of each vehicle model on different lines; the delivery month comparison is classified according to the delivery time to analyze the data characteristic change trend of each month.

10. The multi-dimensional intelligent cleaning and billing method based on network freight data set according to claim 1, characterized in that: Step S5 includes: S5.1, review and classification: statistical analysis of different abnormal reasons, classification of abnormal data; S5.2, data desensitization: different degrees of data desensitization for different abnormal reasons; S5.3, labeling personnel review: according to the same abnormal type, the data is distributed to professional skill labelers for manual review; S5.4, labeling review administrator review: a certain proportion of data of the same abnormal type is randomly selected into the administrator review pool for review by the reviewer whether it is passed, if the abnormal review pass rate of this type of data is lower than the threshold, it is still marked as abnormal, and the labeling personnel review step is repeated; S5.5, review feedback: the labeling review administrator reviews the abnormal data and adjusts the static, dynamic and weight ratio cleaning configuration according to the expert analysis experience to confirm whether to clean the entire data set again.

Images