Logistics waybill anomaly detection method, system and storage medium in data stream environment

Abstract

The present invention relates to a field of logistics waybill data processing, and discloses a logistics waybill anomaly detection method, system and storage medium in a data stream environment, which comprises: pre-processing the original waybill data and original trajectory data of the logistics company respectively, and then selecting positive and negative samples; integrating the negative samples and the positive samples to generate a concept drift data set and a non-concept drift data set, simulating the concept drift of the waybill anomaly by the non-concept drift data set, and selecting features that can fully reflect the relationship between the waybill and the trajectory from each data set; processing the selected features by the EFAT algorithm to obtain anomaly detection results. The present invention has fast detection speed, high accuracy, and the ability to resist concept drift.

Description

Technical Field

[0001] The present invention relates to the technical field of logistics waybill data processing, and in particular to a logistics waybill anomaly detection method, system and storage medium in a data stream environment. Background Art

[0002] With the rapid development of the global economy and the booming e-commerce, the information age has arrived. Nowadays, most retail sales are completed through online stores. At the same time, the emergence of freight platform companies such as Manbang and G7 has made it possible to use big data technology to analyze transportation behavior and improve operational efficiency. Therefore, the status of the logistics industry has grown rapidly and has become a key factor in supporting the operation of modern supply chains. As an important link in the supply chain, logistics transportation undertakes the important task of transporting goods from the place of production to the destination.

[0003] Road freight is an important indicator of social economy. As the core document in the logistics transportation process, the waybill records the starting point, destination and key links of the goods in transportation. The waybill plays a key role in ensuring the smooth delivery of goods from the source to the destination. Therefore, in this fast-paced and efficient environment, it is crucial to accurately and timely identify and handle waybill anomalies. Trajectory data records the transportation trajectory during the start and end of the waybill and is the basis for waybill anomaly detection. Trajectory data usually comes from various channels, such as transport vehicle sensors, global positioning system (GPS), Beidou navigation system, etc. However, with the rapid development of the logistics industry and the continuous expansion of global trade, the volume of transportation and trajectory data has increased dramatically, which has made the anomaly detection algorithm in traditional static scenarios face huge difficulties. Therefore, how to quickly and accurately detect anomalies in a big data environment has become a prominent challenge for logistics waybill anomaly detection. Traditional anomaly detection algorithms are usually trained and tested based on static data sets, and assume that the data distribution and anomaly patterns remain unchanged. However, in a dynamic environment such as the logistics field, the abnormal patterns of waybills are diverse. At the same time, the data distribution and abnormal patterns of waybills may change over time, which is the concept drift phenomenon. This causes the algorithm performance to weaken as the abnormal scenarios continue to change. Therefore, in the actual business scenario of logistics waybill anomaly detection, the ability to adapt to the ever-changing types of waybill anomalies should be one of the necessary capabilities of the anomaly detection algorithm.

[0004] In summary, as the core information in the logistics transportation process, how to detect anomalies in the trajectory data of the waybill is a research topic with great application value and a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a logistics waybill anomaly detection method, system and storage medium in a data stream environment, which have fast detection speed, high accuracy and the ability to resist concept drift.

[0006] To achieve the above-mentioned purpose, in the first aspect, the technical solution adopted by the present invention is: a logistics waybill anomaly detection method in a data stream environment, which comprises: preprocessing the original waybill data and the original trajectory data of the logistics company respectively, and then selecting positive and negative samples; integrating the negative samples and the positive samples to generate a concept drift data set and a non-concept drift data set, simulating the concept drift of the waybill anomaly by the non-concept drift data set, and selecting features that can comprehensively reflect the relationship between the waybill and the trajectory from each data set; processing the selected features with the EFAT algorithm to obtain anomaly detection results.

[0007] Furthermore, the original waybill data of the logistics company is preprocessed, including:

[0008] Filter the number of original waybill data based on geographic information, and limit the waybill to the set range by filtering by latitude and longitude and province, city and district. At the same time, limit the related attributes of the waybill start and end points to non-null values ​​to eliminate invalid or incomplete data, reduce the number of waybill data, and ensure that waybill records are not repeated;

[0009] The number of vehicles and waybills is screened, vehicles with a value less than the first set threshold are excluded, and data with a transportation time and a transportation mileage exceeding the set range are further eliminated, and abnormal travel is excluded to obtain the final number of vehicles and waybill data items;

[0010] Preprocess the original trajectory data of the logistics company, including:

[0011] For multiple track records contained in the original track data, by matching the truck numbers with each other and matching the trigger time with the start time and end time in the waybill data, the track record data related to the valid waybill is screened out to reduce the number of track records;

[0012] Limiting the truck speed to be greater than the second set threshold, filtering out possible worthless redundant trajectory data points, and further reducing the number of trajectory records;

[0013] The remaining trajectory records are filtered by latitude and longitude and province, city and district, and the trajectories are limited to the set range. At the same time, the province, city and district related attributes of the trajectory are limited to non-null values, invalid or abnormal trajectory data are excluded, the number of trajectory records is further reduced, and the trajectory data is deduplicated to ensure the consistency of the data set, and the final non-duplicate trajectory records are obtained.

[0014] Furthermore, the selection of positive and negative samples includes:

[0015] The real waybill-track data collected from the business side of a large logistics company is used as the positive sample. The positive sample includes correct and complete waybill-related information and corresponding track data, which can reflect the normal behavior in the actual transportation process;

[0016] Negative samples are artificially generated to simulate potential abnormal situations. The logic for generating negative samples includes: negative samples of trajectory mismatch, negative samples of delivery timeout, and negative samples of lost waybills.

[0017] Furthermore, the negative samples and positive samples are integrated to generate concept drift datasets and non-concept drift datasets, including:

[0018] By restricting the distribution of positive samples and negative samples, a dataset with no concept drift and a dataset with concept drift are formed; in the dataset with no concept drift, negative samples are randomly distributed at various positions.

[0019] Furthermore, the distribution of concept drift datasets includes:

[0020] In the concept drift dataset, negative samples with mismatched trajectories are normally distributed near the set first positive sample, negative samples with timeouts are normally distributed near the set second positive sample, and negative samples with lost waybills are normally distributed near the third positive sample, to simulate the concept drift of waybill anomalies.

[0021] Furthermore, features that can fully reflect the relationship between waybills and trajectories are selected from each data set, including:

[0022] Confidence: by counting the number of track points a in the starting time period of the waybill and the number of track points b in the corresponding time periods of the starting and ending points, the value of a / b is calculated to reflect the consistency of the waybill information and the track information;

[0023] Start and end point matching: check whether the starting and ending points of the waybill appear in all the track points of the corresponding track to determine whether the starting and ending points of the waybill are completely included in the track;

[0024] The starting point offset distance calculates the distance between the starting point recorded in the waybill and the actual position of the track point corresponding to the waybill start time to reveal the deviation of the actual position of the starting point from the waybill record;

[0025] The end point offset distance is calculated by calculating the distance between the end point of the waybill record and the actual position of the track point corresponding to the waybill end time;

[0026] Waybill distance: using the distance information recorded in the waybill as the driving distance feature of the waybill;

[0027] Straight-line distance between the start and end points: calculate the straight-line distance between the start and end points of the waybill record on the track to provide a global reference for the length of the waybill track.

[0028] Running time, using the time information recorded on the waybill, indicates the running time of the waybill;

[0029] The total track time length calculates the time length recorded by the trigger time of the actual first track point and the trigger time of the last track point in the waybill to provide a perspective on the actual overall time length of the waybill;

[0030] Labels are used to distinguish positive and negative samples.

[0031] Furthermore, the selected features are processed using the EFAT algorithm to obtain anomaly detection results, including:

[0032] Compare whether the difference between the best feature attribute and the second best feature attribute is greater than the Hoeffding limit. If it is greater, split the leaf node on the best feature attribute to generate a new child node; if the difference between the best feature attribute and the second best feature attribute does not meet the condition, further compare the difference between the best feature attribute and the current split feature attribute;

[0033] If the best feature attribute is the most frequent category in the data set, replace the current internal node with a leaf node and delete its subtree;

[0034] If the best feature attribute is not the current split feature attribute, then the split feature attribute is updated, and the current internal node is replaced by a new internal node split on the best feature attribute;

[0035] At the same time, an ADWIN detector is established on each leaf node to monitor changes in the statistical information of the node; when a new data sample arrives, the EFAT node will update its statistical information, and at the same time, these statistical information will be input into the node's ADWIN detector; the ADWIN detector continuously monitors changes in input data. When changes in statistical characteristics are detected, it indicates that concept drift may have occurred, and the window undergoes adaptive changes. Subsequently, the EFAT tree performs adaptive pruning operations to optimize the tree structure and obtain more accurate anomaly detection results.

[0036] In the second aspect, the technical solution adopted by the present invention is: a logistics waybill anomaly detection system in a data flow environment, which includes: a sample selection module, which pre-processes the original waybill data and the original trajectory data of the logistics company respectively, and then selects positive and negative samples; an integration module, which integrates the negative samples and the positive samples to generate a concept drift data set and a non-concept drift data set, and simulates the concept drift of the waybill anomaly by the non-concept drift data set, and selects features from each data set that can comprehensively reflect the relationship between the waybill and the trajectory; an EFAT module, which processes the selected features using the EFAT algorithm to obtain anomaly detection results.

[0037] In a third aspect, the technical solution adopted by the present invention is: a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes any one of the above methods.

[0038] In a fourth aspect, the technical solution adopted by the present invention is: a computing device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0039] The present invention adopts the above technical solution, which has the following advantages:

[0040] 1. The spatiotemporal dataset established by the present invention is composed of real data under the actual business scenarios of a logistics company, covering all the information of many waybills of the logistics company and all the information of the vehicle transportation trajectory during the completion of the waybills. This information has not yet been presented in the field of anomaly detection of logistics waybills. The present invention applies this information to the research on anomaly detection of logistics waybills for the first time. This dataset not only provides sufficient data support for the conclusions of the present invention, but also provides a research basis for subsequent research in the field of anomaly detection algorithms. Compared with artificially generated or public datasets that only represent specific scenarios, actual business data reflects real business scenarios and operating environments, and fully covers the diversity and complexity of real business scenarios, which helps to evaluate the accuracy, adaptability and robustness of algorithms in real environments, making relevant research conclusions closer to actual conditions.

[0041] 2. The present invention is an innovative application in the field of logistics waybills. Currently, there are few studies on electronic waybill data, and even fewer studies on the rationality and correctness of logistics waybill data streams, and there is an urgent need to fill the gap. The present invention applies the data stream anomaly detection algorithm to the anomaly detection of logistics waybills for the first time, focusing on abnormal waybill detection, thereby improving the quality of electronic waybill data and providing more possibilities for other studies on electronic waybill data. In addition, the present invention proposes a new data stream anomaly detection algorithm EFAT (Extremely Fast Adaptive Tree), and applies it to the anomaly detection scenario of logistics waybills. Compared with many existing online learning data stream anomaly detection algorithms, the present invention has better real-time performance, accuracy, and the ability to resist concept drift, so that the logistics waybill data stream can be quickly, in real time, accurately, and stably detected for anomalies.

[0042] 3. The present invention applies the data flow anomaly detection method to the logistics waybill anomaly detection, thereby reducing the possibility of negative situations such as transportation interruption, illegal transportation, commercial fraud, etc. during the transportation process, reducing the economic losses of individuals, enterprises and society, and improving the efficiency and reliability of the supply chain, which has a certain positive impact on the operation and economic development of the enterprise.

[0043] 4. The present invention has universal applicability. The novel online learning data stream anomaly detection method of the present invention has a strong ability to resist concept drift. As the time span increases and the types of new anomalies increase, the method still has the same anomaly detection capability. At the same time, the present invention acts on the preprocessed logistics data feature matrix. Therefore, under reasonable data preprocessing, the present invention also has good effects in other business scenarios, such as fraud detection in the financial field, abnormal electricity consumption observation in the power industry, and other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method for detecting anomalies in logistics waybills in a data stream environment according to an embodiment of the present invention;

[0045] Figure 2 This is a distribution diagram of longitudinal comparison experimental indicators of the same type of algorithms when there is no concept drift in the embodiment of the present invention;

[0046] Figure 3 It is a distribution diagram of experimental indicators of longitudinal comparison of algorithms of the same type when concept drift exists in an embodiment of the present invention;

[0047] Figure 4 This is a distribution diagram of experimental indicators for horizontal comparison of algorithms of the same type when there is no concept drift in an embodiment of the present invention;

[0048] Figure 5 This is a distribution diagram of experimental indicators for horizontal comparison of algorithms of the same type when concept drift exists in an embodiment of the present invention;

[0049] Figure 6 is a distribution diagram of algorithm ablation experiment indicators when there is no concept drift in an embodiment of the present invention;

[0050] Figure 7 It is a distribution diagram of algorithm ablation experiment indicators when concept drift exists in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] With the rapid development of emerging technologies such as the Internet of Things and cloud computing, the application of massive data is entering the era of real-time analysis and value realization. Against the background of the rapid development of the logistics industry and the sharp increase in transportation volume, the anomaly detection algorithm in traditional static scenarios can no longer meet the requirements of real-time and robustness. Therefore, based on the rapid development of the logistics field and the challenge of the surge in waybill data at this stage, the present invention proposes a new data stream anomaly detection algorithm EFAT that can cope with the diversity of waybill abnormal patterns and concept drift and can adapt to real-time environments. According to the experimental results of the real data set of the logistics company in the actual business scenario, the present invention can detect various waybill anomalies in real time in the data stream environment, and has the ability to resist concept drift, which can help logistics companies to timely discover and solve waybill anomalies, thereby improving the reliability and efficiency of the transportation process. The present invention has a certain role in promoting the research progress in the field of data stream anomaly detection, and at the same time provides a valuable reference for the application of data stream anomaly detection algorithms in real business scenarios.

[0054] In the study of logistics electronic waybills, some studies focus on the use of electronic waybills in transportation systems. For example, existing literature analyzes the information synergy between electronic waybill solutions and intelligent transportation system services. There are also existing literatures that design electronic multimodal transport waybills and solutions to be implemented using electronic freight electronic delivery infrastructure. And use rule-based methods to extract micro-logistics data from express waybills, and apply these data to analyze the spatial pattern, structural characteristics and functional organization of China's logistics network. There are also some studies that focus on the prediction of waybill information, such as using machine learning-based methods to predict the type of goods transported between different cities with specific time and weight from key information in the waybill; using waybill data to predict the category of goods moving between specific origins and destinations, and evaluating the performance of different machine learning algorithms. There are also existing literatures that develop electronic waybills combined with blockchain technology to improve the operational efficiency and security of electronic waybills. And use machine learning algorithms to analyze garbage truck electronic waybills and status data to predict important transportation nodes, thereby providing decision support for the management of electronic waybills and garbage truck transportation. However, there is no systematic research in the field of anomaly detection of electronic waybills. Existing research on logistics electronic waybills is based on the same assumption: there is nothing wrong with the waybill data itself and it can be studied directly. However, this is not the case. In actual business scenarios, there are many redundant data such as invalid waybills and abnormal waybills, which will interfere with research results and damage economic interests. Therefore, this invention applies the data stream anomaly detection algorithm to the anomaly detection of logistics waybills for the first time, focusing the research on the electronic waybill data itself.

[0055] Data stream anomaly detection algorithms are divided into three categories: data stream anomaly detection algorithms based on offline learning, semi-online learning, and online learning.

[0056] The data stream anomaly detection algorithm based on offline learning refers to using all historical data in the data set to train the model offline to obtain a model that can process data streams and will not change, so it cannot resist concept drift. In the semi-online data stream anomaly detection algorithm, the model can be re-trained periodically in batches to update the accuracy of the model or adapt to the changes of new anomalies. Therefore, the data stream anomaly detection algorithm based on offline learning and semi-online learning can use traditional offline learning algorithms such as LOF, K-means, KNN, iForest, etc., or improve on them.

[0057] The data stream anomaly detection method based on online learning usually uses an incremental learning algorithm to adapt to the changes in the data stream by constantly updating the model. Compared with the data stream anomaly detection algorithm based on offline learning and semi-online learning, it has better real-time performance and lower storage cost. Therefore, the present invention adopts an online learning method for the anomaly detection of the logistics electronic waybill data stream, and adopts EFAT, which also uses the Hoeffding tree structure in the online shallow learning algorithm. At present, the most commonly used data stream anomaly detection algorithms based on the Hoeffding tree structure are VFDT, HAT, and EFDT. In general, the VFDT algorithm is fast, but cannot resist concept drift and has low accuracy; the HAT algorithm has low storage cost, medium speed, accuracy, and resistance to concept drift; the EFDT algorithm is slow, has poor resistance to concept drift, but has high accuracy. Therefore, the present invention proposes a new data stream anomaly detection algorithm EFAT based on the characteristics of this type of algorithm, which improves the accuracy of the algorithm and the ability to resist concept drift under the premise of rapid detection of anomalies.

[0058] In summary, in order to effectively adapt to the big data environment, logistics waybill anomaly detection technology must have the characteristics of real-time rapid analysis, accurate anomaly detection, and resistance to concept drift. This paper applies a number of data stream anomaly detection algorithms based on online learning to the business scenario of waybill anomaly detection in the logistics field, and proposes a new data stream anomaly detection algorithm, which has better detection speed, accuracy and ability to resist concept drift than most existing algorithms, and verifies it using real data from a logistics company's actual business scenario.

[0059] In one embodiment of the present invention, a method for detecting anomalies in logistics waybills in a data stream environment is provided. Figure 1 As shown, the method comprises the following steps:

[0060] 1) After preprocessing the original waybill data and original trajectory data of the logistics company, positive and negative samples are selected;

[0061] 2) Integrate negative samples and positive samples to generate concept drift datasets and non-concept drift datasets. Use the non-concept drift dataset to simulate the concept drift of waybill anomalies, and select features from each dataset that can fully reflect the relationship between waybill and trajectory;

[0062] 3) The selected features are processed using the EFAT algorithm to obtain anomaly detection results.

[0063] In the above step 1), the original waybill data of the logistics company is pre-processed, including the following steps:

[0064] 1.1.1) Filter the number of original waybill data based on geographic information, and limit the waybill to a set range by filtering by latitude and longitude and province, city, and district. At the same time, limit the attributes related to the starting and ending points of the waybill to non-null values ​​to eliminate invalid or incomplete data, reduce the number of waybill data, and ensure that waybill records are not repeated;

[0065] For example, the original waybill data contained 13,351 records. By filtering by longitude and latitude and province, city, and district, the waybill was restricted to China. At the same time, the attributes related to the starting and ending places of the waybill were restricted to non-null values, thereby eliminating invalid or incomplete data and reducing the number of waybills to 11,687. During the processing, possible data duplication was filtered out to ensure the uniqueness of the data set, and finally 11,687 non-duplicate waybill records were obtained.

[0066] 1.1.2) Filter the number of vehicles and waybills, exclude vehicles with a value less than the first set threshold, further exclude data with transportation time and transportation mileage exceeding the set range, exclude abnormal trips, and obtain the final number of vehicles and waybill data;

[0067] For example, if the first threshold is set to 10, vehicles with less than 10 waybills are excluded, leaving 85 vehicles and 11,530 waybills. After further eliminating data with too long or too short transportation time and transportation mileage, abnormal trips are excluded, and finally 84 vehicles and 9,676 waybills are obtained.

[0068] In the above step 1), the original trajectory data of the logistics company is preprocessed, including the following steps:

[0069] 1.2.1) For multiple track records contained in the original track data, by matching the truck numbers with each other, and matching the trigger time with the start time and end time in the waybill data, the track record data related to the valid waybill is screened out to reduce the number of track records;

[0070] For example, the original trajectory dataset contains 15,834,372 trajectory records; by matching the truck number truckno, and matching the trigger_time with the start_time and end_time in the waybill dataset, the trajectory data related to the valid waybill are filtered out, and 10,026,907 trajectory data are obtained.

[0071] 1.2.2) Limit the truck speed to be greater than the second set threshold, filter out possible worthless redundant trajectory data points, and further reduce the number of trajectory records;

[0072] In this embodiment, the second set threshold is preferably 20, so the truck speed is limited to be greater than 20, and possible worthless redundant trajectory data points are filtered out, and finally 8495119 trajectory data are obtained.

[0073] 1.2.3) Filter the remaining trajectory records by latitude and longitude and province, city and district, limit the trajectory to the set range, and limit the trajectory province, city and district related attributes to non-null values, exclude invalid or abnormal trajectory data, further reduce the number of trajectory records, and deduplicate the trajectory data to ensure the consistency of the data set, and obtain the final non-duplicate trajectory records.

[0074] For example, by filtering by longitude and latitude and province, city, and district, the trajectories were restricted to China. At the same time, the province, city, and district related attributes of the trajectories were restricted to non-null values, excluding invalid or abnormal trajectory data, reducing the number of data items to 4,167,166 (9,462 waybill trajectories). The trajectory data was deduplicated to ensure the consistency of the data set, resulting in 4,167,166 (9,462 waybill trajectories) non-duplicate trajectory records.

[0075] In the above step 1), the selection of positive and negative samples includes the following steps:

[0076] 1.3.1) The real waybill-track data collected from the business side of a large logistics company is used as the positive sample. The positive sample has a high degree of authenticity and reliability. The positive sample includes correct and complete waybill-related information and corresponding track data, which can reflect the normal behavior in the actual transportation process;

[0077] 1.3.2) Since the logistics company does not record abnormal waybill-related data as part of its business in actual business, this embodiment uses artificial generation to generate negative samples to simulate potential abnormal situations; the logic for generating negative samples includes: negative samples of trajectory mismatch, negative samples of delivery timeout, and negative samples of lost waybills.

[0078] Specifically, negative samples with mismatched trajectories: for example, 1,000 positive waybills are randomly selected, and the longitude and latitude of their starting and ending points are randomly offset (within the range of 10km-30km) to generate new starting and ending points to form negative samples with mismatched trajectories, in order to simulate the abnormal situation where the waybill information does not match the actual trajectory.

[0079] Negative samples of delivery timeout: For 1,000 positive sample waybills, the maximum trigger_time (trigger time) of each waybill data is randomly increased, and supplementary data is filled in within the set time range to simulate the abnormal situation of delivery timeout.

[0080] Negative samples of lost waybills: 1,000 positive waybills are randomly selected, and their trajectory-related information is retained, but the waybill-related information is deleted or replaced to simulate the abnormal situation of lost waybills or empty trucks.

[0081] The reason why this embodiment adopts the method of artificially generating negative samples is that the present invention needs to simulate abnormal scenarios that may occur in real business, but these abnormal situations are not recorded in the real data. The three abnormal situations simulated in this embodiment are the three most common and most frequently occurring abnormal modes of waybills in the logistics business layer. Through artificial generation, negative samples with a certain complexity and diversity can be constructed in a targeted manner, ensuring that the model can fully learn and adapt to various abnormal situations during training, which helps to improve the model's generalization ability for potential abnormalities.

[0082] In this embodiment, there are 4,167,166 (9,462 waybills) of trajectory records as positive samples by matching the waybill data with the trajectory data. In addition, 1,334,221 (3,000 waybills) of trajectory records are added as negative samples to form the final waybill-trajectory dataset.

[0083] Carefully processed waybill and trajectory datasets provide a reliable and clean actual business data foundation and ensure accurate modeling of logistics waybill anomaly detection.

[0084] In the above step 2), the negative samples and the positive samples are integrated to generate a concept drift dataset and a non-concept drift dataset. Specifically, the distribution of positive samples and negative samples is restricted to form a non-concept drift dataset data_unconceptdrift and a concept drift dataset data_conceptdrift. In the non-concept drift dataset data_unconceptdrift, negative samples are randomly distributed at various positions.

[0085] In this embodiment, the distribution of the concept drift data set is specifically as follows: in the concept drift data set, the negative samples of trajectory mismatch are normally distributed near the set first positive sample, the negative samples of delivery timeout are normally distributed near the set second positive sample, and the negative samples of lost waybills are normally distributed near the third positive sample, so as to simulate the concept drift of waybill anomalies.

[0086] Preferably, the first positive sample is the 2000th positive sample; the second positive sample is the 5000th positive sample; and the third positive sample is the 8000th positive sample.

[0087] In this embodiment, since the waybill is the core document in the logistics transportation process, it only records the status of the start and end of transportation in a relatively static manner, and lacks relevant information of the corresponding transportation process. This leads to the objective anomalies (such as extreme weather, system errors, etc.) and subjective anomalies (such as driver fraud, hitchhiking, etc.) that occur during the transportation process. It is impossible to accurately judge the relevant information provided by the waybill alone. Therefore, in order to accurately identify the anomalies of the waybill, the trajectory data of the transportation process is used in this embodiment. These trajectory data are real business data collected by the vehicle tracking system (VTS) of a large domestic logistics company, and the trajectory data is combined with the corresponding waybill data to produce a logistics waybill-trajectory data set, so as to accurately identify anomalies.

[0088] The vehicle tracking system (VTS) device can collect relevant data and location information of the vehicle and transmit this data to the service provider or company, so as to realize global tracking of the vehicle. In this embodiment, the real-time location data of the vehicle is collected by the VTS system. The VTS can generate a series of timestamps and location data of the vehicle, that is, spatiotemporal data, and generate the vehicle transportation trajectory with these spatiotemporal data. The trucks of logistics companies usually carry VTS devices, which can collect vehicle operation data received from the vehicle's on-board diagnostic (OBD) interface, as well as time and geographic location information generated by GNSS chips such as GPS and GLONASS.

[0089] Equipment on vehicles and trajectory data collection: The mandatory installation of VTS equipment on Chinese freight vehicles makes it possible to study the operation mode of freight vehicles using trajectory data containing information such as time, space and speed. The mining of trajectory data not only reduces the time and cost of traditional freight data surveys, but also ensures the quality of freight data.

[0090] Dataset structure: This example is based on real business data provided by a logistics company, where the data is divided into two parts: waybill data and track data. The data structure is shown in Table 1 and Table 2.

[0091] Table 1 Data structure of waybill data

[0092] Property Name Data Types Attribute meaning truckno Character Truck number recorded in the waybill start_time Time Type Waybill start time end_time Time Type Waybill expiration time start_province Text Waybill origin (provincial level) start_city Text Waybill origin (city level) start_district Text Waybill origin (county or district level) start_poi_lng Floating point Longitude of the waybill's origin start_poi_lat Floating point Latitude of the waybill's origin end_province Text Waybill Termination Location (Provincial Level) end_city Text Waybill Ending Place (City Level) end_district Text Waybill end place (county or district level) end_poi_lng Floating point Longitude of waybill end point end_poi_lat Floating point Waybill end point latitude run_hours Floating point The transit time recorded in the waybill run_km Floating point The transportation mileage recorded in the waybill

[0093] Table 2 Data structure of trajectory data

[0094]

[0095] After preprocessing, matching, labeling, etc., the waybill data and the trajectory data are subjected to operations such as preprocessing, matching, and labeling, a waybill-trajectory dataset is generated as the only dataset used in this embodiment, and its data structure is shown in Table 3.

[0096] Table 3 Data structure of waybill-track data

[0097]

[0098]

[0099] In the above step 2), constructing a feature matrix is ​​a prerequisite for anomaly detection so that the model can accurately capture the features of waybill anomalies. In order to improve the interpretability of the results and better solve problems for the actual business side, this embodiment adopts a manual selection method to select a series of features with business interpretability. Specifically, features that can fully reflect the relationship between waybills and trajectories are selected from each data set, including:

[0100] 2.1) Confidence: By counting the number of track points a in the starting time period of the waybill and the number of track points b in the corresponding time periods of the starting and ending points, the value of a / b is calculated to reflect the consistency between the waybill information and the track information. A high confidence indicates a high degree of match between the two.

[0101] 2.2) Start and end point matching (res), check whether the start and end points of the waybill appear in all the track points of the corresponding track to determine whether the start and end points of the waybill are completely included in the track;

[0102] 2.3) Origin offset distance (distance_km_x), calculates the distance between the origin recorded in the waybill and the actual location of the track point corresponding to the waybill start time, to reveal the deviation of the actual location of the origin from the waybill record;

[0103] 2.4) End point offset distance (distance_km_y), similar to the start point offset distance, calculates the distance between the end point of the waybill record and the actual position of the track point corresponding to the waybill end time;

[0104] 2.5) Waybill distance (run_km), using the distance information recorded in the waybill as the driving distance feature of the waybill;

[0105] 2.6) Distance between the start and end points (distance_shapping_line_mean), calculates the straight-line distance between the start and end points of the waybill record on the track to provide a global reference for the length of the waybill track;

[0106] 2.7) Run Hours: Use the time information recorded in the waybill to indicate the running time of the waybill;

[0107] 2.8) Time length of the entire track (time_shapping_line_mean), which calculates the time length recorded by the trigger time of the actual first track point in the waybill and the trigger time of the last track point, to provide a perspective on the actual duration of the waybill as a whole;

[0108] 2.9) Tags are used to distinguish positive and negative samples and provide supervised learning targets for subsequent anomaly detection models.

[0109] The features selected in this embodiment can fully reflect the relationship between the waybill and the trajectory. Through these features, the EFAT algorithm can perform in-depth anomaly detection on abnormal situations. At the same time, the selection of these features is to strike a balance between business interpretability and model effect, to ensure that anomaly detection results with strong interpretability and high practical application value can be provided, and to assist enterprises in making relevant decisions.

[0110] In the above step 3), EFAT is a decision tree algorithm based on the Hoeffding tree principle, which has the flexibility of the Hoeffding tree and the rapid adaptability to changing data; EFAT transforms the Hoeffding tree into an adaptive tree structure by referring to the adaptive window principle ADWIN, and optimizes the splitting mechanism of the leaf nodes in the Hoeffding tree algorithm, making the algorithm more efficient and accurate, so as to achieve better performance in high-speed data streams.

[0111] Among them, the decision tree is a classification algorithm. Each internal node of the tree DT contains a test for an attribute, each branch of the node corresponds to a possible result of the test, and each leaf contains a class prediction. For example, the label y=DT(x) of x is obtained by passing the example from the root down to the leaves, testing the appropriate attribute at each node and tracking the branches corresponding to the attribute values ​​in the example. Nodes contain more complex tests and leaves may also contain extended models of more complex classification rules.

[0112] The Hoeffding tree is learned top-down, starting from the root and recursively replacing leaves with test nodes. The attribute to be tested at the node is selected by comparing all available attributes and selecting the best attribute based on a heuristic metric. At the same time, the establishment of the Hoeffding tree is based on the fact that small samples are often sufficient to select the best splitting attribute. The Hoeffding bound quantifies the number of observations required to estimate some statistics within a specified accuracy. More precisely, the Hoeffding bound states that with probability 1-δ, the difference between the true mean of a random variable with range R and the estimated mean after n independent observations will not be greater than:

[0113]

[0114] Where ∈ is the Hoeffding bound and δ is the probability.

[0115] The Hoeffding tree uses the Hoeffding bound to prove that its output is asymptotically identical to the output of a non-incremental learner using infinitely many examples. Therefore, the Hoeffding tree has good performance guarantees. The VFDT algorithm is an implementation of the Hoeffding tree.

[0116] Therefore, in this embodiment, the selected features are processed using the EFAT algorithm to obtain anomaly detection results, including the following steps:

[0117] 3.1) Compare whether the difference between the best feature attribute and the second best feature attribute is greater than the Hoeffding limit. If so, split the leaf node on the best feature attribute to generate a new child node. If the difference between the best feature attribute and the second best feature attribute does not meet the condition, further compare the difference between the best feature attribute and the current split feature attribute.

[0118] 3.2) If the best feature attribute is the most frequent category in the data set, replace the current internal node with a leaf node and delete its subtree;

[0119] 3.3) If the best feature attribute is not the current split feature attribute, update the split feature attribute and replace the current internal node with a new internal node split on the best feature attribute;

[0120] 3.4) At the same time, an ADWIN detector is established on each leaf node to monitor changes in the statistical information of the node; when a new data sample arrives, the EFAT node will update its statistical information, and at the same time, these statistical information will be input into the node's ADWIN detector; the ADWIN detector continuously monitors changes in the input data. When changes in statistical characteristics are detected, it indicates that concept drift may have occurred, and the window changes adaptively. Subsequently, the EFAT tree performs adaptive pruning operations to optimize the tree structure and obtain more accurate anomaly detection results.

[0121] In this embodiment, ADWIN is a change detector and estimator that solves the problem of tracking the average of a bit stream or a real-valued stream in an explicit way. ADWIN maintains a variable-length window of recently seen items, with the property that the maximum length of the window is statistically consistent with the assumption that the average within the window does not change.

[0122] The ADWIN method is as follows: whenever two “large enough” subwindows of the window W show a “significant enough” difference in average value, it can be concluded that the corresponding expected values ​​are different, and the old part of the window is discarded. The meaning of “large enough” and “significant enough” can be further refined by using the Hoeffding bound. The test ultimately boils down to whether the average of the two subwindows is greater than the variable value ∈ calculated as follows cut :

[0123]

[0124]

[0125] ADWIN defines ∈ cut At each time step, the following two steps are performed:

[0126] (1) False positive rate limit: If μ t If ∈ W remains unchanged, the probability that ADWIN shrinks the window in this step is at most δ.

[0127] (2) False negative rate bound: Assume that for two partitions of W, W0 and W1 (where W1 contains the most recent item), if ADWIN will reduce W to W1 or shorter with a probability of 1-δ.

[0128] ADWIN acts as a change detector, in that it shrinks its window if and only if a significant change has occurred recently (with high probability); and as an estimator of the current mean of the sequence it is reading, in that with high probability old parts of the window with significantly different means are automatically removed.

[0129] ADWIN is parameter-free and assumption-free because it can automatically detect and adapt to the current rate of change. Its only parameter is the confidence bound δ, which indicates how confident we are in the output of the algorithm, which is inherent in all algorithms that deal with random processes.

[0130] If the goal is to build an incremental learner with good predictive power at any given point in the stream of instances, it may be necessary to build a structure that exploits information as it becomes available, but then needs to make subsequent corrections when better alternatives are discovered. In scenarios where the distribution of information between attributes is skewed, with some attributes containing more information than others, such a strategy can be very effective because the cost of rebuilding the tree when replacing high-level attributes with highly informative ones is limited. However, the Hoeffding Tree will have difficulty splitting when information is more evenly distributed between attributes, and may have to rely on the number of random variables to enforce the decision threshold, while EFAT will choose one attribute to start with and switch when necessary, allowing for faster learning.

[0131] EFAT uses the Hoeffding bound to determine if the split parameter of the best attribute exceeds the split parameter of the attribute that is not to be split, as well as the split parameter of the current split attribute. If there is no splitting attribute on the node, rather than splitting only when the best candidate splitting attribute is better than the second best candidate attribute, EFAT will split when the information gained due to the best candidate split is non-zero at the required confidence level. At a later stage, EFAT will split when the information gain difference between the current best attribute and the current splitting attribute is non-zero.

[0132] In one embodiment of the present invention, a logistics waybill anomaly detection system in a data stream environment is provided, which includes:

[0133] The sample selection module pre-processes the original waybill data and original trajectory data of the logistics company and selects positive and negative samples;

[0134] The integration module integrates negative samples and positive samples to generate concept drift datasets and non-concept drift datasets. The non-concept drift datasets simulate the concept drift of abnormal waybills, and select features from each dataset that can fully reflect the relationship between waybills and trajectories.

[0135] EFAT module,processes the selected features using EFAT algorithm to obtain anomaly detection results.

[0136] In the above embodiment, the original waybill data of the logistics company is pre-processed, including:

[0137] Filter the number of original waybill data based on geographic information, and limit the waybill to the set range by filtering by latitude and longitude and province, city and district. At the same time, limit the related attributes of the waybill start and end points to non-null values ​​to eliminate invalid or incomplete data, reduce the number of waybill data, and ensure that waybill records are not repeated;

[0138] The number of vehicles and waybills is screened, vehicles with a value less than the first set threshold are excluded, and data with a transportation time and a transportation mileage exceeding the set range are further eliminated, and abnormal travel is excluded to obtain the final number of vehicles and waybill data items;

[0139] Preprocess the original trajectory data of the logistics company, including:

[0140] For multiple track records contained in the original track data, by matching the truck numbers with each other and matching the trigger time with the start time and end time in the waybill data, the track record data related to the valid waybill is screened out to reduce the number of track records;

[0141] Limiting the truck speed to be greater than the second set threshold, filtering out possible worthless redundant trajectory data points, and further reducing the number of trajectory records;

[0142] The remaining trajectory records are filtered by latitude and longitude and province, city and district, and the trajectories are limited to the set range. At the same time, the province, city and district related attributes of the trajectory are limited to non-null values, invalid or abnormal trajectory data are excluded, the number of trajectory records is further reduced, and the trajectory data is deduplicated to ensure the consistency of the data set, and the final non-duplicate trajectory records are obtained.

[0143] In the above embodiment, the selection of positive and negative samples includes:

[0144] The real waybill-track data collected from the business side of a large logistics company is used as the positive sample. The positive sample includes correct and complete waybill-related information and corresponding track data, which can reflect the normal behavior in the actual transportation process;

[0145] Negative samples are artificially generated to simulate potential abnormal situations. The logic for generating negative samples includes: negative samples of trajectory mismatch, negative samples of delivery timeout, and negative samples of lost waybills.

[0146] In the above embodiment, the negative samples and the positive samples are integrated to generate a concept drift data set and a non-concept drift data set, including:

[0147] By restricting the distribution of positive samples and negative samples, a dataset with no concept drift and a dataset with concept drift are formed; in the dataset with no concept drift, negative samples are randomly distributed at various positions.

[0148] In this embodiment, the distribution of the concept drift data set includes:

[0149] In the concept drift dataset, negative samples with mismatched trajectories are normally distributed near the set first positive sample, negative samples with timeouts are normally distributed near the set second positive sample, and negative samples with lost waybills are normally distributed near the third positive sample, to simulate the concept drift of waybill anomalies.

[0150] In the above embodiment, features that can fully reflect the relationship between the waybill and the track are selected from each data set, including:

[0151] Confidence: by counting the number of track points a in the starting time period of the waybill and the number of track points b in the corresponding time periods of the starting and ending points, the value of a / b is calculated to reflect the consistency of the waybill information and the track information;

[0152] Start and end point matching: check whether the starting and ending points of the waybill appear in all the track points of the corresponding track to determine whether the starting and ending points of the waybill are completely included in the track;

[0153] The starting point offset distance calculates the distance between the starting point recorded in the waybill and the actual position of the track point corresponding to the waybill start time to reveal the deviation of the actual position of the starting point from the waybill record;

[0154] The end point offset distance is calculated by calculating the distance between the end point of the waybill record and the actual position of the track point corresponding to the waybill end time;

[0155] Waybill distance: using the distance information recorded in the waybill as the driving distance feature of the waybill;

[0156] Straight-line distance between the start and end points: calculate the straight-line distance between the start and end points of the waybill record on the track to provide a global reference for the length of the waybill track.

[0157] Running time, using the time information recorded on the waybill, indicates the running time of the waybill;

[0158] The total track time length calculates the time length recorded by the trigger time of the actual first track point and the trigger time of the last track point in the waybill to provide a perspective on the actual overall time length of the waybill;

[0159] Labels are used to distinguish positive and negative samples.

[0160] In the above embodiment, the selected features are processed using the EFAT algorithm to obtain anomaly detection results, including:

[0161] Compare whether the difference between the best feature attribute and the second best feature attribute is greater than the Hoeffding limit. If it is greater, split the leaf node on the best feature attribute to generate a new child node; if the difference between the best feature attribute and the second best feature attribute does not meet the condition, further compare the difference between the best feature attribute and the current split feature attribute;

[0162] If the best feature attribute is the most frequent category in the data set, replace the current internal node with a leaf node and delete its subtree;

[0163] If the best feature attribute is not the current split feature attribute, then the split feature attribute is updated, and the current internal node is replaced by a new internal node split on the best feature attribute;

[0164] At the same time, an ADWIN detector is established on each leaf node to monitor changes in the statistical information of the node; when a new data sample arrives, the EFAT node will update its statistical information, and at the same time, these statistical information will be input into the node's ADWIN detector; the ADWIN detector continuously monitors changes in input data. When changes in statistical characteristics are detected, it indicates that concept drift may have occurred, and the window undergoes adaptive changes. Subsequently, the EFAT tree performs adaptive pruning operations to optimize the tree structure and obtain more accurate anomaly detection results.

[0165] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0166] The embodiment is used to introduce in detail the effect of the EFAT algorithm of the present invention in the task of anomaly detection of logistics waybill data. First, the experimental environment and experimental settings are introduced, including the data set used, algorithm parameter settings, etc.; then the experimental results are compared and analyzed from the aspects of vertical comparison of the same type of algorithms, horizontal comparison of different types of algorithms, and self-ablation experiments, to evaluate the performance and effectiveness of the EFAT algorithm in anomaly detection of logistics waybill data.

[0167] (1) Experimental environment and experimental settings:

[0168] (1.1) Experimental environment: This experiment is run on a laptop with 16GB memory, Intel(R) Core(TM) i5-10210U CPU, 500G hard disk and 64-bit Windows 11 operating system. The software environment of the experiment is Dell PowerEdge R740xd server, the programming language is Python, using Python 3.9 version, and the experimental simulation is carried out on Jupyter Notebook.

[0169] (1.2) Experimental setup: The experimental setup mainly includes longitudinal comparison experiments of the same type of algorithms, horizontal comparison experiments of different types of algorithms, and EFAT self-ablation experiments.

[0170] (1.2.1) Longitudinal comparison experiment of the same type of algorithms: In the longitudinal comparison experiment of the same type of algorithms, the EFAT algorithm is compared with the same type of "data stream anomaly detection algorithm based on online learning-based on online shallow learning-based on decision tree principle". The compared algorithms include VFDT, HAT and EFDT algorithms.

[0171] (1.2.2) Horizontal comparison experiment of different types of algorithms: In the horizontal comparison experiment of different types of algorithms, the EFAT algorithm is compared horizontally with different types of data stream anomaly detection algorithms, including three types of algorithms: offline learning, semi-online learning, and online learning. The compared algorithms include KNN, HST, BI, and OB.

[0172] (1.2.3) EFAT self-ablation experiment: The self-ablation experiment aims to verify the effectiveness of the adaptive sliding window ADWIN principle used in the EFAT algorithm. The EFAT algorithm without ADWIN is named EFAT-a, and a comparative experiment is conducted with EFAT itself to evaluate the role of ADWIN in the EFAT algorithm.

[0173] (2) Experimental results and comparative analysis: The data stream anomaly detection algorithm in the experimental setting is applied to the constructed logistics waybill-trajectory dataset, and their performance in the anomaly detection task is evaluated through five evaluation indicators: Accuracy, Precision, Recall, F1 score and running time. By comparing the experimental results of different algorithms, the superiority and adaptability of the EFAT algorithm in logistics waybill anomaly detection are verified.

[0174] (2.1) Longitudinal comparison experiment of the same type of algorithms: The EFAT algorithm is compared with other online learning algorithms based on the Hoeffding tree principle. The VFDT, HAT, EFDT and EFAT algorithms are considered from two aspects: the anomaly detection effect and the ability to resist concept drift. The experiment uses two different data sets, data_unconceptdrift and data_conceptdrift, to compare the ability of the observation algorithm to resist concept drift. Now a brief introduction to many comparison algorithms:

[0175] (2.1.1) VFDT (Very Fast Decision Tree): Hoeffding tree is the first algorithmic principle that can mine from infinite data streams. The algorithm proposed based on this principle is called VFDT, which requires very low computing power. The algorithm generates decision trees in real time and can maintain similar performance to offline decision trees. VFDT constructs Hoeffding trees incrementally to classify new data, and is suitable for processing large-scale, high-speed changing data streams.

[0176] (2.1.2) HAT (Hoeffding Adaptive Tree): The HAT algorithm is also a data stream anomaly detection method based on the Hoeffding tree principle. The algorithm can adaptively learn from the data stream that changes over time. Its principle is to place a frequency statistics estimator instance at each node as an adaptive sliding window to maintain efficient learning performance in the data stream.

[0177] (2.1.3) EFDT (Extremely Fast Decision Tree): EFDT is also a data stream anomaly detection algorithm based on the Hoeffding tree principle. It allows the tree to grow faster by having less restrictive splitting criteria, and tries to get closer to the progressive batch decision tree by re-evaluating the nodes that have been split. EFDT is designed to handle large-scale, high-speed changing data streams, and its characteristic is that it trades high energy consumption for accuracy.

[0178] Table 4. Results of longitudinal comparison experiments on the same type of algorithms

[0179]

[0180] The final results of the experimental indicators are shown in Table 4. From the results in Table 4, it can be observed that, in the case of no concept drift, although the indicators of EFAT are not optimal except for Precision, its Accuracy ranks second among all algorithms. Compared with the EFDT algorithm, EFAT greatly saves the running time, while the F1 score does not decrease significantly. In addition, in the comparison with the HAT algorithm, while ensuring the running time is at the same level, the F1 score is much higher than that of HAT. When facing the scenario with concept drift, EFAT is optimal in almost all indicators. It can be seen that the EFAT algorithm has greatly improved the ability of the algorithm to resist concept drift while ensuring the anomaly detection performance.

[0181] The dynamic distribution of various indicators when data_unconceptdrift is used as the data set (no concept drift) is as follows Figure 2 As shown in Figure 2, the dynamic distribution of various indicators when data_conceptdrift is used as the data set (with concept drift) is as follows Figure 3 As shown. Figure 3 It can be observed from the results that when facing three different concept drifts, the red (EFAT) curve of the Precision and F1 score indicators is always at the top, indicating that the EFAT algorithm can more quickly restore stable and accurate detection results when facing the interference of concept drift.

[0182] (2.2) The EFAT algorithm is compared horizontally with other three types of data stream anomaly detection algorithms including offline learning, semi-online learning, and online learning. The KNN, HST, BI, OB and EFAT algorithms are also considered from the two aspects of anomaly detection effect and the ability to resist concept drift.

[0183] (2.2.1) KNN algorithm: KNN algorithm is a classic anomaly detection algorithm based on offline learning. The anomaly detection principle of KNN algorithm is that for a new data point, the distance between it and all data points in the existing data set is calculated, the K data points with the closest distance are selected, and these K data points are regarded as a local subspace. The average distance between these K data points and the new data point is calculated to determine whether the new data point belongs to the local subspace. If the average distance between the new data point and the subspace is greater than a certain threshold, it is judged as an anomaly point.

[0184] (2.2.2) HST algorithm: HST is a classic data stream anomaly detection algorithm based on offline learning. Its principle is based on dividing the data space into different half-spaces and using these half-spaces to detect anomalies. When a new data point arrives, the algorithm searches down along the tree structure and determines whether it is abnormal based on factors such as the half-space where the data point is located and the distance from the half-space.

[0185] (2.2.3) BI algorithm: BI is a data stream anomaly detection model based on semi-online learning, which uses a fixed sliding window and processes the data stream in a batch processing manner. BI uses a decision tree as the algorithm in the model, stores the features, labels, and sample weights of each instance in the window, and trains a new model and adds it to the ensemble when the window is full, which is achieved by gradually building an ensemble of batch model instances.

[0186] (2.2.4) OB algorithm: OB is an ensemble method based on online learning. They are improvements of the well-known batch setting Bagging ensemble method and can effectively process data streams. It trains multiple models on a constantly changing data stream and then votes or averages their outputs to improve the overall robustness and generalization performance.

[0187] Table 5 Horizontal comparison experimental results of different types of algorithms

[0188]

[0189]

[0190] The final results of the experimental indicators are shown in Table 5. From the results in Table 5, it can be observed that whether facing non-concept drift scenarios or concept drift scenarios, the effect of the EFAT algorithm is the best and it takes the least time.

[0191] The dynamic distribution of various indicators when data_unconceptdrift is used as the data set (no concept drift) is as follows Figure 4 As shown in Figure 2, the dynamic distribution of various indicators when data_conceptdrift is used as the data set (with concept drift) is as follows Figure 5 As shown. Figure 5 It can be observed from the results that when the F1 score indicator faces three different concept drifts, the purple (EFAT) curve is always at the top, indicating that the EFAT algorithm can more quickly restore stable and accurate detection effects when faced with the interference of concept drift.

[0192] (2.3) Algorithm self-ablation experiment: Verify the effectiveness of the adaptive sliding window principle used in the EFAT algorithm. The EFAT algorithm without the adaptive sliding window principle is named EFAT-a, and a comparative experiment is conducted with EFAT itself. The various indicators of the experimental results are shown in Table 6.

[0193] Table 6 Algorithm ablation experiment results

[0194]

[0195] From the results in Table 6, we can observe that, whether facing a non-concept drift scenario or a concept drift scenario, the effect of the EFAT algorithm after adding the adaptive sliding window is optimal. At the same time, the addition of the adaptive sliding window greatly reduces the algorithm time.

[0196] The dynamic distribution of various indicators when data_unconceptdrift is used as the data set (no concept drift) is as follows Figure 6 As shown in Figure 2, the dynamic distribution of various indicators when data_conceptdrift is used as the data set (with concept drift) is as follows Figure 7 As shown. Figure 7 It can be observed from the results that when the Precision and F1 score indicators face three different concept drifts, the yellow (EFAT) curve is always at the top, indicating that the EFAT algorithm can more quickly restore stable and accurate detection effects when faced with the interference of concept drift.

[0197] In summary, the present invention firstly establishes a unique spatiotemporal data set containing the actual business data of logistics companies, which provides sufficient data support for research in this field and provides a basis for subsequent related research. Secondly, the data stream anomaly detection algorithm is applied to logistics waybill anomaly detection, especially the introduction of the new EFAT algorithm, which shows good performance in the experiment, not only ensuring the accuracy and real-time performance of anomaly detection, but also significantly improving the algorithm's ability to resist concept drift.

[0198] From the comparative experiments, we can see that the EFAT algorithm not only shows excellent performance among the same type of algorithms, but also achieves the best results in the horizontal comparison with other types of algorithms. Especially in the case of concept drift, the EFAT algorithm shows better robustness and stability, can quickly adapt to data changes and maintain a high accuracy. In addition, the effectiveness of the adaptive sliding window principle used in the EFAT algorithm is verified through the EFAT algorithm's own ablation experiment. The results show that the EFAT algorithm with the addition of the adaptive sliding window is significantly better than the version without the principle in terms of performance, and the running time of the algorithm is greatly reduced, which further proves the importance and effectiveness of the principle.

[0199] A computing device provided in one embodiment of the present invention may be a terminal, which may include: a processor, a communications interface, a memory, a display screen, and an input device. The processor, the communications interface, and the memory communicate with each other through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory, wherein the non-volatile storage medium stores an operating system and a computer program, and the computer program is executed by the processor to implement the methods in the above embodiments; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computing device, or an external keyboard, touchpad or mouse, etc. The processor may call the logic instructions in the memory.

[0200] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0201] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0202] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores server instructions, and the computer instructions enable a computer to execute the methods provided in the above embodiments.

[0203] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0204] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A logistics waybill anomaly detection method in a data stream environment, characterized in that: include: After preprocessing the original waybill data and original trajectory data of the logistics company, positive and negative samples are selected; Integrate negative samples and positive samples to generate concept drift datasets and non-concept drift datasets. Use the non-concept drift dataset to simulate the concept drift of waybill anomalies, and select features from each dataset that can fully reflect the relationship between waybill and trajectory. The selected features are processed using the EFAT algorithm to obtain anomaly detection results, including: Compare whether the difference between the best feature attribute and the second best feature attribute is greater than the Hoeffding limit. If it is greater, split the leaf node on the best feature attribute to generate a new child node; if the difference between the best feature attribute and the second best feature attribute does not meet the condition, further compare the difference between the best feature attribute and the current split feature attribute; If the best feature attribute is the most frequent category in the data set, replace the current internal node with a leaf node and delete its subtree; If the best feature attribute is not the current split feature attribute, then the split feature attribute is updated, and the current internal node is replaced by a new internal node split on the best feature attribute; At the same time, an ADWIN detector is established on each leaf node to monitor changes in the statistical information of the node; when a new data sample arrives, the EFAT node will update its statistical information, and at the same time, these statistical information will be input into the node's ADWIN detector; the ADWIN detector continuously monitors changes in input data. When changes in statistical characteristics are detected, it indicates that concept drift has occurred and the window has adaptively changed. Subsequently, the EFAT tree performs adaptive pruning operations to optimize the tree structure and obtain more accurate anomaly detection results.

2. The method for detecting anomalies in logistics waybills under a data stream environment as claimed in claim 1, characterized in that: Pre-process the original waybill data of the logistics company, including: Filter the number of original waybill data based on geographic information, and limit the waybill to the set range by filtering by latitude and longitude and province, city and district. At the same time, limit the related attributes of the waybill start and end points to non-null values ​​to eliminate invalid or incomplete data, reduce the number of waybill data, and ensure that waybill records are not repeated; The number of vehicles and waybills is screened, vehicles with a value less than the first set threshold are excluded, and data with a transportation time and a transportation mileage exceeding the set range are further eliminated, and abnormal travel is excluded to obtain the final number of vehicles and waybill data items; Preprocess the original trajectory data of the logistics company, including: For multiple track records contained in the original track data, by matching the truck numbers with each other and matching the trigger time with the start time and end time in the waybill data, the track record data related to the valid waybill is screened out to reduce the number of track records; Limiting the truck speed to be greater than the second set threshold, filtering out possible worthless redundant trajectory data points, and further reducing the number of trajectory records; The remaining trajectory records are filtered by latitude and longitude and province, city and district, and the trajectories are limited to the set range. At the same time, the province, city and district related attributes of the trajectory are limited to non-null values, invalid or abnormal trajectory data are excluded, the number of trajectory records is further reduced, and the trajectory data is deduplicated to ensure the consistency of the data set, and the final non-duplicate trajectory records are obtained.

3. The logistics waybill anomaly detection method in a data stream environment as claimed in claim 1, characterized in that: The selection of positive and negative samples includes: The real waybill-track data collected from the business side of a large logistics company is used as the positive sample. The positive sample includes correct and complete waybill-related information and corresponding track data, which can reflect the normal behavior in the actual transportation process; Negative samples are artificially generated to simulate potential abnormal situations. The logic for generating negative samples includes: negative samples of trajectory mismatch, negative samples of delivery timeout, and negative samples of lost waybills.

4. The method for detecting anomalies in logistics waybills under a data stream environment as claimed in claim 1, characterized in that: Integrate negative samples and positive samples to generate concept drift datasets and non-concept drift datasets, including: By restricting the distribution of positive samples and negative samples, a dataset with no concept drift and a dataset with concept drift are formed; in the dataset with no concept drift, negative samples are randomly distributed at various positions.

5. The method for detecting anomalies in logistics waybills under a data stream environment as claimed in claim 4, characterized in that: The distribution of concept drift datasets, including: In the concept drift dataset, negative samples with mismatched trajectories are normally distributed near the set first positive sample, negative samples with timeouts are normally distributed near the set second positive sample, and negative samples with lost waybills are normally distributed near the third positive sample, to simulate the concept drift of waybill anomalies.

6. The method for detecting anomalies in logistics waybills under a data stream environment as claimed in claim 1, characterized in that: Features that can fully reflect the relationship between waybills and trajectories are selected from each data set, including: Confidence: by counting the number of track points a in the starting time period of the waybill and the number of track points b in the corresponding time periods of the starting and ending points, the value of a / b is calculated to reflect the consistency of the waybill information and the track information; Start and end point matching: check whether the starting and ending points of the waybill appear in all the track points of the corresponding track to determine whether the starting and ending points of the waybill are completely included in the track; The starting point offset distance calculates the distance between the starting point recorded in the waybill and the actual position of the track point corresponding to the waybill start time to reveal the deviation of the actual position of the starting point from the waybill record; The end point offset distance is calculated by calculating the distance between the end point of the waybill record and the actual position of the track point corresponding to the waybill end time; Waybill distance: using the distance information recorded in the waybill as the driving distance feature of the waybill; Straight-line distance between the start and end points: calculate the straight-line distance between the start and end points of the waybill record on the track to provide a global reference for the length of the waybill track. Running time, using the time information recorded on the waybill, indicates the running time of the waybill; The total track time length calculates the time length recorded by the trigger time of the actual first track point and the trigger time of the last track point in the waybill to provide a perspective on the actual overall time length of the waybill; Labels are used to distinguish positive and negative samples.

7. A logistics waybill anomaly detection system in a data stream environment, characterized in that: include: The sample selection module pre-processes the original waybill data and original trajectory data of the logistics company and selects positive and negative samples; The integration module integrates negative samples and positive samples to generate concept drift datasets and non-concept drift datasets. The non-concept drift datasets simulate the concept drift of abnormal waybills, and select features from each dataset that can fully reflect the relationship between waybills and trajectories. The EFAT module processes the selected features using the EFAT algorithm to obtain anomaly detection results, including: Compare whether the difference between the best feature attribute and the second best feature attribute is greater than the Hoeffding limit. If it is greater, split the leaf node on the best feature attribute to generate a new child node; if the difference between the best feature attribute and the second best feature attribute does not meet the condition, further compare the difference between the best feature attribute and the current split feature attribute; If the best feature attribute is the most frequent category in the data set, replace the current internal node with a leaf node and delete its subtree; If the best feature attribute is not the current split feature attribute, then the split feature attribute is updated, and the current internal node is replaced by a new internal node split on the best feature attribute; At the same time, an ADWIN detector is established on each leaf node to monitor changes in the statistical information of the node; when a new data sample arrives, the EFAT node will update its statistical information, and at the same time, these statistical information will be input into the node's ADWIN detector; the ADWIN detector continuously monitors changes in input data. When changes in statistical characteristics are detected, it indicates that concept drift has occurred and the window has adaptively changed. Subsequently, the EFAT tree performs adaptive pruning operations to optimize the tree structure and obtain more accurate anomaly detection results.

8. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 6.

9. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 6.

Images