Abnormal data detection method, device, electronic device and storage medium

Abstract

The present invention discloses an abnormal data detection method, device, electronic device and storage medium, wherein the method comprises: obtaining full process data of cargo transportation; determining corresponding abnormal detection logic according to the transportation data type of the full process data; and detecting the full process data according to the abnormal detection logic to obtain abnormal data detection results. In the embodiment of the present invention, automatic detection is realized through the abnormal detection logic corresponding to the transportation data type of the full process data, reducing the influence of human factors on abnormal detection and improving the accuracy of abnormal transportation data detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer application technology, and in particular, to an abnormal data detection method, device, electronic device and storage medium. Background Art

[0002] With the development of the Internet, the Internet has gradually played an important role in the production, sales, transportation and other links of enterprises. Accurate transportation information is of great significance to enterprises in the transportation link. In order to improve the accuracy of transportation information, most enterprises use information quality control systems to manage abnormal information during transportation. When the waybill information is abnormal during transportation, the transportation personnel manually report the abnormal waybill to the quality control system. However, this processing method has the following technical problems:

[0003] 1) The detection of abnormal waybills mainly relies on the professional knowledge of the transport personnel and is greatly affected by their subjective understanding. The accuracy of the detection cannot be guaranteed, and there are cases where abnormal waybills are missed or misreported.

[0004] 2) After the abnormal waybill is detected, the transportation personnel are required to upload the abnormal waybill to the quality control system through the system website or terminal. The processing efficiency of abnormal waybill is low, and it takes up a lot of normal working time of the transportation personnel, reducing the company's cargo transportation efficiency. Summary of the invention

[0005] The present invention provides an abnormal data detection method, device, electronic device and storage medium to realize automatic detection of cargo transportation data, reduce the influence of human factors on abnormal detection, and improve the accuracy of abnormal transportation data detection.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting abnormal data, the method comprising:

[0007] Obtain full process data of cargo transportation;

[0008] Determine the corresponding anomaly detection logic according to the transport data type of the full process data;

[0009] The whole process data is detected according to the abnormality detection logic to obtain an abnormal data detection result.

[0010] In a second aspect, an embodiment of the present invention provides an abnormal data detection device, the device comprising:

[0011] Data collection module, used to obtain data on the entire process of cargo transportation;

[0012] A logic determination module, used to determine the corresponding anomaly detection logic according to the type of transport data of the full-process data;

[0013] The detection execution module is used to detect the whole process data according to the abnormality detection logic to obtain abnormal data detection results.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a memory for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the abnormal data detection method as described in any one of the embodiments of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the abnormal data detection method as described in any one of the embodiments of the present invention is implemented.

[0019] The embodiments of the present invention collect the whole process data of cargo transportation, determine the anomaly detection logic corresponding to the whole process data according to the type of transportation data, detect the whole process data according to the anomaly detection logic and obtain the anomaly data detection result. By collecting the whole process data of cargo transportation and detecting according to the anomaly data detection logic, the automatic detection of cargo transportation data is realized, the influence of human factors on the detection of cargo transportation data is reduced, and the accuracy of anomaly detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention;

[0021] Figure 2 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention;

[0022] Figure 3 This is an example diagram of full-process data acquisition provided by an embodiment of the present invention;

[0023] Figure 4 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention;

[0024] Figure 5 is an example diagram of an abnormal data detection method provided by an embodiment of the present invention;

[0025] Figure 6 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention;

[0026] Figure 7 This is an example diagram of data supplementation provided by an embodiment of the present invention;

[0027] Figure 8 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention;

[0028] Fig. 9 This is an example diagram of timeliness information detection provided by an embodiment of the present invention;

[0029] Fig.10 is an example diagram of data monitoring provided by an embodiment of the present invention;

[0030] Fig.11 is an example diagram of an abnormal data detection method provided by an embodiment of the present invention;

[0031] Fig.12 is a structural schematic diagram of an abnormal data detection device provided by an embodiment of the present invention;

[0032] Fig.13 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention are shown in the accompanying drawings, rather than all structures. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0034] Embodiment 1

[0035] Figure 1 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention. This embodiment can be applied to the case of detecting abnormal transportation data during cargo transportation. The method can be executed by an abnormal data detection device, which can be implemented in hardware and / or software. Figure 1 The method provided in the embodiment of the present invention specifically includes the following steps:

[0036] Step 110: Obtain the full process data of cargo transportation.

[0037] Among them, the full-process data can be the digital information generated in the process of cargo transportation. The full-process data can cover the digital information of all links of cargo transportation. The full-process data can be the information collected in different data systems of cargo transportation. For example, the full-process data can include order data, express station data, cargo identification data, and delivery status data.

[0038] Specifically, it can be connected with different data systems for cargo transportation, and the digital information collected from each system can be used as the full-process data for cargo transportation. For example, the data system for cargo transportation can send digital information to message channels such as Kafka message queues, RabbitMQ message queues, and ZeroMQ message queues, and the digital information can be collected in the message channels as the full-process data.

[0039] Step 120: Determine the corresponding anomaly detection logic according to the type of transportation data of the entire process data.

[0040] Among them, the transport data type may be information representing the attributes of the full-process data. For example, the full-process data may represent the cargo transportation status or cargo status during cargo transportation. When the full-process data can be divided into two categories according to the cargo transportation status and cargo status, the transport data type may include order information and package status. The anomaly detection logic may be a determination logic for determining whether the full-process data is abnormal data. The anomaly detection logic may be set in advance, and different anomaly detection logic may be set for different transport data types.

[0041] In an embodiment of the present invention, the received full-process data can be classified according to pre-set transport data types, and the corresponding anomaly detection logic can be determined for the transport data type under each classification. It can be understood that the transport data type and the anomaly detection logic can be associated with each other, for example, they can have the same identification number or be stored in an associated storage space.

[0042] Step 130: Detect the entire process data according to the anomaly detection logic to obtain an abnormal data detection result.

[0043] Among them, the abnormal data detection result can be the result information generated after anomaly detection is performed on the whole process data, and the abnormal data detection result can include the whole process data and an identification of whether the whole process data is abnormal data.

[0044] In an embodiment of the present invention, it is possible to determine whether the full-process data under the corresponding transportation data type complies with the acquired exception detection logic. If it is determined that the full-process data complies with the corresponding exception detection logic, the full-process data can be identified as abnormal data as an exception detection result. It can be understood that the method of identifying abnormal data can include logical identification and physical identification. For example, the full-process data determined as abnormal data can be stored in a storage space for abnormal data, and all the full-process data in the storage space can be used as an exception detection result.

[0045] The embodiments of the present invention obtain full-process data of cargo transportation, determine corresponding anomaly detection logic based on the operation data type of the full-process data, detect the full-process data according to the determined anomaly detection logic, and obtain the anomaly detection result, thereby realizing automatic detection of full-process data in the cargo transportation process, reducing the influence of human factors on anomaly detection, and improving the accuracy of abnormal data detection.

[0046] Further, based on the above-mentioned embodiments of the invention, the acquisition of the full-process data of cargo transportation includes: extracting business data from at least one freight transportation business system through a message channel, wherein the business data includes at least one of order data, waybill data, package data, and delivery data; and using each of the business data as the full-process data.

[0047] Among them, the message channel can be a message transmission channel connecting the business system of cargo transportation, which can be one or more of the message channels such as kafka message queue, RabbitMQ message queue and ZeroMQ message queue. The business system can be a digital system for realizing cargo transportation, which can include sorting and delivery system, transportation system, quality control system, etc. Order data can be contract data or document data for cargo transportation; waybill data can be data generated for consigned goods during cargo transportation, which can be filled in by the consignor and used as the basis for the transportation department to carry the goods; package data can be relevant information of consigned goods during freight transportation, such as the weight, size and location of the goods, and delivery data can be information about the transportation status of the consigned goods, which can reflect the progress of cargo transportation.

[0048] Specifically, the message channel can be used as a medium for message transmission between cargo systems in cargo transportation, and business data such as order data, waybill data, package data, delivery data, etc. can be collected in the message channel as full-process data.

[0049] Figure 2 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention. The embodiment of the present invention is a specific implementation based on the above-mentioned embodiment of the invention. Figure 2 The method provided in the embodiment of the present invention specifically includes the following steps:

[0050] Step 210: Obtain the full process data of cargo transportation.

[0051] Step 220: Divide the whole process data into waybill information and package information according to the type of transport data.

[0052] Among them, the transportation data type is divided into waybill information and package information according to the information represented by the whole process data. The waybill information can represent the cargo transportation status information, and the package information can represent the cargo status information.

[0053] In an embodiment of the present application, the full-process data can be classified according to the type of transportation data. One type of full-process data information may be waybill information, and another type of full-process data may be package information.

[0054] Step 230: Send the waybill information and the package information to the waybill processor and the package processor respectively, wherein the waybill processor and the package processor are respectively preset with anomaly detection logic corresponding to the transport data type.

[0055] Among them, the waybill processor can be a software and / or hardware device for processing waybill information, and the package processor can be a software and / or hardware device for processing package information. The waybill processor and the package processor can be respectively provided with anomaly detection logic for processing waybill information and package information, and the anomaly detection logic can be added to the waybill processor and the package processor in a predefined manner or by user input.

[0056] Specifically, the waybill information can be sent to the waybill processor, so that the waybill processor can detect the corresponding waybill information according to the preset exception detection logic. Correspondingly, the package information can be sent to the package processor, so that the package processor can detect the corresponding package information according to the preset exception detection logic.

[0057] Step 240: Determine whether the full process data complies with the anomaly detection logic.

[0058] In an embodiment of the present invention, the abnormality detection logic in the waybill processor and the package processor can be used to detect the full process data to determine whether the full process data in the waybill processor and the package processor meet the corresponding abnormality detection logic. For example, in the package processor, it can be determined whether the full process data is damage report information, or whether the full process data is cancellation of shipment information, or whether the full process data contains a new waybill number string under the premise of being split order printing information.

[0059] Step 250: If the conditions are met, the entire process data is stored as the abnormal data detection result; if not met, the entire process data is lost.

[0060] Specifically, if the full process data in the waybill processor or the package processor conforms to the corresponding abnormal detection logic, the full process data will be considered as abnormal data, and the full process data can be stored as the abnormal detection result. If the full process data in the waybill processor or the package processor does not conform to the corresponding abnormal detection logic, the full process data is not abnormal data, and the full process data can be discarded without processing.

[0061] In an embodiment of the present invention, by collecting full-process data of cargo transportation, the full-process data is divided into waybill information and parcel information according to the type of transportation data, and the waybill information and parcel information are sent to a waybill processor and a parcel processor respectively preset with corresponding different abnormal detection logics. The waybill processor and the parcel processor respectively determine whether the full-process data meets the abnormal detection logic, store the satisfied full-process data as the abnormal data detection result, and discard the unsatisfactory full-process data. The embodiment of the present invention processes the waybill information and the parcel information respectively according to the waybill processor and the parcel processor, which can facilitate the customization of the abnormal detection logic and improve the accuracy of cargo transportation data detection.

[0062] Further, based on the above-mentioned embodiment of the invention, before obtaining the full process data of cargo transportation, it also includes:

[0063] Add an exception detection logic corresponding to at least one business order type to the waybill processor; add an exception detection logic corresponding to at least one abnormal package status to the package processor, wherein the abnormal package transportation status includes at least one of split order printing, successful delivery, damage report, and cancellation of shipment.

[0064] Among them, split order printing can be the process of splitting the original order into multiple sub-orders and printing them, successful delivery can be the process of completing the consignment of goods according to the consignment address and recipient in accordance with the prescribed procedures, damage report can be the process of handling the goods after they are damaged during the consignment process, and cancel shipment can be the process of canceling the consignment of goods.

[0065] In an embodiment of the present invention, the exception handling logic in the waybill processor can be added according to different business order types, and different exception detection logic can be set for the waybill information of different businesses. In the package processor, different exception detection logic can be set according to the different abnormal package status of the package. The abnormal package status can include one or more of the following: split order printing, successful delivery, damage report, and cancellation of shipment.

[0066] Exemplarily, in the abnormal data detection method provided in the embodiment of the present invention, the acquired full-process data can be pre-processed. Figure 3 This is an example diagram of a full-process data acquisition provided by an embodiment of the present invention, see Figure 3, actively capture the full-process data to be processed, and select abnormal data to detect specific business lines, waybill processes and waybill packages according to actual business logic; the identification information of the waybill package can be used to clean useless data, and only focus on the required data; data screening uses waybill information to identify data, and filters reasonable and normal data by status and creation time to ensure data integrity and timeliness; data processing is logical processing, and logical processors are added in a pluggable manner. Logical processors can be divided into two major categories of processors, order type processors and package status processors, and you can also customize your own processors as needed; after the full-process data processing is completed, the abnormal data detection results can be stored in the database ES.

[0067] Figure 4 is a flowchart of an abnormal data detection method provided by an embodiment of the present invention. The embodiment of the present invention is a specific embodiment based on the above invention embodiment, and the abnormal detection result is corrected based on the prediction information. Figure 4 The method provided in the embodiment of the present invention specifically includes the following steps:

[0068] Step 310: Obtain the full process data of cargo transportation.

[0069] Step 320: Determine the corresponding anomaly detection logic according to the type of transportation data of the entire process data.

[0070] Step 330: Detect the entire process data according to the anomaly detection logic to obtain an abnormal data detection result.

[0071] Step 340: Collect prediction information from prediction data sources, wherein the prediction information at least includes sorting and shipping information, routing information, batch information, and quality control hotspot information.

[0072] Among them, the prediction data source can be the data source for collecting prediction information, the prediction data source can be the business system for cargo transportation, the prediction information can be information for predicting and supplementing the existing full-process data, and can include sorting and shipping information, routing information, batch information and quality control hotspot information, among which the sorting and shipping information can be information reflecting different packages in the same package during cargo transportation, the routing information can be the path information of the package transportation during cargo transportation, the batch information can be the information of the batch to which the package belongs, and the quality control hotspot information can be the information of the key focus area or vehicle set by the user.

[0073] In an embodiment of the present invention, different cargo transportation business systems may be used as prediction data sources, and sorting and shipping messages, routing information, batch information, quality control hotspot information, etc. in each business system may be collected as prediction information.

[0074] Step 350: Determine the prediction detection conditions corresponding to the full process data according to the prediction information.

[0075] Among them, the prediction detection condition can be a condition for screening abnormal data detection results based on the prediction information, and the prediction detection condition can select the full-process data that can be mistakenly judged as abnormal data in the abnormal data detection results.

[0076] Specifically, one or more prediction detection conditions can be determined according to the prediction information. For example, the predicted location of the package can be generated according to the sorting and shipping message, and the predicted location can be used as the prediction detection condition. The abnormal route of the package can also be predicted according to the routing information, and the abnormal route can be used as the prediction detection condition. The arrival status of the package can also be predicted according to the batch information, and the arrival status can be used as the prediction detection condition.

[0077] Step 360: Eliminate the full-process data that meets the prediction detection conditions in the abnormal data detection results.

[0078] In an embodiment of the present invention, it can be determined whether the full-process data in the abnormal detection data results meets the prediction detection conditions. If there is full-process data that meets the prediction detection conditions, it is determined that the full-process data is incorrectly judged as abnormal data, and the full-process data can be removed from the abnormal data detection results. If the full-process data does not meet the prediction detection conditions, the prediction information can be added to the corresponding full-process data.

[0079] In an embodiment of the present invention, full-process data of cargo transportation is acquired, corresponding anomaly detection logic is determined based on the operation data type of the full-process data, the full-process data is detected according to the determined anomaly detection logic, and anomaly detection results are obtained, prediction information is collected from a prediction data source, prediction detection conditions corresponding to the prediction information are generated, and full-process information that meets the requirements in the anomaly detection results is eliminated according to the prediction detection conditions, thereby reducing the misjudgment rate of abnormal data detection and improving the accuracy of abnormal data detection during cargo transportation.

[0080] In an exemplary embodiment, after abnormal data detection is performed on the whole process data, the prediction information can also be used to reduce the misjudgment rate of abnormal data detection. Figure 5 is an example diagram of an abnormal data detection method provided by an embodiment of the present invention, see Figure 5, data prediction mainly uses processed data to further improve the accuracy of abnormal data detection and supplement additional data. The current information of the package is obtained through the box number or package number concerned by the sorting and delivery message, and the box number or package number is compared with the full process information to determine whether it complies with the package document and the current location information; the route information obtains the abnormal route, and the site and expected arrival time provided by the abnormal route information are compared with the full process data to determine whether it complies with the abnormal route; the batch information can obtain the package information of the same batch, and the full process data can be judged by reverse checking the package to determine whether the normal flow of the whole process data; the quality control hotspot is manually reported by the quality control system, and the processed data is compared with the hotspot area or vehicle to determine whether the full process data is a hotspot abnormality, and the data is updated in time.

[0081] Figure 6 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention. The embodiment of the present invention is a specific embodiment based on the above-mentioned invention embodiment, and enriches the abnormal data detection result by collecting supplementary information. Figure 6 The method provided in the embodiment of the present invention specifically includes the following steps:

[0082] Step 410: Obtain the full process data of cargo transportation.

[0083] Step 420: Determine the corresponding anomaly detection logic according to the type of transportation data of the entire process data.

[0084] Step 430: Detect the entire process data according to the anomaly detection logic to obtain an abnormal data detection result.

[0085] Step 440: Collect supplementary information from the data supplement source, wherein the supplementary information includes at least one of supplementary waybill information, supplementary abnormal order process information, supplementary abnormal site information, and supplementary abnormal timeliness information.

[0086] Among them, the data supplement source can be the data source for supplementing the full-process data in the anomaly detection results, and can be different business systems in the cargo transportation process. The supplementary information can be the data for supplementing the full-process data. The collection time of the supplementary information can be different from the collection time of the full-process data. For example, the information of the data source can be continued to be collected as supplementary information after the full-process data is collected.

[0087] In an embodiment of the present invention, the business system of the cargo transportation process can be used as a data supplement source for supplementing the full-process data of the abnormal detection results. Supplementary waybill information, supplementary abnormal order process information, supplementary abnormal site information, and supplementary abnormal time limit information in the data supplement source can be collected regularly or continuously as supplementary information. Among them, the supplementary waybill information can be the waybill information used to supplement the full-process data, the supplementary abnormal order process information can be the order process data used to supplement the full-process data, the supplementary abnormal site information can be the cargo transportation site information used to supplement the full-process data, and the supplementary abnormal time limit information can be the time limit information used to supplement the full-process data.

[0088] Step 450: Add the supplementary information to the corresponding full-process data in the abnormal data detection result.

[0089] Specifically, the corresponding full-process data can be searched in the abnormal data detection results according to the unique identification information of each supplementary information, and the supplementary information can be added to the found full-process data to supplement the abnormal data detection results, thereby facilitating the monitoring and processing of abnormal data.

[0090] In an embodiment of the present invention, full-process data of cargo transportation is acquired, corresponding anomaly detection logic is determined based on the operation data type of the full-process data, the full-process data is detected according to the determined anomaly detection logic, and anomaly detection results are obtained, supplementary information of data supplement sources is collected, and the supplementary information is added to the corresponding full-process data in the anomaly detection result, thereby improving the completeness of the full-process data in the anomaly data detection result and facilitating data monitoring and processing during cargo transportation.

[0091] In an exemplary embodiment, after determining the abnormal data detection result, the abnormal data detection method provided in the embodiment of the present invention can also supplement the full process data in the abnormal data detection result. Figure 7 This is an example diagram of data supplementation provided by an embodiment of the present invention, see Figure 7 The data supplement mechanism can supplement the missing data in the entire process by accessing waybill information, sorting and shipping information, transportation site information, and quality control information, update abnormal data in the ES database, make the reported abnormal data more complete, reduce the system pressure of the quality control system, and ensure data integrity.

[0092] Figure 8 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention. The embodiment of the present invention is based on the above-mentioned embodiment of the invention, and screens the full process data in the abnormal data detection result through timeliness information to improve the accuracy of the abnormal data detection result. Figure 8 The method provided in the embodiment of the present invention specifically includes the following steps:

[0093] Step 510: Obtain the full process data of cargo transportation.

[0094] Step 520: Determine the corresponding anomaly detection logic according to the type of transportation data of the entire process data.

[0095] Step 530: Detect the entire process data according to the anomaly detection logic to obtain an abnormal data detection result.

[0096] Step 540: Collect the timeliness information of the timeliness data source, wherein the timeliness information at least includes waybill timeliness information, operation timeliness information, site timeliness information and abnormal timeliness information.

[0097] Among them, the source of timeliness data can be a business system with timeliness information in the cargo transportation system, and the timeliness information can be information representing the effective time range of the whole process data. According to the dimension of cargo transportation, the timeliness information can include waybill timeliness information, operation timeliness information, waybill timeliness information and abnormal timeliness information. Waybill timeliness can be the effective time limit set for the waybill dimension, operation timeliness information can be the effective time limit set for the cargo operation process, waybill timeliness information can be the effective time limit set for the waybill dimension, and abnormal timeliness information can be the effective time limit set for different types of cargo transportation abnormalities.

[0098] In an embodiment of the present invention, a business system with timeliness information can be used as a source of timeliness data, and waybill timeliness information, operation timeliness information, waybill timeliness information and abnormal timeliness information of the timeliness data source can be collected as timeliness information.

[0099] Step 550: Obtain abnormal time limit rules that match the time limit information.

[0100] Among them, the abnormal timeliness rule can be a timeliness rule for determining whether the full process data is abnormal data. For example, it can determine whether the time range of the full process data is not within the time range of the timeliness information.

[0101] Specifically, the corresponding abnormal time limit rules can be determined according to the time limit information. The abnormal time limit rules can set one or more rules according to the different dimensions of the time limit information in the cargo transportation process. Each rule can correspond to one or more of the waybill time limit information, operation time limit information, waybill time limit information and abnormal time limit information. The matching abnormal time limit rules can be determined according to the dimension of the time limit information.

[0102] Step 560: Eliminate the full-process data that does not comply with the abnormal timeliness rules in the abnormal detection results.

[0103] In an embodiment of the present invention, the abnormal time limit rule can be used to judge the full-process data in the abnormal detection result. If the time range of the full-process data meets the abnormal time limit rule, the full-process data is determined to be abnormal data. If the time range of the full-process data does not meet the abnormal time limit rule, the full-process data is determined to be erroneously identified as abnormal data, and the full-process data can be deleted from the abnormal data detection result.

[0104] In an embodiment of the present invention, full-process data of cargo transportation is acquired, corresponding anomaly detection logic is determined based on the operation data type of the full-process data, the full-process data is detected according to the determined anomaly detection logic, and anomaly data detection results are obtained, timeliness information is collected, and anomaly timeliness rules that match the timeliness information are determined, and full-process data that does not match the anomaly timeliness rules in the anomaly data detection results are eliminated, so as to improve the accuracy of the anomaly data detection results through the timeliness information.

[0105] In an exemplary implementation, after performing abnormal data detection, the embodiment of the present invention can also collect timeliness information of the entire process data, and filter the abnormal data detection results according to the timeliness information. Fig. 9 This is an example diagram of a time-sensitive information detection provided by an embodiment of the present invention, see Fig. 9 , control the capture of timeliness information, and perform timeliness control on the entire process data according to the timeliness information. Timeliness control mainly defines the reporting timeliness by obtaining the promised timeliness of the waybill, the estimated arrival time downstream, and the quality control timeliness. The timeliness dispatcher can define different timeliness according to the region, site, abnormal reason, and single document. The classification management is flexible and matched, and different timeliness ranges can be accurately controlled to obtain more accurate reporting data.

[0106] Further, on the basis of the above-mentioned embodiments of the invention, after detecting the full process data according to the anomaly detection logic to obtain the anomaly data detection result, it also includes: obtaining the newly added full process data, and obtaining the target process data in the anomaly data detection result that matches the newly added full process data; encapsulating the newly added full process data and the target process data and adding them to the anomaly detection result.

[0107] In an embodiment of the present invention, newly added full-process data can also be collected, and the newly added full-process data can be matched with the full-process data in the abnormal data detection results to obtain the target full-process data with an associated relationship. The newly added full-process data and the target full-process data can be encapsulated, and the encapsulated data can be added as new full-process data to the abnormal detection results.

[0108] In an exemplary embodiment, the whole process data can also be monitored. Fig.10 This is an example diagram of data monitoring provided by an embodiment of the present invention, see Fig.10, obtain the reported waybill anomalies and package anomalies, among which waybill anomalies and package anomalies are stored in Redis storage, and obtain the completion information, order change information, warehouse return information, and site collection termination information of the whole process data, and compare the obtained new whole process data with the waybill anomalies and package anomalies, and send the logical data package to the quality control system, which will re-detect and process the abnormal data of abnormal documents such as waybill anomalies and package anomalies.

[0109] In an exemplary embodiment, Fig.11 is an example diagram of an abnormal data detection method provided by an embodiment of the present invention, see Fig.11 , an abnormal data detection system mainly accesses the data message channel of the whole process of the package, filters out the data that does not need to be concerned about and the wrong data by cleaning the data, and filters out the status data of concern after reaching the data screening step. The data will have its own processing logic according to different states, and will divide the different status data into different business logics, and store them in ES storage after processing. The asynchronous update data logic updates the ES storage data by accessing new messages to compensate the data and update the status of abnormal packages, reducing false positives and over-reporting; by adding different time models to capture the stored abnormal package information and report it to the quality control system; data prediction uses routing information to timely predict whether the downstream site of the package flow is operating normally; the reported data will continue to be monitored, and when new flow information arrives, it will promptly notify and update the abnormal order information.

[0110] Fig.12 It is a structural diagram of an abnormal data detection device provided in an embodiment of the present invention. It can execute the abnormal data detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and / or hardware, and specifically includes: a data acquisition module 601, a logic determination module 602 and a detection execution module 603.

[0111] The data collection module 601 is used to obtain the whole process data of cargo transportation.

[0112] The logic determination module 602 is used to determine the corresponding anomaly detection logic according to the type of transportation data of the full-process data.

[0113] The detection execution module 603 is used to detect the whole process data according to the abnormality detection logic to obtain abnormal data detection results.

[0114] In the embodiment of the present invention, the full-process data of cargo transportation is acquired through the data acquisition module, the logic determination module determines the corresponding anomaly detection logic based on the operation data type of the full-process data, the detection execution module detects the full-process data according to the determined anomaly detection logic, and obtains the anomaly detection result, thereby realizing the automatic detection of the full-process data in the cargo transportation process, reducing the influence of human factors on anomaly detection, and improving the accuracy of abnormal data detection.

[0115] Further, based on the above-mentioned embodiment of the invention, the data acquisition module 601 includes:

[0116] A data extraction unit is used to extract business data from at least one freight transportation business system through a message channel, wherein the business data includes at least one of order data, waybill data, package data, and delivery data; and each of the business data is used as full-process data.

[0117] Further, based on the above-mentioned embodiment of the invention, the logic determination module 602 includes:

[0118] The data classification unit is used to classify the full-process data into waybill information and package information according to the type of transportation data.

[0119] An information sending unit is used to send the waybill information and the package information to the waybill processor and the package processor respectively; wherein the waybill processor and the package processor are respectively preset with abnormality detection logic corresponding to the transport data type.

[0120] Furthermore, based on the above-mentioned embodiments of the invention, the logic preset module is used to add the exception detection logic corresponding to at least one business order type to the waybill processor; and add the exception detection logic corresponding to at least one abnormal package status to the package processor, wherein the abnormal package transportation status includes at least one of split order printing, completed delivery, damage report, and cancellation of shipment.

[0121] Furthermore, based on the above-mentioned embodiments of the invention, the device also includes: a data prediction unit, used to collect prediction information of the prediction data source, wherein the prediction information at least includes sorting and shipping messages, routing information, batch information and quality control hotspot information; determine the prediction detection conditions corresponding to the full-process data according to the prediction information; and eliminate the full-process data that meets the prediction detection conditions in the abnormal data detection results.

[0122] Furthermore, based on the above-mentioned embodiments of the invention, the device also includes: a data supplement module, which is used to collect supplementary information from the data supplement source, wherein the supplementary information includes at least one of supplementary waybill information, supplementary abnormal order process information, supplementary abnormal site information, and supplementary abnormal time limit information; and the supplementary information is added to the corresponding full-process data in the abnormal data detection result.

[0123] Furthermore, based on the above-mentioned embodiments of the invention, the device also includes: a timeliness processing module, which is used to collect timeliness information from the timeliness data source, wherein the timeliness information at least includes waybill timeliness information, operation timeliness information, site timeliness information and abnormal timeliness information; obtain abnormal timeliness rules that match the timeliness information; and eliminate the full process data in the abnormal detection results that does not comply with the abnormal timeliness rules.

[0124] Furthermore, based on the above-mentioned embodiments of the invention, the data addition module is used to obtain newly added full-process data, and obtain target process data in the abnormal data detection result that matches the newly added full-process data; and encapsulate the newly added full-process data and the target process data and add them to the abnormal detection result.

[0125] Further, based on the above-mentioned embodiment of the invention, the detection execution module 603 includes:

[0126] A detection unit, used to determine whether the full process data conforms to the anomaly detection logic;

[0127] The execution unit is used to store the whole process data as the abnormal data detection result if the conditions are met; if not, the whole process data is lost.

[0128] Fig.13 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.13 As shown, the electronic device includes a processor 70, a memory 71, an input device 72 and an output device 73; the number of processors 70 in the electronic device can be one or more. Fig.13 A processor 70 is taken as an example; the processor 70, the memory 71, the input device 72 and the output device 73 in the electronic device can be connected by a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0129] The memory 71, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the abnormal data detection method in the embodiment of the present invention (for example, the data acquisition module 601, the logic determination module 602 and the detection execution module 603 in the abnormal data detection device). The processor 70 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 71, that is, implements the above-mentioned abnormal data detection method.

[0130] The memory 71 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 71 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 71 may further include a memory remotely arranged relative to the processor 70, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The input device 72 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 73 may include a display device such as a display screen.

[0132] An embodiment of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to perform an abnormal data detection method when executed by a computer processor, the method comprising:

[0133] Obtain full process data of cargo transportation;

[0134] Determine the corresponding anomaly detection logic according to the transport data type of the full process data;

[0135] The whole process data is detected according to the abnormality detection logic to obtain an abnormal data detection result.

[0136] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the abnormal data detection method provided by any embodiment of the present invention.

[0137] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] It is worth noting that in the embodiment of the above-mentioned abnormal data detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0139] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting abnormal data, characterized in that: include: Obtain full process data of cargo transportation; Determine the corresponding anomaly detection logic according to the transport data type of the full process data; Detect the whole process data according to the abnormality detection logic to obtain abnormal data detection results; Wherein, determining the corresponding anomaly detection logic according to the type of transportation data of the whole process data includes: Classifying the received full-process data according to the preset transport data type, and determining the corresponding anomaly detection logic for the transport data type under each classification, wherein different transport data types correspond to different anomaly detection logics; The received full-process data is classified according to the preset transport data type, and the corresponding abnormality detection logic is determined for the transport data type under each classification, including: Dividing the full-process data into waybill information and package information according to the type of transport data; Sending the waybill information and the package information to a waybill processor and a package processor respectively; The waybill processor and the package processor are respectively preset with abnormal detection logic corresponding to the transport data type; The method further comprises: Collecting forecast information from forecast data sources, wherein the forecast information at least includes sorting and shipping information, routing information, batch information, and quality control hotspot information; Determine the prediction detection condition corresponding to the full-process data according to the prediction information; wherein the prediction detection condition refers to the condition that can select the full-process data that is wrongly judged as abnormal data in the abnormal data detection result; Eliminate the full-process data that meets the prediction detection conditions in the abnormal data detection results.

2. The method according to claim 1, characterized in that The whole process data of cargo transportation is obtained, including: Extracting business data from at least one freight transportation business system through a message channel, wherein the business data includes at least one of order data, waybill data, package data, and delivery data; The business data are regarded as the whole process data.

3. The method according to claim 1, characterized in that Before obtaining the full process data of cargo transportation, it also includes: Adding anomaly detection logic corresponding to at least one business order type to the waybill processor; An exception detection logic corresponding to at least one abnormal package status is added to the package processor, wherein the abnormal package transportation status includes at least one of bill splitting and printing, delivery, damage report, and shipment cancellation.

4. The method according to claim 1, characterized in that: Also includes: Collecting supplementary information of the data supplementation source, wherein the supplementary information includes at least one of supplementary waybill information, supplementary abnormal order process information, supplementary abnormal site information, and supplementary abnormal timeliness information; The supplementary information is added to the corresponding full-process data in the abnormal data detection result.

5. The method according to claim 1, characterized in that Also includes: Collecting timeliness information from the timeliness data source, wherein the timeliness information at least includes waybill timeliness information, operation timeliness information, station timeliness information and abnormal timeliness information; Obtaining an abnormal time limit rule that matches the time limit information; Eliminate the full process data that does not comply with the abnormal time limit rules in the abnormal detection results.

6. The method according to claim 1, characterized in that After detecting the whole process data according to the abnormality detection logic to obtain abnormal data detection results, the method further includes: Acquire the newly added full-process data, and acquire the target process data in the abnormal data detection result that matches the newly added full-process data; The newly added full-process data and the target process data are encapsulated and added to the anomaly detection result.

7. The method according to any one of claims 1 to 6, characterized in that: The detecting the whole process data according to the abnormality detection logic to obtain abnormal data detection results includes: Determining whether the full process data complies with the anomaly detection logic; If satisfied, the whole process data is stored as abnormal data detection result; If not satisfied, the entire process data will be lost.

8. An abnormal data detection device, characterized in that: include: Data collection module, used to obtain data on the entire process of cargo transportation; A logic determination module, used to determine the corresponding anomaly detection logic according to the type of transport data of the full-process data; A detection execution module, used to detect the whole process data according to the abnormality detection logic to obtain abnormal data detection results; Wherein, the logic determination module is specifically used for: Classifying the received full-process data according to the preset transport data type, and determining the corresponding anomaly detection logic for the transport data type under each classification, wherein different transport data types correspond to different anomaly detection logics; The logic determination module comprises: A data classification unit, used to classify the full-process data into waybill information and package information according to the type of transport data; An information sending unit, used to send the waybill information and the package information to the waybill processor and the package processor respectively; The waybill processor and the package processor are respectively preset with abnormal detection logic corresponding to the transport data type; The device also includes: A data prediction unit, used to collect prediction information of prediction data sources, wherein the prediction information at least includes sorting and shipping information, routing information, batch information and quality control hot spot information; Determine the prediction detection condition corresponding to the full-process data according to the prediction information; wherein the prediction detection condition refers to the condition that can select the full-process data that is wrongly judged as abnormal data in the abnormal data detection result; Eliminate the full-process data that meets the prediction detection conditions in the abnormal data detection results.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the abnormal data detection method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the abnormal data detection method as described in any one of claims 1 to 7 is implemented.

Images