Abnormal data detection method and device for logistics information
By using a combination of local density algorithm and tag propagation algorithm, the problem of low accuracy of abnormal data detection caused by uneven distribution of logistics information is solved, and the accuracy of abnormal data detection in logistics information is improved.
Patent Information
- Application Number
- CN202310595211.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, due to the uneven distribution of logistics information data, the accuracy of abnormal data detection is low, and abnormal data in logistics information cannot be effectively detected.
By obtaining the first logistics data with the data label and the second logistics data without the label, the local density algorithm is used to calculate the aggregation degree, and the correlation relationship between the data is determined in combination with the tag propagation algorithm to determine whether the target data is abnormal data.
It reduces the impact of uneven data distribution on abnormal data detection, improves the accuracy of abnormal data detection, and can more accurately identify abnormal data in logistics information.
Smart Images

Figure CN119026040A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of logistics technology, and in particular to a method and device for detecting abnormal data of logistics information. Background Art
[0002] In e-commerce transactions, the accuracy of logistics information has an important impact on the service quality of e-commerce. At present, data is generally divided into multiple data sets according to the data distribution density of logistics information, and data far away from the data set is judged as abnormal data.
[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:
[0004] Due to the uneven distribution of logistics information data, the accuracy of data set division based on data distribution density is low, which leads to low accuracy of abnormal data detection. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for detecting abnormal data of logistics information, by acquiring a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, the aggregation corresponding to the first logistics data and the second logistics data is calculated respectively by using a local density algorithm; according to the data label, a reference label indicating whether the second logistics data is abnormal is determined by using a label propagation algorithm; according to the data label and the reference label, the association relationship between any target data to be tested in the second logistics data and other data is determined; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, whether the target data to be tested is abnormal data is determined. Thus, the abnormal data in the logistics information is detected by the aggregation degree of the data and the association relationship between the data, thereby reducing the influence of uneven data distribution on abnormal data detection and improving the accuracy of abnormal data detection.
[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for detecting abnormal data in logistics information is provided.
[0007] A method for detecting abnormal data of logistics information according to an embodiment of the present invention includes: acquiring a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, using a local density algorithm to respectively calculate the clustering corresponding to the first logistics data and the second logistics data; according to the data label, using a label propagation algorithm to determine a reference label indicating whether the second logistics data is abnormal; according to the data label and the reference label, determining the association relationship between any target data to be tested in the second logistics data and other data; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the clustering degree and the association relationship, determining whether the target data to be tested is abnormal data.
[0008] Optionally, determining the association relationship between any target data to be tested in the second logistics data and other data based on the data label and the reference label includes: determining any other target data from the other data, and determining the similarities and differences in labels between the target data to be tested and the other target data; determining the degree of association corresponding to the data pair consisting of the target data to be tested and the other target data based on the similarities and differences in the labels; and determining the association relationship between the target data to be tested and other data based on the degrees of association corresponding to each of the data pairs.
[0009] Optionally, the method provided by the present invention also includes: for each of the data pairs: according to the object characteristics corresponding to the target data to be tested and the object characteristics corresponding to the other data, calculating the difference corresponding to the data pairs respectively; according to the difference and the aggregation degree corresponding to the data pairs respectively, calculating the label association factor corresponding to the data pairs.
[0010] Optionally, for each of the data pairs: according to the item characteristics corresponding to the target data to be tested and the item characteristics corresponding to the other data, calculating the difference degrees respectively corresponding to the data pairs includes: calculating the difference degrees between the target data to be tested and the other data according to the item characteristics corresponding to the target data to be tested and the item characteristics corresponding to the other data, and determining the difference degree as the difference degree corresponding to the target data to be tested; calculating the difference degrees corresponding to the other target data according to the item characteristics corresponding to the other target data and the item characteristics corresponding to the logistics data other than the other target data in the first logistics data and the second logistics data; taking the difference degrees corresponding to the target data to be tested and the difference degrees corresponding to the other target data as the differences respectively corresponding to the data pairs.
[0011] Optionally, the determining, according to the similarities and differences of the labels, the degree of association corresponding to the data pair consisting of the target data to be tested and the other target data includes any one of the following:
[0012] In the case where the label type of the target data to be tested is the same as that of the other target data and both indicate that the corresponding logistics data is not abnormal, determining the association degree of the data pair according to the sum of the label association factor and a preset value;
[0013] In the case where the target data to be tested and the other target data have the same label type and both indicate that the corresponding logistics data is abnormal, determining the association degree of the data pair according to the difference between the preset value and the label association factor;
[0014] In the case that the label types of the target data to be tested are different from those of the other target data, the label association factor is used as the association degree of the data pair.
[0015] Optionally, determining whether the target data to be tested is abnormal data based on the clustering degree and the association relationship includes: for each other target data among the other data: calculating the associated clustering degree corresponding to the other target data based on the association degree corresponding to the data pair consisting of the other target data and the target data to be tested, and the clustering degree corresponding to the other target data; calculating the average clustering degree of the other target data based on the sum of the associated clustering degrees corresponding to each other target data and the number of data of the other data; determining whether the target data to be tested is abnormal data based on the average clustering degree and the clustering degree corresponding to the target data to be tested.
[0016] Optionally, determining whether the target data to be tested is abnormal data based on the average clustering degree and the clustering degree corresponding to the target data to be tested includes: calculating the ratio of the average clustering degree to the clustering degree corresponding to the target data to be tested; when the ratio is greater than a preset threshold, determining that the target data to be tested is abnormal data; when the ratio is not greater than the preset threshold, determining that the target data to be tested is non-abnormal data.
[0017] Optionally, the degree of aggregation corresponding to the first logistics data and the second logistics data is calculated respectively by using a local density algorithm according to the item features included in the first logistics data and the second logistics data, and the corresponding degrees of aggregation include: for any target logistics data in the first logistics data and the second logistics data: according to the item features, determining a neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data; according to the item features included in the neighbor set, calculating respectively the degree of aggregation corresponding to the target logistics data by using a local density algorithm.
[0018] Optionally, determining a neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data according to the item characteristics includes: determining the nearest neighbor data, reverse nearest neighbor data and shared nearest neighbor data of the target logistics data from the first logistics data and the second logistics data according to the item characteristics; and taking the union of the nearest neighbor data, the reverse nearest neighbor data and the shared nearest neighbor data as the neighbor set.
[0019] Optionally, the method of determining a reference label indicating whether the second logistics data is abnormal based on the data label and using a label propagation algorithm includes: determining a neighbor set corresponding to any data in the second logistics data; determining a label type of data labels with a larger number in the neighbor set of any data; and using the determined label type as a reference label for any data.
[0020] Optionally, the method provided by the present invention further includes: when the acquired first logistics data and / or the second logistics data include text data, converting the text data into numerical data.
[0021] To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for detecting abnormal data in logistics information is provided.
[0022] An abnormal data detection device for logistics information according to an embodiment of the present invention includes: a data acquisition module, which is used to acquire multiple first logistics data and multiple second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; a clustering calculation module, which is used to calculate the clustering corresponding to the first logistics data and the second logistics data respectively according to the item features included in the first logistics data and the second logistics data using a local density algorithm; a label supplement module, which is used to determine a reference label indicating whether the second logistics data is abnormal according to the data label using a label propagation algorithm; a relationship determination module, which is used to determine the association relationship between any target data to be tested in the second logistics data and other data according to the data label and the reference label; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; an abnormality judgment module, which is used to determine whether the target data to be tested is abnormal data according to the clustering degree and the association relationship.
[0023] To achieve the above objective, according to another aspect of an embodiment of the present invention, a server is provided.
[0024] A server according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement an abnormal data detection method for logistics information according to an embodiment of the present invention.
[0025] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.
[0026] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements a method for detecting abnormal data in logistics information according to an embodiment of the present invention.
[0027] One embodiment of the above invention has the following advantages or beneficial effects: by acquiring a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, the aggregation corresponding to the first logistics data and the second logistics data is calculated respectively using a local density algorithm; according to the data label, a reference label indicating whether the second logistics data is abnormal is determined using a label propagation algorithm; according to the data label and the reference label, the association relationship between any target data to be tested in the second logistics data and other data is determined; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, whether the target data to be tested is abnormal data is determined. Thus, the abnormal data in the logistics information is detected by the aggregation degree of the data and the association relationship between the data, thereby reducing the impact of uneven data distribution on abnormal data detection and improving the accuracy of abnormal data detection.
[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0030] Figure 1 is a schematic diagram of the main steps of a method for detecting abnormal data in logistics information according to an embodiment of the present invention;
[0031] Figure 2 is a schematic diagram of the main steps of determining the correlation between data in a method for detecting abnormal data of logistics information according to an embodiment of the present invention;
[0032] Figure 3 is a schematic diagram of the main steps of a method for detecting abnormal data in logistics information according to an embodiment of the present invention;
[0033] Figure 4 is a schematic diagram of main modules of a device for detecting abnormal data of logistics information according to an embodiment of the present invention;
[0034] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0035] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0037] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.
[0038] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.
[0039] Figure 1 It is a schematic diagram of the main steps of the abnormal data detection method of logistics information according to an embodiment of the present invention.
[0040] like Figure 1 As shown, the abnormal data detection method of logistics information in the embodiment of the present invention mainly includes the following steps:
[0041] Step S101: Acquire a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data include data labels; the data labels include abnormal data labels and non-abnormal data labels, and the second logistics data do not include the data labels.
[0042] The embodiment of the present invention takes the logistics information entry scenario as an example to elaborate on the abnormal data detection method of logistics information. In the logistics warehouse, the logistics package is placed on the assembly line, and various sensors are used to identify and measure the category of the goods in the package, as well as the weight, volume and size of the package. Due to sensor damage or other external factors, the sensor will have errors or mistakes during operation, which makes the data on the category, weight, volume and size obtained by the sensor deviate from the actual data, thus resulting in abnormal data in the entered logistics information.
[0043] The embodiment of the present invention adopts a semi-supervised learning method, and uses the data information provided by a small amount of data labels as a basis to perform anomaly detection on a large amount of unlabeled logistics data to determine whether there is abnormal data in the unlabeled logistics data. Therefore, it is necessary to obtain first logistics data that have been artificially processed and include data labels, that is, a small amount of logistics data with data labels, and second logistics data, where the second logistics data is logistics data that has not been artificially processed, that is, other logistics data without added labels.
[0044] In order to detect abnormal data in logistics information, a small amount of data is selected from the logistics data to be detected, and a manual method is used to determine whether the selected small amount of data is abnormal data. Data labels are added to the small amount of data according to the judgment result. The data labels are divided into non-abnormal data labels and abnormal data labels. It can be understood that non-abnormal data labels are added to data judged as non-abnormal, and abnormal data labels are added to data judged as abnormal.
[0045] The first logistics data and the second logistics data are both logistics data identified or measured by various sensors on the assembly line, such as electronic products--mobile phones--1 kg, alcoholic beverages--beverages--10 kg, etc. The difference between the first logistics data and the second logistics data is that the first logistics data is data that has been manually detected and added with data labels. When the logistics data such as "electronic products--mobile phones--10 kg" is manually detected, because the weight data of the mobile phone is found to be abnormal, this data is marked as abnormal data, that is, an abnormal data label is added. For logistics data such as "electronic products--mobile phones--1 kg", since all its data indicators are within a reasonable range, it is marked as non-abnormal data, that is, a non-abnormal data label is added to it; the second logistics data is the initial data that has not been manually detected, so it does not have a data label. After the label is added, the first logistics data with data labels is dispersed to other second logistics data without labels.
[0046] Step S102: According to the item features included in the first logistics data and the second logistics data, the aggregation degrees corresponding to the first logistics data and the second logistics data are calculated respectively using a local density algorithm.
[0047] In one embodiment of the present invention, the method provided in this embodiment also includes: when the acquired first logistics data and / or the second logistics data include text data, converting the text data into numerical data to facilitate subsequent data processing.
[0048] The acquired first logistics data and second logistics data both include item characteristics, such as category, weight, volume and / or size. To facilitate subsequent data processing, the text data in the first logistics data and the second logistics data can be converted into numerical data. For example, the category data: home appliances - refrigerators, electronic products - mobile phones, wine and beverages - beverages, food - biscuits and other text data can be converted into numerical data such as 0, 1, 2 or 00, 01, 10, and the set of the converted first logistics data and the second logistics data is determined as the data set U, where the number of categories corresponding to the logistics data in the data set U is n.
[0049] In order to facilitate the execution of subsequent algorithms, the data set U is normalized and U is calculated first. ij with U i,min The difference between i,max with U i,min The difference between ij with U i,min The difference between i,max with U i,min The ratio of the difference between ij The normalized value of U ij The normalization formula is as follows:
[0050]
[0051] Among them U ij is the logistic data value of the jth category of the i-th record in the data set U, such as the weight value, U i,max and U i,min is the maximum and minimum value of the j-th class of data, U′ ij For U ij The logistics data values are obtained after normalization, and the data set U is normalized to obtain the data set U′.
[0052] From the data set U ′ In the above example, we select the target data to be tested (referred to as target data) u without data labels, and calculate the aggregation degree corresponding to the target data u according to the item feature value, such as weight value, that is, the local density of the target data u relative to any other data v, and calculate the relationship between the target data u and the data set U. ′ The Gaussian kernel function value of any data v in is accumulated and divided by the data set U ′ The number of logistics data in the data, divided by h n , so the calculation formula of the local density corresponding to the target data u is as follows:
[0053]
[0054] Among them, ρ(u) is the local density (aggregation) corresponding to the target data u, |U ′ | is the data set U ′ The number of logistics data, u is the target data to be tested, and v is the data set U ′ Any data in, n is the number of categories in the logistics data, is the Gaussian kernel function value corresponding to the data u and v.
[0055] According to the item feature values corresponding to data u and v, such as the difference between weight values, the Gaussian kernel function value is calculated. The Gaussian kernel function value represents the distance between the target data u and any data v. The calculation formula of the Gaussian kernel function value corresponding to data u and v is as follows:
[0056]
[0057] in, is the Gaussian kernel function value corresponding to data u and v, h is the kernel function window width, h represents the discrete degree of data u and v, and the value of h in the embodiment of the present invention can be 1; according to formula (3) and formula (4), the farther the distance between data u and v, the smaller the value of the corresponding Gaussian kernel function, and the smaller the local density of target data u relative to any data v.
[0058] In one embodiment of the present invention, the aggregation corresponding to the first logistics data and the second logistics data is calculated using a local density algorithm according to the item features included in the first logistics data and the second logistics data. The corresponding aggregation includes: for any target logistics data in the first logistics data and the second logistics data: according to the item features, a neighbor set corresponding to the target logistics data is determined from the first logistics data and the second logistics data; according to the item features included in the neighbor set, the aggregation corresponding to the target logistics data is calculated using a local density algorithm. Therefore, the influence on the abnormal data detection result caused by using all data for density estimation without considering the difference in data density is reduced, while reducing the amount of calculation and saving computing resources.
[0059] Since using all logistics data for density estimation will fail to detect local abnormal logistics data due to not considering the difference in data density, and using all logistics data for detection will require more computing resources, when performing local density estimation, only data with a strong correlation with the target data u (i.e., item feature values corresponding to the target data u, such as data with a close weight value, i.e., neighbor data) can be used for abnormal monitoring, where data with a strong correlation with the target data u constitutes data set T, which can also be called the neighbor set of target data u. Based on the neighbor set (i.e., data set T), the local density (clustering degree) formula corresponding to the target data u is:
[0060]
[0061] Wherein, ρ(u) is the local density (aggregation) corresponding to the target data u, |T(u)| is the number of logistics data in the data set T(u), u is the target data to be tested, v is any data in the union of the data set T(u) and the data u, n is the number of categories in the logistics data, h represents the degree of discreteness of the data u and v, and the value of h in the embodiment of the present invention can be 1, and is the Gaussian kernel function value corresponding to the data u and v.
[0062] In one embodiment of the present invention, the determining of the neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data according to the item characteristics may include: determining the nearest neighbor data, the reverse nearest neighbor data, and the shared nearest neighbor data of the target logistics data from the first logistics data and the second logistics data according to the item characteristics; and taking the union of the nearest neighbor data, the reverse nearest neighbor data, and the shared nearest neighbor data as the neighbor set. Therefore, data with a strong correlation with the target logistics data is determined from all the data as the data set for detecting the target logistics data, thereby making the detection result more accurate and saving computing resources.
[0063] The data set T, i.e., the neighbor set, can be determined by determining the union of the nearest neighbor data, the reverse nearest neighbor data, and the shared nearest neighbor data. According to the item feature value corresponding to the target data u, such as the weight value, by determining the k nearest neighbor data KNN(u)={v1,v2,v3} of the target data u, where k can be set as needed, each reverse nearest neighbor data RNN(u)={v2,v3,v4} that regards the target data u as one of the k nearest neighbor data is determined, and the shared nearest neighbor data SNN(u)={v5} that has at least one nearest neighbor data with the target data u is determined, and the union of KNN(u), RNN(u) and SNN(u) is used as the data set T, i.e., the neighbor set of the target data u.
[0064] Step S103: Based on the data label, a label propagation algorithm is used to determine a reference label indicating whether the second logistics data is abnormal.
[0065] In order to determine the association between data based on the additional information provided by the data label of a small amount of data (first logistics data), in one embodiment of the present invention, the reference label indicating whether the second logistics data is abnormal is determined by using the label propagation algorithm based on the data label, including: determining the neighbor set corresponding to any data in the second logistics data; determining the label type of the data label with a large number in the neighbor set of any data; and using the determined label type as the reference label of any data. Therefore, the reference label of the second logistics data can be determined based on the existing data label of the first logistics data, so as to determine the association between data based on the data label and the reference label.
[0066] According to the characteristic values of the items, such as the weight value, determine the data set that is close to the weight value corresponding to any unlabeled logistics data u1, that is, the data set that is strongly associated with the unlabeled data u1, that is, the neighbor set of the unlabeled data u1, and determine the label type and number of labeled data in the neighbor set. For example, the neighbor set L of the unlabeled data u1 includes data A+, B, C-, D, and E+, where A+ and E+ represent data with non-abnormal data labels, and C- represents data with abnormal data labels. According to the data set L, it can be determined that there are 2 non-abnormal data labels and 1 abnormal data label. According to the principle of "obeying the majority", it can be determined that the type of reference label of the unlabeled data u1 is non-abnormal, that is, u1(+), where "+" represents an artificially added non-abnormal data label, "-" represents an artificially added abnormal data label, "(+)" represents a non-abnormal reference label determined according to the label propagation algorithm, and "(-)" represents an abnormal reference label determined according to the label propagation algorithm. It is understandable that when the type of the reference label of the unlabeled data u1 is determined to be abnormal according to the principle of "obeying the majority", u1 with the abnormal reference label can be represented as u1(-). The reference label and the data label provide additional data association information for abnormal data detection during abnormal data detection, making the data detection results more reliable and accurate. It is understandable that when the number of abnormal data labels contained in the data set L is large, the type of the reference label of the unlabeled data u1 can be determined to be abnormal. It is understandable that the reference label can be determined for the unlabeled data in the neighbor set while determining the nearest neighbor data, the reverse nearest neighbor data, the shared nearest neighbor data and their union.
[0067] Step S104: Determine the association relationship between any target data to be tested in the second logistics data and other data based on the data tag and the reference tag; wherein the other data is logistics data other than the target data to be tested in the first logistics data and the second logistics data.
[0068] In order to reduce the impact of uneven data distribution on the accuracy of anomaly detection, the embodiment of the present invention determines whether the logistics data to be tested is abnormal data by combining the local density of the logistics data with the correlation relationship between the data, wherein the correlation relationship between the data is determined based on the manually added data labels and the reference labels determined by the label propagation algorithm. Furthermore, the correlation relationship between the target data to be tested and other data is determined based on the correlation degree between the target data to be tested and each of the other data.
[0069] In order to determine the association relationship between the target data to be tested and other data based on the data label and the reference label, in one embodiment of the present invention, determining the association relationship between any one target data to be tested in the second logistics data and other data based on the data label and the reference label includes: determining any other target data from the other data, and determining the similarities and differences between the labels of the target data to be tested and the other target data; determining the degree of association corresponding to the data pair consisting of the target data to be tested and the other target data based on the similarities and differences in the labels; and determining the association relationship between the target data to be tested and other data based on the degrees of association corresponding to each of the data pairs.
[0070] Determine the target data u from the unlabeled data, determine any data v from other logistics data except u, take the data u and v as a data pair, and determine the label types of the data u and v respectively. According to the label types of the data u and v, determine the association between the data pair u and v; according to the association w(u,v) between the target data u and other logistics data, determine the association relationship between the data u and other data.
[0071] In order to determine the label association factor of the target data to be tested and the other target data according to the object features corresponding to the target data to be tested and the other target data, and the local densities corresponding to each of them, in one embodiment of the present invention, for each of the data pairs: according to the object features corresponding to the target data to be tested and the object features corresponding to the other data, the difference degrees corresponding to the data pairs are calculated; according to the difference degrees and the clustering degrees corresponding to the data pairs, the label association factors corresponding to the data pairs are calculated.
[0072] According to the item feature values corresponding to the logistics data, such as weight value, the distance variance std between each logistics data and other logistics data is calculated, and the distance variance is determined as the difference between each logistics data and other logistics data. Therefore, the distance variance between data u and v and other logistics data except themselves can be determined as std(u) and std(v), that is, the difference corresponding to data u and v is determined as std(u) and std(v). Next, the ratio of the difference corresponding to data u and v, and the ratio of the aggregation (local density) corresponding to data u and v are calculated. The product of the two ratios is used as the label association factor λ(u,v) between data u and v. The calculation formula of the label association factor is as follows:
[0073]
[0074] Among them, λ(u,v) is the label association factor, std(u) and std(v) are the differences (distance variance) between data u and v and other logistics data respectively, and ρ(u) and ρ(v) are the aggregation (local density) of data u and v respectively.
[0075] In order to determine the difference between the target data to be tested and other target data, in one embodiment of the present invention, for each of the data pairs: according to the item characteristics corresponding to the target data to be tested and the item characteristics corresponding to the other data, calculating the difference degrees respectively corresponding to the data pairs includes: calculating the difference degrees between the target data to be tested and the other data according to the item characteristics corresponding to the target data to be tested and the item characteristics corresponding to the other data, and determining the difference degree as the difference degree corresponding to the target data to be tested; calculating the difference degrees corresponding to the other target data according to the item characteristics corresponding to the other target data and the item characteristics corresponding to the logistics data other than the other target data in the first logistics data and the second logistics data; taking the difference degrees corresponding to the target data to be tested and the difference degrees corresponding to the other target data as the difference degrees respectively corresponding to the data pairs.
[0076] The specific process of calculating the differences std(u) and std(v) corresponding to the data u and v respectively is as follows: based on the item characteristic value of the target data u, such as weight value, and the item characteristic value of other logistics data except the target data u, such as weight value, calculate the distance variance std(u) between the target data u and the corresponding other logistics data, and use std(u) as the difference between the target data u and the corresponding other logistics data; based on the item characteristic value of the data v, such as weight value, and the item characteristic value of other logistics data except the target data v, such as weight value, calculate the distance variance std(v) between the target data v and the corresponding other logistics data, and use std(v) as the difference between the target data v and the corresponding other logistics data; std(u) and std(v) are called the differences corresponding to the data u and v respectively.
[0077] In order to determine the degree of association corresponding to a data pair (i.e., the target data to be tested and other target data) according to the data label and the reference label, in one embodiment of the present invention, the degree of association corresponding to the data pair consisting of the target data to be tested and the other target data is determined according to the similarities and differences of the labels, including any one of the following:
[0078] In the case where the label type of the target data to be tested is the same as that of the other target data and both indicate that the corresponding logistics data is not abnormal, determining the association degree of the data pair according to the sum of the label association factor and a preset value;
[0079] In the case where the target data to be tested and the other target data have the same label type and both indicate that the corresponding logistics data is abnormal, determining the association degree of the data pair according to the difference between the preset value and the label association factor;
[0080] In the case that the label types of the target data to be tested are different from those of the other target data, the label association factor is used as the association degree of the data pair.
[0081] The target data to be tested and other target data form a data pair, such as data pair u and v, where the label of u is a reference label and the label of v is a reference label or a data label. The label situations of data pair u and v can be divided into three types: the label types of u and v are both non-abnormal data labels; the label types of u and v are both abnormal data labels; one of the label types of u and v is a non-abnormal data label and the other is an abnormal data label.
[0082] When the label types of data pairs u and v are both non-abnormal data labels, the correlation between data pairs u and v is w(u,v)=λ(u,v)+1;
[0083] When the label types of data pairs u and v are both abnormal data labels, the correlation between data pairs u and v is w(u,v)=1-λ(u,v);
[0084] When one of the label types of data pairs u and v is a non-abnormal data label and the other is an abnormal data label, the correlation between the data pairs u and v is w(u,v) = λ(u,v);
[0085] The correlation degree w(u,v) between the data pair u and v is the correlation degree between the target data to be tested and other target data.
[0086] Figure 2 It is a schematic diagram of the main steps of determining the correlation between data in the method for detecting abnormal data of logistics information according to an embodiment of the present invention.
[0087] like Figure 2 As shown, the method for detecting abnormal data of logistics information according to the embodiment of the present invention mainly includes the following steps:
[0088] Step S201: Determine any unlabeled data u1 and a neighbor set of u1;
[0089] Step S202: Determine the type and quantity of data tags included in the neighbor set;
[0090] Step S203: determining the tag type with a larger number as the reference tag of u1;
[0091] Step S204: determine the next unlabeled data, and execute the above steps S201-S203 for it, until there is no unlabeled data, and then execute step S205;
[0092] Step S205: Calculate the aggregation degree ρ and the difference degree std corresponding to each data, and calculate the label correlation factor λ according to the aggregation degree ρ and the difference degree std;
[0093] Step S206: Determine whether the label type of the target data u to be tested is the same as that of the other data v. If yes, execute step S207; if not, execute step S210;
[0094] Step S207: Determine whether the label type of data u and data v is non-abnormal, if yes, execute step S208; if no, execute step S209;
[0095] Step S208: Determine the correlation between data u and data v as w(u,v)=λ(u,v)+1;
[0096] Step S209: Determine the correlation between data u and data v as w(u,v)=1-λ(u,v);
[0097] Step S210: Determine the correlation between data u and data v as w(u,v)=λ(u,v).
[0098] The "correlation" mentioned in the embodiments of the present invention refers to a data pair, such as the relationship between data u and data v. In addition, the "association relationship" mentioned in the embodiments of the present invention refers to the relationship between one data and a data set, such as the relationship between data u and all other data.
[0099] Step S105: Determine whether the target data to be tested is abnormal data according to the aggregation degree and the association relationship.
[0100] According to the local density corresponding to each data and the correlation between the target data to be tested and other data, it is determined whether the target data to be tested is abnormal data deviating from other data.
[0101] In order to determine the average distribution density of a data set based on the local density of each data in the data set and the correlation between the data, in one embodiment of the present invention, determining whether the target data to be tested is abnormal data based on the clustering and the association relationship may include: for each other target data in the other data: calculating the associated clustering corresponding to the other target data based on the correlation corresponding to the data pair consisting of the other target data and the target data to be tested, and the clustering corresponding to the other target data; calculating the average clustering of the other target data based on the sum of the associated clustering corresponding to each other target data and the number of data of the other data; determining whether the target data to be tested is abnormal data based on the average clustering and the clustering corresponding to the target data to be tested.
[0102] According to the local density (i.e., aggregation) corresponding to each logistics data and the correlation between the target data and the logistics data, the associated local density (i.e., the associated aggregation w(u,v)*ρ(v)) corresponding to each logistics data is determined, and then the average distribution density (i.e., average aggregation) of the logistics data is determined. According to the average distribution density of the data and the local density of the target data, it is determined whether the target data is abnormal data.
[0103] In order to determine whether the target data to be tested is abnormal data when the average distribution density of the data set and the local density of the target data to be tested are known, in one embodiment of the present invention, determining whether the target data to be tested is abnormal data based on the average clustering degree and the clustering degree corresponding to the target data to be tested includes: calculating the ratio of the average clustering degree to the clustering degree corresponding to the target data to be tested; when the ratio is greater than a preset threshold, determining that the target data to be tested is abnormal data; when the ratio is not greater than the preset threshold, determining that the target data to be tested is non-abnormal data.
[0104] The product of the average distribution density and the inverse of the local density of the target data is determined as the anomaly factor LF(u) corresponding to the target data u, where the associated aggregation refers to w(u,v)*ρ(v), and the calculation formula of the anomaly factor is as follows:
[0105]
[0106] Among them, LF(u) is the abnormal factor corresponding to the target data u, w(u,v) refers to the correlation between data u and v, ρ(u) and ρ(v) are the aggregation degrees (local densities) corresponding to data u and v respectively, |T(u)| is the number of logistics data in the data set T(u), data u is the target data to be tested, and data v is any data in the data set T.
[0107] The preset threshold is set based on the historical experience of abnormal data detection. When the abnormal factor is greater than the preset threshold, the target data u is determined to be abnormal data, otherwise the target data u is determined to be non-abnormal data. According to the calculation formula of the abnormal factor, it can be seen that the correlation between the target data and other data and the data distribution density jointly determine whether the target data is abnormal data.
[0108] According to the abnormal data detection method of logistics information of the embodiment of the present invention, it can be seen that by obtaining multiple first logistics data and multiple second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, the local density algorithm is used to calculate the aggregation corresponding to the first logistics data and the second logistics data respectively; according to the data label, the reference label indicating whether the second logistics data is abnormal is determined by the label propagation algorithm; according to the data label and the reference label, the association relationship between any target data to be tested in the second logistics data and other data is determined; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, it is determined whether the target data to be tested is abnormal data. Thus, the abnormal data in the logistics information is detected by the aggregation degree of the data and the association relationship between the data, thereby reducing the impact of uneven data distribution on abnormal data detection and improving the accuracy of abnormal data detection.
[0109] Figure 3 It is a schematic diagram of the main steps of the abnormal data detection method of logistics information according to an embodiment of the present invention.
[0110] like Figure 3 As shown, the abnormal data detection method of logistics information in the embodiment of the present invention mainly includes the following steps:
[0111] Step S301: Acquire multiple logistics data, including labeled data and unlabeled data.
[0112] Acquire a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data include data labels; the data labels include abnormal data labels and non-abnormal data labels, and the second logistics data do not include the data labels.
[0113] Step S302: Convert the text data in the multiple logistics data into numerical data.
[0114] For example, convert category data: home appliances - refrigerators, electronic products - mobile phones, wine and beverages - beverages, food - biscuits and other text data into numerical data such as 0, 1, 2 or 00, 01, 10.
[0115] Step S303: Determine reference labels for unlabeled data.
[0116] Determine a neighbor set corresponding to any data in the second logistics data; determine the label type of data labels with a larger number in the neighbor set of any data; and use the determined label type as a reference label for any data.
[0117] Step S304: Determine a neighbor set, ie, data set T, for the target data to be tested.
[0118] The data set T, ie, the neighbor set of the target data to be tested, is determined by determining the union of the nearest neighbor data, the reverse nearest neighbor data, and the shared nearest neighbor data of the target data to be tested.
[0119] Step S305: Based on the data set T, the aggregation degrees corresponding to the target data to be tested and other data are calculated.
[0120] Calculate the local density corresponding to the target data to be tested and the local density corresponding to each data in the data set T.
[0121] Step S306: Determine the correlation between the target data to be tested and other data.
[0122] According to the similarities and differences in labels between the target data to be tested and each data pair consisting of other data, the corresponding correlation degree of the data pair is determined.
[0123] Step S307: Determine the abnormal factor corresponding to the target data.
[0124] According to the local density (i.e., aggregation) corresponding to each logistics data and the correlation between the target data and the logistics data, the associated local density (i.e., the associated aggregation w(u,v)*ρ(v)) corresponding to each logistics data is determined, and then the average distribution density (i.e., average aggregation) of the logistics data is determined. According to the average distribution density of the data and the local density of the target data, the abnormal factor corresponding to the target data is determined.
[0125] Step S308: Determine whether the abnormal factor is greater than a preset threshold, if yes, execute step S309; if no, execute step S310.
[0126] Generally speaking, abnormal data is data distributed outside the range of non-abnormal data aggregation. Therefore, the local density corresponding to the abnormal data is small. According to the calculation formula of the abnormal factor, the smaller the local density, the larger the abnormal factor.
[0127] Step S309: Determine whether the target data to be tested is abnormal data.
[0128] When the abnormal factor is greater than a preset threshold, the target data to be tested is determined to be abnormal data.
[0129] Step S310: Determine whether the target data to be tested is non-abnormal data.
[0130] When the abnormal factor is not greater than the preset threshold, the target data to be tested is determined as non-abnormal data.
[0131] Figure 4 4 is a schematic diagram of main modules of a device for detecting abnormal data of logistics information according to an embodiment of the present invention.
[0132] like Figure 4 As shown, the abnormal data detection device 200 of logistics information according to the embodiment of the present invention includes:
[0133] The data acquisition module 401 is used to acquire a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label;
[0134] A clustering calculation module 402, configured to calculate the clustering corresponding to the first logistics data and the second logistics data respectively by using a local density algorithm according to the item features included in the first logistics data and the second logistics data;
[0135] A label supplement module 403 is used to determine a reference label indicating whether the second logistics data is abnormal based on the data label by using a label propagation algorithm;
[0136] A relationship determination module 404 is used to determine the association relationship between any target data to be measured in the second logistics data and other data according to the data label and the reference label; wherein the other data is the logistics data other than the target data to be measured in the first logistics data and the second logistics data;
[0137] The abnormality judgment module 405 is used to determine whether the target data to be tested is abnormal data according to the aggregation degree and the association relationship.
[0138] In one embodiment of the present invention, the relationship determination module 404 is further used to determine any other target data from the other data, and determine the similarities and differences between the labels of the target data to be tested and the other target data;
[0139] Determining the correlation between the target data to be tested and the other target data according to the similarities and differences of the labels;
[0140] The association relationship between the target data to be tested and other data is determined according to the association degrees corresponding to the respective data pairs.
[0141] In one embodiment of the present invention, the relationship determination module 404 is also used to calculate, for each of the data pairs: the difference corresponding to the data pairs according to the object features corresponding to the target data to be tested and the object features corresponding to the other data; and the label association factor corresponding to the data pairs according to the difference and the clustering degree corresponding to the data pairs.
[0142] In one embodiment of the present invention, the relationship determination module 404 is also used to calculate the difference between the target data to be tested and the other data based on the item characteristics corresponding to the target data to be tested and the item characteristics corresponding to the other data, and determine the difference as the difference corresponding to the target data to be tested; calculate the difference corresponding to the other target data based on the item characteristics corresponding to the other target data and the item characteristics corresponding to the logistics data other than the other target data in the first logistics data and the second logistics data; and use the difference corresponding to the target data to be tested and the difference corresponding to the other target data as the differences corresponding to the data pairs respectively.
[0143] In one embodiment of the present invention, the relationship determination module 404 is further used for any of the following:
[0144] In the case where the label type of the target data to be tested is the same as that of the other target data and both indicate that the corresponding logistics data is not abnormal, determining the association degree of the data pair according to the sum of the label association factor and a preset value;
[0145] In the case where the target data to be tested and the other target data have the same label type and both indicate that the corresponding logistics data is abnormal, determining the association degree of the data pair according to the difference between the preset value and the label association factor;
[0146] In the case that the label types of the target data to be tested are different from those of the other target data, the label association factor is used as the association degree of the data pair.
[0147] In one embodiment of the present invention, the abnormality judgment module 405 is also used to calculate, for each other target data in the other data: the correlation aggregation degree corresponding to the other target data according to the correlation degree corresponding to the data pair consisting of the other target data and the target data to be tested, and the aggregation degree corresponding to the other target data; calculate the average aggregation degree of the other target data according to the sum of the correlation aggregation degrees corresponding to each other target data and the number of data of the other data; determine whether the target data to be tested is abnormal data according to the average aggregation degree and the aggregation degree corresponding to the target data to be tested.
[0148] In one embodiment of the present invention, the anomaly judgment module 405 is also used to calculate the ratio of the average concentration to the concentration corresponding to the target data to be tested; when the ratio is greater than a preset threshold, the target data to be tested is determined to be abnormal data; when the ratio is not greater than the preset threshold, the target data to be tested is determined to be non-abnormal data.
[0149] In one embodiment of the present invention, the aggregation calculation module 402 is also used for: for any target logistics data among the first logistics data and the second logistics data, according to the item characteristics, determining a neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data; and according to the item characteristics included in the neighbor set, using a local density algorithm to respectively calculate the aggregation corresponding to the target logistics data.
[0150] In one embodiment of the present invention, the aggregation calculation module 402 is also used to determine the nearest neighbor data, reverse nearest neighbor data and shared nearest neighbor data of the target logistics data from the first logistics data and the second logistics data according to the item characteristics; and take the union of the nearest neighbor data, the reverse nearest neighbor data and the shared nearest neighbor data as the neighbor set.
[0151] In one embodiment of the present invention, the label supplement module 403 is also used to determine a neighbor set corresponding to any data in the second logistics data; determine the label type of data labels with a larger number in the neighbor set of any data; and use the determined label type as a reference label for any data.
[0152] In one embodiment of the present invention, the aggregation degree calculation module 402 is further used to convert the text data into numerical data when the acquired first logistics data and / or the second logistics data include text data.
[0153] According to the abnormal data detection device of the logistics information of the embodiment of the present invention, it can be seen that by acquiring a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, the aggregation corresponding to the first logistics data and the second logistics data is calculated respectively by using the local density algorithm; according to the data label, a reference label indicating whether the second logistics data is abnormal is determined by using the label propagation algorithm; according to the data label and the reference label, the association relationship between any target data to be tested in the second logistics data and other data is determined; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, whether the target data to be tested is abnormal data is determined. Thus, the abnormal data in the logistics information is detected by the aggregation degree of the data and the association relationship between the data, thereby reducing the influence of uneven data distribution on abnormal data detection and improving the accuracy of abnormal data detection.
[0154] Figure 5 An exemplary system architecture 500 is shown to which the abnormal data detection method for logistics information or the abnormal data detection device for logistics information according to an embodiment of the present invention can be applied.
[0155] like Figure 5 As shown, system architecture 500 may include sensors 501, 502, 503, a network 504, and a server 505. Network 504 is used to provide a medium for communication links between sensors 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables.
[0156] The user can use the sensors 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. The sensors 501, 502, 503 can be weight sensors, barcode scanners, laser distance sensors, etc.
[0157] The server 505 may be a server that provides various services, such as a background management server that provides support for abnormal data detection of logistics data produced by sensors 501, 502, and 503. The background management server may analyze and process the received logistics data including data tags and logistics data not including tags, and feed back the processing results (e.g., whether the target data to be tested is abnormal data or non-abnormal data) to the sensor.
[0158] It should be noted that the abnormal data detection method for logistics information provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the abnormal data detection device for logistics information is generally set in the server 505 .
[0159] It should be understood that Figure 5 The number of sensors, networks and servers in the embodiment is only for illustration. Any number of sensors, networks and servers may be provided according to the implementation requirements.
[0160] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0161] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0162] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0163] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
[0164] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0166] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a data acquisition module, a concentration calculation module, a label supplement module, a relationship determination module, and an abnormality judgment module. Among them, the names of these modules do not constitute a limitation on the modules themselves under certain circumstances. For example, the abnormality judgment module may also be described as "a module that determines whether the target data to be tested is abnormal data based on the concentration and the association relationship".
[0167] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, respectively calculate the aggregation corresponding to the first logistics data and the second logistics data using a local density algorithm; according to the data label, determine a reference label indicating whether the second logistics data is abnormal using a label propagation algorithm; according to the data label and the reference label, determine the association relationship between any target data to be tested in the second logistics data and other data; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, determine whether the target data to be tested is abnormal data.
[0168] According to the technical solution of the embodiment of the present invention, by obtaining a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; according to the item features included in the first logistics data and the second logistics data, the aggregation corresponding to the first logistics data and the second logistics data is calculated respectively using a local density algorithm; according to the data label, a reference label indicating whether the second logistics data is abnormal is determined using a label propagation algorithm; according to the data label and the reference label, the association relationship between any target data to be tested in the second logistics data and other data is determined; wherein the other data is the logistics data other than the target data to be tested in the first logistics data and the second logistics data; according to the aggregation degree and the association relationship, it is determined whether the target data to be tested is abnormal data. Thus, the abnormal data in the logistics information is detected by the aggregation degree of the data and the association relationship between the data, thereby reducing the impact of uneven data distribution on abnormal data detection and improving the accuracy of abnormal data detection.
[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for detecting abnormal data in logistics information, characterized in that: include: Acquire a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data include data labels; the data labels include abnormal data labels and non-abnormal data labels, and the second logistics data do not include the data labels; Calculate the aggregation degree corresponding to the first logistics data and the second logistics data respectively by using a local density algorithm according to the item features included in the first logistics data and the second logistics data; Determine, according to the data label, a reference label indicating whether the second logistics data is abnormal by using a label propagation algorithm; Determine the association relationship between any target data to be measured in the second logistics data and other data according to the data tag and the reference tag; wherein the other data is the logistics data other than the target data to be measured in the first logistics data and the second logistics data; According to the aggregation degree and the association relationship, it is determined whether the target data to be tested is abnormal data.
2. The method according to claim 1, characterized in that The determining, according to the data tag and the reference tag, the association relationship between any target data to be measured in the second logistics data and other data includes: Determine any other target data from the other data, and determine the similarities and differences between the labels of the target data to be tested and the other target data; Determining the correlation between the target data to be tested and the other target data according to the similarities and differences of the labels; The association relationship between the target data to be tested and other data is determined according to the association degrees corresponding to the respective data pairs.
3. The method according to claim 2, characterized in that For each of the data pairs: calculating the difference degrees respectively corresponding to the data pairs according to the object features corresponding to the target data to be tested and the object features corresponding to the other data; According to the difference degree and the aggregation degree corresponding to the data pair, a label association factor corresponding to the data pair is calculated.
4. The method according to claim 3, characterized in that For each of the data pairs: calculating the difference degree corresponding to the data pairs respectively according to the object feature corresponding to the target data to be tested and the object feature corresponding to the other data, including: Calculating the difference between the target data to be tested and the other data according to the object features corresponding to the target data to be tested and the object features corresponding to the other data, and determining the difference as the difference corresponding to the target data to be tested; Calculate the difference degree corresponding to the other target data according to the item characteristics corresponding to the other target data and the item characteristics corresponding to the logistics data other than the other target data in the first logistics data and the second logistics data; The difference degree corresponding to the target data to be tested and the difference degree corresponding to the other target data are used as the difference degrees respectively corresponding to the data pairs.
5. The method according to claim 3, characterized in that: Determining the correlation degree corresponding to the data pair consisting of the target data to be tested and the other target data according to the similarities and differences of the labels includes any one of the following: In the case where the label type of the target data to be tested is the same as that of the other target data and both indicate that the corresponding logistics data is not abnormal, determining the association degree of the data pair according to the sum of the label association factor and a preset value; In the case where the target data to be tested and the other target data have the same label type and both indicate that the corresponding logistics data is abnormal, determining the association degree of the data pair according to the difference between the preset value and the label association factor; In the case that the label types of the target data to be tested are different from those of the other target data, the label association factor is used as the association degree of the data pair.
6. The method according to claim 2, characterized in that The determining, according to the aggregation degree and the association relationship, whether the target data to be tested is abnormal data includes: For each other target data in the other data: calculating the correlation aggregation degree corresponding to the other target data according to the correlation degree corresponding to the data pair consisting of the other target data and the target data to be tested, and the aggregation degree corresponding to the other target data; Calculate the average aggregation degree of the other target data according to the sum of the associated aggregation degrees corresponding to each other target data and the number of the other data; Whether the target data to be tested is abnormal data is determined according to the average degree of aggregation and the degree of aggregation corresponding to the target data to be tested.
7. The method according to claim 6, characterized in that The determining whether the target data to be tested is abnormal data according to the average concentration degree and the concentration degree corresponding to the target data to be tested includes: Calculating the ratio of the average aggregation degree to the aggregation degree corresponding to the target data to be tested; When the ratio is greater than a preset threshold, determining that the target data to be tested is abnormal data; When the ratio is not greater than a preset threshold, it is determined that the target data to be tested is non-abnormal data.
8. The method according to claim 1, characterized in that The calculating, according to the item features included in the first logistics data and the second logistics data, respectively the aggregation degrees corresponding to the aggregation degrees corresponding to the first logistics data and the second logistics data by using a local density algorithm, comprises: For any target logistics data of the first logistics data and the second logistics data: According to the item characteristics, determining a neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data; According to the features of the items included in the neighbor set, the aggregation degrees corresponding to the target logistics data are calculated respectively using a local density algorithm.
9. The method according to claim 8, characterized in that The step of determining a neighbor set corresponding to the target logistics data from the first logistics data and the second logistics data according to the item characteristics includes: According to the item characteristics, determine the nearest neighbor data, reverse nearest neighbor data, and shared nearest neighbor data of the target logistics data from the first logistics data and the second logistics data; The union of the nearest neighbor data, the reverse nearest neighbor data and the shared nearest neighbor data is taken as the neighbor set.
10. The method according to claim 1, characterized in that The step of determining, according to the data label, a reference label indicating whether the second logistics data is abnormal by using a label propagation algorithm includes: Determine a neighbor set corresponding to any data in the second logistics data; Determine the label type of the data labels with a larger number in the neighbor set of any data; The determined label type is used as a reference label for any of the data.
11. The method according to claim 1, characterized in that: Also includes: In the case where the acquired first logistics data and / or the second logistics data include text data, the text data is converted into numerical data.
12. A device for detecting abnormal data of logistics information, characterized in that: include: A data acquisition module, used to acquire a plurality of first logistics data and a plurality of second logistics data, wherein the first logistics data includes a data label; the data label includes an abnormal data label and a non-abnormal data label, and the second logistics data does not include the data label; a concentration calculation module, configured to calculate the concentrations corresponding to the first logistics data and the second logistics data respectively by using a local density algorithm according to the item features included in the first logistics data and the second logistics data; a label supplement module, configured to determine, based on the data label, a reference label indicating whether the second logistics data is abnormal by using a label propagation algorithm; A relationship determination module, used to determine the association relationship between any target data to be measured in the second logistics data and other data according to the data label and the reference label; wherein the other data is the logistics data other than the target data to be measured in the first logistics data and the second logistics data; The abnormality judgment module is used to determine whether the target data to be tested is abnormal data according to the aggregation degree and the association relationship.
13. A server, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.
14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Data processing method and device and readable storage medium
CN113011646A
New energy operation data anomaly detection method and system based on memory matrix
CN114943272A
Novel data detection method and device
CN115146795A
Express weight abnormity identification method, device and equipment and storage medium
CN115393627A
Field electric energy metering error prediction method and system of gateway electric energy meter, and medium
CN115471361A
