A traffic smart card data cleaning method and system

Abstract

The present application relates to the field of data processing technology, and specifically to a method and system for cleaning traffic smart card data, the method comprising: collecting all traffic smart card data, obtaining the information contribution of traffic smart card data according to the similarity characteristics between the lines of any two traffic smart card data, the time interval between entering and leaving the station, the relationship between the number of stations passed, and the difference in the time of entering the station; adjusting the distance metric of the information contribution in the clustering process; obtaining the abnormal value of each traffic smart card data based on the clustering anomaly detection algorithm; and cleaning the traffic smart card data in combination with the threshold segmentation algorithm. The present application aims to analyze the factors affecting traffic passage, improve the accuracy of anomaly detection and the effectiveness of data cleaning.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for cleaning traffic smart card data. Background Art

[0002] With the improvement and optimization of urban public transportation systems and the increasing popularity of the concept of green travel, the proportion of public transportation has continued to increase, and the passenger flow has increased, which has also generated a large amount of traffic data, which can be reflected by the information of traffic smart cards. By mining the huge amount of traffic smart card data, a lot of useful information can be obtained, which can guide traffic planning and operation, and improve the convenience and service quality of public transportation.

[0003] Since various factors may cause anomalies in passengers' transportation smart card data, anomaly detection algorithms are usually used to identify these anomalies during the data cleaning process. In the clustering-based anomaly detection algorithm, the anomaly detection process first relies on clustering, and then calculates the anomaly value based on the clustering results. However, the algorithm does not consider whether the information carried by each piece of transportation smart card data is universal during the clustering stage. The final clustering result is only generated based on the differences between the data, while ignoring the changes in traffic conditions, which makes the anomaly detection result inaccurate and affects the effectiveness of data cleaning. Summary of the invention

[0004] In view of the above, it is necessary to provide a traffic smart card data cleaning method and system to solve the above problems.

[0005] The first aspect of the present application provides a method for cleaning data of a transportation smart card, the method comprising:

[0006] Collect all traffic smart card data, each piece of traffic smart card data includes at least route information and entry and exit information; the entry and exit information includes at least entry and exit time information and station information;

[0007] According to the similarity characteristics between the routes of any two pieces of traffic smart card data, a first contribution degree between the any two pieces of traffic smart card data is obtained; according to the relationship between the time interval between the any two pieces of traffic smart card data entering and leaving the station and the route information, a second contribution degree between the any two pieces of traffic smart card data is obtained; according to the difference between the entry time of each piece of traffic smart card data and other pieces of traffic smart card data, the information contribution degree of each piece of traffic smart card data is obtained by combining the first contribution degree and the second contribution degree;

[0008] According to the difference characteristics between any two pieces of traffic smart card data entry and exit information, the distance metric of the information contribution in the clustering process is adjusted; based on the clustering results, the outliers of each piece of traffic smart card data are obtained;

[0009] According to the abnormal values ​​and information contribution, combined with the threshold segmentation algorithm, the traffic smart card data is cleaned.

[0010] The process of obtaining the first contribution between any two pieces of traffic smart card data is as follows:

[0011] According to the route information of each piece of traffic smart card data, a set consisting of the digital numbers of the corresponding entry and exit stations and all the stations passed through in the middle is used as the path set of each piece of traffic smart card data;

[0012] Get the intersection and union of any two paths of traffic smart card data;

[0013] According to the number of elements in the intersection and the number of elements in the union of any two pieces of traffic smart card data, a first contribution degree between the any two pieces of traffic smart card data is obtained; wherein the first contribution degree is positively correlated with the number of elements in the intersection and negatively correlated with the number of elements in the union.

[0014] The steps for obtaining the second contribution between any two pieces of traffic smart card data are as follows:

[0015] According to the entry time and exit time of each transportation smart card data, the riding time of each transportation smart card data is obtained;

[0016] Calculate the subtraction result of the number of stations along the way corresponding to each piece of transportation smart card data and a preset value, and use the ratio of the riding time to the subtraction result as the average riding time of each piece of transportation smart card data;

[0017] According to the difference between the average riding time of any two pieces of traffic smart card data, a second contribution degree between the any two pieces of traffic smart card data is obtained.

[0018] The information contribution of each piece of traffic smart card data is obtained as follows:

[0019] According to the difference characteristics of the entry time between any two pieces of traffic smart card data, a third contribution degree between the any two pieces of traffic smart card data is obtained;

[0020] The information contribution of the i-th traffic smart card data is recorded as , its formula form is: ;in, is the total number of traffic smart card data, For the Traffic smart card data and The first contribution between the traffic smart card data, For the Traffic smart card data and The second contribution between the traffic smart card data, For the Traffic smart card data and The third contribution between the traffic smart card data, is the number of all path sets whose intersection with the path set described in the i-th transportation smart card data is not an empty set.

[0021] The process of obtaining the third contribution degree between any two pieces of traffic smart card data includes:

[0022] The absolute value of the difference between the entry times of any two pieces of traffic smart card data is calculated; and the third contribution between the any two pieces of traffic smart card data is obtained according to the positive relationship of the absolute value of the difference in the logarithmic function.

[0023] The process of adjusting the distance metric of the information contribution in the clustering process is as follows:

[0024] According to the distance between the entry sites and the distance between the exit sites in any two pieces of traffic smart card data, the first distance and the second distance of the any two pieces of traffic smart card data are obtained;

[0025] Obtaining the minimum information contribution of any two pieces of traffic smart card data; calculating the proportion of the minimum information contribution in the information contribution of all traffic smart card data;

[0026] Merge the difference in the travel time of any two pieces of transportation smart card data, the first distance and the second distance;

[0027] Based on the proportion and the fusion result, the adjusted distance between the arbitrary two pieces of traffic smart card data is obtained; wherein the adjusted distance is negatively correlated with the proportion and positively correlated with the fusion result.

[0028] The step of obtaining the abnormal value of each piece of traffic smart card data is as follows:

[0029] All the entry and exit information of each piece of traffic smart card data constitutes a feature vector of each piece of traffic smart card data;

[0030] The feature vectors of all traffic smart card data and the preset number of clusters are used as the input of the CBLOF anomaly detection algorithm, and the output is the anomaly value of each traffic smart card data.

[0031] The cleaning of the traffic smart card data is specifically as follows:

[0032] Perform threshold segmentation on the outliers and information contribution of all traffic smart card data to obtain the outlier segmentation threshold and contribution segmentation threshold;

[0033] Abnormal data is obtained according to the abnormal segmentation threshold and the contribution segmentation threshold; and the abnormal data is deleted to obtain cleaned traffic smart card data.

[0034] The abnormal data specifically refers to traffic smart card data whose abnormal value is greater than or equal to the abnormal segmentation threshold and whose information contribution is less than the contribution segmentation threshold.

[0035] In a second aspect, an embodiment of the present application further provides a transportation smart card data cleaning system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0036] In the above scheme, each piece of traffic smart card data is first obtained, and the riding information reflected by each traffic smart card is analyzed based on the station information and entry and exit time information corresponding to each piece of traffic smart card data; according to the similarity between the routes, the first contribution is constructed to measure the similarity of the paths between the traffic smart card data, which is helpful for the subsequent characterization of the universal characteristics of the data; considering the time interval between entering and exiting the station, as well as the station conditions in the middle, the second contribution is obtained, and the similarity of the riding time of each station between the traffic smart card data is further measured. The universality of the traffic smart card data is considered from multiple angles, and the information contribution is obtained in combination with the difference in entry time, which is helpful for accurately measuring the universality of the traffic smart card data, thereby improving the accuracy of subsequent anomaly detection; then the information contribution is used to adjust the distance measurement formula in the clustering-based anomaly detection algorithm to obtain the adjusted distance between any two traffic smart card data, while considering the difference between the two traffic smart card data themselves and the universality of the information carried, which is helpful for improving the rationality of clustering, thereby improving the accuracy of subsequent abnormal data detection and judgment. In this way, when cleaning the traffic smart card data, the differences in the data itself and the impact of changes in traffic conditions on the data are taken into account at the same time, which improves the judgment of the validity of the information carried by the data, thereby helping to improve the accuracy of anomaly detection, so that the detected abnormal data is data carrying less valid information, thereby improving the quality of data cleaning, and also helping to improve the quality and validity of information during subsequent information mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A flow chart of the steps of a method for cleaning data of a transportation smart card provided by an embodiment of the present application;

[0038] Figure 2 A flowchart for obtaining abnormal values ​​provided in one embodiment of the present application. DETAILED DESCRIPTION

[0039] In the description of the embodiments of the present application, words such as "exemplary", "or", "for example" and the like are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "or", "for example" and the like is intended to present related concepts in a concrete manner.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0041] It should also be noted that the terms "first" and "second" in this application and the accompanying drawings are used to distinguish similar objects, rather than to describe a specific order or sequence. The method disclosed in the embodiments of the present application or the method shown in the flow chart includes one or more steps for implementing the method. Without departing from the scope of protection of the present application, the execution order of multiple steps can be interchanged with each other, and some steps can also be deleted.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0043] The following is a detailed description of a method and system for cleaning data of a transportation smart card provided by the present application in conjunction with the accompanying drawings.

[0044] See also Figure 1 , which shows a flow chart of a method for cleaning data of a transportation smart card provided by an embodiment of the present application, the method comprising the following steps:

[0045] The first step is to collect all traffic smart card data, each piece of traffic smart card data includes at least route information and entry and exit information; the entry and exit information includes at least entry and exit time information and station information.

[0046] The data of each transportation smart card is obtained through the transportation card big data platform. Each transportation smart card data includes the card number, route information, entry station and entry time, exit station and exit time.

[0047] Among them, the route information can be obtained through the card number information of each transportation smart card data and the passenger identity information of the one-card big data platform, using the target tracking network to obtain the route information of the passenger corresponding to each transportation smart card data. The route information specifically includes the passenger's entry station, exit station and all stations passed in the middle. The target tracking network is an existing well-known technology and will not be described in detail in this application.

[0048] It should be noted that this embodiment standardizes all station data and time data to obtain the coordinates of each station. In this embodiment, the standardization method adopted is to assign a digital number to each station, and the digital numbers of different stations are not repeated. The coordinates of the station are represented by the longitude and latitude of the station; it should be understood that the route information of each transportation smart card data is specifically expressed as a set of digital numbers of the stations.

[0049] All times are also standardized. In this embodiment, the time data is 24-hour data. The specific operation of standardizing the time data is: starting from 0:00 on the day, the number of hours in a day is converted into a value between 0 and 24 in proportion. For example, if the time is 01:10:10, then starting from 0:00 on the day, the standardized time is Through the above method, each piece of traffic smart card data is standardized and then preprocessed to delete missing values ​​or duplicate traffic smart card data.

[0050] The second step: according to the similarity characteristics between the routes of any two traffic smart card data, obtain the first contribution between the any two traffic smart card data; according to the relationship between the time interval of entering and leaving the station and the route information of the any two traffic smart card data, obtain the second contribution between the any two traffic smart card data; according to the difference in the entry time of each traffic smart card data and other traffic smart card data, combined with the first contribution and the second contribution, obtain the information contribution of each traffic smart card data.

[0051] As one of the public service facilities that facilitate public travel, public transportation has universal characteristics in its decision-making, operation and management, that is, it needs to meet the travel needs of most people. Therefore, the more universal the information provided by the traffic smart card data is, the more it can reflect the public's traffic conditions, and the more guiding significance it has for public transportation decision-making and operation. However, the system anomalies at the transportation station or the personal behavior of passengers lead to abnormal traffic smart card data. Such anomalies are accidental and unique, that is, they cannot represent the travel needs and changes of most people, and have little value in guiding transportation decision-making and operation. For example, the passenger's ride time is calculated based on the entry and exit time of the traffic smart card. When a passenger actively stays at the station for a long time, the ride time will increase. However, such abnormal changes in data are caused by the passenger's personal behavior, which does not have universal characteristics and cannot reflect the travel data of most passengers. It is meaningless for transportation decision-making and operation and has little value.

[0052] In order to encourage the use of public transportation and improve the efficiency of public transportation, "dedicated lanes" will be set up in the construction of urban transportation networks to ensure the operation of public transportation. Because of the existence of "dedicated lanes", the average ride time between any two stations is relatively consistent or close. The riding data in the passenger transportation smart card data is of great value for public transportation decision-making and operation. Due to the stability of public transportation operation, the universality of the information contained can be measured based on the passenger riding data. The more universal the information is, the more valuable it is for public transportation decision-making and operation.

[0053] Due to the distance between different stations and differences in traffic conditions, the ride time often varies. In addition, even if starting from the same station, the ride time may also vary due to the different travel times and traffic routes chosen by passengers, so the information reflected in each transportation smart card data is different.

[0054] Based on this, firstly, according to the similarity characteristics between the routes of any two traffic smart card data, the first contribution between the any two traffic smart card data is obtained:

[0055] According to the route information of each piece of traffic smart card data, a set consisting of the digital numbers of the corresponding entry and exit stations and all the intermediate stations is used as the path set of each traffic smart card data; the intersection and union of the path sets of any two traffic smart card data are obtained; according to the number of elements in the intersection and the number of elements in the union of the any two traffic smart card data, the first contribution degree between the any two traffic smart card data is obtained; wherein the first contribution degree is positively correlated with the number of elements in the intersection and negatively correlated with the number of elements in the union.

[0056] It should be noted that positive correlation means that as one variable increases, the other variable also increases; while negative correlation means that as one variable increases, the other variable decreases.

[0057] In this embodiment, the first contribution is specifically as follows: calculating the ratio of the number of elements in the intersection to the number of elements in the union, using the ratio as the exponent of an exponential function with a natural constant as the base, calculating the sum of the number of elements in the intersection and a set value, using the sum as the true number of a logarithmic function with a natural constant as the base; using the product of the calculation result of the exponential function and the calculation result of the logarithmic function as the first contribution between the two pieces of traffic smart card data. The set value is 1 to ensure that the independent variable value in the logarithmic function is greater than 0.

[0058] In other embodiments, the ratio of the number of elements in the intersection to the number of elements in the union is used as the first contribution between the any two pieces of traffic smart card data.

[0059] It should be understood that the more similar the route information is, the smaller the difference between the two traffic smart card data is, the greater the probability of pointing to the same riding route, the more universal the two traffic smart card data is, and thus the greater the first contribution is.

[0060] Furthermore, according to the relationship between the time interval between the entry and exit of the station and the number of stations passed by the two pieces of traffic smart card data, the second contribution between the two pieces of traffic smart card data is obtained:

[0061] According to the entry time and exit time of each piece of transportation smart card data, the ride time of each piece of transportation smart card data is obtained; according to the ride time and the number of passed stations, the average ride time of each piece of transportation smart card data is obtained; and the average ride time of any two pieces of transportation smart card data is compared to obtain the second contribution degree between the any two pieces of transportation smart card data.

[0062] In this implementation, the riding time refers to the difference between the exit time and the entry time of each piece of traffic smart card data; the average riding time is specifically: the number of passing stations on the corresponding route between the entry station and the exit station minus the preset value, and the ratio of the riding time to the result is calculated, and the preset value is 1; the average riding time of any two pieces of traffic smart card data is obtained, and the ratio of the minimum value to the maximum value is recorded as ,Will As the second contribution between any two pieces of traffic smart card data, is an exponential function with a natural constant as the base. In other embodiments, As the second contribution between the two pieces of traffic smart card data.

[0063] Next, according to the difference in entry time between each piece of traffic smart card data and other traffic smart card data, the information contribution of each piece of traffic smart card data is obtained by combining the second contribution and the first contribution:

[0064] The information contribution of the i-th traffic smart card data is recorded as , its formula form is: ;in, is the total number of traffic smart card data, For the Traffic smart card data and The first contribution between the traffic smart card data, For the Traffic smart card data and The second contribution between the traffic smart card data, For the Traffic smart card data and The third contribution between the traffic smart card data, is the number of all path sets whose intersection with the path set of the i-th traffic smart card data is not an empty set. The third contribution degree is specifically: calculate the absolute value of the difference between the entry time of the i-th and j-th traffic smart card data, record the absolute value of the difference as t, and As the third contribution, is a natural constant.

[0065] It should be understood that the more similar the traffic smart card data is to the information reflected in other traffic smart card data, that is, the closer the entry time is, the closer the average riding time is, and the more similar the routes are, the more universal the traffic smart card data is, and the greater the information contribution of the traffic smart card data is.

[0066] It should be further explained that when the third contribution is large, it means that the traffic conditions of the two traffic smart card data at different times are different. The increase in this difference may lead to a decrease in the ride similarity metric, but the overall information contribution is still increased. Since the public transportation routes are fixed, the time spent between the same two stations will fluctuate under the traffic conditions at different times, so the ride similarity metric may decrease to a certain extent; however, due to the different entry times of passengers, the third contribution will increase. Since the ride similarity metric and the third contribution are scaled by exponential functions and logarithmic functions respectively, combined with the range of data values, the increase in the entry time difference metric is greater than the decrease in the ride similarity metric, so the difference in traffic conditions at different times has little effect on the information contribution, and the information contribution increases.

[0067] Furthermore, when there are no anomalies between the traffic smart card data, the greater the information contribution; otherwise, the smaller the information contribution. Secondly, under normal circumstances, the smaller the difference between the riding station sets, the more similar the riding conditions between the corresponding traffic smart card data should be, and the more it can reflect the common riding conditions between the corresponding stations. At this time, a greater weight is given, that is, the greater the first contribution; the role of the first contribution is to give corresponding weights according to the similarity of the riding paths, so as to enhance the influence weight of more common riding conditions on the information contribution, thereby making the corresponding traffic smart card data more universal.

[0068] The third step: according to the difference characteristics between the entry and exit information of any two pieces of traffic smart card data, the distance measurement of the information contribution in the clustering process is adjusted; based on the clustering results, the outliers of each piece of traffic smart card data are obtained.

[0069] The entry site coordinates, exit site coordinates, entry time, and exit time of each piece of traffic smart card data together constitute the feature vector of each piece of traffic smart card data, and the feature vectors of all traffic smart card data and the preset number of clusters are used as the input of the cluster-based local outlier factor (CBLOF) detection algorithm, and the outlier value of each piece of traffic smart card data, that is, the LOF value, is output. In this embodiment, the number of clusters is preset to 10, and the CBLOF anomaly detection algorithm is a well-known technology, and the specific process is not repeated.

[0070] Among them, the flowchart for obtaining abnormal values ​​is as follows: Figure 2 shown.

[0071] In order to avoid the above problems, it is necessary to consider the universality of the information carried by each piece of traffic smart card data during clustering. When traffic conditions change and affect riding, it often causes universal impact. Clustering based on this can simultaneously consider the changes in riding data and traffic conditions, thereby improving the quality of subsequent abnormal data detection. Based on the above analysis, the distance formula for clustering is adjusted:

[0072] : It is Traffic smart card data and The adjusted distance between the traffic smart card data; It is Information contribution of traffic smart card data; is the total number of traffic smart card data; It is Traffic smart card data and The minimum contribution of traffic smart card data information; is the distance measure between the entry site of the i-th traffic smart card data and the r-th traffic smart card data, recorded as the first distance; is the distance measure between the exit site of the i-th traffic smart card data and the r-th traffic smart card data, recorded as the second distance; It is the difference between the entry and exit time of the i-th traffic smart card data and the r-th traffic smart card data.

[0073] In this embodiment, the distance measurement adopts Manhattan distance, and the difference of entry and exit time refers to the absolute value of the difference of riding time; in other embodiments, Euclidean distance can be used instead of Manhattan distance; in addition, the entry time of each transportation smart card data is used as the first dimension data, and the exit time is used as the second dimension data, together forming a two-dimensional riding time vector of transportation smart card data, and the difference of entry and exit time between transportation smart card data refers to the Euclidean distance of the riding time vectors of two transportation smart card data.

[0074] It should be understood that when the information carried by the two pieces of traffic smart card data is universal and the riding conditions between the two pieces of data are similar, the difference between the two pieces of data is small, i.e. , , The more universal the information carried by two pieces of data is, the greater the information contribution is, and thus the distance between the two pieces of data is On the contrary, when one of the two data carries less universal information, the information contribution is smaller, so that the adjusted distance between the two data is relatively larger.

[0075] The fourth step: according to the abnormal values ​​and information contribution, combined with the threshold segmentation algorithm, the traffic smart card data is cleaned.

[0076] The Otsu threshold segmentation algorithm is used for the LOF value of all traffic smart card data to obtain the abnormal segmentation threshold. Although the universality of the information carried in the traffic smart card data is considered to a certain extent during clustering, it cannot be guaranteed that the universality of the information carried in the abnormal data obtained in the end is poor, so the Otsu threshold segmentation algorithm is used for the information contribution of all traffic smart card data to obtain the contribution segmentation threshold. Traffic smart card data that simultaneously meets the LOF value greater than the abnormal segmentation threshold and the information contribution less than the contribution segmentation threshold are marked as abnormal data and deleted.

[0077] It should be understood that, through the above method, when performing anomaly detection, not only the difference of the data itself is considered, but also the universality of the data is considered. Through these two aspects, the abnormal data screened out during anomaly detection is the traffic smart card data carrying information with less value for traffic decision-making and operation. In this way, the traffic smart card data is cleaned and the quality of the cleaned data is improved. When the cleaned data is used for information mining in the future, the quality and effectiveness of the mined information can also be improved.

[0078] Based on the same inventive concept as the above method, an embodiment of the present application also provides a traffic smart card data cleaning system, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of any one of the above-mentioned traffic smart card data cleaning methods are implemented.

[0079] In summary, the embodiment of the present application first obtains each piece of traffic smart card data, and analyzes the riding information reflected by each traffic smart card based on the station information and entry and exit time information corresponding to each piece of traffic smart card data; constructs a first contribution degree according to the similarity between the routes, measures the similarity of the paths between the traffic smart card data, and helps to subsequently characterize the universal characteristics of the data; considers the time interval between entering and exiting the station, as well as the station conditions in the middle, obtains a second contribution degree, and further measures the similarity of the riding time of each station between the traffic smart card data, considers the universality of the traffic smart card data from multiple angles, and combines the difference in entry time to obtain the information contribution degree, which helps to accurately measure the universality of the traffic smart card data, thereby improving the accuracy of subsequent anomaly detection; then uses the information contribution degree to adjust the distance measurement formula in the clustering-based anomaly detection algorithm to obtain the adjusted distance between any two traffic smart card data, while considering the difference between the two traffic smart card data themselves, as well as the universality of the information carried, which helps to improve the rationality of clustering, thereby improving the accuracy of subsequent abnormal data detection and judgment. In this way, when cleaning the traffic smart card data, the differences in the data itself and the impact of changes in traffic conditions on the data are taken into account at the same time, which improves the judgment of the validity of the information carried by the data, thereby helping to improve the accuracy of anomaly detection, so that the detected abnormal data is data carrying less valid information, thereby improving the quality of data cleaning, and also helping to improve the quality and validity of information during subsequent information mining.

[0080] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two continuous operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0081] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the basic features of the present application. Therefore, no matter from which point of view, the above embodiments of the present application should be regarded as exemplary and non-restrictive; the technical solutions recorded in the above embodiments are modified, or some of the technical features are replaced by equivalents, which does not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for cleaning traffic smart card data, characterized in that: The method comprises the following steps: Collect all traffic smart card data, each piece of traffic smart card data includes at least route information and entry and exit information; the entry and exit information includes at least entry and exit time information and station information; According to the similarity characteristics between the routes of any two pieces of traffic smart card data, a first contribution degree between the any two pieces of traffic smart card data is obtained; according to the relationship between the time interval between the any two pieces of traffic smart card data entering and leaving the station and the route information, a second contribution degree between the any two pieces of traffic smart card data is obtained; according to the difference between the entry time of each piece of traffic smart card data and other pieces of traffic smart card data, the information contribution degree of each piece of traffic smart card data is obtained by combining the first contribution degree and the second contribution degree; According to the difference characteristics between any two pieces of traffic smart card data entry and exit information, the distance metric of the information contribution in the clustering process is adjusted; based on the clustering results, the outliers of each piece of traffic smart card data are obtained; According to the abnormal values ​​and information contribution, combined with the threshold segmentation algorithm, the traffic smart card data is cleaned; The process of obtaining the first contribution between any two pieces of traffic smart card data is as follows: According to the route information of each piece of traffic smart card data, a set consisting of the digital numbers of the corresponding entry and exit stations and all the stations passed through in the middle is used as the path set of each piece of traffic smart card data; Get the intersection and union of any two paths of traffic smart card data; According to the number of elements in the intersection and the number of elements in the union of the two pieces of traffic smart card data, a first contribution degree between the two pieces of traffic smart card data is obtained; wherein the first contribution degree is positively correlated with the number of elements in the intersection and negatively correlated with the number of elements in the union; The information contribution of each piece of traffic smart card data is obtained as follows: According to the difference characteristics of the entry time between any two pieces of traffic smart card data, a third contribution degree between the any two pieces of traffic smart card data is obtained; The information contribution of the i-th traffic smart card data is recorded as , its formula form is: ;in, is the total number of traffic smart card data, For the Traffic smart card data and The first contribution between the traffic smart card data, For the Traffic smart card data and The second contribution between the traffic smart card data, For the Traffic smart card data and The third contribution between the traffic smart card data, is the number of all path sets whose intersection with the path set described in the i-th transportation smart card data is not an empty set; The process of adjusting the distance metric of the information contribution in the clustering process is: According to the distance between the entry sites and the distance between the exit sites in any two pieces of traffic smart card data, the first distance and the second distance of the any two pieces of traffic smart card data are obtained; Obtaining the minimum information contribution of any two pieces of traffic smart card data; calculating the proportion of the minimum information contribution in the information contribution of all traffic smart card data; Merge the difference in the travel time of any two pieces of transportation smart card data, the first distance and the second distance; Based on the proportion and the fusion result, the adjusted distance between the arbitrary two pieces of traffic smart card data is obtained; wherein the adjusted distance is negatively correlated with the proportion and positively correlated with the fusion result.

2. A method for cleaning traffic smart card data as claimed in claim 1, characterized in that: The steps of obtaining the second contribution between any two pieces of traffic smart card data are as follows: According to the entry time and exit time of each transportation smart card data, the riding time of each transportation smart card data is obtained; Calculate the subtraction result of the number of stations along the way corresponding to each piece of transportation smart card data and a preset value, and use the ratio of the riding time to the subtraction result as the average riding time of each piece of transportation smart card data; According to the difference between the average riding time of any two pieces of traffic smart card data, a second contribution degree between the any two pieces of traffic smart card data is obtained.

3. A method for cleaning traffic smart card data as claimed in claim 1, characterized in that: The process of obtaining the third contribution degree between any two pieces of traffic smart card data includes: The absolute value of the difference between the entry times of any two pieces of traffic smart card data is calculated; and the third contribution between the any two pieces of traffic smart card data is obtained according to the positive relationship of the absolute value of the difference in the logarithmic function.

4. A method for cleaning traffic smart card data as claimed in claim 1, characterized in that: The steps of obtaining the abnormal value of each piece of traffic smart card data are as follows: All the entry and exit information of each piece of traffic smart card data constitutes a feature vector of each piece of traffic smart card data; The feature vectors of all traffic smart card data and the preset number of clusters are used as the input of the CBLOF anomaly detection algorithm, and the output is the anomaly value of each traffic smart card data.

5. A method for cleaning traffic smart card data as claimed in claim 1, characterized in that: The cleaning of the traffic smart card data is specifically as follows: Perform threshold segmentation on the outliers and information contribution of all traffic smart card data to obtain the outlier segmentation threshold and contribution segmentation threshold; Obtaining abnormal data according to the abnormal segmentation threshold and the contribution segmentation threshold; The abnormal data is deleted to obtain cleaned traffic smart card data.

6. A method for cleaning traffic smart card data as claimed in claim 5, characterized in that: The abnormal data specifically refers to traffic smart card data whose abnormal value is greater than or equal to the abnormal segmentation threshold and whose information contribution is less than the contribution segmentation threshold.

7. A traffic smart card data cleaning system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images