A method for cleaning mobile phone signaling data and related equipment

Abstract

The present invention discloses a method for cleaning mobile phone signaling data and related equipment. The method includes: obtaining a mobile data table, a resident data table, and a path node table; calculating a first speed of each movement in the mobile data table according to a first user requirement, and deleting abnormal movements therein to obtain a first mobile data table; determining the type of interruption in the first mobile data table according to a second user requirement, and selecting whether to modify or retain the interruption to obtain a second mobile data table; retaining abnormal movements according to a third user requirement, calculating a second speed of each movement in the mobile data table according to the mobile data table and the path node table, deleting abnormal movements corresponding to the second speed whose residence time is lower than a preset threshold, and obtaining a processed third mobile data table. The present invention formulates a targeted cleaning solution for mobile phone signaling data according to different user requirements, and has higher flexibility in responding to different data cleaning requirements.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, system, terminal and computer-readable storage medium for cleaning mobile phone signaling data. Background Art

[0002] Mobile phone signaling data has the characteristics of large data volume, high temporal and spatial resolution, and good representativeness. It has become an important data source for conducting business work or scientific research in current fields such as human geography, urban and rural planning, and transportation planning. Mobile phone signaling data also has the common characteristics of big data, such as large data volume and high noise. Due to various technical reasons such as signaling drift, ping-pong effect, and timestamp lag, a large number of errors will be generated during the collection and recording, which greatly affects the promotion and application of mobile phone signaling data.

[0003] Current technologies mostly focus on cleaning the raw data of mobile phone signaling, believing that after the formation of residence and movement records, there is theoretically no noise and error. However, in fact, due to certain technical reasons, the "residence-movement" alternating form of data formed after preliminary cleaning often contains abnormal records. When it is necessary to study the fine characteristics of residents' travel activities, further processing is required to reduce noise. To this end, designing a reasonable and effective data preprocessing and data cleaning method to reduce data noise and improve data quality has become an important part of the workflow based on mobile phone signaling data.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for cleaning mobile phone signaling data, aiming to solve the problem in the prior art that mobile phone signaling data cannot be cleaned according to the different needs of users.

[0006] To achieve the above object, the present invention provides a method for cleaning mobile phone signaling data, the method comprising the following steps:

[0007] Acquire a signaling data table, wherein the signaling data table includes a mobile data table, a resident data table, and a path node table;

[0008] Obtaining a first demand of a user, and calculating a first speed of each movement in the movement data table according to the first demand, and deleting abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table;

[0009] Obtaining a second requirement of the user, and determining a type of discontinuity in the first mobile data table according to the second requirement, and selecting whether to modify or retain the discontinuity according to the type of the discontinuity to obtain a processed second mobile data table;

[0010] The user's third requirement is obtained, and the abnormal movement is retained according to the third requirement. The second speed of each movement in the movement data table is calculated based on the movement data table and the path node table. The abnormal movement corresponding to the residence time below the preset threshold in the second speed is deleted to obtain a processed third movement data table.

[0011] Optionally, the method for cleaning mobile phone signaling data includes obtaining a third requirement of the user, retaining the abnormal movement according to the third requirement, calculating a second speed of each movement in the movement data table based on the movement data table and the path node table, deleting abnormal movements corresponding to the second speeds having a residence time below a preset threshold to obtain a processed third movement data table, and then further comprising:

[0012] When the mobile data cleaning is completed, the corresponding resident points are extracted from the resident data table according to the first mobile data table, the second mobile data table and the third mobile data table to form resident records matching the first mobile data table, the second mobile data table and the third mobile data table.

[0013] Optionally, the method for cleaning mobile phone signaling data, wherein obtaining a user's first demand, calculating a first speed of each movement in the movement data table according to the first demand, and deleting abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table, specifically includes:

[0014] Obtaining a first demand of the user, and obtaining the positions and times of the starting and ending points of each movement in the movement data table according to the first demand, and calculating a first speed of each movement;

[0015] The calculation formula for the first speed of each movement is: Where V is the first velocity of each movement, r is the displacement between the starting and ending points of each movement, O represents the starting point, D represents the end point, λ and φ represent the longitude and latitude respectively, and t is the time corresponding to the starting and ending points of each movement;

[0016] Abnormal movements in the movement data table are deleted to obtain a processed first movement data table, wherein the abnormal movements are overspeed movements with a speed greater than a preset speed and zero-speed movements with a speed of 0.

[0017] Optionally, the method for cleaning mobile phone signaling data, wherein obtaining a second requirement of the user, determining a type of interruption in the first mobile data table according to the second requirement, and selecting to modify or retain the interruption according to the type of the interruption to obtain a processed second mobile data table, specifically includes:

[0018] After obtaining the first mobile data table, pre-processing the first mobile data table is performed, and discontinuities in the first mobile data table are marked. When a discontinuity occurs, it is marked as intrpt=1, otherwise intrpt=0;

[0019] Obtaining a second requirement of the user, and determining a type of discontinuity in the first mobility data table based on the second requirement, where the types of discontinuity in the first mobility data table include: discontinuity caused by abnormal movement deletion, natural discontinuity, discontinuity caused by signaling leaving the study area, and continuous discontinuity;

[0020] When the discontinuity is caused by deletion of the abnormal movement, the abnormal movement is deleted, and the time-space parameters of the endpoint of the normal movement preceding the abnormal movement are modified to the time-space parameters of the endpoint of the normal movement following the abnormal movement, so that the abnormal movement, the previous movement, and the next movement are merged into one movement, and the previous movement and the next movement are deleted, and the speed value of the merged movement is the weighted average of the three movements;

[0021] When the discontinuity is a natural discontinuity or a discontinuity caused by the signal leaving the study area, the discontinuity is retained;

[0022] When the discontinuity is a continuous discontinuity, determine whether the continuous discontinuity is a discontinuity caused by the abnormal movement deletion and a natural discontinuity. If so, mark it as adjust=-1, and modify the adjust=1 of the normal movement before the discontinuity caused by the abnormal movement deletion to adjust=0;

[0023] When a move is not connected to any other moves at the beginning or end, the move will be deleted;

[0024] Adjust = 1 means deleting the new discontinuity after the abnormal movement, indicating that the next normal movement needs to be connected. Adjust = 0 means that in cases other than Adjust = -1 and Adjust = 1, there is no need to connect the movement.

[0025] Repeatedly polling the first movement table for the discontinuities caused by the abnormal movement deletion until the discontinuities caused by the abnormal movement deletion in the first movement table no longer occur, thereby obtaining a processed second movement data table.

[0026] Optionally, in the method for cleaning mobile phone signaling data, the preprocessing of the first mobile data table specifically includes:

[0027] Obtaining the coordinates of all movement start points and end points in the first movement data table and the coordinates of the resident points in the resident data table, and clustering, grouping, de-duplicating, and sorting the coordinates;

[0028] Marking the natural breaks in the first movement data table, and checking whether the starting point of each movement of the next row is consistent with the end point of the current movement through a shift operation; if not, a natural break is generated after the current movement;

[0029] When a natural discontinuity occurs, it is marked as ntrl_intrpt=1, otherwise ntrl_intrpt=0;

[0030] The shift operation refers to group shifting, each group in the first movement data table moves independently, and the next row corresponding to the last movement of each user on each day is a null value.

[0031] Optionally, the method for cleaning mobile phone signaling data, wherein obtaining a third requirement of the user, retaining the abnormal movement according to the third requirement, calculating a second speed of each movement in the movement data table based on the movement data table and the path node table, deleting abnormal movements corresponding to the second speeds having a residence time lower than a preset threshold, and obtaining a processed third movement data table, specifically includes:

[0032] Obtaining a third requirement of the user, retaining abnormal movements in the movement data table according to the third requirement, and performing a comparison and connection between the movement data table and the path node table;

[0033] If the transportation mode is highway, the number of nodes passed and the path distance are calculated by searching in the path node table;

[0034] If the traffic mode is not highway, or if the traffic mode is highway but no corresponding node is found in the path node table, the straight-line distance is calculated directly from the movement data table according to the latitude and longitude coordinates, and the number of nodes passed is recorded as 0;

[0035] Obtaining a second speed of each movement in the movement data table according to the path distance and the movement time, and creating a first pseudo movement data table, wherein the fields of the first pseudo movement data table are the same as those of the movement data table;

[0036] Filling the data in the resident data table into the first pseudo mobile data table, and vertically connecting the first pseudo mobile data table with the newly added data and the mobile data table;

[0037] The residence time in the first pseudo movement data table of the newly added data is obtained, and the movements with overspeed and zero speed and a residence time lower than a preset threshold in the movement data table are deleted to obtain a processed third movement data table.

[0038] Optionally, the method for cleaning mobile phone signaling data, wherein after the mobile data cleaning is completed, extracting corresponding resident points from the resident data table according to the first mobile data table, the second mobile data table, and the third mobile data table to form resident records matching the first mobile data table, the second mobile data table, and the third mobile data table, specifically includes:

[0039] When the mobile data table is cleaned to become a first mobile data table, directly matching the first mobile data table with the resident data table to extract corresponding first resident data;

[0040] When the mobile data table is cleaned to become a second mobile data table or a third mobile data table, the user data and date data in the second mobile data table or the third mobile data table are grouped and sorted, and labels are added through a shift operation to construct a first resident data table;

[0041] If it is the first move in a day or the start and end codes of the previous move are different, then choose not to connect with the previous move, otherwise connect;

[0042] If it is the last move in a day or the end point and the starting point code of the next move are different, then it is not connected with the next move; otherwise, it is connected, and the first resident sub-table is obtained according to the processed result;

[0043] Obtain the residency data table after encoding the residency point, and set a time threshold τ0. When the current residency is at the same position as the next residency, and the difference between the end time of the current residency and the start time of the next residency does not exceed τ0, set the residency mark to cnt=1. Cnt=1 indicates that the current residency is merged with the next residency, otherwise cnt=0 indicates that they are not merged.

[0044] Delete the records in the current row where cnt=1 and the previous row where cnt=1, and replace the end time of the current row with the end time of the next row through a shift operation. Then delete the next row where cnt=0 to construct a second resident data table.

[0045] Converting the second resident data table into a second pseudo mobile data table, and vertically connecting the second pseudo mobile data table with the second mobile data table or the third data table, filtering out data representing resident data, and obtaining a second resident sub-table;

[0046] Summarizing and counting the dates and individual numbers in the second resident data table, and extracting the resident data corresponding to the date from the second resident data table to obtain a third resident sub-table;

[0047] The first resident sub-table, the second resident sub-table and the third resident sub-table are merged and sorted by time to obtain an extracted resident point data set.

[0048] In addition, to achieve the above-mentioned purpose, the present invention further provides a mobile phone signaling data cleaning system, wherein the mobile phone signaling data cleaning system comprises:

[0049] A signaling data acquisition module is used to acquire a signaling data table, wherein the signaling data table includes a mobile data table, a resident data table, and a path node table;

[0050] a movement data cleaning module, configured to obtain a first requirement of a user, calculate a first speed of each movement in the movement data table according to the first requirement, and delete abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table;

[0051] a movement discontinuity modification module, configured to obtain a second requirement of the user, determine a type of discontinuity in the first movement data table according to the second requirement, and select whether to modify or retain the discontinuity according to the type of the discontinuity, thereby obtaining a processed second movement data table;

[0052] The abnormal movement screening module is used to obtain the user's third requirement, retain the abnormal movements according to the third requirement, calculate the second speed of each movement in the movement data table based on the movement data table and the path node table, delete the abnormal movements corresponding to the second speed whose residence time is lower than a preset threshold, and obtain a processed third movement data table.

[0053] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a mobile phone signaling data cleaning program stored in the memory and runnable on the processor, and when the mobile phone signaling data cleaning program is executed by the processor, the steps of the mobile phone signaling data cleaning method described above are implemented.

[0054] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a cleaning program for mobile phone signaling data, and when the cleaning program for mobile phone signaling data is executed by the processor, the steps of the mobile phone signaling data cleaning method as described above are implemented.

[0055] In the present invention, a signaling data table is obtained, and the signaling data table includes a mobile data table, a resident data table, and a path node table; the user's first requirement is obtained, and the first speed of each movement in the mobile data table is calculated according to the first requirement, and the abnormal movement in the mobile data table is deleted according to the first speed to obtain a processed first mobile data table; the user's second requirement is obtained, and the type of interruption in the first mobile data table is determined according to the second requirement, and the interruption is modified or retained according to the type of the interruption to obtain a processed second mobile data table; the user's third requirement is obtained, and the abnormal movement is retained according to the third requirement, and the second speed of each movement in the mobile data table is calculated based on the mobile data table and the path node table, and the abnormal movement corresponding to the residence time below the preset threshold in the second speed is deleted to obtain a processed third mobile data table. The present invention performs targeted cleaning of mobile phone signaling data according to the different needs of users in different scenarios, which can not only reduce data noise and improve data quality, but also improve flexibility when meeting different data cleaning requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of a preferred embodiment of the method for cleaning mobile phone signaling data of the present invention;

[0057] Figure 2 This is a schematic diagram of an overspeed intermittent connection of a preferred embodiment of the method for cleaning mobile phone signaling data of the present invention;

[0058] Figure 3 Schematic diagram of non-study interval interruption and natural interruption of the method for cleaning mobile phone signaling data of the present invention;

[0059] Figure 4 It is a schematic diagram of a new discontinuous generation of a method for cleaning mobile phone signaling data of the present invention;

[0060] Figure 5 It is a schematic diagram of an intermittent type of a method for cleaning mobile phone signaling data of the present invention;

[0061] Figure 6 It is a schematic diagram of the natural interruption and overspeed continuous misconnection of the method for cleaning mobile phone signaling data of the present invention;

[0062] Figure 7 It is a schematic diagram of multiple interrupted connections of the method for cleaning mobile phone signaling data of the present invention;

[0063] Figure 8 It is a schematic diagram of short-stay overspeed movement and patrol and signal interruption of the method for cleaning mobile phone signaling data of the present invention;

[0064] Figure 9This is a schematic diagram of extracting the first resident point of the method for cleaning mobile phone signaling data of the present invention;

[0065] Figure 10 This is a schematic diagram of extracting the second resident point of the method for cleaning mobile phone signaling data of the present invention;

[0066] Figure 11 This is a schematic diagram of extracting the third resident point of the mobile phone signaling data cleaning method of the present invention;

[0067] Figure 12 This is a structural diagram of a preferred embodiment of the mobile phone signaling data cleaning system of the present invention;

[0068] Figure 13 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] Mobile phone signaling data, with its large data volume, high spatiotemporal resolution, and good representativeness, has become an important data source for conducting business work or scientific research in fields such as human geography, urban and rural planning, and transportation planning. Mobile phone signaling data also shares the common characteristics of large data volume and high noise. Due to various technical reasons such as signaling drift, ping-pong effect, and timestamp lag during data collection and recording, a large number of errors can be generated, which greatly affects the promotion and application of mobile phone signaling data. Therefore, designing a set of reasonable and effective data preprocessing and data cleaning methods to reduce data noise and improve data quality has become an important part of the workflow based on mobile phone signaling data. This application designs a set of cleaning methods for mobile phone signaling data in the "resident-mobile" form to complete data preprocessing in a modular manner according to different application requirements. The modularization refers to the three strategies and one subsidiary step proposed in this invention, where the first strategy can be used as a step of the second strategy, and the second strategy as a step of the third strategy. Each strategy can be combined with the subsidiary steps, hence the term "modularization", and the method steps can be reused.

[0071] The method for cleaning mobile phone signaling data according to the preferred embodiment of the present invention is as follows: Figure 1 As shown, the method for cleaning mobile phone signaling data includes the following steps:

[0072] Step S10: Acquire a signaling data table, where the signaling data table includes a mobile data table, a resident data table, and a path node table.

[0073] The data formats applicable to this technology are as follows: The database should contain the following data tables:

[0074] Mobility data table: A table that records the user's movement within the area, including user code, start time, end time, starting location coordinates, ending location coordinates, and transportation method. The data sample is as follows:

[0075]

[0076] All movement trajectories will not cross days, that is, there is no movement with a start and end time that spans midnight. If any of the start and end points is outside the scope of the study area, it will be recorded as a missing value.

[0077] Residency data table: A table that records the user's stay in the area, including user code, start time, end time and location coordinates. The data sample is as follows:

[0078]

[0079] All dwell records do not span multiple days; that is, there are no dwells with a start and end time that straddles midnight. Theoretically, the movement and dwell data tables can complement each other to form daily activity trajectories, creating a "stay-move" alternating trajectory recording method. In practice, due to data recording and other factors, trajectories may occasionally be interrupted, resulting in missing records.

[0080] Path node table (optional): records all the road nodes that the user passes through each time they move, as well as their corresponding location information. Its format is as follows:

[0081]

[0082] All path node codes (rn_id) uniquely correspond to a geographic coordinate.

[0083] The goal of data cleaning is to remove abnormal and unreasonable activity traces from mobile phone signaling data. This paper proposes different cleaning strategies based on different data quality requirements. Before cleaning, users should intercept three data tables, select qualified individuals and study time periods based on application requirements, and then select one of the following three cleaning strategies based on actual application needs.

[0084] Step S20: Obtain the user's first demand, calculate the first speed of each movement in the movement data table according to the first demand, and delete abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table.

[0085] The operation is performed based on the movement data table, and the speed of each movement is calculated through the position and time of the starting and ending points, wherein the movement refers to the displacement with direction and speed from the starting point to the ending point within a certain movement time in the movement data table.

[0086] Specifically, the user's first demand is obtained, and the positions and times of the start and end points of each movement in the movement data table are obtained according to the first demand, and the first speed of each movement is calculated.

[0087] The first requirement of the user is: high requirements for travel speed and distance, high requirements for original data fidelity, and no requirement to form a complete travel chain.

[0088] The calculation formula for the first speed of each movement is: Where V is the first velocity of each movement, r is the displacement between the starting and ending points of each movement, O represents the starting point, D represents the end point, λ and φ represent the longitude and latitude respectively, and t is the time corresponding to the starting and ending points of each movement.

[0089] The numerator is the straight-line distance between the two points calculated using the start and end points (strictly speaking, this is the spherical distance on the Earth's surface, but for urban applications, the effect of Earth's curvature can be ignored and the difference in distances can be disregarded). The denominator is the time difference between the start and end points, or the travel time. If the start and end points are missing, the speed is marked as missing.

[0090] According to the user's first demand, a first strategy is formulated. The specific steps of the first strategy are: deleting abnormal movements in the movement data table to obtain a processed first movement data table, wherein the abnormal movements are overspeeding movements with a speed greater than a preset speed and zero-speed movements with a speed of 0.

[0091] In the present invention, the overspeed movement is preferably set as a movement record with a speed exceeding 120 km / h. The speed is missing and is not deleted. The displacement above 120 km / h is deleted because the displacement with too fast speed may be the drift that occurs when the base station collects the signal; the displacement of zero speed is deleted because such record is not a real displacement in essence, thereby obtaining the processed first movement data table.

[0092] Step S30: Obtain the user's second demand, and determine the type of the discontinuity in the first mobile data table according to the second demand, and select whether to modify or retain the discontinuity according to the type of the discontinuity to obtain a processed second mobile data table.

[0093] The second demand of the user is: an application scenario that requires individuals to form a complete travel chain, such as the need to roughly count the number of trips and total distance at the individual level, and pay attention to scenarios such as multimodal transport and activity patterns.

[0094] A second strategy is established based on the user's second need. After obtaining the first mobile data table according to the first strategy, some movements exceeding 120 km / h are deleted, which will cause the travel chain within a day to be interrupted and disconnected. Based on the first strategy, the second strategy hopes to complete the connectivity and construction of the travel chain within a single day.

[0095] The specific method is: modify the normal movement before the speeding movement so that the time and space parameters of its endpoint are the time and space parameters of the endpoint of the normal movement after the speeding movement, and then delete the original record of the normal movement after the speeding movement. This is equivalent to merging the movement with a speed exceeding 120km / h and the two normal movements before and after into one movement, whose speed value is the weighted average of the three movements.

[0096] Specifically, after obtaining the first mobile data table, preprocessing is performed on the first mobile data table. The specific steps of the preprocessing are:

[0097] Obtain the coordinates of all movement start points and end points in the first movement data table and the coordinates of the resident points in the resident data table, and cluster, group, de-duplicate, and sort the coordinates. The implementation steps of the second strategy are as follows:

[0098] Step 1: Group the coordinates of all mobile starting points, ending points, and resident points involved in the mobile data table and the resident data table according to the mobile phone user number, and then perform de-duplication encoding (the object of de-duplication is the coordinate point, which means that all the starting points, ending points, and resident points of a user are digitally encoded as 1, 2, 3, ..., N. The digital codes of coordinate points with the same position are the same, and the digital codes of coordinate points with different positions are different. However, the coordinate points with the same digital code for user A and user B are not necessarily in the same position. In other words, a user code plus a position code can uniquely lock a position coordinate). Each code corresponds to a coordinate one by one.

[0099] In step one, coordinates can be clustered (all coordinate points can be clustered, using a user-defined clustering algorithm such as hierarchical clustering, K-means, or OPTICS. The only requirement is to ensure that all points are assigned to a specific cluster. The cluster center can be the weighted centroid of all points in the same cluster. Regardless of the clustering method, as long as the same code and coordinate parameters are the same, the number of coordinates must be the same). This reduces the number of coordinates after deduplication. If multiple coordinates are clustered into the same cluster during the clustering phase and coded with the same number, the corresponding coordinate fields must be modified to match. Missing values, representing points outside the study area, are uniformly coded with a special value (such as 999). The coded fields are orig_id (the starting point location number in the moving data table) and dest_id (the ending point location number in the moving data table) in the moving table and poi_id (the location number in the resident data table) in the resident table.

[0100] Among them, the signaling leaving the study area is uniformly coded, which is equivalent to designing a specially designed "out-of-area" activity point. The situation of signaling entering and leaving the study area is thus simplified to a problem within the study area and handled according to general circumstances.

[0101] Step 2: The mobile data table obtained in step 1 is grouped according to user codes and sorted in chronological order within each group.

[0102] The natural breaks in the first movement data table are marked, and a shift operation is performed to find out whether the starting point of each movement of the next row is consistent with the end point of the current movement. If not, a natural break is generated after the current movement.

[0103] When a natural discontinuity occurs, it is marked as ntrl_intrpt=1, otherwise ntrl_intrpt=0.

[0104] The shift operation refers to group shifting, each group in the first movement data table moves independently, and the next row corresponding to the last movement of each user on each day is a null value.

[0105] After pre-processing the first mobile data table, discontinuities in the first mobile data table are marked. When a discontinuity occurs, it is marked as intrpt=1, otherwise intrpt=0.

[0106] Step 3: Mark natural breaks. First, group the data by user code and date (preserving the original chronological order). Each group uses a shift operation to find the starting point of each move in the next row (the purpose of finding the next row is to find the next move in the chronological order, but it must not span days). If the dest_id of the next row is inconsistent with the orig_id of the current row, it means that there is a disconnection in the original data. This means that a natural break has occurred after this move, and it is marked as ntrl_intrpt = 1. Otherwise, ntrl_intrpt = 0.

[0107] Step 4: Implement the steps of the first strategy to obtain the speed of each move and delete the moves with excessive speed and zero speed.

[0108] Step 5: Mark discontinuities. Use the shift operation to find the starting point of the next shift in each row. If the dest_id of the next row is different from the orgi_id of the current row, mark it as a discontinuity: intrpt = 1; otherwise, intrpt = 0.

[0109] Obtain the user's second demand, and determine the type of interruptions existing in the first mobile data table based on the second demand. The types of interruptions in the first mobile data table include: interruptions caused by abnormal movement deletion, natural interruptions, interruptions caused by signaling leaving the study area, and continuous interruptions.

[0110] like Figure 2 As shown, when the interruption is the interruption caused by the deletion of the abnormal movement, the abnormal movement is deleted, and the end-point time-space parameters of the normal movement before the abnormal movement are modified to the end-point time-space parameters of the normal movement after the abnormal movement, so that the abnormal movement and the previous movement and the next movement are merged into one movement, and the previous movement and the next movement are deleted, and the speed value of the merged movement is the weighted average of the three movements.

[0111] Figure 2 The middle movement b is an overspeed movement. After the disconnection, movement a and purple movement c are not connected, and the end position of movement a is changed.

[0112] After deleting all overspeed and zero-speed moves, this method can connect the interruptions of the trip chain. However, there are several situations in which this method should not be used rashly:

[0113] like Figure 3 As shown, when the discontinuity is a natural discontinuity or a discontinuity caused by the signaling leaving the study area, the discontinuity is retained.

[0114] like Figure 3As shown in the lower part, the natural gaps are gaps that originally existed in the mobile data table. Such gaps may be caused by technical reasons, such as signal loss or power off, resulting in the two adjacent mobiles being disconnected. Such gaps should not be connected as described above to restore the original data to the greatest extent possible.

[0115] like Figure 3 As shown in the upper part, the discontinuity caused by the signaling leaving the study area refers to two adjacent normal movements, one leaving the study area and the other entering the study area, resulting in the missing position coordinates of the end point of the previous movement and the starting point of the latter movement. The above processing cannot solve this situation, and it is even impossible to determine whether there is a discontinuity. The above two situations are leaving the study area and natural discontinuity, and they should not be connected.

[0116] Step 6: Compare the results of Step 3 and Step 5 and mark the movement records:

[0117] a. If a movement record ntrl_intrpt = 0 and intrpt = 1, it means that after deleting the overspeed zero speed movement, the "newly generated" interruption (such as Figure 5 (as shown in B in the figure), recorded as adjust=1.

[0118] b. If a movement record has ntrl_intrpt = 1 but no intrpt tag (no intrpt tag means this row is an overspeed or zero speed movement. This is because step 5 has already labeled all non-overspeed or zero speed movements with either 1 or 0), it means that the movement before the natural break was deleted due to overspeed or zero speed, and is marked as adjust = -1.

[0119] c. In other cases, mark adjust = 0; where ntrl_intrpt represents whether it is a natural discontinuity feature. When a natural discontinuity occurs (ntrl_intrpt = 1), the movement should not be connected (adjust = 0), such as Figure 5 As shown in A.

[0120] The other situations include: intrpt = 0 (no disconnection), intrpt = 1 and ntrl_intrpt = 1 (it is disconnected, but due to a natural break), intrpt has no label, but ntrl_intrpt = 1 (there is an overspeed before the natural break, that is, after the overspeed, the signal is lost. The correct handling of this situation is: delete the overspeeding movement, but do not connect the travel chain before and after.).

[0121] Step 7: For the result of step 6, delete the record with ntrl_intrpt=0 but no intrpt tag (indicating that there is no natural break after overspeed or zero speed movement), and then change the adjust tag of the movement record that meets the following conditions to 0: adjust is 1 and the movement corresponding to the next row of records adjust=-1 (indicating that it is known that the end point of the current movement is different from the starting point of the next movement (adjust=1 in this row), but there is at least one natural break between the two movements (adjust=-1 in the corresponding record in the next row). Therefore, the end point of the current movement b does not need to be adjusted to retain the natural break contained in the original data); Among them, steps 6 and 7 mainly solve the problem of break loss, that is, the natural break is lost because the movement to which the tag is attached is overspeed or zero speed and is deleted, such as Figure 6 As shown ( Figure 6 Moving the dotted line in c is an incorrect connection method, ignoring the natural break A which needs to be retained).

[0122] When the interruption is a continuous interruption, determine whether the continuous interruption is the interruption caused by the abnormal movement deletion and the natural interruption. If so, mark it as adjust=-1, and modify the adjust=1 marked by the previous normal movement of the interruption caused by the abnormal movement deletion to adjust=0.

[0123] like Figure 6 As shown, if the previous step of the natural break A is an overspeed move a, then this move a should be deleted. The previous normal movement will lose the natural break mark and be mistaken for a break caused by overspeed and choose to connect (that is, the above Figure 6 The solution is to change Figure 7 The adjust value of the second movement a is changed from 1 to 0 (e.g. Figure 7 As shown). In the figure, you can see three orange moves corresponding to: Natural break A is retained ( Figure 7 Upper left part), followed by a continuous trip with speeding b and speeding c + natural interruption A1 ( Figure 7 Central), speeding leading to discontinuous travel ( Figure 7 right).

[0124] When a move is not connected to any other moves at the beginning or end, the move will be deleted;

[0125] like Figure 4As shown, discontinuity occurs. Due to the averaging connection between the discontinuity and the two normal movements preceding and following it, when discontinuity occurs, the connection may cause a new discontinuity. In this case, the movement records of movement a (solid line pointing to the upper right) and movement b (solid line pointing to the lower right) need to be deleted first. If they are not deleted, according to the interruption connection method mentioned in the present invention, the end point of the previous movement is adjusted to the end point of the next movement, and the movements represented by the solid lines c, a, and b will be changed to the dotted lines d, e, and f. Obviously, after changing to dotted lines, the travel chain still does not meet the requirement of being disconnected from each other.

[0126] The adjust=1 means deleting the new discontinuity after the abnormal movement, indicating that the next normal movement needs to be connected. The adjust=0 means that in cases other than adjust=-1 and adjust=1, the movement does not need to be connected.

[0127] Step 8: 1. Delete the movement records that meet the following conditions: adjust = -1, or;

[0128] 2. Through the shift operation, adjust = 1 in this row and adjust = 1 in the previous row. When interruptions occur successively, if a shift is not connected to the previous and next ones, it will be deleted.

[0129] Step 9: After obtaining the result of step 8, for the movement of adjust=1: use the shift operation to move the end point code (dest_id) and spatiotemporal parameters (t_end residence end time, lon_d residence point longitude, lat_d residence point latitude) of the next row, replace the corresponding fields of the current row, and then delete the next row.

[0130] Step 10: For the result of step 9, repeat steps 4-9 until the number of data rows no longer decreases (all previous steps are to delete overspeed and zero-speed movements, and make connections in the resulting interruptions. The movement after connection is from the starting point of the previous movement to the end point of the next movement. Its speed is always slower than the three-segment movement (this, previous, next), or zero. Even so, it is still possible that the movement speed after connection still exceeds 120Km / h or is zero. Therefore, when the number of data rows no longer decreases, all movement speeds in the data table meet the requirements of neither overspeed nor zero speed. The cleaning step is completed). The iterative repetition is to prevent the movement after connection from still exceeding the speed.

[0131] Repeatedly polling the first movement table for the discontinuities caused by the abnormal movement deletion until the discontinuities caused by the abnormal movement deletion in the first movement table no longer occur, thereby obtaining a processed second movement data table.

[0132] Step S40: Obtain the user's third requirement, retain the abnormal movement according to the third requirement, calculate the second speed of each movement in the movement data table based on the movement data table and the path node table, delete the abnormal movement corresponding to the second speed with a residence time lower than a preset threshold, and obtain a processed third movement data table.

[0133] The third requirement of the user is: it is applicable to application scenarios with path node tables, travel process description, and high distance accuracy requirements.

[0134] Based on the user's third need, a third strategy is formulated. The third strategy improves on the second strategy in the following aspects:

[0135] 1. Allowing some excessive speeding: Excessive speeding is likely due to low signaling time resolution. The base station detects a delay in the time it takes for the mobile phone user to leave the area, shortening the movement time and systematically increasing the speed.

[0136] 2. Allowing some zero-speed moves: If the start and end points of a move are the same, it may be an "interpolation" or "filling" move caused by the interruption of the dwell activity, or it may be a tour with the same start and end points. The two need to be distinguished.

[0137] 3. Precise movement distance: Replace simple surface straight-line distance with road network distance (this table is essentially a resident data table, the purpose of which is to facilitate unified time sorting of movement and residence. Because the general SQL language can only sort a table by attributes, the format of the resident table is different from that of the movement table. Only by rewriting the "resident table" into the "movement table" can the two tables be merged into one and then sorted).

[0138] This strategy implementation requires a path node table. The steps are as follows:

[0139] Step 1: Implement steps 1 to 3 of the second strategy.

[0140] Specifically, obtaining the user's third requirement, retaining the abnormal movement in the movement data table according to the third requirement, and performing a comparison and connection between the movement data table and the path node table;

[0141] Step 2: Connect the movement data table and the path node table.

[0142] If the traffic mode of movement is road, the number of nodes passed and the path distance are calculated by corresponding search in the path node table.

[0143] If the traffic mode is not a highway, or if the corresponding node cannot be found in the path node table, the straight-line distance is calculated directly from the movement data table according to the latitude and longitude coordinates, and the number of nodes passed is recorded as 0; the distance obtained is called the journey, and the speed is then calculated according to the travel time.

[0144] Step 3. Create a new "pseudo-mobility data table" (Python's pandas package corresponds to the pd.concat function (axis = 0), R corresponds to the rbind function, and sql corresponds to the insert command, which is vertical connection. That is, vertically connect the two tables to increase the number of rows. Generally, the column names of the two tables are the same. If the column names are different, the data corresponding to the row of the table without the column is null). All fields are the same as the mobile data table obtained in step 2; fill the resident data table obtained in step 2 into the pseudo-mobility data table, where the starting and ending point coordinates and codes are the resident point coordinates and codes, and the speed, distance, transportation method, and natural break signs are missing.

[0145] The second speed of each movement in the movement data table is obtained according to the path distance and the movement time, and a first pseudo movement data table is newly created. The fields of the first pseudo movement data table are the same as those of the movement data table.

[0146] Step 4: Vertically connect the pseudo mobile data table and the mobile data table obtained in step 2, then group them according to user codes and sort them by time.

[0147] The data in the resident data table is filled into the first pseudo mobile data table, and the first pseudo mobile data table with the newly added data is vertically connected with the mobile data table.

[0148] Step 5. From the results of step 4, delete the records that meet the following conditions.

[0149] It represents movement (from the movement table obtained in step 2) and the speed value exceeds 120 km / h, and one of the following conditions is met:

[0150] a. Both the previous record and the next record represent a stay (from the pseudo-movement table), and neither time exceeds 30 minutes. The starting point of this movement has the same number as the previous stay point, and the ending point is the same as the next stay point.

[0151] b. Both the previous record and the next record represent a stay (from the pseudo-move table), but the starting point of the move is different from the number of the previous stay point or the ending point is different from the next stay point.

[0152] c. At least one of the previous and next records represents movement. The three reasons for deletion are essentially that both ends of the overspeed movement have not formed a valid stop point.

[0153] The formation of a dwell point is a mark formed by the telecommunications company on the short-term static state of mobile phone signaling. The signaling switching between dwells is identified as travel. If the travel speed is too fast and there are very short and unstable signaling traces at both ends of the trip, then this movement is likely to be drift.

[0154] In the first case, it is believed that a stay shorter than 30 minutes is not considered an effective stay. The choice of 30 minutes is a subjective standard, because many activities themselves require more than 30 minutes to carry out. Some mobile phone signaling data uses the standard of stable signaling for 30 minutes to identify stay. Users can select the standard based on research and practical needs combined with the characteristics of mobile phone signaling data.

[0155] Obtain the residence time in the first pseudo movement data table of the newly added data, and remove the movement (such as Figure 8 The movement a, movement b and movement c) on the left are deleted to obtain the processed third movement data table.

[0156] like Figure 8 As shown on the left, the 30-minute criterion ensures that the start and end points of the overspeed movement are both time-dwelling activities, thus distinguishing them from data drifts with very short stays.

[0157] Step 6. From the result of step 5, delete all the records representing residency, that is, from the pseudo-mobile table.

[0158] Step 7: Delete the records with the same starting and ending points and 0 nodes passed. This step distinguishes between tours and signal interruptions for movements with the same starting and ending points, such as Figure 8 Shown on the right.

[0159] Step 8: Based on the result of step 7, implement steps 5 to 8 of the second strategy to obtain the cleaning result of the third strategy.

[0160] Extract dwell points:

[0161] The first, second, and third strategies mentioned above all belong to the cleaning of movement. After selecting any of the above strategies for implementation, a resident record paired with the processed movement table must be generated. If the movement data is processed using the first strategy, since there is no need to ensure the integrity of the individual daily travel chain, in order to maintain the original data form as much as possible, the original resident data table can be directly used as a matching set.

[0162] When the mobile data cleaning is completed, the corresponding resident points are extracted from the resident data table according to the first mobile data table, the second mobile data table and the third mobile data table to form resident records matching the first mobile data table, the second mobile data table and the third mobile data table.

[0163] When the mobile data table is cleaned to become a first mobile data table, the first mobile data table is directly matched with the resident data table to extract corresponding first resident data.

[0164] If the mobile data is processed using the second or third strategy, it is necessary to further extract and organize the paired resident data from the resident data table. The general steps are as follows:

[0165] When the mobile data table is cleaned to become the second mobile data table or the third mobile data table, the user data and date data in the second mobile data table or the third mobile data table are grouped and sorted, and tags are added through a shift operation to construct a first resident data table.

[0166] Step 1: Select the mobile data table processed by the second or third strategy, group by user and date, sort by time, and add labels through shift operations. If it is the first movement in a day, or the start and end codes of the previous movement are different, then bfr = 0, otherwise it is 1. The bfr determines whether it is connected to the previous movement, 1 for yes, 0 for no.

[0167] If it is the last movement in a day, or the end point and the starting point code of the next movement are different, then nxt=0, otherwise it is 1. The nxt determines whether it is connected to the next movement, 1 for yes, 0 for no. The nxt determines whether it is connected to the next movement, 1 for yes, 0 for no.

[0168] If it is the first movement in a day or the start point and the end point code of the previous movement are different, choose not to connect with the previous movement, otherwise connect.

[0169] If it is the last movement in a day or the end point and the starting point code of the next movement are different, it is not connected with the next movement, otherwise it is connected, and the first resident sub-table is obtained according to the processed result.

[0170] Step 2: Select the residency data table after encoding the residency points, sort them by time according to user and date groups, set a time threshold τ0, and mark the residency that meets the following conditions as cnt=1 through shift operation (the cnt determines whether the current residency should be merged with the next residency because there is a disconnection of the residency), the position of this residency and the next residency are the same, and the difference between the end time of this residency and the start time of the next residency does not exceed τ0, and cnt=0 for other residencies.

[0171] Then, delete the records in this row where cnt=1 and the previous row also have cnt=1. At this time, records with cnt=1 will not appear continuously in the table, and the next row must have cnt=0. Replace the end time t_end of this row with t_end of the next row by shifting, and then delete the next row with cnt=0. The purpose of step 2 is to merge the fragmented dwells due to short signal interruptions into a complete dwell. In the application, τ0 can be selected as 5 minutes.

[0172] Step 3. For the result of step 1, construct a resident data table through shift operation, in which the longitude and latitude coordinates and numbers are the coordinates and numbers of the end point of nxt=1, the start time is the time when nxt=1 reaches the end point, and the end time is the time when the next row of nxt=1 starts from the starting point. The result is called the "first resident sub-table".

[0173] Obtain the residency data table after encoding the residency point, and set a time threshold τ0. When the current residency is at the same position as the next residency, and the difference between the end time of the current residency and the start time of the next residency does not exceed τ0, set the residency mark to cnt=1. Cnt=1 indicates that the current residency is merged with the next residency, otherwise cnt=0 indicates no merging.

[0174] Delete the records in the current row where cnt=1 and the previous row where cnt=1, and replace the end time of the next row with the end time of the current row through a shift operation. Then delete the next row where cnt=0 to construct a second resident data table.

[0175] Step 4: Similar to steps 3 and 4 of the third strategy, the residence table obtained in step 2 is converted into a "pseudo-mobile data table" with both the starting and ending points being residence points. Then, it is vertically connected with the mobile table obtained in step 1, grouped by individual number and sorted by time. Afterwards: for records that satisfy bfr=0, the previous row represents a residence point and the position is the same as the starting point of this row, select their respective previous rows. For records that satisfy nxt=0, the next row represents a residence point and the position is the same as the ending point of this row, select their respective next rows and then filter these two groups of data representing residence, and then convert them back to the form of the residence table, named "second residence sub-table".

[0176] The second resident data table is converted into a second pseudo mobile data table, and the second pseudo mobile data table is vertically connected with the second mobile data table or the third data table to filter out data representing residence to obtain a second resident sub-table.

[0177] The dates and individual numbers in the second resident data table are summarized and counted, and the resident data corresponding to the date is retrieved from the second resident data table to obtain a third resident sub-table.

[0178] Step 5: By summarizing statistics by date and individual number, select the dates with residence records but no movement records, and then take out the residence data corresponding to these dates from the residence table obtained in step 2 and name it the "third residence sub-table".

[0179] The first resident sub-table, the second resident sub-table and the third resident sub-table are merged and sorted by time to obtain an extracted resident point data set.

[0180] Step 6: Figure 9 As shown, the resident sub-tables 1, 2, and 3 in the merge step are grouped by individuals (the individual grouping means that the resident data has individual numbers, and can be grouped by individual numbers, corresponding to the pandas terminal pd.GroupBy method and the group_by function in R) and sorted by time to obtain the extracted resident point dataset. Among them, steps 3, 4, and 5 correspond to the following respectively Figure 9 、 Figure 10 、 Figure 11 The extraction process of the three cases of the stationary point, the stationary point is Figure 9 、 Figure 10 、 Figure 11 The part enclosed in square brackets.

[0181] The effects of the present invention are:

[0182] Pioneering: Current technologies often focus on cleaning raw mobile phone signaling data, assuming that after the dwell and movement records are formed, there is theoretically no noise or error. However, due to certain technical reasons (such as delays in dwell signals and signal loss during interruptions), the "dwell-move" alternating data formed after initial cleaning often contains abnormal records (such as some movements being too fast, clearly not part of normal human activity, or a pause of only a few seconds between two dwells, which is mistakenly identified as travel). When it is necessary to study the detailed characteristics of residents' travel activities, further processing is required to reduce noise. This technology provides a method for cleaning "dwell-move" signaling data.

[0183] Adaptability: This technology can provide targeted cleaning solutions based on different user scenarios and the resulting varying data cleaning requirements. This provides greater flexibility in addressing diverse data cleaning requirements (in summary, if high fidelity is required for the original data, use the first strategy; if individual trajectories are required to form a relatively complete travel chain, use the second strategy; if the specific details of the travel process (such as transportation methods and road network distances) must be as realistic as possible, use the third strategy).

[0184] Efficiency: The key cleaning process in this technology does not include a loop process, which means that the time complexity of the algorithm developed based on this data will be very low, making it suitable for application in large-capacity mobile phone signaling data.

[0185] Furthermore, if Figure 12 As shown, based on the above-mentioned method for cleaning mobile phone signaling data, the present invention also provides a mobile phone signaling data cleaning system, wherein the mobile phone signaling data cleaning system includes:

[0186] A signaling data acquisition module 51 is configured to acquire a signaling data table, wherein the signaling data table includes a mobile data table, a resident data table, and a path node table;

[0187] a movement data cleaning module 52 for obtaining a first requirement of a user, calculating a first speed of each movement in the movement data table according to the first requirement, and deleting abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table;

[0188] a movement discontinuity modification module 53 for obtaining a second requirement of the user, determining the type of discontinuity in the first movement data table according to the second requirement, and selecting whether to modify or retain the discontinuity according to the type of the discontinuity to obtain a processed second movement data table;

[0189] The abnormal movement screening module 54 is used to obtain the user's third requirement, retain the abnormal movements according to the third requirement, calculate the second speed of each movement in the movement data table based on the movement data table and the path node table, delete the abnormal movements corresponding to the second speed with a residence time lower than a preset threshold, and obtain a processed third movement data table.

[0190] Furthermore, if Figure 13 As shown, based on the above-mentioned method and system for cleaning mobile phone signaling data, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 13 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0191] In some embodiments, the memory 20 can be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 can also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a cleaning program 40 for mobile phone signaling data is stored on the memory 20, and the cleaning program 40 for mobile phone signaling data can be executed by the processor 10, thereby realizing the cleaning method of mobile phone signaling data in the present application.

[0192] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the method for cleaning mobile phone signaling data.

[0193] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0194] In one embodiment, when the processor 10 executes the mobile phone signaling data cleaning program 40 in the memory 20 , the steps of the mobile phone signaling data cleaning method described above are implemented.

[0195] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a cleaning program for mobile phone signaling data, and when the cleaning program for mobile phone signaling data is executed by a processor, the steps of the mobile phone signaling data cleaning method described above are implemented.

[0196] In summary, the present invention provides a method for cleaning mobile phone signaling data and related equipment, the method comprising: obtaining a signaling data table, the signaling data table comprising a mobile data table, a resident data table, and a path node table; obtaining a user's first requirement, and calculating a first speed of each movement in the mobile data table according to the first requirement, and deleting abnormal movements in the mobile data table according to the first speed, to obtain a processed first mobile data table; obtaining a user's second requirement, and determining the type of interruption in the first mobile data table according to the second requirement, and selecting to modify or retain the interruption according to the type of the interruption, to obtain a processed second mobile data table; obtaining a user's third requirement, and retaining the abnormal movement according to the third requirement, calculating a second speed of each movement in the mobile data table according to the mobile data table and the path node table, deleting abnormal movements corresponding to the second speed with a residence time lower than a preset threshold, to obtain a processed third mobile data table. The present invention performs targeted cleaning of mobile phone signaling data according to the different needs of users in different scenarios, which not only reduces data noise and improves data quality, but also improves flexibility in meeting different data cleaning requirements.

[0197] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes the element.

[0198] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0199] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for cleaning mobile phone signaling data, characterized in that: The method for cleaning mobile phone signaling data includes: Acquire a signaling data table, wherein the signaling data table includes a mobile data table, a resident data table, and a path node table; Obtaining a first demand of a user, and calculating a first speed of each movement in the movement data table according to the first demand, and deleting abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table; Obtaining a second requirement of the user, and determining a type of discontinuity in the first mobile data table according to the second requirement, and selecting whether to modify or retain the discontinuity according to the type of the discontinuity to obtain a processed second mobile data table; The acquiring of the second requirement of the user, determining the type of the discontinuity in the first mobile data table according to the second requirement, and selecting to modify or retain the discontinuity according to the type of the discontinuity to obtain the processed second mobile data table specifically includes: After obtaining the first mobile data table, pre-processing the first mobile data table is performed, and discontinuities in the first mobile data table are marked. When a discontinuity occurs, it is marked as intrpt=1, otherwise intrpt=0; Obtaining a second requirement of the user, and determining a type of discontinuity in the first mobility data table based on the second requirement, where the types of discontinuity in the first mobility data table include: discontinuity caused by abnormal movement deletion, natural discontinuity, discontinuity caused by signaling leaving the study area, and continuous discontinuity; When the discontinuity is caused by deletion of the abnormal movement, the abnormal movement is deleted, and the time-space parameters of the endpoint of the normal movement preceding the abnormal movement are modified to the time-space parameters of the endpoint of the normal movement following the abnormal movement, so that the abnormal movement, the previous movement, and the next movement are merged into one movement, and the previous movement and the next movement are deleted, and the speed value of the merged movement is the weighted average of the three movements; When the discontinuity is a natural discontinuity or a discontinuity caused by the signal leaving the study area, the discontinuity is retained; When the discontinuity is a continuous discontinuity, determine whether the continuous discontinuity is a discontinuity caused by the abnormal movement deletion and a natural discontinuity. If so, mark it as adjust=-1, and modify the adjust=1 of the normal movement before the discontinuity caused by the abnormal movement deletion to adjust=0; When a move is not connected to any other moves at the beginning or end, the move will be deleted; Adjust = 1 means deleting the new discontinuity after the abnormal movement, indicating that the next normal movement needs to be connected. Adjust = 0 means that in cases other than Adjust = -1 and Adjust = 1, there is no need to connect the movement. Repeatedly polling the discontinuities caused by the abnormal movement deletion in the first movement table until the discontinuities caused by the abnormal movement deletion in the first movement table no longer occur, thereby obtaining a processed second movement data table; The user's third requirement is obtained, and the abnormal movement is retained according to the third requirement. The second speed of each movement in the movement data table is calculated based on the movement data table and the path node table. The abnormal movement corresponding to the residence time below the preset threshold in the second speed is deleted to obtain a processed third movement data table.

2. The method for cleaning mobile phone signaling data according to claim 1, characterized in that: The method further includes obtaining a third requirement of the user, retaining the abnormal movement according to the third requirement, calculating a second speed of each movement in the movement data table based on the movement data table and the path node table, deleting abnormal movements corresponding to the second speed with a dwell time lower than a preset threshold, and obtaining a processed third movement data table. When the mobile data cleaning is completed, the corresponding resident points are extracted from the resident data table according to the first mobile data table, the second mobile data table and the third mobile data table to form resident records matching the first mobile data table, the second mobile data table and the third mobile data table.

3. The method for cleaning mobile phone signaling data according to claim 1, characterized in that: The obtaining of the user's first requirement, calculating a first speed of each movement in the movement data table according to the first requirement, and deleting abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table specifically includes: Obtaining a first demand of the user, and obtaining the positions and times of the starting and ending points of each movement in the movement data table according to the first demand, and calculating a first speed of each movement; The calculation formula for the first speed of each movement is: Where V is the first velocity of each movement, r is the displacement between the starting and ending points of each movement, O represents the starting point, D represents the end point, λ and φ represent the longitude and latitude respectively, and t is the time corresponding to the starting and ending points of each movement; Abnormal movements in the movement data table are deleted to obtain a processed first movement data table, wherein the abnormal movements are overspeed movements with a speed greater than a preset speed and zero-speed movements with a speed of 0.

4. The method for cleaning mobile phone signaling data according to claim 1, characterized in that: The preprocessing of the first mobile data table specifically includes: Obtaining the coordinates of all movement start points and end points in the first movement data table and the coordinates of the resident points in the resident data table, and clustering, grouping, de-duplicating, and sorting the coordinates; Marking the natural breaks in the first movement data table, and checking whether the starting point of each movement of the next row is consistent with the end point of the current movement through a shift operation; if not, a natural break is generated after the current movement; When a natural discontinuity occurs, it is marked as ntrl_intrpt=1, otherwise ntrl_intrpt=0; The shift operation refers to group shifting, each group in the first movement data table moves independently, and the next row corresponding to the last movement of each user on each day is a null value.

5. The method for cleaning mobile phone signaling data according to claim 1, characterized in that: The method of obtaining the user's third requirement, retaining the abnormal movement according to the third requirement, calculating a second speed for each movement in the movement data table based on the movement data table and the path node table, deleting abnormal movements corresponding to the second speeds with a dwell time lower than a preset threshold, and obtaining a processed third movement data table specifically includes: Obtaining a third requirement of the user, retaining abnormal movements in the movement data table according to the third requirement, and performing a comparison and connection between the movement data table and the path node table; If the transportation mode is highway, the number of nodes passed and the path distance are calculated by searching in the path node table; If the traffic mode is not highway, or if the traffic mode is highway but no corresponding node is found in the path node table, the straight-line distance is calculated directly from the movement data table according to the latitude and longitude coordinates, and the number of nodes passed is recorded as 0; Obtaining a second speed of each movement in the movement data table according to the path distance and the movement time, and creating a first pseudo movement data table, wherein the fields of the first pseudo movement data table are the same as those of the movement data table; Filling the data in the resident data table into the first pseudo mobile data table, and vertically connecting the first pseudo mobile data table with the newly added data and the mobile data table; The residence time in the first pseudo movement data table of the newly added data is obtained, and the movements with overspeed and zero speed and a residence time lower than a preset threshold in the movement data table are deleted to obtain a processed third movement data table.

6. The method for cleaning mobile phone signaling data according to claim 2, characterized in that: After the mobile data cleaning is completed, the corresponding resident points are extracted from the resident data table according to the first mobile data table, the second mobile data table, and the third mobile data table to form resident records that match the first mobile data table, the second mobile data table, and the third mobile data table, specifically including: When the mobile data table is cleaned to become a first mobile data table, directly matching the first mobile data table with the resident data table to extract corresponding first resident data; When the mobile data table is cleaned to become a second mobile data table or a third mobile data table, the user data and date data in the second mobile data table or the third mobile data table are grouped and sorted, and labels are added through a shift operation to construct a first resident data table; If it is the first move in a day or the start and end codes of the previous move are different, then choose not to connect with the previous move, otherwise connect; If it is the last move in a day or the end point and the starting point code of the next move are different, then it is not connected with the next move; otherwise, it is connected, and the first resident sub-table is obtained according to the processed result; Obtain the residency data table after encoding the residency point, and set a time threshold τ0. When the current residency is at the same position as the next residency, and the difference between the end time of the current residency and the start time of the next residency does not exceed τ0, set the residency mark to cnt=1. Cnt=1 indicates that the current residency is merged with the next residency, otherwise cnt=0 indicates that they are not merged. Delete the records in the current row where cnt=1 and the previous row where cnt=1, and replace the end time of the current row with the end time of the next row through a shift operation. Then delete the next row where cnt=0 to construct a second resident data table. Converting the second resident data table into a second pseudo mobile data table, and vertically connecting the second pseudo mobile data table with the second mobile data table or the third data table, filtering out data representing resident data, and obtaining a second resident sub-table; Summarizing and counting the dates and individual numbers in the second resident data table, and extracting the resident data corresponding to the date from the second resident data table to obtain a third resident sub-table; The first resident sub-table, the second resident sub-table and the third resident sub-table are merged and sorted by time to obtain an extracted resident point data set.

7. A mobile phone signaling data cleaning system, characterized in that: The mobile phone signaling data cleaning system is applied to the mobile phone signaling data cleaning method according to any one of claims 1 to 6, and the mobile phone signaling data cleaning system includes: A signaling data acquisition module is used to acquire a signaling data table, wherein the signaling data table includes a mobile data table, a resident data table, and a path node table; a movement data cleaning module, configured to obtain a first requirement of a user, calculate a first speed of each movement in the movement data table according to the first requirement, and delete abnormal movements in the movement data table according to the first speed to obtain a processed first movement data table; a movement discontinuity modification module, configured to obtain a second requirement of the user, determine a type of discontinuity in the first movement data table according to the second requirement, and select whether to modify or retain the discontinuity according to the type of the discontinuity, thereby obtaining a processed second movement data table; The abnormal movement screening module is used to obtain the user's third requirement, retain the abnormal movements according to the third requirement, calculate the second speed of each movement in the movement data table based on the movement data table and the path node table, delete the abnormal movements corresponding to the second speed whose residence time is lower than a preset threshold, and obtain a processed third movement data table.

8. A terminal, characterized in that: The terminal includes: a memory, a processor, and a mobile phone signaling data cleaning program stored in the memory and runnable on the processor. When the mobile phone signaling data cleaning program is executed by the processor, the steps of the mobile phone signaling data cleaning method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a mobile phone signaling data cleaning program, and when the mobile phone signaling data cleaning program is executed by the processor, the steps of the mobile phone signaling data cleaning method according to any one of claims 1 to 6 are implemented.

Images