Efficient stay point identification method based on NSD data

Abstract

The embodiment of the invention relates to the technical field of stay point identification, in particular to an efficient stay point identification method based on NSD (Non-Secure Digital) data, which comprises the following steps: partitioning the NSD data according to a hash value of a device identifier; sorting according to the timestamps to form an original base station sequence, and de-duplicating in combination with the timestamps and the geographic coordinates to obtain a de-duplicated base station sequence; calculating the short duration of the equipment in each attached base station, and determining the accumulated duration of the equipment in each attached base station; dividing a static base station and a dynamic base station based on a first time threshold, and grouping and numbering the de-duplicated base station sequence according to the state of the base station to obtain a static sequence and dynamic sequence staggered state sequence; processing the abnormal dynamic sequence, and converting the static sequence which stays for a short time into a dynamic sequence; and the weighted geometric center point of each static sequence is respectively calculated as the position of each stay point, so that the stay points are efficiently identified in a distributed manner, and the accuracy and robustness of stay point identification are improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of stay point identification, and in particular, to an efficient stay point identification method based on NSD data. Background Art

[0002] With the development of mobile communication technology and location services, a large amount of NSD data (Network Signaling Data) is continuously generated and collected. When faced with a huge amount of data, the traditional method of calculating the stay point often seems to be inadequate, especially in terms of data processing efficiency, algorithm calculation accuracy, and abnormal data processing. There are many challenges. Therefore, how to efficiently extract useful information from this data, especially to identify the stay points of the device, has become an urgent problem to be solved. There are currently two commonly used stay point recognition methods proposed, namely, the recognition method based on the cumulative stay time and the recognition method based on the time window.

[0003] The identification method based on the cumulative dwell time is a direct way to identify the dwelling point. This method requires setting a time threshold T (for example, set to 30 minutes). If the cumulative duration of the mobile device staying at a base station location exceeds this threshold, the user is considered to be staying at the base station location. This method first sorts the NSD data in chronological order, calculates the short duration of each attached base station, and then performs aggregation and summation operations according to the attached base stations to obtain the cumulative duration of each attached base station, and regards the attached base stations with a cumulative duration exceeding T as candidate dwelling points. This method also has an improved algorithm, which is to sort the attached base stations according to the cumulative duration, and regard the top N base stations as dwelling points.

[0004] The recognition method based on the time window also needs to sort the NSD data in chronological order, calculate the short duration of each attached base station, and then define a time window W (for example, set to 20 minutes) and a spatial radius R (for example, set to 800 meters). During the calculation process, each time a piece of data is put in, the distance between the data and the base station location of each existing data in the window is calculated to determine whether it exceeds R. If it exceeds R, the data is considered not a stop point and the window is reset to empty. If it does not exceed R, continue to put in the next data until the time window W is reached. At this time, the time difference between the first data and the last data in the window is 20 minutes, and the distance between each data in the window is less than 800 meters, then all the data in this window are stop points. According to this logic, the stop points are saved, the window is reset to empty, and the next batch of stop points are iteratively calculated.

[0005] However, the inventors of the present application have found that both methods have certain limitations.

[0006] The limitation of the recognition method based on time threshold is that it assumes that the user's stay behavior is static, that is, the user stays at a certain location for more than a certain time. However, the actual device behavior may be more complicated, such as a short stay, a short leave and then return, etc. If the time threshold T is set inappropriately, it may lead to misjudgment. For example, if the time threshold T is set too long, a short stay behavior may be missed. If the time threshold T is set too short, a short stay may be misjudged as a stay. Another limitation of this method is that the device stay point obtained is the location of the base station where the device is attached, which cannot reflect the actual location of the device. The algorithm has poor accuracy and a large overall error.

[0007] Although the time window-based recognition method can identify more complex dwelling behaviors, it relies on the selection of multiple parameters, including the radius threshold R, the time window W, etc. This method still cannot eliminate the problem of dwelling point calculation errors caused by the common behavior pattern of "short stay, short leave and return". In addition, this method requires sorting and sliding window processing of NSD data, which has very high computational complexity and is extremely inefficient when processing large-scale data. This method also does not process abnormal data and will mistakenly identify abnormal data as dwelling points. In addition, this method also does not further calculate the true location of the device. The dwelling point location used is still the location of the attached base station, which has poor accuracy. Summary of the invention

[0008] In view of this, an embodiment of the present application proposes an efficient stop point identification method based on NSD data, which can identify stop points in an efficient and distributed manner, effectively improving the accuracy and robustness of stop point identification.

[0009] In the first aspect, an embodiment of the present application proposes an efficient stop point identification method based on NSD data, comprising the following steps: partitioning the NSD data according to the hash value of the device identifier through the Spark computing engine to ensure that the NSD data with the same device identifier is assigned to the same partition; sorting the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and deduplicating the original base station sequence based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence; based on the deduplicated base station sequence, calculating the time difference between the device at the current attached base station and the next attached base station as the short duration of the device at the current attached base station, and respectively performing a deduplication on the base station sequence for each attached base station. The base station performs aggregation operations to determine the cumulative duration of the device at each attached base station; the attached base station whose cumulative duration exceeds the first time threshold is determined as a static base station, and the attached base station whose cumulative duration does not exceed the first time threshold is determined as a dynamic base station, and the deduplicated base station sequence is grouped and numbered according to the different base station states to obtain a state sequence in which static sequences and dynamic sequences are interlaced; the abnormal dynamic sequences in the state sequence are removed according to the abnormal discrimination conditions, and the static sequences whose cumulative duration does not exceed the second time threshold are converted into dynamic sequences; all static sequences in the state sequence are filtered out, and the positions of the weighted geometric center points of each static sequence are calculated respectively as the positions of each stay point.

[0010] Optionally, the NSD data is partitioned according to the hash value of the device identifier through the Spark computing engine, and the partition function is implemented. The partition function is f(id)=hash(id) / k, id is the device identifier, k is the number of partitions, and the partitioning process specifically includes: for each item of NSD data, calculate the hash value hash(id) of its device identifier; take hash(id) modulo k to obtain the partition number p, and allocate the NSD data to partition p.

[0011] Optionally, the original base station sequence is deduplicated based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence, including: traversing each NSD data in the original base station sequence, and checking whether the timestamp and geographic coordinates of the current NSD data item are consistent with those of the previous NSD data item; if the timestamp and geographic coordinates of the current NSD data item are consistent with those of the previous NSD data item, the two are regarded as duplicate NSD data, and only the current NSD data item is retained; if the timestamp of the current NSD data item is consistent with that of the previous NSD data item, but the geographic coordinates are inconsistent, the previous NSD data item is regarded as erroneous data, and only the current NSD data item is retained; if the geographic coordinates of the current NSD data item are consistent with those of the previous NSD data item, but the timestamps are inconsistent, the current NSD data item is regarded as erroneous data, and only the previous NSD data item is retained; if the timestamp and geographic coordinates of the current NSD data item are inconsistent with those of the previous NSD data item, both the current NSD data item and the previous NSD data item are retained.

[0012] Optionally, the time difference between the device being at the current attached base station and the next attached base station is calculated as the short duration of the device being at the current attached base station, which is implemented by the following formula:

[0013] L i =t i+1 -t i ;

[0014] Among them, t i and t i+1 are two consecutive timestamps in the base station sequence after deduplication, L i Indicates the short duration of the device at the current attached base station; perform aggregation calculations on each attached base station to determine the cumulative duration of the device at each attached base station, which is achieved through the following formula:

[0015]

[0016] in, It indicates the cumulative duration of the device attaching to the bth base station, and n indicates the number of short durations of the device attaching to the bth base station.

[0017] Optionally, abnormal dynamic sequences in the state sequence are removed according to abnormal judgment conditions, including: traversing each dynamic sequence in the state sequence, determining whether the number of position points in the current dynamic sequence is less than a preset position number threshold, determining whether the duration of the current dynamic sequence is less than a preset duration threshold, and determining whether the current dynamic sequence overlaps with the previous and next static sequences on the base station; if the number of position points in the current dynamic sequence is less than the position number threshold, or the duration of the current dynamic sequence is less than the duration threshold, or the current dynamic sequence overlaps with the previous and next static sequences on the base station, then the current dynamic sequence is marked as an abnormal dynamic sequence and removed.

[0018] Optionally, the position of the weighted geometric center point of each static sequence is calculated respectively, including: traversing each static sequence to determine the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence; performing weighted summation based on the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence to calculate the position of the weighted geometric center point of the current static sequence; after completing the traversal, the position of the weighted geometric center point of each static sequence is obtained.

[0019] Optionally, a weighted sum is performed based on the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence to calculate the position of the weighted geometric center point of the current static sequence, which is implemented by the following formula:

[0020]

[0021] Among them, assuming that the current static sequence is the i-th static sequence, G(S i ) represents the position of the weighted geometric center point of the current static sequence, (lat ij ,lon ij ) represents the longitude and latitude coordinates corresponding to the jth time point in the current static sequence, n i Indicates the number of time points in the current static sequence, w ij Represents the weight corresponding to the j-th time point in the current static sequence.

[0022] The efficient stop point identification method based on NSD data proposed in this application has the following beneficial effects:

[0023] First, data distribution and load balancing are optimized, improving processing efficiency.

[0024] This application uses the Spark computing engine and hash value partitioning technology to effectively improve the data processing speed and realize the rapid analysis of large-scale data sets, so that data with the same identifier is assigned to the same partition, avoiding data skew, reducing task execution time and reducing task failure rate.

[0025] Second, the ability to handle abnormal data has been enhanced.

[0026] This application can effectively identify and process abnormal data by introducing data deduplication, outlier removal, and abnormal state sequence detection and removal mechanisms, thereby improving the accuracy and reliability of stay point identification. In particular, in the abnormal sequence detection and removal steps, the process is repeated and iterated until no abnormal sequence appears, ensuring that the data has high quality.

[0027] Third, the stay point identification criteria are flexible.

[0028] This application calculates the duration of attachment to the base station, analyzes the characteristics of the business scenario, and distinguishes between static base stations and dynamic base stations by setting a reasonable time threshold. The time threshold can be adjusted according to specific business needs, thereby improving the flexibility of stay point identification and making stay point identification more accurate.

[0029] Fourth, the state sequence processing is highly refined.

[0030] This application processes the state sequence in detail, and accurately divides the complete NSD data of the day into dynamic sequence and static sequence through the dynamic and static recognition algorithm, converts the short static sequence into a dynamic sequence and merges it with the adjacent dynamic sequence, which can more accurately distinguish between repeated waypoints and real stop points, and improves the accuracy of stop point recognition.

[0031] In summary, the efficient stop point identification method based on NSD data proposed in this application not only improves the computational efficiency when processing large-scale NSD data, but also enhances the accuracy and reliability of data processing, thereby providing a more efficient, accurate and reliable stop point identification solution.

[0032] In the second aspect, an embodiment of the present application proposes an efficient stop point identification system based on NSD data, the system comprising: an acquisition module, a partitioning module, a sorting and deduplication module, a duration calculation module, a status marking module, an exception handling module and a location calculation module; the acquisition module is used to acquire NSD data; the partitioning module is used to partition the NSD data according to the hash value of the device identifier through the Spark computing engine to ensure that the NSD data with the same device identifier is assigned to the same partition; the sorting and deduplication module is used to sort the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and deduplicate the original base station sequence based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence; the duration calculation module is used to calculate the distance between the current attached base station and the next attached base station based on the deduplicated base station sequence. The time difference is used as the short duration of the device at the current attached base station, and aggregation operations are performed on each attached base station to determine the cumulative duration of the device at each attached base station; a state marking module is used to determine the attached base station whose cumulative duration exceeds the first time threshold as a static base station, and the attached base station whose cumulative duration does not exceed the first time threshold as a dynamic base station, and group and number the deduplicated base station sequence according to the different base station states to obtain a state sequence in which static sequences and dynamic sequences are interlaced; an exception handling module is used to remove abnormal dynamic sequences in the state sequence according to the exception judgment conditions, and convert static sequences whose cumulative duration does not exceed the second time threshold into dynamic sequences; a position calculation module is used to filter out all static sequences in the state sequence, and calculate the position of the weighted geometric center point of each static sequence as the position of each stop point.

[0033] In a third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an efficient stop point identification method based on NSD data as described in the first aspect above.

[0034] In a fourth aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an efficient method for identifying stop points based on NSD data as described in the first aspect above.

[0035] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of an efficient method for identifying stay points based on NSD data provided in one embodiment of the present application;

[0038] Figure 2 is a schematic diagram of hash partitioning provided in one embodiment of the present application;

[0039] Figure 3 is a schematic diagram of data deduplication provided in an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of a state sequence provided in one embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of the principle of removing abnormal dynamic sequences in state sequences according to abnormal discrimination conditions provided in one embodiment of the present application;

[0042] Figure 6 is a schematic diagram of state sequence merging provided in one embodiment of the present application;

[0043] Figure 7 is a schematic diagram of all static sequences in the filtered state sequence provided in one embodiment of the present application;

[0044] Figure 8 is a schematic diagram of the position of a weighted geometric center point of a static sequence provided in an embodiment of the present application;

[0045] Fig. 9 is a structural schematic diagram of an efficient stop point identification system based on NSD data provided in another embodiment of the present application;

[0046] Fig.10 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0048] An embodiment of the present application proposes an efficient identification method for stay points based on NSD data, which is applied to electronic devices, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described by taking the server as an example. The implementation details of the efficient identification method for stay points based on NSD data proposed in this embodiment are described in detail below. The following content is only provided for the convenience of understanding and is not necessary for the implementation of this solution. The specific process of the efficient identification method for stay points based on NSD data proposed in this embodiment is as follows: Figure 1 As shown, including:

[0049] Step 101 , partitioning the NSD data according to the hash value of the device identifier by using the Spark computing engine, ensuring that the NSD data with the same device identifier is allocated to the same partition.

[0050] In the specific implementation, the server obtains NSD data collected on a daily basis. The collected NSD data not only contains the location information of the device's attached base station, but also contains the timestamp and other metadata of the attached base station. The timestamp of the attached base station refers to the moment when the device is attached to the nearby base station. The server determines the device's stay point based on the NSD data, that is, determines the location where the device stays for a long time in a day. The server must first partition the NSD data based on the hash value of the device identifier through the Spark computing engine to ensure that NSD data with the same device identifier is assigned to the same partition, thereby effectively avoiding the common data skew problem in distributed computing, and then preventing problems caused by long task execution time or high failure rate.

[0051] It is understandable that data partitioning can split large-scale data sets according to certain dimensions (such as geographic location, timestamp or other identifiers). This split helps to evenly distribute them across multiple computing nodes, thereby accelerating distributed computing and improving data retrieval efficiency.

[0052] In one example, the NSD data collected by the server for one day constitutes a data set D. Each item of NSD data contains a device identifier, that is, id. The server uses the partition function f(id) = hash(id) / k to partition all NSD data in D, where k is the number of partitions. For each item of NSD data, the hash value hash(id) of its device identifier is calculated, and then hash(id) modulo k is used to obtain the partition number p, and the NSD data is assigned to partition p.

[0053] After partitioning, NSD data with the same identifier is assigned to the same partition, so that subsequent algorithms are executed in a single partition without data omission. At the same time, the partitioning operation also avoids data skew and can effectively utilize the advantages of distributed computing, improve task execution efficiency, and reduce task failure rate.

[0054] In an example, Figure 2 As shown in the figure, hash partitioning is more scientific and reasonable than original data partitioning.

[0055] Step 102, sorting the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and deduplicating the data of the original base station sequence based on the timestamp and the geographic coordinates to obtain a deduplicated base station sequence.

[0056] In the specific implementation, after completing the data partition, the server can sort the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and then deduplicate the original base station sequence based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence. The sorting operation helps to calculate the time difference between adjacent attached base stations, and also helps to improve the data storage efficiency and the speed of subsequent processing. The deduplication operation saves storage space and speeds up the processing of subsequent tasks. At the same time, it can also remove duplicate or dirty data to ensure the integrity and accuracy of the sequence.

[0057] In one example, when the server deduplicates the data of the original base station sequence, it needs to traverse each NSD data in the original base station sequence to check whether the timestamp and geographic coordinates of the current NSD data are consistent with those of the previous NSD data. The following four situations may occur in this check. If the timestamp and geographic coordinates of the current NSD data are consistent with those of the previous NSD data, the server regards the two as duplicate NSD data and only retains the current NSD data. If the timestamp of the current NSD data is consistent with that of the previous NSD data, but the geographic coordinates are inconsistent, the server regards the previous NSD data as erroneous data and only retains the current NSD data. If the geographic coordinates of the current NSD data are consistent with those of the previous NSD data, but the timestamps are inconsistent, the server regards the current NSD data as erroneous data and only retains the previous NSD data. If the timestamp and geographic coordinates of the current NSD data are inconsistent with those of the previous NSD data, it means that the two are different NSD data, and the server needs to retain both the current NSD data and the previous NSD data.

[0058] In an example, Figure 3 As shown, the NSD data of items 2, 3, and 4 were removed due to duplication.

[0059] Step 103, based on the deduplicated base station sequence, calculate the time difference between the device at the current attached base station and the next attached base station as the short duration of the device at the current attached base station, and perform aggregation operations on each attached base station to determine the cumulative duration of the device at each attached base station.

[0060] In a specific implementation, after the server obtains the deduplicated base station sequence, it can calculate the time difference between the device's current attachment base station and the next attachment base station as the short duration of the device at the current attachment base station, and perform aggregation operations on each attachment base station to determine the cumulative duration of the device at each attachment base station.

[0061] In one example, the time difference between the current attached base station and the next attached base station of the device is calculated as the short duration of the device at the current attached base station, which is implemented by the following formula:

[0062] L i =t i+1 -t i ;

[0063] Among them, t i and t i+1 are two consecutive timestamps in the base station sequence after deduplication, L i Indicates the brief duration that the device is currently attached to the base station.

[0064] In one example, an aggregation operation is performed for each attached base station to determine the cumulative duration of the device at each attached base station, which is achieved by the following formula:

[0065]

[0066] in, It indicates the cumulative duration of the device attaching to the bth base station, and n indicates the number of short durations of the device attaching to the bth base station.

[0067] Step 104, determine the attached base stations whose cumulative duration exceeds the first time threshold as static base stations, and determine the attached base stations whose cumulative duration does not exceed the first time threshold as dynamic base stations, and group and number the deduplicated base station sequences according to the different base station states to obtain a state sequence in which static sequences and dynamic sequences are interlaced.

[0068] In a specific implementation, after calculating the cumulative duration of the device at each attached base station, the server can mark the state of the attached base station. The server needs to mark the cumulative duration exceeding the first time threshold T th1 The attached base station is determined as a static base station, and the attached base station whose cumulative duration does not exceed the first time threshold is determined as a dynamic base station. The server then needs to group and number the deduplicated base station sequence according to the different base station states to obtain a state sequence that interleaves the static sequence and the dynamic sequence. The static sequence is also called the S sequence, and the dynamic sequence is also called the M sequence.

[0069] In one example, T th1 Set to 1200 seconds.

[0070] In an example, Figure 4 As shown, the states of the attached base stations corresponding to the timestamps t3, t4 and t5 in the deduplicated base station sequence are all static base stations, so they can be merged into a static sequence, and the states of the attached base stations corresponding to the timestamps t6, t7 and t8 are all static and dynamic base stations, so they can be merged into a dynamic sequence. After completing the grouping and numbering, Figure 4 The state sequence in can be expressed as: {S1, M1, S2, M2, S3}.

[0071] It is worth noting that traditional stop point identification methods often only focus on one dimension in time or space, or use a time window method to identify stop points, such as simply relying on the length of time to stay or based on the change of geographical location to determine whether to stay, ignoring the analysis of user behavior characteristics, resulting in large errors in stop point identification. This embodiment innovatively proposes a dynamic and static sequence identification method, comprehensively considers time and space factors, and distinguishes between dynamic and static to perform more accurate stop point identification. This method can better reflect the behavior patterns of mobile devices in the real world, more accurately capture the activity patterns of mobile devices, and significantly increase the accuracy of stop point calculation.

[0072] Step 105 , removing abnormal dynamic sequences from the state sequence according to the abnormality discrimination condition, and converting static sequences whose cumulative duration does not exceed the second time threshold into dynamic sequences.

[0073] In one example, after obtaining the state sequence, the server can remove the abnormal dynamic sequence in the state sequence according to the abnormal discrimination condition, and convert the static sequence whose cumulative duration does not exceed the second time threshold into a dynamic sequence. The traditional stay point identification method does not have an abnormal detection and removal mechanism, which may lead to low accuracy and robustness of stay point identification. However, this embodiment sets detailed rules to detect, remove, convert and merge abnormal sequences, and ensures that all abnormal sequences are correctly processed through iterative execution, which greatly improves the accuracy and reliability of stay point identification.

[0074] It is worth noting that the second time threshold T th2 Greater than the first time threshold T th1 .

[0075] In one example, when the server removes the abnormal dynamic sequence in the state sequence according to the abnormal judgment condition, it traverses each dynamic sequence in the state sequence and determines in turn whether the number of position points in the current dynamic sequence is less than the preset position number threshold N. err , determine whether the duration of the current dynamic sequence is less than the preset duration threshold T err , and determine whether the current dynamic sequence overlaps with the previous and next static sequences on the base station. If the number of position points in the current dynamic sequence is less than the position number threshold N err , or the duration of the current dynamic sequence is less than the duration threshold T err , or if the current dynamic sequence overlaps with the previous and next static sequences on the base station, the server needs to mark the current dynamic sequence as an abnormal dynamic sequence and remove it.

[0076] In one example, N err Set to 3, T err Set to 10 seconds.

[0077] In one example, the principle of abnormal dynamic sequence detection and removal can be as follows Figure 5 shown.

[0078] In an example, Figure 6 As shown, if the duration of a static sequence is less than T th2 , then it indicates that he is a short stay rather than a long stay, so it should be converted into a dynamic sequence and merged with the adjacent dynamic sequence.

[0079] Assume there is a state sequence: {S1, M1, S2, M2, S3, M3, S4, M4}, where the duration of S2 is less than T th2 , then the server needs to merge it with M1 and M2, and the new state sequence is (S1, S3, M3, S4, M4}.

[0080] At this point, all dynamic sequences and static sequences have been processed, and the position of the stop point can be calculated next.

[0081] Step 106: Filter out all static sequences in the state sequence, and calculate the position of the weighted geometric center point of each static sequence as the position of each stay point.

[0082] In the specific implementation, Figure 7 As shown, after the server removes abnormal dynamic sequences and merges short-stay static sequences, it can filter out all static sequences in the state sequence and calculate the position of the weighted geometric center point of each static sequence as the position of each stay point.

[0083] In an example, Figure 8 As shown, when calculating the position of the weighted geometric center point of each static sequence, the server can traverse each static sequence to determine the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence. Then, based on the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence, a weighted sum is performed to calculate the position of the weighted geometric center point of the current static sequence. After completing the traversal of all static sequences, the position of the weighted geometric center point of each static sequence can be obtained.

[0084] In one example, the server calculates the position of the weighted geometric center point of the current static sequence by performing weighted summation based on the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence. This can be achieved by the following formula:

[0085]

[0086] Among them, assuming that the current static sequence is the i-th static sequence, G(S i ) represents the position of the weighted geometric center point of the current static sequence, (lat ij ,lon ij ) represents the longitude and latitude coordinates corresponding to the jth time point in the current static sequence, n i Indicates the number of time points in the current static sequence, w ij Represents the weight corresponding to the j-th time point in the current static sequence.

[0087] The efficient stop point identification method based on NSD data proposed in this application has the following beneficial effects:

[0088] First, data distribution and load balancing are optimized, improving processing efficiency.

[0089] This application uses the Spark computing engine and hash value partitioning technology to effectively improve the data processing speed and realize the rapid analysis of large-scale data sets, so that data with the same identifier is assigned to the same partition, avoiding data skew, reducing task execution time and reducing task failure rate.

[0090] Second, the ability to handle abnormal data has been enhanced.

[0091] This application can effectively identify and process abnormal data by introducing data deduplication, outlier removal, and abnormal state sequence detection and removal mechanisms, thereby improving the accuracy and reliability of stay point identification. In particular, in the abnormal sequence detection and removal steps, the process is repeated and iterated until no abnormal sequence appears, ensuring that the data has high quality.

[0092] Third, the stay point identification criteria are flexible.

[0093] This application calculates the duration of attachment to the base station, analyzes the characteristics of the business scenario, and distinguishes between static base stations and dynamic base stations by setting a reasonable time threshold. The time threshold can be adjusted according to specific business needs, thereby improving the flexibility of stay point identification and making stay point identification more accurate.

[0094] Fourth, the state sequence processing is highly refined.

[0095] This application processes the state sequence in detail, and accurately divides the complete NSD data of the day into dynamic sequence and static sequence through the dynamic and static recognition algorithm, converts the short static sequence into a dynamic sequence and merges it with the adjacent dynamic sequence, which can more accurately distinguish between repeated waypoints and real stop points, and improves the accuracy of stop point recognition.

[0096] In summary, the efficient stop point identification method based on NSD data proposed in this application not only improves the computational efficiency when processing large-scale NSD data, but also enhances the accuracy and reliability of data processing, thereby providing a more efficient, accurate and reliable stop point identification solution.

[0097] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0098] Another embodiment of the present application proposes an efficient stop point identification system based on NSD data. The implementation details of the efficient stop point identification system based on NSD data proposed in this embodiment are described in detail below. The following content is only the implementation details provided for easy understanding and is not necessary for the implementation of this example.

[0099] Fig. 9 It is a structural diagram of an efficient stop point identification system based on NSD data proposed in this embodiment, which includes: an acquisition module 201, a partitioning module 202, a sorting and deduplication module 203, a duration calculation module 204, a state marking module 205, an exception handling module 206 and a position calculation module 207.

[0100] The acquisition module 201 is used to acquire NSD data.

[0101] The partitioning module 202 is used to partition the NSD data according to the hash value of the device identifier through the Spark computing engine to ensure that the NSD data with the same device identifier is allocated to the same partition.

[0102] The sorting and deduplication module 203 is used to sort the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and to dedupe the original base station sequence based on the timestamp and the geographic coordinates to obtain a deduplicated base station sequence.

[0103] The duration calculation module 204 is used to calculate the time difference between the device at the current attached base station and the next attached base station based on the deduplicated base station sequence as the short duration of the device at the current attached base station, and perform aggregation operations on each attached base station to determine the cumulative duration of the device at each attached base station.

[0104] The state marking module 205 is used to determine the attached base stations whose cumulative duration exceeds the first time threshold as static base stations, and determine the attached base stations whose cumulative duration does not exceed the first time threshold as dynamic base stations, and group and number the deduplicated base station sequences according to the different base station states to obtain a state sequence in which static sequences and dynamic sequences are interlaced.

[0105] The exception processing module 206 is used to remove abnormal dynamic sequences in the state sequence according to the abnormal judgment condition, and convert static sequences whose cumulative duration does not exceed the second time threshold into dynamic sequences.

[0106] The position calculation module 207 is used to filter out all static sequences in the state sequence, and calculate the position of the weighted geometric center point of each static sequence as the position of each stop point.

[0107] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0108] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0109] Another embodiment of the present application provides an electronic device, whose specific structure is as follows: Fig.10 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute an efficient stop point identification method based on NSD data described in the above method embodiments.

[0110] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0111] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0112] Another embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an efficient stop point identification method based on NSD data as described in the above method embodiments.

[0113] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. An efficient method for identifying stay points based on NSD data, characterized in that: include: The Spark computing engine partitions the NSD data based on the hash value of the device ID to ensure that the NSD data with the same device ID is assigned to the same partition. The NSD data in the partition is sorted according to the timestamp to form an ordered original base station sequence, and the original base station sequence is deduplicated based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence; Based on the deduplicated base station sequence, the time difference between the device at the current attached base station and the next attached base station is calculated as the short duration of the device at the current attached base station, and aggregation operations are performed for each attached base station to determine the cumulative duration of the device at each attached base station; Determine the attached base station whose cumulative duration exceeds the first time threshold as a static base station, determine the attached base station whose cumulative duration does not exceed the first time threshold as a dynamic base station, and group and number the deduplicated base station sequence according to different base station states to obtain a state sequence in which the static sequence and the dynamic sequence are interlaced; According to the abnormal discrimination condition, the abnormal dynamic sequence in the state sequence is removed, and the static sequence whose cumulative duration does not exceed the second time threshold is converted into a dynamic sequence; All static sequences in the state sequence are filtered out, and the positions of the weighted geometric center points of each static sequence are calculated respectively as the positions of each stay point.

2. The method for efficiently identifying stay points based on NSD data according to claim 1, characterized in that: The Spark computing engine partitions the NSD data according to the hash value of the device identifier. This is achieved through a partition function, which is f(id)=hash(id) / k, where id is the device identifier and k is the number of partitions. The partition process specifically includes: For each item of NSD data, calculate the hash value of its device identifier hash(id); Take hash(id) modulo k to get partition number p, and distribute the NSD data to partition p.

3. The method for efficiently identifying stay points based on NSD data according to claim 1, characterized in that: The original base station sequence is deduplicated based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence, including: Traverse each NSD data in the original base station sequence and check whether the timestamp and geographic coordinates of the current NSD data are consistent with those of the previous NSD data; If the timestamp and geographic coordinates of the current NSD data are consistent with those of the previous NSD data, the two are considered duplicate NSD data and only the current NSD data is retained; If the timestamps of the current NSD data and the previous NSD data are consistent, but the geographic coordinates are inconsistent, the previous NSD data is considered to be erroneous data, and only the current NSD data is retained; If the geographical coordinates of the current NSD data are consistent with those of the previous NSD data, but the timestamps are inconsistent, the current NSD data is considered to be erroneous data, and only the previous NSD data is retained; If the timestamp and geographic coordinates of the current NSD data item are inconsistent with those of the previous NSD data item, both the current NSD data item and the previous NSD data item are retained.

4. The method for efficiently identifying stay points based on NSD data according to claim 1, characterized in that: The time difference between the current attached base station and the next attached base station is calculated as the short duration of the device at the current attached base station, which is implemented by the following formula: L i =t i+1 -t i ; Among them, t i and t i+1 are two consecutive timestamps in the base station sequence after deduplication, L i Indicates the short duration of the device's current attachment to the base station; Aggregate operations are performed on each attached base station to determine the cumulative duration of the device at each attached base station, which is achieved through the following formula: in, It indicates the cumulative duration of the device attaching to the bth base station, and n indicates the number of short durations of the device attaching to the bth base station.

5. The method for efficiently identifying stay points based on NSD data according to claim 1, characterized in that: Remove abnormal dynamic sequences from the state sequence according to abnormal discrimination conditions, including: Traverse each dynamic sequence in the state sequence, determine in turn whether the number of position points in the current dynamic sequence is less than a preset position number threshold, determine whether the duration of the current dynamic sequence is less than a preset duration threshold, and determine whether the current dynamic sequence overlaps with the previous and next static sequences on the base station; If the number of position points in the current dynamic sequence is less than the position number threshold, or the duration of the current dynamic sequence is less than the duration threshold, or the current dynamic sequence overlaps with the previous and next static sequences on the base station, the current dynamic sequence is marked as an abnormal dynamic sequence and removed.

6. A method for efficiently identifying stay points based on NSD data according to any one of claims 1 to 5, characterized in that: The positions of the weighted geometric center points of each static sequence are calculated respectively, including: Traverse each static sequence to determine the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence; The position of the weighted geometric center point of the current static sequence is calculated by performing weighted summation based on the duration of the current static sequence, the number of time points in the current static sequence, and the longitude and latitude coordinates corresponding to each time point in the current static sequence; After the traversal is completed, the position of the weighted geometric center point of each static sequence is obtained.

7. The method for efficiently identifying stay points based on NSD data according to claim 6, characterized in that: Based on the duration of the current static sequence, the number of time points in the current static sequence, and the latitude and longitude coordinates corresponding to each time point in the current static sequence, the position of the weighted geometric center point of the current static sequence is calculated, which is achieved by the following formula: Among them, assuming that the current static sequence is the i-th static sequence, G(S i ) represents the position of the weighted geometric center point of the current static sequence, (lat ij ,lon ij ) represents the longitude and latitude coordinates corresponding to the jth time point in the current static sequence, n i Indicates the number of time points in the current static sequence, w ij Represents the weight corresponding to the j-th time point in the current static sequence.

8. An efficient stop point identification system based on NSD data, characterized in that: include: Acquisition module, used to acquire NSD data; The partitioning module is used to partition the NSD data according to the hash value of the device identifier through the Spark computing engine to ensure that the NSD data with the same device identifier is assigned to the same partition; A sorting and deduplication module is used to sort the NSD data in the partition according to the timestamp to form an ordered original base station sequence, and to dedupe the original base station sequence based on the timestamp and geographic coordinates to obtain a deduplicated base station sequence; A duration calculation module, used to calculate the time difference between the current attached base station and the next attached base station of the device based on the deduplicated base station sequence as the short duration of the device at the current attached base station, and perform aggregation calculations on each attached base station to determine the cumulative duration of the device at each attached base station; A state marking module, used to determine the attached base stations whose cumulative duration exceeds the first time threshold as static base stations, and determine the attached base stations whose cumulative duration does not exceed the first time threshold as dynamic base stations, and group and number the deduplicated base station sequences according to different base station states to obtain a state sequence in which the static sequence and the dynamic sequence are interlaced; An exception processing module, used to remove abnormal dynamic sequences in the state sequence according to the abnormal discrimination condition, and convert static sequences whose cumulative duration does not exceed the second time threshold into dynamic sequences; The position calculation module is used to filter out all static sequences in the state sequence, and respectively calculate the position of the weighted geometric center point of each static sequence as the position of each stay point.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an efficient stop point identification method based on NSD data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is possible to implement an efficient method for identifying stay points based on NSD data as described in any one of claims 1 to 7.

Images