A method for fusing congestion events from different batches based on single-batch fused congestion data

Abstract

This invention discloses a method for fusing congestion events from different batches based on single-batch fused congestion data. The method includes: extracting a historical congestion data table corresponding to the selected city code from a historical database; if the historical congestion data table is empty, storing the current batch of congestion data in the historical database, and setting the initial ID of the fused congestion event and the ID of the previous batch of congestion events as the event IDs corresponding to the current batch of congestion data; if the historical congestion data table is not empty, matching the original congestion event IDs in the current batch of congestion data with the associated event IDs in the historical congestion data table, assigning values ​​to the initial ID of the fused congestion event and the ID of the previous batch of congestion events based on the matching results, and storing them in the historical database. This invention optimizes the data tracking problem caused by changes in event IDs, ensuring that the same event can be correctly matched in different batches of data, and enhancing the consistency of traffic analysis.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method for fusing congestion events from different batches based on single-batch fused congestion data. Background Technology

[0002] Existing technologies have already implemented methods for fusing congestion data within the current batch. For example, they capture congestion data interfaces from major map providers such as Baidu, Gaode, and Tencent every 10 minutes to obtain raw congestion data from each provider and perform event fusion. The final result is an `eventid` that concatenates the original IDs of each event source and its corresponding congestion event information for the same congestion event. However, existing methods only achieve the fusion of congestion data within the current batch. With increasing management demands, it is necessary to track and handle the same congestion event, thus creating a management requirement to fuse congestion data from different batches (e.g., captured every 10 minutes), that is, to associate the same congestion event across different batches. Therefore, this paper proposes a method for fusing congestion events from different batches based on single-batch fused congestion data.

[0003] Existing methods retain the original event ID, which can be used for cross-batch fusion processes, but this event ID cannot be used directly, and there are several issues:

[0004] (1) The eventid is simply a concatenation of IDs that are considered to belong to the same congestion event in the order they are crawled. For example, if the parsing results show that Baidu event ID congestion01 and Gaode event ID congestion02 are the same event, then the concatenated eventid would be expressed as bd_congestion01&gd_congestion02. However, during subsequent crawling, Gaode congestion02 may be crawled first, and the original data may not be crawled in the original order. Therefore, for congestion events with the same original data source, the eventids formed in different batches may be different.

[0005] (2) For the same event, map providers such as Baidu and Gaode may generate different congestion event IDs in different batches for the same congestion event.

[0006] (3) In addition, even if the congestion event still exists, in the next batch, the event ID of a certain map provider that was detected last time may not appear in the next batch, and it will be replaced by another event. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a method for fusing congestion events from different batches based on single-batch fused congestion data, thereby resolving the issues present in the prior art.

[0008] To achieve the above objectives, the present invention provides a method for fusing congestion events from different batches based on single-batch fused congestion data, comprising:

[0009] Build a historical database and classify and store congestion data in the historical database based on city codes;

[0010] Extract historical congestion data tables corresponding to the selected city codes from the historical database;

[0011] If the congestion data history table is empty, the current batch of congestion data will be stored in the historical database, and the initial ID of the merged congestion event and the ID of the previous batch of congestion events will be set as the event ID corresponding to the current batch of congestion data.

[0012] If the congestion data history table is not empty, then parse the event IDs in the current batch of congestion data and extract the original congestion event IDs. Based on the original congestion event IDs and the associated event IDs in the congestion data history table, match the current batch of congestion data with the congestion data history table. Based on the matching results, assign values ​​to the initial ID of the fused congestion event in the current batch of congestion data and the ID of the previous batch of congestion event, and store the current batch of congestion data in the historical database.

[0013] Optionally, the historical database is a temporary historical database that stores congestion data within 30 minutes, with an independent data table created for each city code.

[0014] Optionally, parse the event IDs in the current batch of congestion data and extract the original congestion event IDs; based on the original congestion event IDs, determine whether the event IDs in the current batch of congestion data are from a single source. If they are from a single source, assign empty values ​​to both the initial ID of the merged congestion event and the ID of the previous batch of congestion events, and obtain the latest time; if they are not from a single source, split the event IDs to obtain a list of associated event IDs.

[0015] Optionally, if it is a single source, the event ID is matched with the associated event ID in the congestion data history table. If the match is successful, the event ID corresponding to the matched data is assigned to the initial ID of the merged congestion event. It is then determined whether the request_time of the matched data is less than the current latest time. If so, the current latest time is assigned to request_time, the event ID corresponding to the matched data is assigned to the previous batch of congestion event IDs, and the associated event IDs are extracted to complete the initial data processing.

[0016] Optionally, after the initial processing is completed, it is determined whether the initial ID of the merged congestion event and the ID of the previous batch of congestion events are empty. If they are not empty, the processed data is stored in the historical database. If they are empty, the event ID in the current batch of congestion data is assigned to the corresponding associated event ID, the initial ID of the merged congestion event, and the ID of the previous batch of congestion events, until the congestion data historical table is traversed.

[0017] Optionally, if it is not a single source, it is segmented based on the event ID generation rule to obtain a list of associated event IDs. The list of associated event IDs is then matched with the associated event IDs in the congestion data history table. If there is overlap, the match is successful. After a successful match, the traversal of the congestion data history table is stopped. The initial ID of the fused congestion event of the current batch of congestion data is the initial ID of the fused congestion event of the matched data, and the ID of the congestion event of the previous batch is the event ID corresponding to the matched data.

[0018] Optionally, after successful matching and assignment, obtain the associated event ID based on the initial ID of the merged congestion event, determine whether the event ID of the current batch of congestion data exists in it, if it exists, store it in the historical database, if it does not exist, concatenate the event ID of the current batch of congestion data into the associated event ID obtained based on the initial ID of the merged congestion event, and update all data with the same initial ID of the merged congestion event to the concatenated associated event ID.

[0019] Optionally, when matching the current batch of congestion data with the historical congestion data table, the historical data in the historical congestion data table can be sorted in reverse chronological order.

[0020] Compared with the prior art, the present invention has the following advantages and technical effects:

[0021] This invention utilizes the correlation between historical data eventids to fuse traffic congestion events from different batches. By using correlation analysis of event IDs for congestion data from different batches, event fusion can be performed more efficiently, avoiding the computational resource overhead and delay caused by latitude and longitude calculation and matching.

[0022] (1) Achieve data consistency across batches: optimize the data tracking problem caused by eventid changes.

[0023] (2) Improve data availability: Ensure that the same event can be correctly matched in different batches of data to enhance the consistency of traffic analysis. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a general flowchart of an embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating the process where the event ID in the current batch of congestion data is from a single source, according to an embodiment of the present invention.

[0027] Figure 3 This is a flowchart illustrating the process where the event ID in the current batch of congestion data comes from multiple sources, as described in this embodiment of the invention. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0030] Example 1

[0031] like Figure 1-3 As shown, this embodiment provides a method for fusing congestion events from different batches based on single-batch fused congestion data, including:

[0032] (1) In order to integrate congestion events from different batches, the event ID should include the original event ID information obtained from each batch;

[0033] (2) Construct an initial ID (hereinafter referred to as lid) for the fusion congestion event. For each batch of congestion events, analyze the lid of each event to determine whether the congestion events belong to the same event. At the same time, construct the ID (hereinafter referred to as pid) of the previous batch of congestion events under this event to associate it with the fusion event ID collected last time under this congestion event.

[0034] (3) An association analysis method based on eventid was constructed, thereby connecting different batches of congestion events.

[0035] The parameters and their meanings are shown in Table 1:

[0036] Table 1

[0037]

[0038]

[0039] like Figure 1 As shown, the main steps include:

[0040] (1) Data classification and storage: Congestion data is classified and stored according to space, such as storing each batch of data according to city (corresponding to the city_code data field), and constructing a temporary.db historical table that stores the historical data table of each city_code to store the data within the last 30 minutes.

[0041] (2) Historical data acquisition: Arrange historical events in reverse chronological order and extract all historical events in reverse chronological order for subsequent retrieval and association.

[0042] If the historical database is empty, meaning that congestion has occurred for the first time in the city within 30 minutes, the currently received data will be directly stored in the historical database, and its own eventid will be assigned to lid and pid, indicating that the current event is the first congestion event in this road segment area of ​​the city.

[0043] (3) Event matching and fusion:

[0044] 1) Obtain the current eventid and determine whether it is an eventid from a single source or a combination of multiple sources (concatenation rule: &).

[0045] 2) Extract all original congestion event IDs contained in the currently acquired single-batch fusion congestion event `current_combine_congestion_data_1` from its `eventid`, construct a `related_id` list, and perform data matching. Calculate the matching relationship based on `lid` and `pid`. From the historical database, perform correlation calculations with the `related_id` of the currently acquired single-batch fusion congestion event `current_combine_congestion_data_1` for each item. The first data item matched from the historical database (referred to as `history_data_1`) has its event ID as the `pid` of the current single-batch fusion congestion event, and the `lid` corresponding to `history_data_1` is the cross-batch tracking event ID of the congestion event corresponding to the current congestion event `current_combine_congestion_data_1`.

[0046] 3) If the eventid already exists in the historical data, update the historical database; otherwise, generate a new eventid and store it in the historical database.

[0047] (4) Data storage and output:

[0048] 1) Maintain the latest_related_congestion_time variable to ensure the integrity of the data time series.

[0049] 2) The relationship between lid and pid is matched, and finally stored in the combine_city table (a temporary data table that stores the current batch and the current city's congestion data) to achieve data fusion.

[0050] The specific processing procedure is as follows: Figure 2 and Figure 3 As shown, the description is as follows:

[0051] (1) Set up storage space according to spatial regions:

[0052] 1) A spatial partitioning storage method is adopted, and congestion data is stored according to the city code (city_code);

[0053] 2) Establish a temporary.db historical data repository to maintain the historical data table corresponding to each city_code;

[0054] 3) Store congestion data for the past 30 minutes to control data validity using time windows;

[0055] (2) Set a unique identifier for the same congestion data in different batches for cross-batch event fusion:

[0056] An initial ID (hereinafter referred to as lid) for merging congestion events is constructed. For each batch of congestion events, the lid of each event is analyzed to determine whether the congestion events belong to the same event. At the same time, the ID of the previous batch of congestion events under this event (hereinafter referred to as pid) is constructed to associate it with the ID of the last congestion event collected under this congestion event.

[0057] (3) Obtain all historical data within the current spatial region in reverse chronological order:

[0058] 1) Extract historical congestion event data in reverse chronological order, and then search for correlations between the historical data and the congestion data obtained in the current batch.

[0059] 2) If the historical data table is empty, that is, the first time congestion has occurred in the city, the current data is stored in the database, and the eventid of the current data is assigned to lid and pid to mark the first congestion event in the road segment area.

[0060] 3) Parse the eventid in each historical data record and extract the original event ID from each eventid. For example, it can be constructed into a list of related_id and passed into the computer system for calculation.

[0061] (4) Construct a related_id list containing the original event IDs collected in different batches under the current event by parsing the eventid of each data item:

[0062] 1) Parse the eventid in the current batch of congestion event data to extract the original congestion event information contained therein;

[0063] 2) For each piece of data in the historical database, a related_id field is also added. All original congestion event IDs associated with the current event are stored in this field, and can be stored in the database in list form.

[0064] (5) Parse the original event IDs contained in the current event and each data entry in the historical database, and calculate the correlation between different batches of congestion data:

[0065] 1) Based on all the original event information parsed from the current eventid, perform association matching with the related_id list of each record in the historical database. If a match is found, it proves that the current event is related to the event in the historical database. If no match is found, it is considered that the current event does not exist in the historical database and is the first occurrence of a congestion event.

[0066] 2) Use the eventid corresponding to the first matched data as the pid of the current event to mark the congestion state expressed by the last detected data in the current congestion event;

[0067] 3) The lid corresponding to the first matched data is used as the lid of the current event. In other words, data from different batches belonging to the same congestion event will all have the same lid. This lid is the eventid when the current event was first detected by the system.

[0068] (5) Rolling update of the original event IDs of the data:

[0069] 1) After supplementing the current congestion event data with information such as lid and pid, store it in the historical database for the next batch of data association calculation;

[0070] 2) Write the original event IDs parsed from the eventid of this batch into the related_id list in the historical database that are detected as the same related event. If there are duplicate elements, perform deduplication.

[0071] 3) Output the current batch of congestion event information with the lid and pid fields to the business system for congestion data processing and cross-batch congestion event analysis. Specifically, the lid can be used as an index to track and display the changing relationship of congestion event data in different batches under that lid.

[0072] eventids are matched by cross-batch data association. The matching rules and processes include:

[0073] (1) In the current batch and historical data, the eventid of each congestion event is formed by concatenating event identifiers from different sources using the '&' symbol to create a unique event identification structure. For example, if the current congestion event is formed by merging the two events bd_congestion_01 and gd_congestion_02, then the final eventid should be "bd_congestion_01&gd_congestion_02". This concatenation method ensures that the eventid of the current event contains the id characteristics of the original data, thus laying the foundation for event association in different batches.

[0074] (2) Construct a list of all original event IDs under the current event, represented as the related_id list, and dynamically update the list. Store the original event IDs captured in each batch into this list. If there are duplicate elements, perform deduplication. For example, if a congestion event is captured for the first time and its eventid is "bd_congestion_01&gd_congestion_02", then the corresponding related_id is ['bd_congestion_01', 'gd_congestion_02']. If the eventid of the same event is captured for the second time and is "bd_congestion_01&tencent_congestion_03", then after merging, store tencent_congestion_03 into the related_id and update it to ['bd_congestion_01', 'gd_congestion_02', 'tencent_congestion_03'], and so on.

[0075] (3) Use historical data reverse order matching to prioritize obtaining the most recent historical event data for request_time;

[0076] (4) Based on the data association in the historical table, assign values ​​to the newly added fields of the congestion events in the current batch. If the original event ID in the eventid of the current event A already exists in the related_id list of the historical event B, then A inherits the lid of B because the two are the same event and should both enjoy the same lid (initial ID of the merged congestion event); and the eventid in the historical event B is assigned to the pid of A. Because the historical data is output in reverse chronological order, the first match of the association proves that the current event is the most recently captured event.

[0077] (5) If the eventid is not matched in the historical database, that is, the current congestion event is occurring for the first time, a new eventid is created and stored, and its eventid is assigned to lid and pid.

[0078] The technical features of the matching strategy for congestion event identifiers (eventid) during data fusion include:

[0079] (1) In the process of facilitating the association and matching of historical congestion data, the historical data is output in reverse time order, and the data that is closest to request_time in the current batch is prioritized for matching. The most recent historical data that is finally matched is used as the previous data of the current congestion event data.

[0080] (2) Set a unique lid inheritance rule to ensure that events remain consistent when matching across batches;

[0081] (3) Use a for loop to iterate through the calculation of historical data, but terminate the iteration immediately when the matching condition is met for the first time to improve matching efficiency;

[0082] (4) During the eventid data storage process, ensure that the related_id association range includes all original data sources.

[0083] Example 2

[0084] This embodiment provides a method for fusing congestion events from different batches based on single-batch fused congestion data, including:

[0085] Build a historical database and classify and store congestion data in the historical database based on city codes;

[0086] Extract historical congestion data tables corresponding to the selected city codes from the historical database;

[0087] If the congestion data history table is empty, the current batch of congestion data will be stored in the historical database, and the initial ID of the merged congestion event and the ID of the previous batch of congestion events will be set as the event ID corresponding to the current batch of congestion data.

[0088] Specifically, if the congestion data history table is empty, the current batch of congestion data will be stored in the historical database, and the initial ID of the merged congestion event will be used, by default, as the congestion event ID of the current batch of congestion data stored in the historical database. In other words, newly received data will, by default, be the initial congestion event, and the corresponding initial congestion event ID will be the ID of that event.

[0089] Example:

[0090] The current congestion event information is shown in Table 2:

[0091] Table 2

[0092] bd_001 June 1, 2025, 9:00 AM

[0093] When storing data in the historical database, since this congestion data is the first record of this congestion event, the event ID is used as the initial ID by default, as shown in Table 3:

[0094] Table 3

[0095] bd_001 June 1, 2025, 9:00 AM bd_001

[0096] If the congestion data history table is not empty, then parse the event IDs in the current batch of congestion data and extract the original congestion event IDs. Based on the original congestion event IDs and the associated event IDs in the congestion data history table, match the current batch of congestion data with the congestion data history table. Based on the matching results, assign values ​​to the initial ID of the fused congestion event in the current batch of congestion data and the ID of the previous batch of congestion event, and store the current batch of congestion data in the historical database.

[0097] The historical database is a temporary historical database that stores congestion data for the past 30 minutes, with a separate data table created for each city code.

[0098] Parse the event IDs in the current batch of congestion data to extract the original congestion event IDs; based on the original congestion event IDs, determine whether the event IDs in the current batch of congestion data are from a single source. If they are from a single source, assign null values ​​to both the initial ID of the merged congestion event and the ID of the previous batch of congestion events, and obtain the latest time; if they are not from a single source, split the event IDs to obtain a list of associated event IDs.

[0099] Specifically, if it is a single source, the event ID is matched with the associated event ID in the congestion data history table. If the match is successful, the event ID corresponding to the matched data is assigned to the initial ID of the merged congestion event. It is then determined whether the request_time of the matched data is less than the current latest time. If so, the current latest time is assigned to request_time. The event ID corresponding to the matched data is assigned to the previous batch of congestion event IDs, and the associated event IDs are extracted. It is then determined whether the initial ID of the merged congestion event and the ID of the previous batch of congestion event are empty. If they are not empty, the processed data is stored in the historical database. If they are empty, the event IDs in the current batch of congestion data are simultaneously assigned to the corresponding associated event IDs, the initial ID of the merged congestion event, and the ID of the previous batch of congestion event, until the congestion data history table is traversed.

[0100] Specifically, if it is not a single source, the event ID is segmented based on the event ID generation rules to obtain a list of associated event IDs. The list of associated event IDs is then matched with the associated event IDs in the congestion data history table. If there is overlap, the match is successful, and the traversal of the congestion data history table stops after a successful match. The initial ID of the merged congestion event of the current batch of congestion data is the initial ID of the merged congestion event of the matched data, and the event ID of the previous batch of congestion data is the event ID corresponding to the matched data. The associated event ID is obtained based on the initial ID of the merged congestion event. It is then determined whether the event ID of the current batch of congestion data exists in the list. If it exists, it is stored in the historical database. If it does not exist, the event ID of the current batch of congestion data is concatenated into the associated event ID obtained based on the initial ID of the merged congestion event. The concatenated associated event ID is then updated to all data with the same initial ID of the merged congestion event.

[0101] After a successful match and assignment, the newly received data is stored in the historical data table. Before writing to the database, the associated event ID list field in the newly received data is integrated with the list of successfully matched associated event IDs in the historical data table, and duplicate elements in the list are removed.

[0102] See Table 4 for specific examples:

[0103] Table 4

[0104] Newly received data t1 bd_001&gd_002 ['bd_001','gd_002'] Historical data t0 bd_001&tencent_001 ['bd_001','tencent_001'] New data stored in the history table t1 bd_001&gd_002 ['bd_001','tencent_001','gd_002']

[0105] Specifically, when matching the current batch of congestion data with the historical congestion data table, the historical data in the historical congestion data table is arranged in reverse chronological order.

[0106] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for fusing congestion events from different batches based on single-batch fused congestion data, characterized in that, Includes the following steps: Build a historical database and classify and store congestion data in the historical database based on city codes; Extract historical congestion data tables corresponding to the selected city codes from the historical database; If the congestion data history table is empty, the current batch of congestion data will be stored in the historical database, and the initial ID of the merged congestion event and the ID of the previous batch of congestion events will be set as the event ID corresponding to the current batch of congestion data. If the congestion data history table is not empty, then parse the event IDs in the current batch of congestion data and extract the original congestion event IDs included; based on the original congestion event IDs and the associated event IDs in the congestion data history table, match the current batch of congestion data with the congestion data history table, assign values ​​to the initial ID of the fused congestion event of the current batch of congestion data and the ID of the previous batch of congestion event based on the matching results, and store the current batch of congestion data in the historical database; Parse the event IDs in the current batch of congestion data and extract the original congestion event IDs; based on the original congestion event IDs, determine whether the event IDs in the current batch of congestion data are from a single source. If they are from a single source, insert the current batch of congestion data into the congestion data history table, set the initial ID of the merged congestion event and the ID of the previous batch of congestion event to be empty, and extract the acquisition time of the current event. If it is not a single source, the event ID is split to obtain a list of associated event IDs; If it is a single source, the event ID is matched with the list of associated event IDs for each congestion data entry in the congestion data history table. If a match is found, the current batch of congestion data is stored in the congestion data history table according to the field format in the history table. If the previous batch of congestion event IDs for the current batch of congestion data is empty, the event ID of the matching data in the congestion data history table is assigned to it. It is then determined whether the acquisition time of the matching data in the congestion data history table is less than the acquisition time of the current event. If so, the latest time is assigned to the acquisition time of the matching data in the history table. The event ID of the matching data in the congestion data history table is assigned to the initial ID of the fused congestion event for the current batch of congestion data, and the associated event IDs are extracted, completing the initial data processing. If it is not a single source, it is segmented based on the event ID generation rules to obtain a list of associated event IDs. The list of associated event IDs is then matched with the list of associated event IDs for each congestion data entry in the congestion data history table. If an inclusion relationship exists, the match is successful. Once a match is successful, the traversal of the congestion data history table is stopped. The initial ID of the fused congestion event for the current batch of congestion data is the initial ID of the fused congestion event for the matching data in the congestion data history table. The event ID of the previous batch of congestion data for the current batch of congestion data is the event ID corresponding to the matching data in the congestion data history table. After successful matching and assignment, the associated event ID is obtained based on the initial ID of the merged congestion event. It is then determined whether the event ID of the current batch of congestion data exists in the database. If it does, the current batch of congestion data is stored in the historical database. If it does not exist, the event ID of the current batch of congestion data is concatenated to the associated event ID obtained based on the initial ID of the merged congestion event. The concatenated associated event ID is then updated in the congestion data with the same initial ID of the merged congestion event. If there are duplicate elements, deduplication is performed, and the concatenated associated event ID is updated in the congestion data with the same initial ID of the merged congestion event.

2. The method for fusing congestion events from different batches based on single-batch fused congestion data according to claim 1, characterized in that, The historical database is a temporary historical database that stores congestion data within 30 minutes, with an independent data table created for each city code.

3. The method for fusing different batches of congestion events based on single-batch fused congestion data according to claim 1, characterized in that, After the initial processing is completed, it is determined whether the initial ID of the merged congestion event and the ID of the previous batch of congestion event in the current batch of congestion data stored in the congestion data history table are empty. If they are not empty, they are stored in the congestion data history table. If they are empty, the event ID in the current batch of congestion data is assigned to the corresponding associated event ID, the initial ID of the merged congestion event, and the ID of the previous batch of congestion event, and then stored in the congestion data history table.

4. The method for fusing congestion events from different batches based on single-batch fused congestion data according to claim 1, characterized in that, When matching the current batch of congestion data with the historical congestion data table, the historical data in the historical congestion data table is sorted in reverse chronological order.

Images