Different-batch congestion event fusion method based on single-batch fusion congestion data

Abstract

The invention discloses a different-batch congestion event fusion method based on single-batch fusion congestion data. The method comprises the following steps: extracting a congestion data historical table corresponding to a selected city code from a historical database; if the congestion data historical table is empty, storing the current batch of congestion data into a historical database, and setting the fused congestion event initial ID and the previous batch of congestion event ID as an event ID corresponding to the current batch of congestion data; and if the congestion data historical table is not empty, matching the original congestion event ID in the current batch of congestion data with the associated event ID in the congestion data historical table, assigning values to the fused congestion event initial ID of the current batch of congestion data and the previous batch of congestion event ID based on a matching result, and storing the values to a historical database. According to the method, the data tracking problem caused by the change of the event ID is optimized, the same event can be correctly matched in different batches of data, and the continuity of traffic analysis is enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a different batch congestion event fusion method based on single batch congestion data fusion. BACKGROUND

[0002] The prior art has realized a technical method for fusing congestion data under the current batch. For example, congestion data interfaces of mainstream map vendors such as Baidu, Gaode and Tencent are captured every 10 minutes, original congestion data of each map vendor is obtained, and event fusion is performed. Finally, for the same congestion event, an eventid formed by splicing original IDs of each event source and corresponding congestion event information are obtained. However, the existing method only realizes the fusion of congestion data under the current batch. With the continuous improvement of management requirements, it is necessary to track and dispose the same congestion event, and therefore, there is a management requirement for fusing congestion data under different batches (such as captured every 10 minutes), that is, the same congestion event under different batches is associated. Therefore, a different batch congestion event fusion method based on single batch congestion data fusion is proposed.

[0003] In the existing method, the original event ID is retained and can be used for cross-batch fusion process, but the eventid cannot be directly used and has the following problems:

[0004] (1) The eventid is simply spliced according to the order of the captured ID which is considered to belong to the same congestion event, such as bd_congestion01&gd_congestion02. However, in the subsequent capture, congestion02 of Gaode may be captured first, and the original data may not be captured in the original order, so for the congestion event with the same original data source, the eventid formed by different batches may be different.

[0005] (2) For the same event, Baidu, Gaode and other map vendors may generate different congestion event IDs for the same congestion event in different batches.

[0006] (3) In addition, even if the congestion event still exists, the event ID of a certain map vendor detected in the previous batch may not appear in the next batch, and another event may replace it. SUMMARY

[0007] To solve the above technical problems, the application provides a different batch congestion event fusion method based on single batch fusion congestion data to solve the problems existing in the prior art.

[0008] To achieve the above object, the application provides a different batch congestion event fusion method based on single batch fusion congestion data, comprising:

[0009] A historical database is constructed, and congestion data is classified and stored in the historical database based on city codes;

[0010] A congestion data history table corresponding to the selected city code is extracted from the historical database;

[0011] If the congestion data history table is empty, the current batch congestion data is stored in the historical database, and the fusion congestion event initial ID and the last batch congestion event ID are set as the event ID corresponding to the current batch congestion data.

[0012] If the congestion data history table is not empty, the event ID in the current batch congestion data is parsed, and the original congestion event ID is extracted; the current batch congestion data and the congestion data history table are matched based on the original congestion event ID and the associated event ID in the congestion data history table, the fusion congestion event initial ID and the last batch congestion event ID of the current batch congestion data are valued based on the matching result, and the current batch congestion data is stored in the historical database.

[0013] Optionally, the historical database is a temporary historical database, and stores congestion data within 30 minutes, and an independent data table is created for each city code.

[0014] Optionally, the event ID in the current batch congestion data is parsed, and the original congestion event ID is extracted; whether the event ID in the current batch congestion data is single source is judged based on the original congestion event ID, if it is single source, the fusion congestion event initial ID and the last batch congestion event ID are both valued as empty, and the current latest time is obtained; if it is not single source, the event ID is segmented to obtain an associated event ID list.

[0015] Optionally, if it is single source, the event ID is matched with the associated event ID in the congestion data history table, if the matching is successful, the event ID corresponding to the matching data is valued as the fusion congestion event initial ID, whether the matching data request_time is less than the current latest time is judged, if it is, the current latest time is valued as request_time, the event ID corresponding to the matching data is valued as the last batch congestion event ID, and the associated event ID is extracted, and the data preliminary processing is completed.

[0016] Optionally, after the preliminary processing is completed, it is judged whether the fusion congestion event initial ID and the last batch congestion event ID are empty, if not empty, the processed data is stored in the historical database, if empty, the event ID in the current batch congestion data is assigned to the corresponding associated event ID, the fusion congestion event initial ID and the last batch congestion event ID, until the end of the congestion data history table traversal.

[0017] Optionally, if it is not a single source, it is segmented based on the generation rule of the event ID to obtain an associated event ID list, the associated event ID list is matched with the associated event ID in the congestion data history table, if there is a coincidence, the matching is successful, the congestion data history table is stopped after the matching is successful, the fusion congestion event initial ID of the current batch congestion data is the fusion congestion event initial ID of the matching data, and the last batch congestion event ID is the event ID corresponding to the matching data.

[0018] Optionally, after the matching is successful and the assignment is completed, the associated event ID is obtained based on the fusion congestion event initial ID, it is judged whether the event ID of the current batch congestion data exists in the associated event ID, if exists, it is stored in the historical database, if not exists, the event ID of the current batch congestion data is spliced into the associated event ID obtained based on the fusion congestion event initial ID, and the spliced associated event ID is updated to all data with the same fusion congestion event initial ID.

[0019] Optionally, when the current batch congestion data and the congestion data history table are matched, the historical data in the congestion data history table is arranged in time reverse order.

[0020] Compared with the prior art, the present application has the following advantages and technical effects:

[0021] The present application utilizes the association between historical data eventid, and performs fusion of different batch traffic congestion events. For different batch congestion data, the association of event ID is analyzed, the event fusion can be more efficient, and the calculation resource consumption and delay caused by the use of longitude and latitude calculation matching are avoided.

[0022] (1) Realize cross-batch data consistency: optimize the data tracking problem caused by eventid change.

[0023] (2) Improve data availability: ensure that the same event can be correctly matched in different batch data, and enhance the coherence of traffic analysis. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and do not limit the present application. In the drawings:

[0025] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0026] Figure 2 This is a flow chart of an embodiment of the present invention in which the event ID in the current batch of congestion data is from a single source;

[0027] Figure 3 This is a flow chart of an embodiment of the present invention in which event IDs in a current batch of congestion data come from multiple sources. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Example 1

[0031] like Figures 1-3 As shown, this embodiment provides a method for fusing congestion events of different batches based on single-batch fusion congestion data, including:

[0032] (1) In order to merge congestion events from different batches, the event ID should contain the original event ID information obtained from each batch;

[0033] (2) Construct the initial ID of the fused congestion event (hereinafter referred to as lid). For each batch of congestion events, analyze the lid of each event and use the lid to determine whether the congestion events belong to the same event. At the same time, construct the ID of the previous batch of congestion events (hereinafter referred to as pid) under the event to associate it with the fused event ID collected last time under the congestion event.

[0034] (3) An eventid-based correlation analysis method was constructed to connect different batches of congestion events.

[0035] The parameters and their meanings are shown in Table 1:

[0036] Table 1

[0037]

[0038]

[0039] like Figure 1 As shown, the main steps include:

[0040] (1) Data classification storage: congestion data is classified and stored by space, such as classified and stored by city (corresponding to the city_code data field) for each batch of data, and a temporary.db history table containing a history data table for each city_code is constructed to store data within 30 minutes.

[0041] (2) Historical data acquisition: arrange historical events in reverse chronological order, and extract all historical events in reverse chronological order for step-by-step retrieval and correlation.

[0042] If the historical database is empty, that is, the city range appears congestion within 30 minutes for the first time, the current received data is directly stored in the historical database, and its own eventid is assigned to lid and pid, representing the current event is the first congestion event of the city and the road area.

[0043] (3) Event matching and fusion:

[0044] 1) Get the current eventid and determine whether it is a single source or multi-source fusion eventid (concatenation rule: &).

[0045] 2) Extract all original congestion event ids contained in the current single batch fusion congestion event current_combine_congestion_data_1, construct a related_id list, and perform data matching according to lid and pid to calculate the matching relationship. From the historical database, perform relevance calculation with the related_id of the current single batch fusion congestion event current_combine_congestion_data_1, and the event id of the first data (history_data_1) matched from the historical database is the pid of the current single batch fusion congestion event, and the lid corresponding to the history_data_1 is the cross-batch tracking event id of the congestion event corresponding to the current congestion event current_combine_congestion_data_1.

[0046] 3) If the eventid has appeared in the historical data, update the historical database; otherwise, generate a new eventid and store it in the historical database.

[0047] (4) Data storage and output:

[0048] 1) Maintain the latest_related_congestion_time variable to ensure the integrity of the data time sequence.

[0049] 2) lid and pid relationship matching, finally stored in combine_city table (point to store the current batch, the current city congestion data temporary data table), realize data fusion.

[0050] The specific processing process is shown in Figure 2 and Figure 3 , described as follows:

[0051] (1) according to the space region setting storage space:

[0052] 1) using spatial partition storage method, according to city code city_code classification storage congestion data;

[0053] 2) set up temporary.db history data storage library, maintain each city_code corresponding history data table;

[0054] 3) store congestion data within 30 minutes, to time window control data validity;

[0055] (2) the same congestion data of different batches is set with unique identifier for cross batch event fusion:

[0056] Construct fusion congestion event initial ID (hereinafter referred to as lid), for each batch of congestion events, analyze the lid of each event, so as to judge whether each congestion event belongs to the same event through lid. At the same time, construct the last batch congestion event ID (hereinafter referred to as pid) under the event, which is used to associate the last congestion event id collected under the congestion event.

[0057] (3) according to time in reverse order to obtain all the history data in the current space region:

[0058] 1) according to time in reverse order to extract historical congestion event data, and compare the historical data with the congestion data obtained in the current batch;

[0059] 2) if the history data table is empty, that is, the first time congestion occurs in the city range, the current data is stored in the database, and the eventid of the current data is assigned to lid and pid, marking the first congestion event of the road section area.

[0060] 3) analyze the eventid in each historical data, and extract the original event id in each eventid, such as can be constructed as related_id list, input into computer system for calculation.

[0061] (4) through the analysis of each data eventid to construct related_id list containing the original event id collected in different batches under the current event:

[0062] 1) Parse the eventid in the current batch of congestion event data, parse the original congestion event information contained in it;

[0063] 2) Each data in the history library also adds a related_id field, and saves all the original congestion event ids associated with the current event in this field, which can be stored in the database in list form.

[0064] (5) The original event ID contained in the current event and each data in the history library is parsed to calculate the correlation of congestion data in different batches:

[0065] 1) According to the original event information parsed by the current eventid, associate and match with the related_id list of each data in the history database. If it matches, it proves that the current event is associated with the event in the history library, and if it does not match, it is considered that the current event does not exist in the history library, and it is the first congestion event;

[0066] 2) The eventid corresponding to the first matched data is used as the pid of the current event, marking the congestion state expressed by the last detected data in the current congestion event.

[0067] 3) The lid corresponding to the first matched data is used as the lid of the current event, that is, the data detected in different batches belonging to the same congestion event all have the same lid. The lid is the eventid when the current event is first detected by the system.

[0068] (5) Update the original event ID of the data:

[0069] 1) After supplementing the lid, pid and other field information of the current congestion event data, store it in the history library for the next batch of data correlation calculation;

[0070] 2) Write each original event id parsed by the eventid in this batch to the related_id list corresponding to the same associated event detected in the history library, and do the de-duplication processing if there are the same elements.

[0071] 3) The current batch of congestion event information with lid and pid fields is output to the business system to carry out congestion data disposal and cross-batch congestion event analysis. According to the lid as an index, the change relationship of congestion event data in different batches under the lid can be tracked and displayed.

[0072] eventid is matched through cross-batch data association, and the matching rules and processes include:

[0073] (1) In the current batch and historical data, the event ID of each congestion event data is concatenated with the & symbol to form a unique event identification structure. For example, if the current congestion event is formed by the fusion of two events, bd_congestion_01 and gd_congestion_02, the final event ID should be "bd_congestion_01&gd_congestion_02". The concatenation method is used to ensure that the event ID of the current event contains the ID characteristics of the original data, thus laying a foundation for the association of events in different batches.

[0074] (2) Construct a list of all original event IDs under the current event, expressed as a related_id list, and dynamically update the list. Store the original event IDs of each batch captured in the list. If there are duplicate elements, perform deduplication processing. For example, if a congestion event is captured for the first time, its eventID is "bd_congestion_01&gd_congestion_02", then the corresponding related_id is ['bd_congestion_01', 'gd_congestion_02']. If the event is captured for the second time, its eventID is "bd_congestion_01&tencent_congestion_03". After fusion, tencent_congestion_03 is stored in related_id and updated to ['bd_congestion_01', 'gd_congestion_02', 'tencent_congestion_03'], and so on.

[0075] (3) Using the reverse order matching method of historical data, the historical event data with the most recent request_time is obtained first;

[0076] (4) Based on the data association in the historical table, assign the new fields of the congestion events in the current batch. If the original event id in the eventid of the current event A already exists in the related_id list of the historical event B, then A inherits the lid of B because the two are the same event and should have the same lid (the initial ID of the fused congestion event); and the eventid in the historical event B is assigned to the pid of A. Because historical data is output in reverse chronological order, the first matching association proves that the current event is the most recently captured event.

[0077] (5) If eventid is not matched in the history database, that is, the current congestion event occurs for the first time, a new eventid is created for storage, and its eventid is assigned to lid and pid.

[0078] The technical features of the matching strategy of the congestion event identifier eventid in the data fusion process include:

[0079] (1) In the process of facilitating the association matching of historical congestion data, the historical data is output in a time reverse order, the data closest to the current batch request_time is matched preferentially, and the last historical data matched is taken as the last data of the current congestion event data.

[0080] (2) Set a unique lid inheritance rule to ensure that the event remains consistent when matching across batches;

[0081] (3) Use a for loop to traverse the calculation of historical data, but terminate the traversal immediately when the matching condition is met for the first time, thereby improving the matching efficiency;

[0082] (4) In the eventid data storage process, ensure that the related_id association range contains all original data sources.

[0083] Embodiment Two

[0084] The embodiment provides a different batch congestion event fusion method based on single batch fusion congestion data, which includes:

[0085] Construct a historical database, and store the congestion data corresponding to the selected city code in the historical database based on city code;

[0086] Extract the congestion data history table corresponding to the selected city code in the historical database;

[0087] If the congestion data history table is empty, store the current batch congestion data in the historical database, and set the initial ID of the fusion congestion event and the last batch congestion event ID as the event ID corresponding to the current batch congestion data;

[0088] Specifically, if the congestion data history table is empty, store the current batch congestion data in the historical database, and set the initial ID of the fusion congestion event, which is the congestion event ID of the current batch congestion data stored in the historical database by default. That is, the newly received data is the initial congestion event by default, and the initial ID of the corresponding congestion event is the ID of the event.

[0089] Example:

[0090] The current received congestion event information is shown in Table 2:

[0091] Table 2

[0092] Congestion event ID Acquisition time bd_001 2025-06-01 09:00

[0093] When stored in the historical database, since the congestion data is the first data of the congestion event, the event ID is the initial ID by default, as shown in Table 3:

[0094] Table 3

[0095] Event ID Acquisition time Fused congestion event initial ID bd_001 2025-06-01 09:00 bd_001

[0096] If the congestion data history table is not empty, the event ID in the current batch of congestion data is parsed to extract the original congestion event ID. Based on the original congestion event ID and the associated event ID in the congestion data history table, the current batch of congestion data and the congestion data history table are matched. Based on the matching result, the initial ID of the fusion congestion event of the current batch of congestion data and the last batch of congestion event ID are valued, and the current batch of congestion data is stored in the historical database.

[0097] The historical database is a temporary historical database, which stores congestion data within 30 minutes, and creates an independent data table for each city code.

[0098] The event ID in the current batch of congestion data is parsed to extract the original congestion event ID. Based on the original congestion event ID, it is judged whether the event ID in the current batch of congestion data is single source. If it is single source, the initial ID of the fusion congestion event and the last batch of congestion event ID are both valued as empty, and the current latest time is obtained. If it is not single source, the event ID is segmented to obtain an associated event ID list.

[0099] Specifically, if it is single source, the event ID is matched with the associated event ID in the congestion data history table. If the matching is successful, the event ID corresponding to the matching data is valued to the initial ID of the fusion congestion event. It is judged whether the matching data request_time is less than the current latest time. If it is, the current latest time is valued as request_time. The event ID corresponding to the matching data is valued to the last batch of congestion event ID and the associated event ID is extracted. It is judged whether the initial ID of the fusion congestion event and the last batch of congestion event ID are empty. If they are not empty, the processed data is stored in the historical database. If they are empty, the event ID in the current batch of congestion data is valued to the corresponding associated event ID, the initial ID of the fusion congestion event and the last batch of congestion event ID, until the end of the congestion data history table traversal.

[0100] Specifically, if not a single source, the event ID-based generation rule is used to segment the same, obtain a list of associated event IDs, match the list of associated event IDs with the associated event IDs in the congestion data history table, if there is a coincidence, the matching is successful, and the matching is successful. Stop traversing the congestion data history table, the initial ID of the fusion congestion event of the current batch of congestion data is the initial ID of the fusion congestion event of the matching data, the last batch of congestion event ID is the event ID corresponding to the matching data, the associated event ID is obtained based on the initial ID of the fusion congestion event, and it is judged whether the event ID of the current batch of congestion data exists. If it exists, it is stored in the historical database, if it does not exist, the event ID of the current batch of congestion data is spliced into the associated event ID obtained based on the initial ID of the fusion congestion event, and the spliced associated event ID is updated to all data with the same initial ID of the fusion congestion event.

[0101] After the matching is successful and the assignment is completed, the newly received data is stored in the historical data table. Before writing into the database, the associated event ID list field in the newly received data is integrated with the associated event ID list in the historical data table that is matched successfully, and the elements in the list are de-duplicated.

[0102] Specific examples are shown in Table 4:

[0103] Table 4

[0104] Data source Acquisition time Event ID Associated event ID Newly received data t1 bd_001 & gd_002 ['bd_001', 'gd_002'] Data in the history table t0 bd_001 & tencent_001 ['bd_001', 'tencent_001'] New data stored in the history table t1 bd_001 & gd_002 ['bd_001', 'tencent_001', 'gd_002']

[0105] Specifically, when matching the current batch of congestion data with the congestion data history table, the historical data in the congestion data history table is arranged in reverse order of time.

[0106] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fusing congestion events from different batches based on single-batch fusion congestion data, characterized in that: The following steps are involved: Build a historical database and classify and store congestion data based on city codes in the historical database; Extracting a congestion data history table corresponding to the selected city code from a history database; If the congestion data history table is empty, the current batch of congestion data is stored in the history database, and the initial ID of the fused congestion event and the ID of the previous batch of congestion events are set as the event ID corresponding to the current batch of congestion data; If the congestion data history table is not empty, parse the event ID in the current batch of congestion data and extract the original congestion event ID. Based on the original congestion event ID and the associated event ID in the congestion data history table, match the current batch of congestion data with the congestion data history table. Based on the matching result, assign the initial ID of the fused congestion event of the current batch of congestion data and the ID of the previous batch of congestion events, and store the current batch of congestion data in the history database.

2. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 1 is characterized in that: The historical database is a temporary historical database that stores congestion data within 30 minutes and creates an independent data table for each city code.

3. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 1 is characterized in that: Parse the event ID in the current batch of congestion data and extract the original congestion event ID; based on the original congestion event ID, determine whether the event ID in the current batch of congestion data is from a single source. If it is from a single source, insert the current batch of congestion data into the congestion data history table, set the initial ID of the fused congestion event and the ID of the previous batch of congestion events to empty, and extract the acquisition time of the current event; if it is not from a single source, split the event ID to obtain a list of associated event IDs.

4. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 3 is characterized in that: If it is a single source, the event ID is matched with the list of associated event IDs for each congestion data in the congestion data history table. If the match is successful, the current batch of congestion data is stored in the congestion data history table according to the field format in the history data table. If the congestion event ID of the previous batch of the current batch of congestion data is empty, it is assigned to the event ID of the matching data in the congestion data history table; determine whether the acquisition time of the matching data in the congestion data history table is less than the acquisition time of the current event. If so, the current latest time is assigned to the acquisition time of the matching data in the history data table, and the event ID of the matching data in the congestion data history table is assigned to the initial ID of the fused congestion event of the current batch of congestion data, and the associated event ID is extracted to complete the preliminary data processing.

5. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 4 is characterized in that: After the preliminary processing is completed, it is determined whether the initial ID of the fused congestion event of the current batch of congestion data and the ID of the previous batch of congestion events stored in the congestion data history table are empty. If they are not empty, they are stored in the congestion data history table. If they are empty, the event ID in the current batch of congestion data is assigned to the corresponding associated event ID, the initial ID of the fused congestion event and the ID of the previous batch of congestion events, and stored in the congestion data history table.

6. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 3 is characterized in that: If it is not a single source, it will be split based on the event ID generation rule to obtain a list of associated event IDs, and the associated event ID list will be matched with the associated event ID list of each congestion data in the congestion data history table. If there is an inclusion relationship, the match is successful. After the match is successful, the traversal of the congestion data history table will be stopped. The initial ID of the fused congestion event of the current batch of congestion data is the initial ID of the fused congestion event of the matching data in the congestion data history table, and the ID of the previous batch of congestion event of the current batch of congestion data is the event ID corresponding to the matching data in the congestion data history table.

7. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 6 is characterized in that: After the match is successful and the assignment is completed, the associated event ID is obtained based on the initial ID of the fused congestion event, and it is determined whether the event ID of the current batch of congestion data exists in it. If it exists, the current batch of congestion data will be stored in the historical database. If it does not exist, the event ID of the current batch of congestion data will be spliced ​​into the associated event ID obtained based on the initial ID of the fused congestion event, and the spliced ​​associated event ID will be updated to the congestion data with the same initial ID of the fused congestion event.

8. The method for fusion of congestion events of different batches based on single-batch fusion congestion data according to claim 1 is characterized in that: When matching the current batch of congestion data with the congestion data history table, the historical data in the congestion data history table are arranged in reverse chronological order.

Images