Data processing method and device, computer device and storage medium

Abstract

The invention relates to a data processing method and device, a computer device and a storage medium. The method relates to a traffic data processing and intelligent algorithm technology, and solves the problem of lack of a traffic data management mechanism. The method comprises the steps of performing data extraction on collected data through a plurality of sampling strategies to obtain an extraction result corresponding to each sampling strategy; and integrating the plurality of extraction results to obtain comprehensive extraction data. The technical scheme provided by the invention is suitable for mass data processing of a traffic network, high-efficiency and high-business-value data processing is realized, and the storage and operation pressure is effectively relieved.

Description

Technical Field

[0001] This disclosure relates to traffic data processing and intelligent algorithm technology, and in particular to a data processing method, apparatus, computer device and storage medium. Background Technology

[0002] With the rapid development of vehicle-road-cloud integration and intelligent connected vehicle technologies, the amount of traffic-related data, such as roadside perception, traffic light status, roadside video, connected vehicle bus, in-vehicle video, and cloud events, is growing exponentially. A typical city intersection can generate hundreds of gigabytes of data daily, creating enormous storage and maintenance pressures and increasing the cost and computational cycle of subsequent analysis.

[0003] Processing the original data using methods such as random sampling, proportional sampling, and single-dimensional sampling (based on time or space only) can reduce the amount of data to be retained. However, sampling processing can also lead to the loss of key data and information on the correlation between multidimensional features, which affects the subsequent use of data and makes it impossible to achieve intelligent connected management of the transportation network. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a data processing method, apparatus, computer device, and storage medium. By employing multiple sampling strategies to extract data from collected data, and then integrating the multiple extraction results to obtain comprehensive extracted data, this solves the problem of the lack of a traffic data management mechanism, achieves high-efficiency and high-business-value data processing, and effectively alleviates storage and maintenance pressures.

[0005] According to a first aspect of the embodiments of the present disclosure, a data processing method is provided, comprising: Data is extracted from the collected data using multiple sampling strategies, and the extraction results corresponding to each sampling strategy are obtained. The extraction results from multiple sources are integrated to obtain comprehensive extracted data.

[0006] Furthermore, before the step of extracting data from the collected dataset using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy, the method further includes: Based on the data type of the collected data, one or more matching sampling strategies are selected for data extraction, wherein the data type includes at least one or more of the following: Roadside raw data, connected vehicle data, and cloud event data.

[0007] Furthermore, the sampling strategy is any one of the following: Data-driven sampling strategy, business-driven sampling strategy, whitelist sampling strategy, event-driven sampling strategy. The step of extracting data from the collected dataset using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy includes: Traffic pattern data is obtained by extracting data from the collected data using the data-driven sampling strategy, wherein the data-driven sampling strategy includes at least one or more of the following conditions: Prioritize intersections with high traffic volume, intersections with high variability, and intersections with high incident rates; The business-driven sampling strategy is used to extract data from the collected data to obtain business requirement data. The business-driven sampling strategy includes at least one or more of the following conditions: Prioritize key intersections, retain access during special weather conditions, and provide support for specific scenarios; The collected data is extracted using the whitelist sampling strategy to obtain vehicle-dimensional balanced data. The whitelist sampling strategy includes at least one or more of the following conditions: Manufacturer-level quotas, vehicle-level balancing, and mandatory whitelisting; Data is extracted from the collected data according to the event-driven sampling strategy to obtain key event data. The event-driven sampling strategy includes at least one or more of the following conditions: Accident window, congestion-related data, and violation-related data.

[0008] Furthermore, the step of integrating multiple extraction results to obtain comprehensive extracted data includes: For the intersection data between different extraction results, the extraction result is selected to be added to the candidate data according to the priority of the corresponding sampling strategy and / or the data retention priority. The duplicate data in the candidate data are deduplicated based on the fingerprint information of each data point to obtain the comprehensive extracted data.

[0009] Furthermore, the method also includes: The extracted data is retained in a tiered manner according to a preset data retention priority, with different data retention priorities corresponding to different retention periods.

[0010] Furthermore, before the step of extracting data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy, the method further includes: Raw traffic data is obtained by collecting data from different data sources that meet preset data quality conditions. Based on the equipment quality of the data source, the original traffic data is sampled to obtain the collected data.

[0011] Furthermore, the step of sampling the original traffic data based on the equipment quality of the data source to obtain the collected data includes: The quality score for each device is obtained based on one or more of the following evaluation criteria: Position jump rate, heading jump rate, speed jump rate, type jump rate, frequency jump rate; Based on the quality score, a corresponding data filtering strategy is matched, and the original traffic data of the device is sampled according to the data filtering strategy. The sampling results are then added to the collected data.

[0012] According to a second aspect of the embodiments of the present disclosure, a data processing apparatus is provided, comprising: The parallel extraction module uses multiple sampling strategies to extract data from the collected data, and obtains the extraction results corresponding to each sampling strategy. The data fusion module is used to integrate multiple extraction results to obtain comprehensive extracted data.

[0013] According to a third aspect of the embodiments of this disclosure, a computer apparatus is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the above-described data processing method.

[0014] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a computer device, the computer device is enabled to perform the data processing method described above.

[0015] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: First, data is extracted from the collected data using multiple sampling strategies to obtain extraction results corresponding to each sampling strategy. Then, the multiple extraction results are integrated to obtain comprehensive extracted data. Different sampling strategies independently screen data that meets the conditions, ensuring full extraction of the target data. By integrating multiple extraction results into comprehensive extracted data, data quality is further improved, providing a representative and highly practical data foundation for storage and subsequent data analysis and application. This solves the problem of lacking a traffic data management mechanism, achieves high-efficiency and high-business-value data processing, and effectively alleviates storage and maintenance pressure.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating yet another data processing method according to an exemplary embodiment.

[0020] Figure 3 This is a flowchart illustrating yet another data processing method according to an exemplary embodiment.

[0021] Figure 4 This is a flowchart illustrating yet another data processing method according to an exemplary embodiment.

[0022] Figure 5 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment.

[0023] Figure 6 This is a structural block diagram of a data fusion module 502 according to an exemplary embodiment.

[0024] Figure 7 This is a block diagram illustrating yet another data processing apparatus according to an exemplary embodiment. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] A typical city intersection can generate hundreds of gigabytes of data per day, creating enormous storage and maintenance pressures and increasing the cost and computation cycle of subsequent analysis.

[0027] Processing the original data using methods such as random sampling, proportional sampling, and single-dimensional sampling (based on time or space only) can reduce the amount of data to be retained. However, sampling processing can also lead to the loss of key data and information on the relationship between multidimensional features.

[0028] To address the aforementioned issues, embodiments of this disclosure provide a data processing method, apparatus, computer device, and storage medium. By employing multiple sampling strategies to extract data from collected data, and then integrating the multiple extraction results to obtain comprehensive extracted data, this method solves the problem of the lack of a traffic data management mechanism, achieves highly efficient and high-business-value data processing, and effectively alleviates storage and maintenance pressures.

[0029] An exemplary embodiment of this disclosure provides a data processing method, the process of which involves extracting and integrating traffic data as follows: Figure 1 As shown, it includes: Step 101: Extract data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy.

[0030] When selecting a sampling strategy, one or more of the sampling strategies can be selected to extract data based on the data type of the collected data, so as to obtain the extraction results corresponding to each sampling strategy.

[0031] The data type includes at least one or more of the following: Roadside raw data, connected vehicle data, and cloud event data.

[0032] According to one exemplary implementation, the roadside raw data includes: fused perception, traffic lights, roadside video, etc.; connected vehicle data includes vehicle bus, in-vehicle video, etc.; cloud event data includes perception determination events, vehicle bus determination events, etc.

[0033] The sampling strategy is any one of the following: Data-driven sampling strategy, business-driven sampling strategy, whitelist sampling strategy, event-driven sampling strategy.

[0034] In this step, data extraction is performed on the collected dataset using multiple sampling strategies, including: 1. Traffic pattern data is obtained by extracting data from the collected data using the data-driven sampling strategy, wherein the data-driven sampling strategy includes at least one or more of the following conditions: Prioritize intersections with high traffic volume, high variation, and high incident rate.

[0035] The spatial distribution and temporal variation of urban traffic flow are extremely uneven. Traffic flow at some key intersections during peak hours can be more than ten times that at night; intersections prone to incidents have a particularly significant impact on overall traffic flow. If random or uniform sampling is used, the characteristics of high-flow and high-event areas are diluted, resulting in data samples that cannot accurately depict the true traffic conditions and the probability of extreme events, thus affecting subsequent prediction and scheduling decisions. Therefore, a data-driven sampling strategy is adopted to overcome the problem that samples cannot reflect the true state and the probability of extreme events.

[0036] According to one exemplary implementation, the data-driven sampling strategy includes the following conditions: (1) Prioritize high-flow intersections: Based on historical traffic statistics windows, calculate the average and peak traffic of each intersection, and determine high-flow intersections based on the average and / or peak traffic.

[0037] For example, the system calculates the average and peak traffic flow for each intersection based on historical traffic statistics from the past 4–8 weeks, and sorts them in descending order of total traffic flow. The top q% (e.g., 10%) of intersections are selected to form a set L_high. During peak hours T_peak (e.g., 7:00–9:00, 17:00–19:00), all data from this set are retained, while during off-peak hours, data is sampled proportionally p_non (e.g., 20%).

[0038] (2) Prioritize intersections with high variability: Calculate the variance, coefficient of variation, or entropy value of the unit-time flow series for each intersection to identify areas with drastic flow fluctuations. Increase the sampling ratio for intersections with high fluctuation rankings (e.g., the top k intersections in terms of flow fluctuation).

[0039] (3) Prioritize intersections with high event rates: Calculate the event occurrence rate λ within a unit of time. For data collected from intersections with high event occurrence rates within λ, appropriately increase the sampling level or retain all data directly.

[0040] According to one exemplary implementation, dynamic parameter updates are performed: the thresholds for q, p_non, k, and λ are periodically updated on a rolling basis to ensure that the data-driven sampling strategy always reflects the latest traffic conditions.

[0041] Compared to random sampling, the data-driven strategy improves traffic coverage by approximately 30%, event capture rate by approximately 20%, and reduces storage requirements by more than 50%. Congestion prediction models trained on samples extracted using the data-driven sampling strategy show a 15%–25% reduction in prediction error.

[0042] 2. Extract business requirement data from the collected data using the business-driven sampling strategy, wherein the business-driven sampling strategy includes at least one or more of the following conditions: Prioritize key intersections, retain access during special weather conditions, and provide support for specific scenarios.

[0043] Urban traffic management involves numerous key business scenarios requiring focused attention, such as traffic management for large-scale events, long-term monitoring of key intersections, and emergency management during extreme weather. Simple random or proportional sampling methods are insufficient to meet the specific needs of these critical scenarios. Therefore, a business-driven sampling strategy is employed to extract data from the collected data, yielding results that reflect the targeted requirements of these key scenarios.

[0044] According to one exemplary implementation, the business-driven sampling strategy includes at least one or more of the following conditions: (1) Prioritize key intersections: Based on the key list L_key provided by the traffic management department, mark the relevant intersections as higher or highest priority, and ensure that they are fully or maximally retained during peak hours, and the sampling ratio during off-peak hours is ≥p_base (e.g., 30%).

[0045] (2) Special weather retention: When the meteorological monitoring system issues warnings for heavy rain, heavy snow, fog, typhoon, etc., all data of the intersections in the affected area are retained and weather tags (such as {weather category, start and end time}) are added to the metadata to facilitate subsequent retrieval and modeling.

[0046] (3) Special scenario support: For special events such as large-scale international conferences, marathons, and holidays, the system allows full retention of routes and time windows as needed, and can temporarily increase the priority of other strategies.

[0047] The business-driven sampling strategy ensures high-fidelity data for critical business scenarios. Compared to random sampling, the data missing rate in Blizzard scenarios is reduced to below 3%, and the average vehicle speed prediction error on key road sections is reduced by 20% during major events.

[0048] 3. Extract data from the collected data using the whitelist sampling strategy to obtain vehicle-dimensional balanced data. The whitelist sampling strategy includes at least one or more of the following conditions: Manufacturer-level quotas, vehicle-level balancing, and mandatory whitelisting.

[0049] The distribution of vehicle manufacturers, models, and key vehicles in the transportation network is extremely uneven, with a high proportion of some mainstream models and very little data on niche models and special-purpose vehicles (such as buses and hazardous materials transport vehicles). Without restrictions, random sampling would ignore this scarce but important vehicle data, leading to insufficient model generalization ability. Therefore, a whitelist sampling strategy is used to balance the vehicle model dimension, ensuring that the sampling results reflect the objective reality of vehicles operating on the road, thereby guaranteeing the accuracy and representativeness of the sampled data.

[0050] The whitelist sampling strategy includes at least one or more of the following conditions: (1) Manufacturer-level quota: Set a minimum quota min_quota (e.g., 5%) and a maximum quota max_cap (e.g., 60%) for each automaker. When the random sampling result does not reach the minimum quota, the corresponding automaker's vehicles will be automatically sampled; when the maximum quota is exceeded, the sampling of the corresponding automaker's vehicles will be stopped.

[0051] (2) Balanced vehicle type dimensions: Set coverage thresholds for categories such as sedans, SUVs, trucks, buses, and special vehicles to ensure that each type of vehicle is represented in the sample.

[0052] (3) Mandatory whitelist: Test vehicles, demonstration vehicles, important maintenance vehicles, etc. are directly included in the mandatory whitelist V_imp and retained unconditionally to lock key vehicles.

[0053] The whitelist strategy increased the vehicle model coverage of the sample from 75% to 98%, significantly improving the generalization ability and diagnostic accuracy of the subsequent model under different manufacturers and vehicle models.

[0054] 4. Extract key event data from the collected data according to the event-driven sampling strategy. The event-driven sampling strategy includes at least one or more of the following conditions: Accident window, congestion-related data, and violation-related data.

[0055] Traffic accidents, severe congestion, and illegal activities are often of great significance to traffic management and safety analysis. Traditional sampling strategies based on time or space cannot guarantee the complete retention of relevant data at the time of the event.

[0056] Event-driven sampling strategies must include at least one or more of the following conditions: (1) Accident window: Once a traffic accident is detected, all relevant intersections and adjacent road segments are fully retained within the time window Δt (e.g., 20 minutes) before and after the time t0 of the accident, including perception, traffic light and vehicle bus data.

[0057] (2) Congestion-related data: used to reflect the congestion process. Through average speed or congestion index threshold, the entire process of congestion formation, maintenance and resolution is identified, and full-cycle data is continuously recorded.

[0058] (3) Violation-related data: Triggered by violations, all sensor and traffic light data of the associated intersections are automatically retained for violations such as running red lights, driving against traffic, and occupying lanes. The time window is extended to record the causes before the incident and the recovery process afterward.

[0059] The event-driven sampling strategy ensures complete data throughout the entire event process, providing crucial data support for incident review, liability determination, signal optimization, and emergency management. Evaluation results show that the completeness of event data has improved by more than 40%.

[0060] According to one exemplary implementation, differentiated sampling strategies are adopted for different data types. For example, for roadside raw data, peak hours and key intersections are the main screening criteria; for connected vehicle data, the diversity of manufacturers and models is emphasized; and for cloud event data, the entire event window is retained.

[0061] According to an exemplary implementation, in order to further ensure the accuracy and representativeness of the extraction results, time constraints and / or spatial constraints are imposed on the extraction process during data extraction. The time constraints include at least one time period, and the spatial constraints include a multi-level spatial model of intersection set, road segment set, and region set.

[0062] Traffic flow and event distribution exhibit significant spatiotemporal coupling characteristics. For example, morning and evening rush hours are highly concentrated in the central urban area, while during holidays, they tend to shift to suburban tourist attractions. Considering this spatiotemporal coupling characteristic, single temporal or spatial filtering is insufficient to simultaneously guarantee the representativeness and accuracy of the samples in both spatiotemporal dimensions.

[0063] Therefore, the following constraints are imposed on the data extraction process: (1) Time constraints.

[0064] Define peak periods T_peak (e.g., 7:00–9:00, 17:00–19:00), off-peak periods T_off, periods with special weather conditions T_wea, and periods for holidays or special events T_spec. By continuously analyzing the traffic flow curves over the past n weeks, dynamically adjust T_peak and related thresholds to ensure that time constraints adapt to seasonal and holiday changes.

[0065] (2) Spatial constraints.

[0066] A multi-level spatial model is established, consisting of intersection set L, road segment set R, and region set A. The system supports single-point, region, and multi-level combination filtering, and can perform weight calculations on the traffic transmission relationship between regions.

[0067] (3) Multidimensional superposition: Time constraints and spatial constraints can be arbitrarily intersected and merged, such as (T_peak∧ L_key) or (T_spec ∧ A_tourist), to achieve precise spatiotemporal joint control.

[0068] By employing time and / or spatial constraints, biases caused by sampling based solely on time or space are effectively avoided. Joint time and space constraints can increase coverage to over 95% during peak periods and in key areas, while reducing low-value samples by approximately 35%.

[0069] Step 102: Integrate the multiple extraction results to obtain comprehensive extraction data.

[0070] Because multiple different sampling strategies are used to extract data in parallel, data overlap or conflict may occur among the various extraction results, affecting the accuracy of the extraction. For example, an intersection may be a high-traffic intersection, experiencing severe weather, and listed as a key intersection. Without control, this can easily lead to duplicate storage and oversampling.

[0071] In this step, deduplication and conflict resolution are performed to address data overlap and conflict issues, including: 1. For the intersection data between different sampling results, select the sampling result to add to the candidate data according to the priority of the corresponding sampling strategy and / or the data retention priority.

[0072] The system performs set operations on the candidate data sets generated by all strategies to identify the intersection I and the union U.

[0073] Then priority arbitration is performed. According to one exemplary implementation, each sampling strategy can be assigned a priority based on the importance of the sampling strategy (e.g., defining the priority of the sampling strategies as P0>P1>P2).

[0074] When a data point is selected by multiple sampling strategies, the more sampling strategies that select that data point, the higher its data retention priority. According to one exemplary implementation, the highest and most stringent data retention priority is automatically applied (e.g., defining four data retention priorities R0>R1>R2>R3).

[0075] In practical applications, overlapping or conflicting sampling strategies often occur. By prioritizing sampling strategies and / or prioritizing data retention, data redundancy can be reduced by 30%–45%, effectively alleviating storage pressure and maintaining sample integrity, thus providing a consistent and conflict-free data source for subsequent statistical analysis.

[0076] 2. Based on the fingerprint information of each data point, duplicate data in the candidate data are deduplicated to obtain the comprehensive extracted data.

[0077] According to one exemplary implementation, data deduplication is achieved through uniqueness processing. Specifically, stable fingerprint information can be generated for each data entry, and the fingerprint information includes at least one or more of the following: Data source identifier (ID), time slice, key field hash value.

[0078] Deduplication is performed before data is stored to avoid duplicate storage space usage.

[0079] After integrating conflicting and overlapping data, the comprehensive extracted data is obtained. This comprehensive extracted data can be stored for later retrieval. It can be queried directly or used to build a training set to train a large model.

[0080] An exemplary embodiment of this disclosure also provides a data processing method for storing and managing comprehensively extracted data, including hierarchical data retention and expiration management to prevent the unlimited growth of massive amounts of data. The specific process is as follows: Figure 2 As shown, it includes: Step 201: Extract data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy.

[0081] Step 202: Integrate the multiple extraction results to obtain comprehensive extraction data.

[0082] The implementation principles of steps 201 and 202 are the same as those of steps 101 to 102, and will not be repeated here.

[0083] Step 203: The comprehensive extracted data is retained in a hierarchical manner according to a preset data retention priority, and different data retention priorities correspond to different retention periods.

[0084] If massive amounts of data grow indefinitely over a long period, it will cause a sharp increase in storage costs and a decline in query performance. This disclosure addresses this by establishing a scientific hierarchical retention and expiration management mechanism to achieve layered data storage and lifecycle management. This includes: 1. Data is retained in a tiered manner according to its retention priority.

[0085] Different data retention priorities correspond to different retention periods. According to one exemplary implementation, a hierarchical retention system comprising four data retention priorities is defined, including: R0: Data on major events or strategic samples, to be retained long-term or permanently (≥24 months); R1: Important business samples, retained for 12 months; R2: Standard business sample, retained for 6 months; R3: Low-value or high-frequency data, retained for 3 months.

[0086] 2. Expiration strategy.

[0087] After acquiring and saving the comprehensive extracted data, the stored data can be further cleaned up. According to an exemplary implementation, R0 data expiration is only alerted and manually reviewed; R1 data expiration is alerted and cleanup is suggested; R2 and R3 data expiration is automatically cleaned up directly.

[0088] 3. Dynamic monitoring.

[0089] For data storage systems, the data growth rate and storage capacity can be monitored in real time. When the data approaches the threshold, an early warning will be issued and suggestions will be made to adjust the sampling ratio or upgrade the storage level.

[0090] The tiered retention and expiration management mechanism keeps the storage growth rate within a controllable range, reduces long-term storage costs by more than 40%, and ensures the long-term traceability of critical data.

[0091] An exemplary embodiment of this disclosure also provides a data processing method that, after completing parallel data extraction using multiple strategies and integrating it to obtain comprehensive extracted data, can further evaluate the comprehensive extracted data and verify its quality.

[0092] Regardless of the sampling strategy used, the core objective of sampling is to ensure that the sample accurately represents the characteristics of the population. If the sample deviates in terms of coverage, distribution pattern, or dimensionality balance, it will directly affect downstream modeling and decision-making.

[0093] In this embodiment of the disclosure, the process for evaluating the representativeness and balance of the comprehensively extracted data includes: 1. Coverage Assessment: Calculate the coverage of each dimension (e.g., space, event, vehicle) in the comprehensive extracted data and compare it with the overall distribution. Coverage requirements can be set according to factors such as system accuracy requirements and hardware capabilities; for example, a coverage rate of ≥95% may be required.

[0094] 2. Distribution fit evaluation: The difference between the sample distribution and the population distribution of the comprehensively extracted data is evaluated by statistical test methods such as KS or KL distance, and the KS value is required to be ≤0.1.

[0095] 3. Balance Assessment: Use the Theil index or information entropy to measure the balance of the extracted data across different subcategories to prevent over-concentration. For example, measure the balance between different data types.

[0096] 4. Automatic rebalancing mechanism: When coverage, fit, or balance does not meet the standards, the sampling ratio of the relevant dimensions is automatically increased or the strategy parameters are adjusted until the standards are met.

[0097] Through continuous evaluation and dynamic rebalancing, the representativeness of the extracted data in terms of statistical characteristics is ensured, avoiding performance degradation of the model in scarce sample dimensions. In actual deployment, the distribution difference of the extracted data in each major dimension is controlled within 5%.

[0098] An exemplary embodiment of this disclosure also provides a data processing method for preprocessing raw traffic data collected from different traffic data sources, the specific process of which is as follows: Figure 3 As shown, it includes: Traffic data comes from diverse sources, including roadside sensing, traffic light status, roadside video, connected vehicle bus, in-vehicle video, and cloud events. These data sources differ significantly in terms of collection frequency, data volume, information density, and business value. Using a single sampling strategy may result in insufficient key data and redundant low-value data. Therefore, it is essential to scientifically classify data sources and develop specific strategies for different types before sampling.

[0099] According to one exemplary implementation, in this embodiment of the disclosure, data types are divided into three categories: (1) Roadside raw data: fused sensing, traffic lights, roadside video, etc.; (2) Connected vehicle data: vehicle bus, in-vehicle video, etc.; (3) Cloud event data: perception judgment events, vehicle bus judgment events, etc.

[0100] Different sampling strategies are adopted for different data types. For example, for roadside raw data, a sampling strategy is constructed with peak hours and key intersections as the main screening criteria; for connected vehicle data, a sampling strategy that focuses on ensuring the diversity of manufacturers and models is adopted; and for cloud event data, a sampling strategy that retains all data with the event window as the core is adopted.

[0101] Through systematic classification and strategy matching, the system ensures an optimal balance between representativeness, value density, and storage cost for data from different data sources. Experimental results show that compared to a uniform sampling method that does not classify data types, data utilization is improved by approximately 30%, and storage pressure is reduced by approximately 50%.

[0102] Step 301: Perform quality gating on the collected data.

[0103] During data acquisition, factors such as equipment malfunction, network jitter, and environmental interference can lead to missing, delayed, or outlier data. If these data are not filtered before sampling, low-quality samples will enter the analysis system, affecting the reliability of model training and decision-making. Therefore, a quality gating mechanism must be established before sampling to control both the quality of the data itself and the quality of the equipment used to collect it.

[0104] 1. Data quality control.

[0105] Raw traffic data is obtained by collecting data from different data sources that meet preset data quality conditions.

[0106] The preset data quality conditions may include any one or more of the following conditions: Completeness (key field missing rate ≤ threshold), timeliness (reporting delay ≤ threshold), validity (field values ​​are within the legal range), and consistency (difference between multi-source data ≤ threshold).

[0107] 2. Equipment quality control of the data source.

[0108] Based on the equipment quality of the data source, the original traffic data is sampled to obtain the collected data. Roadside equipment inevitably suffers from hardware aging, environmental interference, or improper maintenance during long-term operation, resulting in location drift, frequency fluctuations, or data anomalies. Data quality cannot be guaranteed solely by field integrity; therefore, equipment-level quality indicators need to be introduced to ensure the reliability of the sampled data. The specific process is as follows: Figure 4 As shown, it includes: Step 401: Obtain the quality score for each device.

[0109] The quality score for each device is obtained based on one or more of the following evaluation criteria: Position jump rate, heading jump rate, speed jump rate, type jump rate, frequency jump rate.

[0110] According to one exemplary implementation, five hop rates are calculated from the real-time data of each roadside device: Position jump rate:

[0111] Heading jump rate:

[0112] Velocity jump rate:

[0113] Type jump rate:

[0114] Frequency jump rate:

[0115] Calculate the overall score based on the jump rate. :

[0116] in, ~ The weights are for each transition rate. ~ Adjust dynamically based on historical statistics or business needs.

[0117] Step 402: Based on the quality score, match the corresponding data filtering strategy, sample the original traffic data of the device according to the data filtering strategy, and add the sampling results to the collected data.

[0118] According to one exemplary implementation, quality is graded based on a quality score, for example: Grade A equipment: Q ≥ 0.9, high quality, all data enters the sampling pool.

[0119] Class B equipment: 0.7 ≤ Q<0.9, normal sampling but with a reduced proportion. Class C equipment: Q<0.7, sampled only for special needs or directly rejected.

[0120] Different quality-driven sampling rules are set for different quality levels. For example: Data from Class A equipment should be prioritized during sampling.

[0121] When multiple devices exist at the same intersection, only the data from the device with the highest quality rating is retained, or the sampling ratio is allocated according to quality weighting. When considering peak hours or special weather conditions, data from devices with higher quality ratings are prioritized for sampling.

[0122] According to one exemplary implementation, a dynamic quality monitoring mechanism can be used for equipment quality scoring. For example, the system recalculates the equipment's Q value every 5 to 15 minutes. When equipment quality declines due to a malfunction, the sampling ratio is automatically reduced or data collection is paused. Upon resumption, the rating is automatically increased. Only data that meets the above conditions can enter the subsequent sampling pool.

[0123] By implementing quality gating, the probability of anomalous data entering the sampling pool was significantly reduced. Actual deployment results show that the data anomaly rate was reduced by approximately 40%, and the stability of model training and the reliability of downstream analysis were significantly improved.

[0124] Step 302: Extract data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy.

[0125] Step 303: Integrate the multiple extraction results to obtain comprehensive extraction data.

[0126] Step 304: The comprehensive extracted data is retained in a hierarchical manner according to a preset data retention priority, and different data retention priorities correspond to different retention periods.

[0127] The implementation principles of steps 302 to 304 are the same as those of steps 201 to 203, and will not be repeated here.

[0128] An exemplary embodiment of this disclosure provides a data processing apparatus, the structure of which is as follows: Figure 5 As shown, it includes: The parallel extraction module 501 extracts data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy. The data fusion module 502 is used to integrate multiple extraction results to obtain comprehensive extracted data.

[0129] Furthermore, the parallel extraction module 501 is used to select one or more matching sampling strategies to extract data according to the data type of the collected data, and obtain the extraction results corresponding to each sampling strategy. The data type includes at least any one or more of the following: Roadside raw data, connected vehicle data, and cloud event data.

[0130] The sampling strategy is any one of the following: Data-driven sampling strategy, business-driven sampling strategy, whitelist sampling strategy, event-driven sampling strategy.

[0131] The parallel extraction module 501 is used to extract traffic pattern data from the collected data using the data-driven sampling strategy. The data-driven sampling strategy includes at least one or more of the following conditions: Prioritize intersections with high traffic volume, intersections with high variability, and intersections with high incident rates; The business-driven sampling strategy is used to extract data from the collected data to obtain business requirement data. The business-driven sampling strategy includes at least one or more of the following conditions: Prioritize key intersections, retain access during special weather conditions, and provide support for specific scenarios; The collected data is extracted using the whitelist sampling strategy to obtain vehicle-dimensional balanced data. The whitelist sampling strategy includes at least one or more of the following conditions: Manufacturer-level quotas, vehicle-level balancing, and mandatory whitelisting; Data is extracted from the collected data according to the event-driven sampling strategy to obtain key event data. The event-driven sampling strategy includes at least one or more of the following conditions: Accident window, congestion-related data, and violation-related data.

[0132] The structure of the further data fusion module 502 is as follows: Figure 6 As shown, it includes: The conflict handling submodule 601 is used to select the extraction result to be added to the candidate data based on the priority of the corresponding sampling strategy and / or the data retention priority for the intersection data between different extraction results. The overlap processing submodule 602 is used to perform deduplication processing on the duplicate data in the candidate data according to the fingerprint information of each data to obtain the comprehensive extracted data.

[0133] Furthermore, the device is as follows Figure 7 As shown, it includes: The constraint module 503 is used to impose time constraints and / or spatial constraints on the data extraction process during data extraction. The time constraints include at least one time period, and the spatial constraints include a multi-level spatial model of intersection set, road segment set, and region set.

[0134] The hierarchical storage module 504 is used to retain the comprehensive extracted data in a hierarchical manner according to a preset data retention priority, with different data retention priorities corresponding to different retention periods.

[0135] The quality gating module 505 is used to collect data that meets preset data quality conditions from different data sources to obtain raw traffic data. Based on the equipment quality of the data source, the original traffic data is sampled to obtain the collected data.

[0136] The quality gating module 505 is specifically used to obtain the quality score of each device based on any one or more of the following evaluation criteria: Position jump rate, heading jump rate, speed jump rate, type jump rate, frequency jump rate; Based on the quality score, a corresponding data filtering strategy is matched, and the original traffic data of the device is sampled according to the data filtering strategy. The sampling results are then added to the collected data.

[0137] The aforementioned device can be integrated into computer equipment such as servers, and the corresponding functions can be implemented by the computer equipment. Regarding the device in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0138] An exemplary embodiment of this disclosure also provides a computer apparatus, including: processor; Memory used to store processor-executable instructions; The processor is configured to execute the data processing method provided in the embodiments of this disclosure.

[0139] An exemplary embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of a computer device, enable the computer device to perform the data processing method provided by an embodiment of this disclosure.

[0140] Embodiments of this disclosure provide a data processing method, apparatus, computer device, and storage medium. First, multiple sampling strategies are used to extract data from collected data, yielding extraction results for each strategy. Then, these extraction results are integrated to obtain comprehensive extracted data. Different sampling strategies independently filter data that meets the criteria, ensuring full extraction of the target data. Integrating multiple extraction results into comprehensive extracted data further improves data quality, providing a representative and highly practical data foundation for storage and subsequent data analysis and application. This solves the problem of a lack of traffic data management mechanisms, achieves highly efficient and high-business-value data processing, and effectively alleviates storage and maintenance pressures.

[0141] Adopting a multi-dimensional approach that combines data-driven and business-driven methods: integrating comprehensive data quality with multi-dimensional business-driven requirements, including traffic flow, event frequency, key intersections, and special weather conditions.

[0142] The comprehensive extracted data obtained through the technical solutions provided by the embodiments of this disclosure has the following advantages: 1. Significantly improved representativeness, covering key spatiotemporal scenarios and extreme weather, with peak traffic data completeness reaching over 95%.

[0143] 2. Enhanced business value: Retain data on key intersections and major events to meet the needs of business presentation, decision-making, and emergency review.

[0144] 3. Data diversity guarantee: The whitelist ensures balanced coverage of different manufacturers and models.

[0145] 4. Redundancy control: Intersection deduplication reduces duplicate samples by 30% to 45%, saving storage and computing resources.

[0146] 5. Improved quality stability: The equipment quality-driven strategy reduces the proportion of low-quality abnormal data by 30% to 50%, improving the reliability of subsequent analysis.

[0147] 6. Governability and Scalability: Tiered retention and expiration policies ensure long-term control over storage resources, and policy parameters can be dynamically adjusted according to city, season, and business needs.

[0148] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0149] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0150] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0151] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0152] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: Data is extracted from the collected data using multiple sampling strategies, and the extraction results corresponding to each sampling strategy are obtained. The extraction results from multiple sources are integrated to obtain comprehensive extracted data.

2. The data processing method according to claim 1, characterized in that, Before the step of extracting data from the collected dataset using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy, the method further includes: Based on the data type of the collected data, one or more matching sampling strategies are selected for data extraction, wherein the data type includes at least one or more of the following: Roadside raw data, connected vehicle data, and cloud event data.

3. The data processing method according to claim 1, characterized in that, The sampling strategy is any one of the following: Data-driven sampling strategy, business-driven sampling strategy, whitelist sampling strategy, event-driven sampling strategy. The step of extracting data from the collected dataset using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy includes: Traffic pattern data is obtained by extracting data from the collected data using the data-driven sampling strategy, wherein the data-driven sampling strategy includes at least one or more of the following conditions: Prioritize intersections with high traffic volume, intersections with high variability, and intersections with high incident rates; The business-driven sampling strategy is used to extract data from the collected data to obtain business requirement data. The business-driven sampling strategy includes at least one or more of the following conditions: Prioritize key intersections, retain access during special weather conditions, and provide support for specific scenarios; The collected data is extracted using the whitelist sampling strategy to obtain vehicle-dimensional balanced data. The whitelist sampling strategy includes at least one or more of the following conditions: Manufacturer-level quotas, vehicle-level balancing, and mandatory whitelisting; Data is extracted from the collected data according to the event-driven sampling strategy to obtain key event data. The event-driven sampling strategy includes at least one or more of the following conditions: Accident window, congestion-related data, and violation-related data.

4. The data processing method according to claim 1, characterized in that, The step of integrating multiple extraction results to obtain comprehensive extracted data includes: For the intersection data between different extraction results, the extraction result is selected to be added to the candidate data according to the priority of the corresponding sampling strategy and / or the data retention priority. The duplicate data in the candidate data are deduplicated based on the fingerprint information of each data point to obtain the comprehensive extracted data.

5. The data processing method according to claim 1, characterized in that, The method further includes: The extracted data is retained in a tiered manner according to a preset data retention priority, with different data retention priorities corresponding to different retention periods.

6. The data processing method according to claim 1, characterized in that, Before the step of extracting data from the collected data using multiple sampling strategies to obtain the extraction results corresponding to each sampling strategy, the method further includes: Raw traffic data is obtained by collecting data from different data sources that meet preset data quality conditions. Based on the equipment quality of the data source, the original traffic data is sampled to obtain the collected data.

7. The data processing method according to claim 6, characterized in that, The step of sampling the raw traffic data based on the equipment quality of the data source to obtain the collected data includes: The quality score for each device is obtained based on one or more of the following evaluation criteria: Position jump rate, heading jump rate, speed jump rate, type jump rate, frequency jump rate; Based on the quality score, a corresponding data filtering strategy is matched, and the original traffic data of the device is sampled according to the data filtering strategy. The sampling results are then added to the collected data.

8. A data processing apparatus, characterized in that, include: The parallel extraction module uses multiple sampling strategies to extract data from the collected data, and obtains the extraction results corresponding to each sampling strategy. The data fusion module is used to integrate multiple extraction results to obtain comprehensive extracted data.

9. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the data processing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to perform the data processing method as described in any one of claims 1 to 7.

Images