Adaptive and dynamically adjusted multi-channel logistics position data acquisition method

Abstract

The invention belongs to the technical field of big data acquisition and processing, and particularly relates to a self-adaptive and dynamically-adjusted multi-channel logistics position data acquisition method, which comprises the following steps of S1, establishing a position data storage warehouse; s2, converting the data into a unified data format and protocol, and realizing seamless connection between the data of different data sources; s3, performing quality evaluation on the acquired data source, wherein an evaluation result is the quality of the data source; and S4, calculating and adjusting the acquisition frequencies of different data sources based on a pre-formulated adaptive position data acquisition strategy. According to the method, a feedback mechanism of data acquisition and data processing is established, an acquisition strategy is adjusted according to a data processing result, and dynamic optimization of the system is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data collection and processing, and in particular relates to an adaptive and dynamically adjusted multi-channel logistics location data collection method. Background Art

[0002] In the existing technology, the collection of big data involves the integration and coordination of multi-source data, but the collection of location data from different channels often relies on fixed time intervals or manual triggering. The frequency and method of collecting data from each channel are different. In scenarios that require rapid response and decision support, this delay may cause the information to become outdated or lose its value. At the same time, the existing data collection strategies are relatively fixed and lack flexibility, making it difficult to adapt to changes in data sources, system status or business needs. In addition, the data formats, structures and quality standards of different channels differ, making the data integration process complicated and time-consuming. Summary of the invention

[0003] The purpose of the present invention is to provide an adaptive and dynamically adjustable multi-channel logistics location data collection method to solve the technical problems that the data collection strategy of the prior art is relatively fixed, lacks flexibility, is difficult to adapt to changes in data sources, system status or business needs, and there are differences in the collection frequency and method of data from each channel. In addition, the data format, structure and quality standards of different channels are different, which makes the data integration process complicated and time-consuming.

[0004] The self-adaptive and dynamically adjusted multi-channel logistics location data collection method comprises the following steps.

[0005] S1. Establish a storage warehouse for location data.

[0006] S2. Convert data into a unified data format and protocol to achieve seamless connection between data from different data sources.

[0007] S3. Perform a quality assessment on the collected data source, and the assessment result is the quality of the data source.

[0008] S4. Based on the pre-established adaptive location data collection strategy, the collection frequencies of different data sources are calculated and adjusted.

[0009] In step S4, a priority is assigned to each data source according to the accuracy and stability of the data source. The better the accuracy and stability of the data source, the higher the priority. When real-time location data is collected, the collection frequency of the data source is dynamically adjusted. In the case of reduced quality of the data source, this step dynamically adjusts the collection frequency of different data sources according to the historical quality data and priority of the data source. The calculation formula for the initial collection frequency of the data source dynamically adjusted is as follows:

[0010]

[0011] Among them, ∫new is the new acquisition frequency, ∫old is the original acquisition frequency, Q current is the quality score of the current data source, Q avg is the historical average quality score of the data source, Q max It is the highest historical quality score of the data source. PriorityFactor is an adjustment factor based on the priority of the data source. The higher the priority, the larger the factor, which reflects the relative importance of the data source in the system.

[0012] Preferably, step S3 includes: adopting a scoring mechanism to analyze each data in terms of accuracy, completeness, timeliness, consistency, and relevance to achieve a score evaluation of data quality in each dimension, and finally derive a data quality score by combining the scores of each dimension to achieve a quality evaluation of the collected data source; multiplying the score of each dimension by the corresponding weight, and then summing them to obtain a final data quality score. The calculation formula of the data quality score is:

[0013] Score DQ =w acc Accuracy+w com Completeness+w tim Timeliness

[0014] +w con Consistency+w rel Relevance

[0015] Among them, Score DQ represents the data quality score. Accuracy, Completeness, Timeliness, Consistency and Relevance represent the scores of the five dimensions of accuracy, completeness, timeliness, consistency and relevance respectively. acc 、w com 、w tim 、w con and w rel It represents the weights of the scores of the five dimensions: accuracy, completeness, timeliness, consistency and relevance.

[0016] Preferably, in step S3, the score of each dimension is equal to the average score of each sub-item under the dimension; the corresponding specific evaluation method includes.

[0017] 1) Accuracy evaluation: Accuracy includes position accuracy and speed accuracy.

[0018] Location Accuracy: This score is determined by comparing location data from various sources, calculating the location difference and comparing it to a threshold.

[0019] Speed ​​Accuracy: Compares speed data from various data sources, calculates speed differences, and compares them with corresponding thresholds to determine the score based on the comparison results.

[0020] 2) Completeness assessment: Accuracy includes field completeness and data missingness.

[0021] Field completeness: Check whether each record contains all required fields. If the record contains all required fields, full points will be awarded; for each missing field, points will be deducted.

[0022] Missing data: Count the proportion of missing values ​​in each field. If the proportion of missing values ​​exceeds the set threshold, points will be deducted according to the excess amount, otherwise full marks will be given.

[0023] 3) Timeliness assessment: Timeliness includes timestamp verification and data freshness.

[0024] Timestamp verification: Determine whether the timestamp data is valid, calculate the difference between the timestamp data and the actual time when the data was acquired, and compare the difference with the set threshold to score.

[0025] Data freshness: Calculate the time difference from when the data is generated to when it is processed, and compare the time difference with the set threshold to score it.

[0026] 4) Consistency assessment: Consistency includes internal consistency, external consistency and data format consistency;

[0027] Internal consistency: Check whether there are logical conflicts between different fields in the same record. If there is no logical conflict, full marks will be given, otherwise marks will be deducted.

[0028] External consistency: When there is more than one data source providing data at the same location, compare the data from these data sources. If the data from each data source is consistent, full marks will be given, otherwise points will be deducted.

[0029] Data format consistency: Score based on whether the data format and encoding rules of each record are consistent. If they are consistent, full marks will be given, otherwise points will be deducted.

[0030] 5) Correlation evaluation: Correlation includes the correlation between location and geographic area, the correlation between speed and location, and the correlation between data source and device.

[0031] Correlation between location and geographic area: Check whether the location data and geographic area data match each other. If they match, full points will be awarded; otherwise, points will be deducted.

[0032] Correlation between speed and position: Calculate the rate of change between the position data and the timestamp data, and analyze whether the speed data matches the rate of change. If so, full marks will be awarded, otherwise marks will be deducted.

[0033] Correlation between data source and device: Verify whether the IDs of the data source and device correspond to the data source and device actually used. If so, full marks will be awarded, otherwise marks will be deducted.

[0034] Preferably, the step S4 specifically includes: calculating the initial acquisition frequency according to the quality of the data source; specifically includes the following steps.

[0035] a. Define monitoring indicators, including CPU usage, memory usage, disk I / O, network bandwidth, and data source quality.

[0036] b. Set thresholds. According to system load capacity and business requirements, set warning thresholds and critical thresholds for each monitoring indicator.

[0037] c. Set the data source priority. This step assigns a priority to each data source based on its accuracy and stability.

[0038] d. Alarm adjustment: When a monitoring indicator exceeds the warning threshold, the adjustment of the data collection strategy is triggered. First, the collection frequency of low-priority data sources is reduced to release system resources, so that the location data output by higher-priority data sources can be better processed. If the monitoring indicator continues to rise and exceeds the critical threshold, the collection of some or all low-priority data sources is suspended.

[0039] e. When collecting location data in real time, dynamically adjust the collection frequency of the data source.

[0040] Preferably, step S4 also includes: monitoring the network status and scoring, adjusting the initial acquisition frequency based on the network status score, and the network status scoring method includes: comprehensively considering multiple performance indicators such as bandwidth, delay, packet loss rate, etc., and using an evaluation method to obtain a comprehensive network quality score, and the calculation formula is: network quality comprehensive score = Σ(indicator weight × indicator score), where the symbols have the following meanings: Σ represents summation, the indicator score represents the performance of the indicator, and the indicator weight represents the importance of the performance indicator in network quality evaluation; judging the network status corresponding to the network quality comprehensive score according to the threshold range of the network quality comprehensive score, and when the trigger condition is met, adjusting the acquisition frequency according to the acquisition frequency adjustment strategy, and the acquisition frequency adjustment strategy based on the network status includes: when the network status is good, keeping the current acquisition frequency unchanged; when the network status is medium: appropriately reducing the acquisition frequency, reducing it by 10%-20%; when the network status is poor: significantly reducing the acquisition frequency, reducing it by more than 30%.

[0041] Preferably, the step S4 further includes: setting a business adjustment method according to the business requirements of the logistics system, thereby adjusting the collection frequency obtained in the previous step; the correspondence between the business requirements of the logistics system and the business adjustment method is as follows.

[0042] a. For urban distribution business, the collection frequency of location data needs to be increased accordingly to meet the minimum collection frequency requirements for this type of business. If it is lower than the requirement, the collection frequency should be increased by a certain proportion to meet the requirement.

[0043] b. For trunk transport business, choose a moderate sampling frequency, which is lower than that for urban distribution business but higher than that for bulk business;

[0044] c. Bulk business: adopt a lower sampling frequency while ensuring that the location data of key locations can be accurately collected and recorded. The sampling frequency range is relatively lower than that of trunk transport business.

[0045] Preferably, the step S1 comprises.

[0046] S1.1. The data table structure needs to be designed.

[0047] S1.2. Design indexing strategy.

[0048] S1.3. Partition the data according to its characteristics and distribute the data to different physical storage devices. Partition the data specifically based on the two dimensions of time and geographic location.

[0049] Preferably, in step S2, the data quality of the coordinate system conversion is monitored, and the accuracy and reliability of the conversion algorithm are evaluated by comparing the coordinate values ​​before and after the conversion, and the conversion algorithm is used for verification. The types of errors involved in the verification include coordinate difference, root mean square error, maximum error and minimum error; among the above errors, a single coordinate difference value can reflect the degree of deviation of a single coordinate point after conversion. After the coordinate difference value of each point is obtained, this step can further calculate the root mean square error and find the maximum error and minimum error. The root mean square error comprehensively considers the errors of all points and gives an overall error estimate. The maximum error provides information on the maximum degree of deviation that the conversion algorithm may produce, and determines the performance boundary of the algorithm; and the minimum error reflects the performance of the algorithm under the best conditions.

[0050] The present invention has the following advantages: 1. By evaluating the quality of the data source and adopting an adaptive location data collection strategy, the method constructs a feedback mechanism, establishes a feedback mechanism for data collection and data processing, adjusts the collection strategy according to the data processing results, and realizes dynamic optimization of the system. This enables the method to adjust the corresponding collection frequency according to the data quality provided by the data source, adapts to the dynamic changes in the quality of the data source, supports more diversified data sources and more types of coordinate system conversions, and improves the accuracy and flexibility of data processing.

[0051] 2. This method establishes a comprehensive data quality assessment system, including multiple dimensions such as data integrity, accuracy, consistency, and timeliness, to provide a reliable basis for data management and application and ensure the quality and value of data.

[0052] 3. The collection strategy adjustment algorithm designed by this method can automatically adapt to business changes and changes in data quality, realize the intelligence and automation of data collection, improve the efficiency and accuracy of data collection, and take into account the cost, efficiency, stability and accuracy of location data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a schematic diagram of the basic flow of a method for collecting multi-channel logistics location data with adaptive and dynamic adjustment.

[0054] Figure 2 The present invention is a flowchart of the operation of an adaptive and dynamically adjusted multi-channel logistics location data collection method in the system. DETAILED DESCRIPTION

[0055] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0056] like Figure 1 , Figure 2 As shown, the present invention provides an adaptive and dynamically adjusted multi-channel logistics location data collection method, comprising the following steps.

[0057] S1. Establish a storage warehouse for location data.

[0058] This step requires selecting a suitable data storage method to store the collected location data in a storage warehouse. The data storage method can be a relational database (such as MySQL, PostgreSQL) or a non-relational database (such as MongoDB), and the location data can be coordinate data.

[0059] The specific steps to establish a storage warehouse are as follows.

[0060] S1.1. It is necessary to design the data table structure, including field names, data types, constraints, etc. For example, design a table named "bs_vhc_his", which contains fields such as id, source_id (data source identifier), location (latitude and longitude), timestamp (timestamp), speed (speed), gps_type (coordinate types include GPS, Beidou, APP, base station, WIFI), gps_x (coordinate longitude), gps_y (coordinate latitude), device_id (device number), province (province), city (city), area (area), etc.

[0061] S1.2. Design index strategies and create indexes for fields that are frequently used for queries, such as location, gps_type, gps_x, gps_y, etc., to improve query efficiency. Partition data according to its characteristics (such as time and geographic location) and distribute data to different physical storage devices to improve query performance and management efficiency.

[0062] S1.3. Partition the data according to its characteristics and distribute the data to different physical storage devices to improve query performance and management efficiency. Specifically, the data can be partitioned according to the two dimensions of time and geographic location. This step is explained using MySQL as an example, because MySQL provides rich partitioning support. The specific partitioning method for the "bs_vhc_his" table is as follows.

[0063] 1) Time partitioning: Since the table contains the timestamp field, this step can be partitioned by time range. For example, if you want to partition by day, use RANGE partitioning; set the partition key: timestamp; the partitioning scheme is as follows.

[0064] CREATE TABLE bs_vhc_his(

[0065] id INT NOT NULL AUTO_INCREMENT,

[0066] source_id INT NOT NULL,

[0067] location VARCHAR(255),

[0068] timestamp DATETIME NOT NULL,

[0069] speed FLOAT,

[0070] gps_type ENUM('GPS','Beidou','APP','Base Station','WIFI'),

[0071] gps_x DECIMAL(9,6),

[0072] gps_y DECIMAL(8,6),

[0073] device_id VARCHAR(255),

[0074] province VARCHAR(50),

[0075] city ​​VARCHAR(50),

[0076] area VARCHAR(50),

[0077] PRIMARY KEY(id,timestamp) )

[0079] PARTITION BY RANGE(UNIX_TIMESTAMP(timestamp))(

[0080] PARTITION p0 VALUES LESS THAN(UNIX_TIMESTAMP('2023-01-01')),PARTITIONp1 VALUES LESS THAN(UNIX_TIMESTAMP('2023-01-02')),

[0081] --...(continue to add partitions for other dates)

[0082] PARTITION pN VALUES LESS THAN MAXVALUE );

[0084] 2) Geographical location partitioning: partition by geographic location, using latitude and longitude ranges or administrative region codes. It may not be easy to determine the appropriate partition boundaries due to the latitude and longitude ranges. For example, the province field stores the administrative code of the province. Taking the administrative region code (such as China's administrative code) as an example for partitioning, use LIST partitioning; set the partition key: province; the partitioning scheme is as follows.

[0085] CREATE TABLE bs_vhc_his(

[0086] --Field definitions are the same as before )

[0088] PARTITION BY LIST(province)(

[0089] PARTITION p_beijing VALUES IN('110000'),--Beijing

[0090] PARTITION p_shanghai VALUES IN('310000'),--Shanghai

[0091] --...(Continue to add subdivisions for other provinces)

[0092] PARTITION p_others VALUES IN (DEFAULT) -- other provinces not explicitly specified );

[0094] S2. Convert data into a unified data format and protocol to achieve seamless connection between data from different data sources.

[0095] There are multiple data sources from different channels, and the data formats and protocols provided by multiple data sources need to be unified. For example: when the data sources include satellite positioning systems, APP positioning systems, base station positioning systems, and WIFI positioning systems. For GPS and Beidou data, since both usually use the WGS84 coordinate system, they are converted to the Baidu coordinate system (BaiDu09) as needed; for APP data, if the WGS84 or GCJ02 coordinate system is used, the data is also converted similarly; for base station and WIFI data, if the longitude and latitude information is provided, the data is also converted to the coordinate system; the conversion of various data is achieved by using coordinate conversion algorithms or calling existing coordinate conversion libraries. In order to monitor the data quality of coordinate system conversion, the accuracy and reliability of the conversion algorithm are evaluated by comparing the coordinate values ​​before and after the conversion, and the conversion algorithm is used for verification. The error types and calculation formulas involved in the verification are as follows.

[0096] 1) Coordinate difference (or error): Calculate the coordinate difference before and after the conversion, using the Euclidean distance or geodetic distance formula. The specific calculation formula is as follows:

[0097]

[0098] Among them, (x 1 ,y 1 ) is the original coordinate longitude and latitude, (x 2 ,y 2 ) is the converted coordinate longitude and latitude, and Error is the coordinate difference between the two coordinate data.

[0099] 2) Root mean square error: For a set of coordinate points, this method uses the root mean square error (RMSE) to evaluate the accuracy of the conversion algorithm. The specific calculation formula is as follows:

[0100]

[0101] Where N is the number of coordinate points, and is the coordinate value of the i-th original coordinate (corresponding x value and y value), and is the converted coordinate value (corresponding x value and y value) corresponding to the i-th original coordinate.

[0102] 3) Maximum error and minimum error: used to determine the performance boundary of the conversion algorithm. The specific calculation formula is as follows:

[0103]

[0104] Among them, Max Error is the maximum error, Min Error is the minimum error, and is the coordinate value of the i-th original coordinate (corresponding x value and y value), and is the converted coordinate value (corresponding x value and y value) corresponding to the i-th original coordinate.

[0105] Among the above errors, the Error value of a single coordinate difference can reflect the degree of deviation after the conversion of a single coordinate point. If the Error value is small, it means that the conversion result of the point is close to the expectation; if the Error value is large, it may indicate that the conversion algorithm has a large deviation at this point. After obtaining the Error of each point, this method can further calculate the root mean square error RMSE. RMSE is a statistic that comprehensively considers the errors of all points and gives an overall error estimate; the smaller the RMSE, the better the performance of the conversion algorithm on the entire data set, and the smaller the average error of the conversion result. By comparing the Error values ​​of all points, this step can find the maximum error and the minimum error; the maximum error provides information on the maximum degree of deviation that the conversion algorithm may produce and determines the performance boundary of the algorithm; while the minimum error reflects the performance of the algorithm under the best conditions. Based on indicators such as Error, RMSE, maximum error and minimum error, this method can comprehensively evaluate the conversion results, and these indicators can not only help understand the accuracy of the conversion algorithm, but also guide how to improve the algorithm or adjust parameters to obtain better conversion effects. In order to intuitively display the performance and error of the algorithm, this method plots the Error or RMSE value into a scatter plot for better decision-making.

[0106] S3. Perform a quality assessment on the collected data source, and the assessment result is the quality of the data source.

[0107] This step adopts a scoring mechanism. By analyzing each data in terms of accuracy, completeness, timeliness, consistency, and relevance, it implements a score assessment of data quality in each dimension, and finally derives a data quality score by combining the scores of each dimension to achieve a quality assessment of the collected data source.

[0108] The score of each dimension is equal to the average of the scores of each sub-item under the dimension. Both weighted average and arithmetic average are acceptable. When deducting points, the deducted points are calculated based on the size of the difference, such as: the difference from the threshold, consistency difference or correlation difference. The following is an explanation of the specific evaluation method of each dimension using the previous example.

[0109] 1) Accuracy evaluation: Accuracy includes position accuracy and speed accuracy. The corresponding evaluation methods are as follows.

[0110] Location accuracy: Compare the location data of each data source, calculate the location difference (such as distance) and set a threshold, compare the location difference with the corresponding threshold and determine the score based on the comparison result. For example: compare the location field (which may contain the original longitude and latitude string) or the gps_x and gps_y fields (clear longitude and latitude values) with the known accurate location (such as obtained through other high-precision devices or map services). Calculate the location difference (such as distance) and set a threshold (such as 10 meters) to judge the accuracy. If the difference is less than the threshold, give the record a higher score (such as 9 points out of 10 points), otherwise deduct points based on the size of the difference.

[0111] Speed ​​accuracy: Compare the speed data of each data source, calculate the speed difference and set a threshold, compare the speed difference with the corresponding threshold and determine the score based on the comparison result. The calculation method of this accuracy is similar to that of location accuracy, except that the data source must provide actual speed data (such as the data from the vehicle speedometer), and the score is scored by comparing the speed field with the threshold (such as the reference speed).

[0112] 2) Completeness assessment: Accuracy includes field completeness and data missingness. The corresponding assessment methods are as follows.

[0113] Field completeness: Check whether each record contains all the required fields, such as id, source_id, timestamp, gps_type, gps_x, gps_y, device_id, province, city, area. If the record contains all the required fields, full marks will be given; for each missing field, a certain number of points will be deducted, thus obtaining a score for field completeness.

[0114] Missing data: Count the proportion of missing values ​​in each field, especially key fields such as gps_x, gps_y and timestamp, set a threshold, and if the proportion of missing values ​​exceeds the set threshold (such as 10%), deduct points according to the excess amount, otherwise give full marks.

[0115] 3) Timeliness evaluation: Timeliness includes timestamp verification and data freshness. The corresponding evaluation methods are as follows.

[0116] Timestamp verification: Make sure the timestamp data is valid, calculate the difference between the timestamp data and the actual data acquisition time, and compare the difference with the set threshold to score. For example: Make sure the timestamp field of each record is valid, calculate the difference between the field data and the actual data acquisition time, if the timestamp is valid and within the acceptable range (i.e. the threshold range), give full marks; otherwise, deduct points according to the size of the difference.

[0117] Data freshness: Calculate the time difference from data generation to data processing, and compare the time difference with the set threshold to score. If the time difference from data generation to data processing is within a reasonable range (i.e. within the set threshold, such as within 24 hours), full marks will be given; otherwise, points will be deducted based on the time difference. This indicator is used to ensure that data is processed within a reasonable time.

[0118] 4) Consistency assessment: Consistency includes internal consistency, external consistency and data format consistency. The corresponding assessment methods are as follows.

[0119] Internal consistency: Check whether there are logical conflicts between different fields in the same record, and score based on the check results. If there is no logical conflict within the record, full marks will be given; otherwise, marks will be deducted. For example, if gps_type is "GPS" but gps_x and gps_y are empty, it means there is a logical conflict and marks should be deducted.

[0120] External consistency: When there is more than one data source providing data at the same location, external consistency is scored by comparing the data from these data sources. If the data from each data source is consistent, full marks are awarded; otherwise, marks are deducted.

[0121] Data format consistency: Scoring is based on whether the data format and encoding rules of each record are consistent. If all records follow the same data format and encoding rules, full marks will be awarded; otherwise, marks will be deducted.

[0122] 5) Correlation evaluation: Correlation includes the correlation between location and geographic area, the correlation between speed and location, and the correlation between data source and device. The corresponding evaluation methods are as follows.

[0123] Correlation between location and geographic area: Check whether the location data and geographic area data match each other for scoring. For example, check whether the province, city, and area fields (indicating geographic areas) match the latitude and longitude positions (indicating locations) represented by the gps_x and gps_y fields. If they match, full marks are awarded; otherwise, marks are deducted.

[0124] Correlation between speed and location: Calculate the rate of change between the location data and the timestamp data, analyze whether the speed data matches the rate of change, and score based on the degree of match. For example, first calculate the rate of change between the data in the location (or gps_x, gps_y) field and the data in the timestamp field, then analyze whether the data in the speed field matches the rate of change. If they match, give full marks; otherwise, deduct points.

[0125] Correlation between data source and device: Verify whether the IDs of the data source and device correspond to the data source and device actually used, and score them. For example, verify whether the data in the source_id and device_id fields correspond to the data source and device actually used. If the data in the source_id and device_id fields correspond to the data source and device actually used, full marks will be given; otherwise, marks will be deducted.

[0126] Finally, the score of each dimension is multiplied by the corresponding weight and then summed to obtain the final data quality score. To this end, this method needs to assign a weight to each dimension of data quality and obtain the calculation formula of the data quality score:

[0127] Score DQ =w acc Accuracy+w com Completeness+w tim Timeliness

[0128] +w con Consistency+w rel Relevance

[0129] Among them, Score DQ represents the data quality score. Accuracy, Completeness, Timeliness, Consistency and Relevance represent the scores of the five dimensions of accuracy, completeness, timeliness, consistency and relevance respectively. acc 、w com 、w tim 、w con and w relIt represents the weights of the scores of the five dimensions: accuracy, completeness, timeliness, consistency and relevance.

[0130] Example: The weight distribution is as follows: the weight of accuracy is 0.3, the weight of completeness is 0.2, the weight of timeliness is 0.2, the weight of consistency is 0.15, and the weight of relevance is 0.15. The score for each dimension is set as follows: if the accuracy of the position difference and the speed difference is less than the threshold, the value of accuracy is 9; if all fields are complete and not missing, the value of completeness is 9; if the timestamp is valid and the data is fresh, the value of timeliness is 9; if the internal, external and data formats are consistent, the value of consistency is 9; if all relevances are matched, the value of relevance is 10. Calculated as follows: Data quality score = 9×0.3+9×0.2+9×0.2+9×0.15+10×0.15=9.15. The data quality score indicates the comprehensive quality of the corresponding data source in the five dimensions of the evaluation. In practical applications, the threshold and scoring criteria can also be adjusted according to actual needs to obtain more accurate or more business-oriented evaluation results.

[0131] S4. Based on the pre-established adaptive location data collection strategy, the collection frequencies of different data sources are calculated and adjusted.

[0132] The adaptive location data collection strategy includes an adaptive adjustment strategy. This step forms a complete adaptive adjustment strategy through real-time monitoring of system status, data source quality, network status, business requirements of the logistics system, etc., which is used to adjust the location data collection strategy. The purpose is to dynamically adjust the data collection strategy according to the real-time monitoring data to optimize the efficiency, quality and cost of data collection. Therefore, this method can regularly evaluate various indicators and adjust the strategy according to preset rules. This step specifically includes the following steps.

[0133] S4.1. Calculate the initial collection frequency based on the quality of the data source.

[0134] a. Define monitoring indicators, including CPU usage, memory usage, disk I / O, network bandwidth, and data source quality, where the data source quality is the evaluation result of step S3.

[0135] b. Set thresholds. According to system load capacity and business requirements, set warning thresholds and critical thresholds for each monitoring indicator. For example, set the warning threshold of CPU usage to 80% and the critical threshold to 90%.

[0136] c. Set the data source priority. This step assigns a priority to each data source based on its accuracy and stability. Data sources with higher priorities have higher acquisition priorities when resources are tight. For example, for GPS / Beidou positioning system (i.e. satellite positioning system), APP positioning system, base station positioning system and WIFI positioning system, the priorities decrease in turn. The reasons are as follows.

[0137] In an open environment with good GPS / Beidou signals, GPS / Beidou positioning systems should be used for positioning due to their high accuracy and stability. As for the APP positioning system, it technically relies on the built-in positioning module and network connection of the mobile phone, and its stability and accuracy are slightly inferior to the satellite positioning system, so its priority is lower. Indoors or in environments with poor GPS / Beidou signals, the satellite positioning system is used in combination with the APP positioning system. The base station positioning system and WIFI positioning system have lower accuracy and stability, so they are used as alternative options for auxiliary positioning with lower priority. Among them, the base station positioning system performs better indoors or in areas with densely populated high-rise buildings in cities, but has lower accuracy in open areas. The accuracy and stability of the WIFI positioning system are usually low, especially when there are not enough WIFI hotspots, so its priority is lower than the base station data source. However, the base station positioning system and WIFI positioning system still have certain application value in specific scenarios.

[0138] d. Alarm adjustment: When a monitoring indicator exceeds the warning threshold, the adjustment of the data collection strategy is triggered. First, the collection frequency of low-priority data sources is reduced to free up system resources so that the location data output by higher-priority data sources can be better processed. If the monitoring indicator continues to rise and exceeds the critical threshold, the collection of some or all low-priority data sources is suspended.

[0139] e. When collecting location data in real time, dynamically adjust the collection frequency of the data source.

[0140] In the case of deteriorating quality of data sources, this method dynamically adjusts the acquisition frequency of different data sources based on the historical quality data and priority of the data sources.

[0141] The calculation formula for dynamically adjusting the collection frequency of the data source is as follows:

[0142]

[0143] Among them, ∫new is the new acquisition frequency, ∫old is the original acquisition frequency, Q current is the quality score of the current data source, Q avg is the historical average quality score of the data source, Q maxIt is the highest historical quality score of the data source. PriorityFactor is an adjustment factor based on the priority of the data source. The higher the priority, the larger the factor, which reflects the relative importance of the data source in the system. This embodiment assigns a fixed value to the adjustment factor PriorityFactor corresponding to each data source priority based on historical experience. The setting of the above adjustment factors is based on the rule that the priorities of the corresponding data sources decrease in sequence. When setting them in practice, the impact of the acquisition frequency on the system performance and the accuracy and precision of the generated positioning results should be comprehensively considered. For example, the PriorityFactor values ​​of different data sources are as follows: Beidou data source>GPS data source>APP data source>base station data source>WIFI data source.

[0144] S4.2. Monitor and score the network status, and adjust the initial collection frequency based on the network status score.

[0145] In this method, the system monitors the network status in real time, including performance indicators such as bandwidth, delay, and packet loss rate. When the network status deteriorates, according to the preset rules and thresholds, the system should be able to automatically reduce the transmission frequency of low-priority data to ensure that high-priority data can be transmitted first. At the same time, when the network bandwidth is limited, data compression technology should be considered to reduce the network transmission volume.

[0146] The scoring method for network status includes: comprehensively considering multiple performance indicators such as bandwidth, delay, packet loss rate, etc., and using the evaluation method to obtain a comprehensive network quality score. The calculation formula of the evaluation method is: comprehensive network quality score = Σ (indicator weight × indicator score), where the symbols have the following meanings.

[0147] Σ: Indicates summation, that is, all calculation results in the brackets are added together.

[0148] Metric score (Pi): For each evaluation metric, a score is required to represent the performance of the metric, namely the metric score (Pi). This score is usually based on the actual measured value and is obtained through a certain conversion or normalization process. For the latency metric, a lower latency value may be converted to a higher score.

[0149] Indicator weight (Wi): Each performance indicator (such as bandwidth, delay, packet loss rate, etc.) has a corresponding weight value (Wi), which represents the importance of the performance indicator in network quality assessment. The weight value is adjusted according to actual needs and is usually determined through expert evaluation or historical data analysis.

[0150] For example: define the indicator weights W1, W2 and W3 corresponding to bandwidth, latency and packet loss rate as 0.4, 0.3 and 0.3 respectively, and the sum of the weights is 1, representing 100% importance. The measured value of bandwidth is 100Mbps, and the conversion score is 90 (out of 100). The measured value of latency is 50ms, and the conversion score is 80 (out of 100, because the lower the latency, the better). The measured value of packet loss rate is 0.1%, and the conversion score is 95 (out of 100, because the lower the packet loss rate, the better). Then, the comprehensive network quality score = 0.4×90+0.3×80+0.3×95=88.5.

[0151] This method sets a threshold range of the comprehensive network quality score based on historical experience to determine the corresponding network status. The corresponding relationship between the network status and the threshold range is as follows.

[0152] Good: A network status score of 80 or above indicates that the network status is good and the data transmission efficiency is high.

[0153] Moderate: The network condition score is between 60 and 80 points, indicating that the network condition is average, with possible slight delay or packet loss.

[0154] Poor: If the network condition score is below 60 points, it means that the network condition is poor and data transmission may be greatly affected.

[0155] Preset trigger conditions: When the network status score is lower than a certain threshold for N consecutive times (such as 3 times), the collection frequency adjustment is triggered. Use network monitoring tools to monitor the network status score in real time, and adjust the collection frequency according to the collection frequency adjustment strategy based on the real-time network status score. The collection frequency adjustment strategy based on network status is as follows.

[0156] When the network condition is good: keep the current collection frequency unchanged to make full use of the efficient transmission capacity of the network.

[0157] When the network condition is moderate: appropriately reduce the acquisition frequency by 10%-20% to reduce the amount of data transmission and reduce the possibility of network congestion.

[0158] When the network condition is poor: significantly reduce the acquisition frequency by 30%-50% or even more to ensure that critical data can be transmitted stably and avoid data loss or transmission delay.

[0159] S4.3. Set the business adjustment method according to the business requirements of the logistics system, so as to adjust the collection frequency obtained in the previous step to obtain the final collection frequency.

[0160] The correspondence between the business requirements of the logistics system and the business adjustment methods is as follows.

[0161] a. City distribution business: City distribution business usually requires quick response, high-frequency distribution and accurate location tracking to ensure that the goods can be delivered in time. In order to meet the needs of high-frequency distribution and accurate location tracking, the frequency of location data collection needs to be increased accordingly to more accurately capture the real-time location of vehicles or goods.

[0162] Real-time and immediate responsiveness: In urban distribution business, the real-time and immediate responsiveness of orders are key. Therefore, the adjustment of sampling frequency needs to ensure that the system can respond quickly to order changes and provide real-time location information. Therefore, it is necessary to meet the minimum collection frequency requirements for this type of business. If it is lower than the requirement, the collection frequency should be increased by a certain proportion to meet the requirement.

[0163] b. Trunk line transportation business: Trunk line transportation usually involves long-distance, large-volume cargo transportation, which requires balancing the requirements of timeliness and cost-effectiveness, and has high requirements for both.

[0164] Impact on sampling frequency: In trunk transportation, due to the long transportation distance and the long time the goods are in transit, it is not necessary to sample the location information as frequently as in the city distribution business. Therefore, choosing a moderate sampling frequency can both meet the timeliness requirements and control costs. The sampling frequency range is lower than that of the city distribution business but higher than that of the bulk business.

[0165] Consider route optimization and cost-effectiveness: In trunk transportation, in addition to the location sampling frequency, route optimization and cost-effectiveness also need to be considered. Through reasonable route planning and scheduling, transportation costs can be reduced while ensuring timeliness.

[0166] c. Bulk business: Bulk business mainly involves the transportation of bulk commodities, which has high requirements on transportation volume, safety and cost-effectiveness.

[0167] Impact on sampling frequency: Due to the large volume and high value of bulk commodity transportation, the safety requirements are also high, so there are higher requirements for the safety and stability of the system. In this case, using a lower sampling frequency can reduce the amount of data transmission and system burden, while ensuring that the location data of key locations can be accurately collected and recorded. The sampling frequency range is relatively lower than that of trunk transportation business.

[0168] Emphasis on security and monitoring: In bulk business, in addition to the sampling frequency of location data, it is also necessary to strengthen security and monitoring measures. By installing sensors and monitoring systems, the transportation status and safety of goods can be tracked in real time to ensure the safety of goods on the way.

[0169] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the inventive concept and technical solution of the present invention, or the inventive concept and technical solution are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. An adaptive and dynamically adjusted multi-channel logistics location data collection method, characterized by: The following steps are involved: S1. Establish a storage warehouse for location data; S2. Convert data into a unified data format and protocol to achieve seamless connection between data from different data sources; S3, conduct quality assessment on the collected data source, and the assessment result is the quality of the data source; S4. Calculate and adjust the collection frequency of different data sources based on a pre-established adaptive location data collection strategy; In step S4, a priority is assigned to each data source according to the accuracy and stability of the data source. The better the accuracy and stability of the data source, the higher the priority. When real-time location data is collected, the collection frequency of the data source is dynamically adjusted. In the case of reduced quality of the data source, this step dynamically adjusts the collection frequency of different data sources according to the historical quality data and priority of the data source. The calculation formula for the initial collection frequency of the data source dynamically adjusted is as follows: Among them, ∫new is the new acquisition frequency, ∫old is the original acquisition frequency, Q current is the quality score of the current data source, Q avg is the historical average quality score of the data source, Q max It is the highest historical quality score of the data source. PriorityFactor is an adjustment factor based on the priority of the data source. The higher the priority, the larger the factor, which reflects the relative importance of the data source in the system.

2. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 1, characterized in that: The step S3 includes: adopting a scoring mechanism, analyzing each data in terms of accuracy, completeness, timeliness, consistency, and relevance, to achieve a score evaluation of data quality in each dimension, and finally obtaining a data quality score by combining the scores of each dimension to achieve a quality evaluation of the collected data source; multiplying the score of each dimension by the corresponding weight, and then summing them to obtain a final data quality score. The calculation formula of the data quality score is: Score DQ =w acc Accuracy+w com Completeness+w tim Timeliness +w con Consistency+w rel Relevance Among them, Score DQ represents the data quality score. Accuracy, Completeness, Timeliness, Consistency and Relevance represent the scores of the five dimensions of accuracy, completeness, timeliness, consistency and relevance respectively. acc 、w com 、w tim 、w con and w rel It represents the weights of the scores of the five dimensions: accuracy, completeness, timeliness, consistency and relevance.

3. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 2 is characterized by: In step S3, the score of each dimension is equal to the average score of each sub-item under the dimension; the corresponding specific evaluation method includes: 1) Accuracy evaluation: Accuracy includes position accuracy and speed accuracy; Location accuracy: This score is determined by comparing the location data from each data source, calculating the location difference and comparing it to the corresponding threshold; Speed ​​Accuracy: Compares the speed data from each data source, calculates the speed difference and compares it with the corresponding threshold to determine the score based on the comparison results; 2) Completeness assessment: accuracy includes field completeness and data missing; Field completeness: Check whether each record contains all required fields. If the record contains all required fields, full marks will be awarded; for each missing field, marks will be deducted; Missing data: Count the proportion of missing values ​​in each field. If the proportion of missing values ​​exceeds the set threshold, points will be deducted according to the excess amount, otherwise full marks will be given; 3) Timeliness evaluation: Timeliness includes timestamp verification and data freshness; Timestamp verification: Determine whether the timestamp data is valid, calculate the difference between the timestamp data and the actual time when the data was acquired, and compare the difference with the set threshold to score; Data freshness: Calculate the time difference between data generation and data processing, and compare the time difference with the set threshold to score; 4) Consistency assessment: Consistency includes internal consistency, external consistency and data format consistency; Internal consistency: Check whether there are logical conflicts between different fields in the same record. If there is no logical conflict, full marks will be given, otherwise marks will be deducted; External consistency: When there is more than one data source providing data at the same location, compare the data from these data sources. If the data from each data source is consistent, full marks will be given, otherwise points will be deducted; Data format consistency: score based on whether the data format and encoding rules of each record are consistent. If they are consistent, full marks will be given, otherwise points will be deducted; 5) Correlation assessment: Correlation includes the correlation between location and geographic area, the correlation between speed and location, and the correlation between data source and device; Correlation between location and geographic area: Check whether the location data and geographic area data match each other. If they match, give full points; otherwise, deduct points. Correlation between speed and position: Calculate the rate of change between the position data and the timestamp data, and analyze whether the speed data matches the rate of change. If so, give full marks, otherwise deduct marks. Correlation between data source and device: Verify whether the IDs of the data source and device correspond to the data source and device actually used. If so, full marks will be awarded, otherwise marks will be deducted.

4. A self-adaptive and dynamically adjusted multi-channel logistics location data collection method according to any one of claims 1 to 3, characterized in that: The step S4 specifically includes: calculating the initial acquisition frequency according to the quality of the data source; specifically includes the following steps: a. Define monitoring indicators, including CPU usage, memory usage, disk I / O, network bandwidth, and data source quality; b. Set thresholds. According to system load capacity and business requirements, set warning thresholds and critical thresholds for each monitoring indicator; c. Setting data source priority: This step assigns priority to each data source based on its accuracy and stability; d. Alarm adjustment: When a monitoring indicator exceeds the warning threshold, the adjustment of the data collection strategy is triggered. First, the collection frequency of low-priority data sources is reduced to release system resources, so that the location data output by higher-priority data sources can be better processed. If the monitoring indicator continues to rise and exceeds the critical threshold, the collection of some or all low-priority data sources is suspended. e. When collecting location data in real time, dynamically adjust the collection frequency of the data source.

5. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 4, characterized in that: The step S4 also includes: monitoring the network status and scoring, adjusting the initial acquisition frequency based on the network status score, the network status scoring method includes: comprehensively considering multiple performance indicators such as bandwidth, delay, packet loss rate, etc., using an evaluation method to obtain a comprehensive network quality score, the calculation formula is: network quality comprehensive score = Σ (indicator weight × indicator score), where the symbols have the following meanings: Σ represents summation, the indicator score represents the performance of the indicator, and the indicator weight represents the importance of the performance indicator in network quality evaluation; judging the network status corresponding to the network quality comprehensive score according to the threshold range of the network quality comprehensive score, and when the trigger condition is met, adjusting the acquisition frequency according to the acquisition frequency adjustment strategy, the acquisition frequency adjustment strategy based on the network status includes: when the network status is good, keeping the current acquisition frequency unchanged; when the network status is medium: appropriately reducing the acquisition frequency, reducing it by 10%-20%; when the network status is poor: significantly reducing the acquisition frequency, reducing it by more than 30%.

6. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 5, characterized in that: The step S4 further includes: setting a business adjustment method according to the business requirements of the logistics system, thereby adjusting the collection frequency obtained in the previous step; the corresponding relationship between the business requirements of the logistics system and the business adjustment method is as follows: a. For urban distribution business, the collection frequency of location data needs to be increased accordingly to meet the minimum collection frequency requirements of this type of business. If it is lower than the requirement, the collection frequency can be increased by a certain proportion to meet the requirement; b. For trunk transport business, select an appropriate sampling frequency. The sampling frequency range is lower than that for urban distribution business but higher than that for bulk business. c. For bulk business: adopt a lower sampling frequency, while ensuring that the location data of key locations can be accurately collected and recorded. The sampling frequency range is relatively lower than that for trunk transport business.

7. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 1, characterized in that: The step S1 comprises: S1.

1. Data table structure needs to be designed; S1.

2. Design indexing strategy; S1.

3. Partition the data according to its characteristics and distribute the data to different physical storage devices. Partition the data specifically based on the two dimensions of time and geographic location.

8. The adaptive and dynamically adjusted multi-channel logistics location data collection method according to claim 1, characterized in that: Step S2 also includes: monitoring the data quality of the coordinate system conversion, evaluating the accuracy and reliability of the conversion algorithm by comparing the coordinate values ​​before and after the conversion, and using the conversion algorithm for verification. The types of errors involved in the verification include coordinate difference, root mean square error, maximum error and minimum error; among the above errors, a single coordinate difference value can reflect the degree of deviation of a single coordinate point after conversion. After the coordinate difference value of each point is obtained, this step can further calculate the root mean square error and find the maximum error and minimum error. The root mean square error comprehensively considers the errors of all points and gives an overall error estimate. The maximum error provides information on the maximum degree of deviation that the conversion algorithm may produce, and determines the performance boundary of the algorithm; while the minimum error reflects the performance of the algorithm under the best conditions.

Images