Distributed storage method of vehicle OBD data

Abstract

The present application relates to the technical field of distributed storage, in particular to a distributed storage method for vehicle OBD data, comprising the following steps: receiving original data blocks uploaded by a vehicle OBD terminal, parsing DTC fault codes, vehicle speeds, engine speeds, gear information and time stamps, assigning data blocks associated with DTC fault codes and sudden acceleration and deceleration events with high key level identifiers, and assigning hot data or cold data identifiers according to the difference between the time stamps and the current time, and establishing a data fidelity and key level scoring set. The present application correlates and processes DTC fault codes and sudden acceleration and deceleration events by parsing original data blocks uploaded by a vehicle OBD terminal, and further assigns key levels to the data in combination with vehicle speeds, engine speeds and gear information, and divides the data temperature according to the difference between the data time stamps and the current time, thereby forming a data fidelity and key level scoring set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a distributed storage method for vehicle OBD data. BACKGROUND

[0002] Distributed storage technology refers to storing data on multiple independent devices connected through a network, achieving flexible expansion and efficient utilization of data storage resources, and improving data reliability and access performance by dividing, redundancy encoding and managing multiple copies of data.

[0003] The prior art adopts static redundancy sharding and copy management when performing distributed storage of data, ignoring the differences in data content and access state, resulting in long-term occupation of storage space by non-important data and expired data, causing waste of storage resources; in the data positioning and retrieval process, only simple static data sharding location is relied on for data addressing, and the data access speed is slow, and as the data size continues to increase, the data search difficulty increases, reducing the overall performance of the storage system. Therefore, improvement is needed. SUMMARY

[0004] The purpose of the present application is to solve the shortcomings in the prior art and provide a distributed storage method for vehicle OBD data.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme, a distributed storage method for vehicle OBD data, comprising the following steps:

[0006] Receiving the original data block uploaded by the vehicle OBD terminal, parsing the DTC fault code, vehicle speed, engine speed, gear information and timestamp, assigning the data block associated with the sudden acceleration and deceleration event and the DTC fault code to a high key level identifier, and assigning the hot data or cold data identifier according to the difference between the timestamp and the current time, establishing a data fidelity and key level score set;

[0007] Based on the data fidelity and key level score set, the key level identifier, data temperature identifier and fidelity score of each data block are extracted, combined into a standardized multi-dimensional data feature vector, and then the multi-dimensional data feature vector is mapped to a predefined and hierarchical index space address according to the preset fidelity threshold, key level type and temperature type, to establish a data hierarchical composite index;

[0008] According to the data hierarchical composite index, obtain the data block coding strategy selection instruction, dispatch the distributed storage system to execute the data block coding strategy selection instruction, perform data sharding and redundancy encoding on the data block, and write the encoded shards into the distributed storage nodes in different storage pools, to generate a distributed storage node layout table;

[0009] The background timing task is started to scan timestamp information in the data grading composite index, filter index entries cooled from hot data, and generate a list of to-be-migrated data blocks. An updated storage structure index is established based on the list of to-be-migrated data blocks and the distributed storage node layout table.

[0010] Preferably, the data fidelity and criticality score set acquisition step is:

[0011] The corresponding vehicle speed value, engine speed value, gear information, sensor self-check state value, and data source reputation score in each original data block are extracted. The engine speed value and gear information are input into a gear ratio function to calculate a theoretical vehicle speed. The theoretical vehicle speed and the vehicle speed value are analyzed by difference, and the abnormal count of the sensor self-check state value is accumulated. Data blocks associated with sudden acceleration and deceleration events and DTC fault codes are assigned a high criticality level identifier to generate a set of to-be-scored parameters.

[0012] Based on the set of to-be-scored parameters, the fidelity score of each data block is calculated.

[0013] Based on the fidelity score, the criticality level identifier, and the data temperature identifier of each data block, a data item identifier set is formed in sequence, and the data fidelity and criticality score set is generated based on the index number of the original data block.

[0014] Preferably, the multi-dimensional data feature vector acquisition step is:

[0015] Based on the index number of each data item in the data fidelity and criticality score set, the criticality level identifier, data temperature identifier, and fidelity score in the data item corresponding to the index number are extracted one by one. The criticality level identifier is converted by binary encoding, and the data temperature identifier is converted into a standard value by a time effectiveness weight table to form an initial data feature value set.

[0016] Based on the initial data feature value set, the fidelity threshold, criticality level type classification table, and temperature type classification table are called respectively. The value difference between the fidelity score and the fidelity threshold is classified. The criticality level identifier value is matched with the index position of the criticality level type classification table, and the data temperature identifier value is matched with the index position of the temperature type classification table. The matching results are combined to form a multi-dimensional data feature vector.

[0017] Preferably, the data grading composite index acquisition step is:

[0018] Based on the multi-dimensional data feature vector, the corresponding index space address is retrieved one by one according to the preset mapping rule table. The multi-dimensional data feature vector is mapped and positioned to the address in the index space in a step-by-step mapping manner to establish a data grading composite index.

[0019] Preferably, the data block encoding strategy selection instruction obtaining step is:

[0020] According to the index space address of each index entry in the data hierarchical composite index, the data items corresponding to the addresses are retrieved one by one, the key level identifier and the data temperature identifier contained in the data items are extracted, the index matching condition is formed according to the encoding value of the key level identifier and the classification value of the data temperature identifier, and the encoding strategy query condition set is generated;

[0021] Based on the encoding strategy query condition set, the preset erasure code strategy library is called one by one, the applicable key level identifier range and the data temperature identifier range of each strategy in the erasure code strategy library are compared one by one, the erasure code strategies that meet the query conditions are filtered, the strategy index number of the erasure code strategies that meet the conditions is determined, and the erasure code strategy index set is formed;

[0022] Based on the erasure code strategy index set, the erasure code strategy entry corresponding to the strategy index number in the erasure code strategy library is called one by one, the encoding rule in the erasure code strategy entry is parsed, the corresponding erasure code encoding instruction is output one by one according to the encoding rule, and the data block encoding strategy selection instruction is obtained.

[0023] Preferably, the distributed storage node layout table obtaining step is:

[0024] According to the data block encoding strategy selection instruction, the data block is equally divided into fragments according to the number of erasure code fragments, each fragment identifier is mapped and bound with the original data block identifier, and the total capacity value, the available capacity value, the current write rate value, the historical write rate standard deviation value, and the current write task number of each candidate distributed storage node are extracted to generate the to-be-scheduled node resource state set;

[0025] According to the to-be-scheduled node resource state set, the node placement score of each storage node is calculated;

[0026] According to the node placement score of each storage node, all candidate nodes are arranged in descending order of node placement score, and the node with the highest node placement score is selected in sequence according to the data fragment number order, each data fragment is allocated to the matching node, and the binding relationship between the data fragment and the storage node is recorded to generate the distributed storage node layout table.

[0027] Preferably, the to-be-migrated data block list obtaining step is:

[0028] According to the data hierarchical composite index, index entries are scanned piece by piece, data temperature identifiers and corresponding data timestamp values in each index entry are extracted, the difference between the data timestamp value and the current system time value is calculated one by one, and if the data temperature identifier is hot data and the time difference value is greater than the preset cooling threshold, the entry is marked as a to-be-migrated index entry, and a to-be-migrated data block list is generated.

[0029] Preferably, the obtaining step of the updated storage structure index is:

[0030] Based on the to-be-migrated data block list, data block index numbers in the to-be-migrated data block list are extracted one by one, the distributed storage node layout table is queried piece by piece according to the data block index number, the data shard corresponding to each data block index number and the storage node binding relationship are determined, the storage node is located through the binding relationship, and all data shard contents are read in sequence, the complete data content is reorganized according to the data shard sequence number, and a complete original data block is generated.

[0031] Based on the complete original data block, according to a preset cold data encoding strategy, each original data block is re-divided into a plurality of data shards, and distributed storage nodes of a cold data storage pool are selected one by one to write data shard contents, and the binding mapping relationship between each data shard and the written storage node is recorded, and an updated storage structure index is generated.

[0032] Compared with the prior art, the advantages and positive effects of the present application are:

[0033] The present application analyzes the original data block uploaded by the vehicle-mounted OBD terminal, associates the DTC fault code with the sudden acceleration and deceleration event, and further assigns the key level of the data in combination with the vehicle speed, engine speed and gear information. According to the time stamp of the data and the current time difference, the data temperature is divided, and the data fidelity and key score set are formed; based on the score set, a standardized multi-dimensional data feature vector is constructed, and a hierarchical index space address is established according to the preset threshold and classification mapping rule, and a hierarchical composite index is realized; according to the index information, the most suitable data encoding strategy is further selected, the data shards and redundant encoding processing are performed on the data block, the shard storage position is dynamically adjusted, and the periodic task is executed in the background to monitor the data hot and cold state in real time, realizing the automatic migration of data and the dynamic update of the storage structure index. In this way, the waste of storage space caused by blind data storage is avoided, the data access efficiency and response speed are improved through data temperature management, and the encoding strategy is flexibly executed for different data states, effectively avoiding important data loss and performance degradation. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The present application is a step schematic diagram. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0036] Please refer to Figure 1 The present application provides a technical scheme, a distributed storage method of vehicle OBD data, comprising the following steps:

[0037] Receiving the original data block uploaded by the vehicle OBD terminal, parsing the DTC fault code, vehicle speed, engine speed, gear information and timestamp, assigning the data block associated with the sudden acceleration and deceleration event and the DTC fault code with a high key level identifier, and assigning the hot data or cold data identifier according to the difference between the timestamp and the current time, establishing a data fidelity and keyness score set;

[0038] Based on the data fidelity and keyness score set, the key level identifier, data temperature identifier and fidelity score of each data block are extracted to form a standardized multi-dimensional data feature vector, and then the multi-dimensional data feature vector is mapped to a pre-defined and hierarchical index space address according to the pre-set fidelity threshold, key level type and temperature type, to establish a data hierarchical composite index;

[0039] According to the data hierarchical composite index, the data block coding strategy selection instruction is obtained, the distributed storage system is dispatched to execute the data block coding strategy selection instruction, the data block is subjected to data sharding and redundancy coding, and the coded shards are written into the distributed storage nodes in different storage pools to generate a distributed storage node layout table;

[0040] Starting a background timing task, scanning the timestamp information in the data hierarchical composite index, screening the index entries that are cooled from hot data to cold data, and generating a to-be-migrated data block list, establishing an updated storage structure index according to the to-be-migrated data block list and the distributed storage node layout table.

[0041] The data fidelity and keyness score set is obtained by:

[0042] The corresponding vehicle speed value, engine speed value, gear information, sensor self-check state value and data source credit score in each original data block are extracted, the engine speed value and gear information are input into the gear ratio function and the theoretical vehicle speed is calculated, the difference between the theoretical vehicle speed and the vehicle speed value is analyzed, the abnormal count of the sensor self-check state value is accumulated, and the data block associated with the sudden acceleration and deceleration event and the DTC fault code is assigned with a high key level identifier, to generate a to-be-scored parameter set;

[0043] According to the to-be-scored parameter set, the fidelity score of each data block is calculated, and the calculation formula is:

[0044] ;

[0045] in, Indicates the first The fidelity score of each data block. Indicates the first The normalized result of the data source reputation score for each data block. Indicates the first The sensor self-test status anomaly count value in each data block, This represents the impact coefficient of self-test failure. Indicates the first The vehicle speed values ​​for each data block. Indicates the first The engine speed values ​​for each data block. Indicates the first The gear information of each data block, Indicates gear position The corresponding total gear ratio function result, Indicates the reference vehicle speed constant. This represents the logical consistency sensitivity coefficient.

[0046] Based on the fidelity score, the criticality identifier of each data block, and the data temperature identifier, a set of data item identifiers is formed by combining them in sequence, and then grouped according to the index number of the original data blocks to generate a set of data fidelity and criticality scores.

[0047] Specifically, first, from each raw data block uploaded by the vehicle-mounted OBD terminal, the vehicle speed value, the engine speed value, the gear information, the sensor self-check state value and the data source reputation score are extracted one by one, wherein the vehicle speed value is directly read from the OBD-II standard PID 0D, the engine speed value is read from PID 0C, the gear information is obtained by analyzing the vehicle private CAN bus protocol, the sensor self-check state value is read from PID 01, and the data source reputation score is obtained by querying the terminal ID from a reputation rating database that maintains the historical reliability of each terminal. Then, the system inputs the engine speed value and the gear information into a preset gear ratio function for processing. The function internally includes a lookup table extracted from the vehicle manufacturer's technical manual and solidified into the program. The table maps the gear information (such as 'P', 'R', 'N', 'D', '1', '2') to specific gear ratio values. Combined with the final gear ratio and the dynamic circumference of the tire, the theoretical vehicle speed is calculated. The calculation method is that the theoretical vehicle speed is equal to the engine speed divided by the total gear ratio, multiplied by the dynamic circumference of the tire, and then converted from meters per minute to kilometers per hour. Then, the calculated theoretical vehicle speed is subtracted from the actual vehicle speed value extracted from the data block, and the absolute value of the difference is recorded. At the same time, the system analyzes the sensor self-check state value, which is a multi-bit binary number. Each bit corresponds to the health status of a specific sensor. The system will traverse the preset list of key sensors (including but not limited to oxygen sensor, air flow sensor, throttle position sensor) to check if the corresponding state bit is an abnormal flag (such as "1"). The number of abnormal flags is accumulated to obtain the sensor self-check abnormal count value of the data block. In addition, the system analyzes the vehicle speed and timestamp sequence in the data block. By calculating the acceleration between consecutive data points (acceleration equals the change in speed divided by the change in time), and comparing it with the preset threshold of sudden acceleration and deceleration, which is set by statistical analysis of driving data on more than 1000 hours of urban, highway, and rural road conditions, for example, if the acceleration is greater than 3.5 meters per second or the absolute value of the deceleration is greater than 4.0 meters per second, it is determined as a sudden acceleration or deceleration event. If the event occurs, or the data block contains a non-empty DTC fault code list, the system assigns the data block a high key level identifier, otherwise a low key level identifier. Finally, the raw data, the calculated theoretical vehicle speed, the vehicle speed difference, the sensor self-check abnormal count value and the key level identifier are integrated into a structured data set to form a set of parameters to be scored.

[0048] Formula: The formula has the advantages that: not only relies on a single data validity test, but also realizes accurate quantification of OBD data quality by organically combining the credibility of the data source itself, the health status of the vehicle-mounted sensor hardware, and the logical self-consistency between multiple key dynamic data. Specifically, by introducing a data source reputation score , the system can make prior judgments about the data source based on historical performance, and suppress contaminated data from poor or faulty terminals; by integrating sensor self-test abnormality count value and impact coefficient , the health status of the hardware level is directly taken into account, and data distortion caused by sensor failure is punished; most importantly, by exponential decay term , the physical correlation between vehicle speed, engine speed, and gear is tested, which can identify abnormal data that is inconsistent with the vehicle dynamics model. This design makes the fidelity score highly sensitive to logical inconsistency, greatly improving the accuracy and intelligence of data cleaning, and providing high-quality decision basis for subsequent distributed storage strategies.

[0049] represents the data source reputation score of the th data block, and the obtaining step is: the system maintains a reputation file for each registered OBD terminal on the backend server, and the initial score is 100. The background analysis program of the data center will periodically sample and check the stored data, for example, cross-verify the specific trip data uploaded by a terminal with high-precision GPS track and vehicle-mounted video record. If serious deviation is found in the data block (such as speed error continuously exceeding 20%), the reputation score of the terminal will be deducted, and the calculation formula is: . On the contrary, if a terminal continuously uploads 10,000 data blocks without finding logical inconsistency or conflict with the verification data, it will be rewarded, and the calculation formula is: . The lower limit of the score is 0, and the upper limit is 100. For example, a terminal with ID "SN202308" has a current reputation score of 92, then .

[0050] represents the normalized result of the data source reputation score of the th data block, and the obtaining step is: in order to map the reputation score in the range of to the interval , the maximum and minimum normalization method is adopted to facilitate multiplication with other factors. The normalization formula is: . Wherein is the set maximum reputation score 100, The minimum reputation score is set to 0. This normalization process ensures that the weight of the data source reputation in the formula is on the same scale as other factors. For example, for the terminal with a reputation score of 92 above, the normalization result is: .

[0051] This represents the impact coefficient of a self-test failure, obtained through the following steps: This coefficient is used to adjust the penalty for sensor failure on the fidelity score. Its value was set based on statistical analysis of over 500,000 historical data records containing DTC fault codes. The analysis showed that when a vehicle experiences a failure of one critical sensor (such as the crankshaft position sensor), there is approximately a 40% probability that related data streams (such as engine speed) will become completely unreliable. To quantify this impact, a target was set: when one sensor fails (i.e.,...)... When considering this, the desired penalty factor in the fidelity score should be... The value is approximately 0.6. Calculations are performed based on this objective: Solving for Therefore, setting It is 0.67.

[0052] Indicates the first The abnormal count value of sensor self-test status in each data block is obtained by directly calculating the value when generating the "set of parameters to be scored" in the previous step. The status of specific bits associated with core sensors (such as engine coolant temperature sensor, intake manifold absolute pressure sensor, vehicle speed sensor, etc.) is checked, and for each bit with a "failed" status, the count value is... Add 1. For example, if the system detects that both the intake manifold pressure sensor and the oxygen sensor report a self-test failure, then... .

[0053] The logical consistency sensitivity coefficient is determined by the following steps: This coefficient reflects the sensitivity of the fidelity score to logical deviations in vehicle speed. To determine its value, OBD data from 10 mainstream vehicle models were collected under standard test cycles (such as NEDC), totaling 100 hours. The deviation rate between theoretical and actual vehicle speeds in this data was analyzed. It was found that under fault-free conditions, 95% of the deviation rate was within 0.05. The design goal is to reduce the decay factor of the exponential part when the deviation rate reaches the critical value of 0.05. It should decrease significantly to To form an effective punishment. Calculations are performed based on this objective: Solving for Therefore, a logical consistency sensitivity coefficient is set. It is 20.

[0054] Vehicle Speed The vehicle speed value in km / h for the

[0055] Engine RPM The engine speed value in RPM for the

[0056] Gear Ratio The result of the transmission total ratio function corresponding to the gear The gear information is parsed from the data block, for example, 'D4' (4th gear). The transmission total ratio function integrates these parameters to output a coefficient that converts RPM to km / h. The formula is:

[0057] Reference Speed Constant The constant is set to 120 km / h, which is a representative value for vehicle high-speed operation conditions, ensuring that the deviation evaluation is within a reasonable range.

[0058] Calculation Process:

[0059] Now let's calculate the fidelity score for a specific OBD data block. The parameter values for this data block are as follows:

[0060] Data Source Credibility Score Normalized to .

[0061] Sensor Self-Test Abnormality Count .

[0062] Actual Vehicle Speed km / h.​​​

[0063] Engine speed RPM.

[0064] Gear information is 'D4', the corresponding transmission overall ratio function result .

[0065] Each coefficient is set to: , , .

[0066] First, calculate the theoretical vehicle speed:

[0067] ;

[0068] Then, calculate the absolute deviation of vehicle speed:

[0069] ;

[0070] Next, substitute all parameters into the fidelity score formula :

[0071] ;

[0072] ;

[0073] Calculate step by step:

[0074] Calculate the sensor penalty factor:

[0075] ;

[0076] Calculate the logic consistency index term:

[0077] ;

[0078] Finally, multiply each part to get the final fidelity score:

[0079] ;

[0080] The result shows that: the fidelity score of this data block is about 0.018, which is a very low score (the score range is 0 to 1). This low score is mainly caused by the huge vehicle speed logic deviation (actual vehicle speed 80 km / h, while the theoretical vehicle speed is as high as 98.4 km / h) and two sensor failures. In the fidelity grading, scores below 0.3 are usually classified as "unreliable" level, meaning that the data block contains serious errors, and its internal values should not be used for any precise analysis or model training.

[0081] Based on the fidelity score of each data block calculated in the previous step, and the key identifiers and data temperature identifiers extracted or determined from the data blocks, the system initiates a data integration and structuring process. First, for determining the data temperature identifier, the system extracts the UTC timestamp recorded within each data block and compares it with the current system's standard time, calculating the time difference. The system has a data cooling time threshold set at 2,592,000 seconds (30 days). This value is based on statistical analysis of data query logs over the past year. The analysis shows that over 95% of data access requests are concentrated within 30 days of data generation. Therefore, if the time difference is less than or equal to this threshold, the data block is designated as "hot data." The system assigns a "high" critical level identifier to data blocks with a high value and a "cold" identifier to data blocks with a low value. Then, it creates a structured data item for each raw data block. This data item contains four core metadata fields: the data block's unique index number, the calculated fidelity score, the critical level identifier ('high' or 'low'), and the data temperature identifier ('hot' or 'cold'). For example, for a data block with index number "BLK_20240520_1130_001", its calculated fidelity score is 0.018. Because it is associated with a DTC fault code, it is assigned a "high" critical level identifier, and because its timestamp is within 30 days, it is assigned a "hot" data identifier. The system will then generate a corresponding data item, represented in memory as a data block containing {index: For records like "BLK_20240520_1130_001", fidelity: 0.018, criticality: "high", temperature: "hot", after all data blocks are processed, a set of data item identifiers containing tens of thousands of such records will be formed. Finally, to facilitate subsequent fast retrieval and index construction, the system transforms this flattened set of records into a hash map or similar key-value data structure with the original data block index number as the key. This process involves traversing the set of data item identifiers, using the index number in each record as the key, and storing the remaining part, which includes the fidelity score, criticality identifier, and data temperature identifier, as a composite value in the hash map, ultimately generating a set of data fidelity and criticality scores.

[0082] The steps for obtaining feature vectors from multidimensional data are as follows:

[0083] Based on the index number of each data item in the data fidelity and criticality scoring set, the criticality identifier, data temperature identifier, and fidelity score are extracted from the data item corresponding to the index number. The criticality identifier is converted through binary encoding, and the data temperature identifier is converted into a standard value through the timeliness weight table to form an initial set of data feature values.

[0084] Based on the initial data feature value set, the fidelity threshold, the key level type classification table and the temperature type classification table are called respectively, classified by the numerical difference between the fidelity score value and the fidelity threshold value, the key level identification value is matched with the index position of the key level type classification table, the data temperature identification value is matched with the index position of the temperature type classification table, and the matching results are combined to form a multi-dimensional data feature vector.

[0085] Specifically, according to the data fidelity and criticality score set generated by the foregoing steps, the system starts an iterative processing flow to process each data item in the score set one by one. Specifically, the system first extracts the corresponding key level identification (for example, "high"), data temperature identification (for example, "hot") and fidelity score (for example, 0.018) from the score set through the index number (for example, "BLK_20240520_1130_001") of the data item. Then, the system quantitatively converts the extracted non-numeric identification. For the key level identification, a fixed binary coding rule is used for conversion. The specific rule is that the "high" key level identification is mapped to the value 1, and the "low" key level identification is mapped to the value 0. This coding directly reflects the priority of the data. For the data temperature identification, the system calls a preset "time effectiveness weight table" for conversion. The construction of this table is based on the statistical analysis of the data access log in the past year. The analysis result shows that the average access frequency of data generated within 30 days (i.e. "hot" data) is 18.5 times that of data generated more than 30 days (i.e. "cold" data). In order to quantify this difference, the time effectiveness weight table maps the "hot" identification to the standard value 1.0 and the "cold" identification to the standard value 0.2. The value setting logic is based on the relative relationship of access frequency and is normalized. For example, for data identified as "hot", its time effectiveness weight is 1.0. This conversion process converts discrete temperature classification into continuous numerical weight. Finally, the original fidelity score, the key level value converted by binary coding and the data temperature standard value converted by the time effectiveness weight table are combined as a tuple, for example, for the data item with index number "BLK_20240520_1130_001", the numerical tuple obtained after processing is (0.018, 1, 1.0). The system repeats this process for all data items in the score set to form the initial data feature value set.

[0086] Based on the initial data feature value set formed in the previous process, the system classifies and matches the value tuple of each data block in multiple dimensions. First, the system discretizes and classifies the fidelity score by calling a set of preset fidelity threshold values. The threshold set contains three values: 0.3, 0.6, and 0.9. These threshold values are set based on statistical analysis of the fidelity score distribution of more than one million historical data blocks. Scores below 0.3 are considered unreliable, accounting for 5% of the total. Scores between 0.3 and 0.6 are considered low fidelity, accounting for 20%. Scores between 0.6 and 0.9 are considered medium fidelity, accounting for 60%. Scores above 0.9 are considered high fidelity, accounting for 15%. The classification rules are as follows: if the fidelity score is less than or equal to 0.3, it is classified as level 0; if the score is between 0.3 and 0.6, it is classified as level 1; if the score is between 0.6 and 0.9, it is classified as level 2; if the score is greater than 0.9, it is classified as level 3. Next, the system calls a "key level type classification table," which is a simple key-value mapping structure: {0: 0, 1: 1}. The system uses the key level identification value (0 or 1) in the initial data feature value set as the key to query the table to obtain the corresponding classification index position. For example, if the key level value is 1, the matching index position is 1. Then, the system calls the "temperature type classification table," which is also based on threshold classification rules. The threshold is set to 0.5, which is determined based on the midpoint of the "hot" data weight 1.0 and the "cold" data weight 0.2 determined in the previous process. The classification rules are as follows: if the data temperature identification value is greater than 0.5, the matching index position is 1 (representing hot data); if the value is less than or equal to 0.5, the matching index position is 0 (representing cold data). Finally, the system combines the fidelity classification level, key level type index position, and temperature type index position obtained through the above three steps into a three-dimensional integer vector in order. For example, an initial data feature value of a data block is (0.85, 1, 1.0). The fidelity score 0.85 is between 0.6 and 0.9, classified as level 2. The key level value 1 matches index 1. The data temperature value 1.0 is greater than 0.5, matching index 1. Therefore, the final combined multi-dimensional data feature vector is (2, 1, 1).

[0087] The data hierarchical composite index acquisition steps are:

[0088] Based on the multi-dimensional data feature vector, the corresponding index space address is retrieved according to the preset mapping rule table. The index space is located step by step using the step-by-step mapping method. The multi-dimensional data feature vector is mapped and located to the address in the index space step by step to establish the data hierarchical composite index.

[0089] Specifically, based on the multi-dimensional data feature vector corresponding to each data block generated in the previous step, the system performs a deterministic, hierarchical address mapping process, which is based on an embedded, structured "mapping rule table". The rule table is not a physical table, but a set of path construction rules implemented through program logic. The core idea is to analyze the three-dimensional feature vector dimension by dimension and map it to an index space address with a hierarchical structure. The index space is designed as / Temperature / Criticality / Fidelity, forming a three-level directory structure. The mapping process uses a step-by-step positioning method. Specifically, for a given multi-dimensional data feature vector, such as (2, 1, 1), the system first analyzes the third element of the vector (temperature type index), which is mapped to the first-level directory: index 1 corresponds to "hot_data", and index 0 corresponds to "cold_data". Therefore, the first-level path is determined as / hot_data / . Then, the system analyzes the second element of the vector (criticality type index), which is mapped to the second-level directory: index 1 corresponds to "high_criticality", and index 0 corresponds to "low_criticality". The first-level path is concatenated to form the path / hot_data / high_criticality / . Finally, the system analyzes the first element of the vector (fidelity classification level), which is mapped to the third-level directory: level 3 corresponds to "fidelity_high", level 2 corresponds to "fidelity_medium", level 1 corresponds to "fidelity_low", and level 0 corresponds to "fidelity_untrusted". The second-level path is concatenated to form the final index space address / hot_data / high_criticality / fidelity_medium / . The system associates the original index number of the data block being processed (e.g., "BLK_20240520_1130_001") with this generated index space address and adds this association to a global index data structure. This data structure is usually implemented as a nested hash table or tree structure, with the key path being the generated index space address and the value being a list containing all data block index numbers mapped to this address. By repeatedly performing this mapping process on the multi-dimensional data feature vectors of all data blocks, a data hierarchical composite index is finally established.

[0090] The data block encoding strategy selection instruction acquisition step is:

[0091] According to the index space address of each index entry in the data hierarchical composite index, the data items corresponding to the address are retrieved one by one, the key level identifier and the data temperature identifier contained in the data items are extracted, the index matching condition is formed according to the coding value of the key level identifier and the classification value of the data temperature identifier, and the coding strategy query condition set is generated;

[0092] Based on the coding strategy query condition set, the preset erasure code strategy library is called one by one, the applicable key level identifier range and the data temperature identifier range of each strategy in the erasure code strategy library are compared one by one, the erasure code strategy that meets the query condition is filtered, the strategy index number of the erasure code strategy that meets the condition is determined, and the erasure code strategy index set is formed.

[0093] Based on the erasure code strategy index set, the erasure code strategy entry corresponding to the strategy index number in the erasure code strategy library is called one by one, the coding rule in the erasure code strategy entry is parsed, and the corresponding erasure code coding instruction is output one by one according to the coding rule to obtain the data block coding strategy selection instruction.

[0094] Specifically, according to the data hierarchical composite index established in the foregoing steps, a traversal program is started by the system, which scans all entries in the index in a depth-first or breadth-first manner until the leaf node. For each leaf node, its path itself contains the key level and data temperature classification of the corresponding data block. For example, for the index entry with the path / hot_data / high_criticality / fidelity_medium / , the system directly parses the data temperature classification as "hot data" and the key level classification as "high key level". The system then converts these classification information into predefined numerical values according to a fixed mapping rule: for the key level, "high" is converted to the numerical value 1 and "low" is converted to the numerical value 0; for the data temperature, "hot" is converted to the numerical value 1 and "cold" is converted to the numerical value 0. This process does not need to extract the original data item again, but directly uses the information of the index structure itself. Therefore, for the above index path, the system obtains a two-tuple (1, 1) composed of the key level coding numerical value and the data temperature classification numerical value. This two-tuple constitutes the index matching condition. The system binds this matching condition with the index numbers of all data blocks associated with this index path (such as "BLK_20240520_1130_001", "BLK_20240520_1132_005", etc.) to form a temporary query task. This process is repeated for all unique path combinations in the index. For example, another path / cold_data / low_criticality / fidelity_high / will generate a matching condition (0, 0). Finally, the system aggregates all these temporary query tasks to generate the coding strategy query condition set.

[0095] Based on the encoding strategy query condition set generated in the previous process, the system starts a strategy matching process, which processes the query conditions in the set one by one, and calls a preset erasure code strategy library for comparison. The erasure code strategy library is a static data structure configured during system initialization, and its construction is based on a comprehensive consideration of data importance, access frequency, storage cost, and system reliability requirements. Specifically, the strategy library contains four core strategies: Strategy One (Strategy Index Number "EC_HP_HT"), which is for data with a key level of 1 and a temperature of 1 (high key, hot data), uses a highly reliable RS(6,3) encoding, i.e., 6 data shards and 3 parity shards, to provide high redundancy to cope with potential failures in a frequent read-write environment. Its setting basis is that such data is the most valuable and most frequently accessed, and its availability must be guaranteed; Strategy Two (Strategy Index Number "EC_LP_HT"), which is for data with a key level of 0 and a temperature of 1 (low key, hot data), uses a more cost-effective RS(10,2) encoding, i.e., 10 data shards and 2 parity shards, to reduce storage overhead while ensuring a certain reliability; Strategy Three (Strategy Index Number "EC_HP_CT"), which is for data with a key level of 1 and a temperature of 0 (high key, cold data), uses a long-term archiving optimal RS(4,4) encoding to provide extremely high redundancy to combat possible silent errors in long-term storage; Strategy Four (Strategy Index Number "EC_LP_CT"), which is for data with a key level of 0 and a temperature of 0 (low key, cold data), uses a storage efficiency highest RS(12,2) encoding to store for a long time at the lowest cost. For each query condition in the encoding strategy query condition set, such as (1, 1), the system will traverse all strategies in the erasure code strategy library, compare it with the applicable key level identifier range and data temperature identifier range of each strategy, and when the query condition completely matches the applicable range of a certain strategy, such as (1, 1) matching the applicable range of Strategy One (key level 1, temperature 1), the system will filter out the strategy and extract its strategy index number "EC_HP_HT", which will be associated with all data block index numbers associated with the query condition to form a strategy mapping entry. This process is repeated for all query conditions, and finally an erasure code strategy index set is formed.

[0096] Based on the erasure coding policy index set formed in the previous process, the system starts the instruction generation program to process each policy mapping entry in the set one by one. For an entry, for example, containing a policy index number "EC_HP_HT" and a list of associated data block index numbers, the system first uses the policy index number to query the preset erasure coding policy library to retrieve the complete policy details. The details clearly record the encoding rule as "RS(6,3)". Then, the system's internal instruction parser parses the string "RS(6,3)", identifies the encoding algorithm as "Reed-Solomon", and extracts two core parameters: the number of data fragments is 6, and the number of redundant (checksum) fragments is 3. After obtaining these parameters, the system generates a structured encoding instruction for each data block index number contained in the policy mapping entry according to these specific encoding rules. This instruction is a data object containing multiple key-value pairs. For example, for the data block "BLK_20240520_1130_001", the specific content of the generated instruction is {"target_block_id": The instruction is generated as follows: “BLK_20240520_1130_001”, “encoding_algorithm”: “Reed-Solomon”, “data_shards_k”: 6, “parity_shards_m”: 3, “chunk_size”: 1048576}, where chunk_size is a preset chunk size used uniformly for all data blocks, such as 1MB. The system repeats this instruction generation process for all data block index numbers in the list to ensure that each data block obtains a clear and executable encoding task description. After all policy mapping entries have been processed, all generated individual encoding instructions are aggregated into a queue or list, ultimately yielding the data block encoding policy selection instruction.

[0097] The steps to obtain the distributed storage node layout table are as follows:

[0098] Based on the data block encoding strategy, select instructions, perform equal-length splitting of data blocks according to the number of erasure coding fragments, map and bind each fragment identifier to the original data block identifier, and extract the total capacity value, available capacity value, current write rate value, historical write rate standard deviation value and current write task number of each candidate distributed storage node to generate a set of resource status of nodes to be scheduled.

[0099] Based on the set of resource statuses of the nodes to be scheduled, calculate the node placement score for each storage node using the following formula:

[0100] ;

[0101] in, Indicates candidate storage nodes Node placement score Indicates storage node The current available capacity, Indicates storage node Total capacity Indicates storage node Current I / O write rate, This represents the maximum I / O write rate across all nodes in the system. This is the coefficient for the percentage of capacity score. This is the I / O performance score ratio coefficient. Indicates storage node Current number of concurrent write tasks, As the load penalty factor, Indicates storage node Historical I / O write rate standard deviation As a stability penalty factor, To prevent division by zero for extremely small constants;

[0102] Based on the node placement score of each storage node, all candidate nodes are sorted in descending order of node placement score. The nodes with the highest node placement scores are selected in sequence according to the data shard numbers. Each data shard is assigned to a matching node, and the binding relationship between the data shard and the storage node is recorded to generate a distributed storage node layout table.

[0103] Specifically, according to the data block encoding strategy selection instruction, the system starts a data distribution preprocessing procedure, which first takes out the instructions one by one from the instruction queue for analysis, for example, for the instruction {“target_block_id”: “BLK_20240520_1130_001”, “encoding_algorithm”: “Reed-Solomon”, “data_shards_k”: 6, “parity_shards_m”: 3, “chunk_size”: 1048576}, the system locates and reads the content of the original data block according to the target_block_id, then, according to the data shard number k = 6 and the block size chunk_size = 1MB in the instruction, the data block is divided into 6 equal length 1MB data shards from beginning to end, if the size of the original data block is not an integer multiple of 6MB, the last shard will be padded with zero bytes to 1MB, then the system creates a unique global shard identifier for each generated data shard, the generation rule is to concatenate the original data block identifier with an underscore and the shard index, for example, generate BLK_20240520_1130_001_ds_0 to BLK_20240520_1130_001_ds_5 a total of 6 data shard identifiers, while reserving BLK_20240520_1130_001_ps_0 to BLK_20240520_1130_001_ps_2 a total of 3 redundant shard identifiers, and establish a temporary mapping relationship table in memory with the original data block identifier BLK_20240520_1130_001, at the same time, a parallel monitoring agent program broadcasts a state query request to all candidate storage nodes in the distributed storage cluster, each node will immediately return its current resource state information after receiving the request, including: the total storage capacity value of the node, which is a static configuration value, for example 10TB; the current available capacity value of the node, which is obtained by querying the file system in real time, for example 4.2TB; the current I / O write rate value of the node, which is calculated by collecting the write byte number per second of the disk in the past 5 seconds, for example 85MB / s; the historical I / O write rate standard deviation value of the node, which is obtained by calculating the standard deviation of the I / O write rate sampling point sequence of each minute in the past hour, for example 12.5MB / s; and the current number of concurrent write tasks of the node, which is obtained by querying the task scheduling queue length inside the node, for example 5 tasks, all the state information returned by the nodes is aggregated to form a set of resource state of the nodes to be scheduled.

[0104] Formula: The formula has the advantage of intelligently selecting the optimal storage location for data shards. Its core advantage lies in the comprehensive consideration of multiple dimensions through linear weighting terms Balancing the node's storage capacity and immediate I / O performance, two basic resource indicators, avoids resource bottlenecks caused by focusing on only one of them. More importantly, two non-linear penalty factors are introduced: the load penalty factor effectively punishes nodes with high current load, preventing write delays from increasing dramatically due to task accumulation. The stability penalty factor focuses on the long-term performance of the node, reducing the weight of nodes with unstable I / O rates. This design makes the scoring system not only focus on the current state of the node, but also consider its historical stability and future predictability, thus placing data shards on nodes with ample resources, balanced load, and stable performance, improving the write throughput and reliability of the entire distributed storage system.

[0105] represents the current available capacity of storage node , with a unit of TB. This data is obtained by real-time querying the available space of the file system through the monitoring agent deployed on each storage node by calling the underlying interface of the operating system (such as the statfs system call in Linux system), and is a dynamic field in the node resource state set to be scheduled. For example, the available capacity of node is 4TB, then .

[0106] represents the total capacity of storage node , with a unit of TB. This is a static configuration value set by the administrator when the node joins the storage cluster, and is recorded in the metadata service of the cluster. It represents the total size of the node's physical storage medium. For example, the total capacity of node is configured as 10TB, then .

[0107] represents the current I / O write rate of storage node , with a unit of MB / s. This value is calculated by the monitoring agent by continuously collecting disk I / O statistical information (such as the sectors_written field in the Linux system / proc / diskstats file). Specifically, the total number of written sectors in the last 5 seconds is multiplied by the sector size and then divided by 5 to reflect the node's instantaneous write capability. For example, the current write rate of node is 50 MB / s, then .

[0108] represents the maximum I / O write rate in all nodes in the system, in units of MB / s, which is dynamically determined by traversing all nodes in the node resource state set to be scheduled at each scheduling calculation and taking the maximum value, for the purpose of normalizing the I / O performance of each node. For example, if the of three nodes in the cluster are 50 MB / s, 80 MB / s, and 65 MB / s, respectively, .

[0109] and are the capacity score proportion coefficient and the I / O performance score proportion coefficient, respectively, which are two dimensionless weight parameters and satisfy . The setting of these two coefficients aims to balance the importance of storage capacity and write performance in node selection, and their values are determined based on regression analysis of historical system operation data. By simulating data placement strategies under different weight combinations and evaluating their impact on average write delay and storage balance, the weight combination with the best overall performance is selected. For the OBD data on vehicles, which continuously flows in and is sensitive to write performance, it is found through analysis that setting the I / O performance weight slightly higher (i.e. ) can achieve better system response, so and are set.

[0110] represents the number of concurrent write tasks of the storage node , which is a dimensionless integer, and is obtained in real time by the task queue maintained by the central scheduler or the node itself, representing the current load pressure of the node. For example, if node has 3 write tasks being processed, then .

[0111] represents the load penalty factor, which is a dimensionless coefficient used to adjust the negative impact of the number of concurrent tasks on the node score. Its value is set based on benchmark testing by gradually increasing the number of concurrent write tasks on a single node and recording the change in I / O throughput. When the number of concurrent tasks reaches 5 and the throughput drops to 50% of the peak value, the penalty target is established as . By solving the equation, , we get , and is calculated.

[0112] represents the number of concurrent write tasks of the storage node The historical I / O write rate standard deviation of the node, in MB / s, which measures the stability of the node performance. The monitoring agent calculates the standard deviation of the sequence of I / O write rate data points collected every minute in the past 1 hour. For example, if the standard deviation of the write rate samples in the past 1 hour is 5 MB / s, then . .

[0113] represents the stability penalty factor, which is a dimensionless coefficient used to adjust the intensity of the penalty of I / O performance fluctuations on the score. Its value is determined by analyzing the correlation between historical fault data and . The goal is to set the value of the stability penalty term to decay to (about 0.368) when the relative volatility of the node reaches 0.2 (a higher fluctuation level determined according to operation and maintenance experience). Accordingly, the equation is listed as: , and the solution is .

[0114] is a very small constant used to prevent division by zero when is 0. Its value is much smaller than the normal I / O rate value, and is set to .

[0115] Calculation process:

[0116] Now calculate the node placement score of the candidate storage node . Get the parameters of the node from the set of to-be-scheduled node resource states as follows:

[0117] TB, TB, MB / s, , MB / s.

[0118] Get from the system global state: MB / s.

[0119] Use the preset coefficients: , , , , .

[0120] Substitute the above values into the formula:

[0121] ;

[0122] Calculate step by step:

[0123] Compute the base resource score term:

[0124] ;

[0125] Compute the load penalty factor:

[0126] ;

[0127] Compute the stability penalty factor:

[0128] ;

[0129] Finally, multiply the three parts together to get the final score:

[0130] ;

[0131] The result shows that the candidate storage node has a node placement score of 0.203, which is a relative, dimensionless suitability measure in itself, used to rank among all candidate nodes, with a higher score representing that the node is more suitable to receive new data shards at the current time. The score integrates the node's remaining space, write performance, current load, and historical stability, and a lower score (as in this example) can mean that the node is underperforming in one or more dimensions (for example, while the base resources are acceptable, the load and stability penalties are large).

[0132] According to the node placement score of each storage node calculated in the foregoing steps, the system starts a deterministic data shard placement scheduling process. First, the system organizes all candidate storage nodes and their corresponding node placement scores into a list, and sorts the list in descending order according to the score values, thereby obtaining a node priority queue from the best to the worst, for example, if there are five nodes, their scores are {node-02: 0.85, node-04: 0.76, node-01: 0.51, node-05: 0.33, node-03: 0.20}, and the sorted priority queue is [node-02, node-04, node-01, node-05, node-03]. Then, the system takes a batch from the data shard queue to be placed, which contains all data shards and redundant shards belonging to the same original data block, for example, 9 shards (6 data shards and 3 redundant shards). Then, the system assigns storage nodes to each shard in order according to the shard number (from _ds_0 to _ds_5, and then to _ps_0 to _ps_2). When assigning, the system takes a node from the head of the sorted node priority queue, assigns the current shard to it, and in order to ensure the high availability of data, a key constraint is that different shards of the same original data block must be stored on different physical nodes. Therefore, once a node is assigned to a shard, it will be removed from the candidate node list for this batch assignment. For example, after BLK_..._ds_0 is assigned to node-02, node-02 will no longer be used to assign the remaining 8 shards of the data block. Then, BLK_..._ds_1 will be assigned to the next node node-04 in the queue, and so on, until all 9 shards are assigned to 9 different nodes with the highest scores. After each assignment is completed, the system immediately records the binding relationship between the shard identifier and the assigned node identifier, for example, {"shard_id": "BLK_..._ds_0", "node_id": "node-02"}. When all shards in a batch are assigned, these binding relationships are submitted and persisted together, constituting part of the distributed storage node layout table.

[0133] The obtaining step of the to-be-migrated data block list is:

[0134] According to the data hierarchical composite index, the index entries are scanned one by one, the data temperature identifier and the corresponding data timestamp value in each index entry are extracted, the difference between the data timestamp value and the current system time value is calculated one by one, and if the data temperature identifier is hot data and the time difference value is greater than the preset cooling threshold, the entry is marked as a to-be-migrated index entry, and a to-be-migrated data block list is generated.

[0135] Specifically, according to the data hierarchical composite index, a timing task of the system background is triggered periodically, for example, once every hour, after the task is started, it systematically traverses all index entries starting from the root node of the index, in particular, it focuses on scanning entries containing " / hot_data / " in the path, for each such entry, the task extracts its associated data block index number, and queries an independent metadata database according to the number to obtain the timestamp value of the data block when it was originally generated, the timestamp is a Unix timestamp accurate to milliseconds, then the task obtains the standard UTC time value of the current system, and calculates the difference between the two timestamps to obtain a time span in seconds, then the task compares the calculated time difference with a preset cooling threshold, which is set to 2592000 seconds, i.e. 30 days, the setting of this threshold is based on the statistical analysis of the historical data access pattern in the foregoing, which clearly indicates that the data has the highest access frequency within 30 days after generation, and then it drops sharply, therefore, if the data temperature identifier parsed in the current index entry is indeed "hot data", and the calculated time difference is strictly greater than 2592000 seconds, the system determines that the data block has met the cooling condition, then the task marks the index entry (i.e. the index number of the data block) as a to-be-migrated object, and adds it to a temporary list stored in memory, when the timing task traverses all related index entries, the temporary list is solidified to form a to-be-migrated data block list.

[0136] The obtaining step of the index of the updated storage structure is:

[0137] Based on the to-be-migrated data block list, the data block index numbers in the to-be-migrated data block list are extracted one by one, the distributed storage node layout table is queried according to the data block index number, the binding relationship between each data block index number and the data shard and the storage node is determined, the storage node is located through the binding relationship and all data shard contents are read in sequence, the complete data content is reorganized according to the data shard sequence number, and the complete original data block is generated;

[0138] Based on the complete original data block, according to the preset cold data encoding strategy, each original data block is re-divided into multiple data shards, and the distributed storage nodes of the cold data storage pool are selected one by one to write the data shard contents, and the binding mapping relationship between each data shard and the written storage node is recorded, and the index of the updated storage structure is generated.

[0139] Specifically, based on the to-be-migrated data block list generated in the previous process, the system starts a data reorganization and recovery process, which processes each data block index number in the list one by one. For a number such as "BLK_20231201_1000_001", the system first uses this number as a query key to initiate a query request to the distributed storage node layout table. The layout table is a key-value storage, with the key being the data block index number and the value being a list containing all shards (including data shards and redundant shards) of the data block and their corresponding storage node identifiers. The query operation returns a structure similar to [{"shard_id": "...", "node_id": "..."},...]. The system parses this structure to determine the locations of all shards needed to reconstruct the data block, for example, it determines that data shards _ds_0 to _ds_5 are stored in nodes node-01 to node-06, and redundant shards _ps_0 to _ps_2 are stored in nodes node-07 to node-09. Then, the system initiates data read requests to these nodes in parallel, each request containing a specific shard identifier. After receiving the request, each storage node reads the corresponding shard content from the local disk and transmits it back to the data reorganization service through the network. The service waits for all shard data to arrive, or in the case of partial node failure, it uses redundant shards to recover the missing data shards through erasure code decoding algorithms. Once all six original data shards are collected, the system strictly follows the shard number order (from 0 to 5) to concatenate the contents of these shards in memory, for example, the contents of _ds_0 are placed at the head of the buffer, followed by the contents of _ds_1, and so on, finally seamlessly combining into a complete original data block identical to that before migration.

[0140] Based on the complete raw data block generated in the previous process, the system starts a data recoding and cold storage archiving process. First, the recovered raw data block is processed according to a preset cold data coding strategy, which is specified as RS(12, 2) in the system configuration, i.e., using the Reed-Solomon algorithm, the data block is divided into 12 data shards and 2 redundant shards. The selection of this strategy is based on cost-benefit considerations, because the cold data access frequency is very low, and appropriate reduction of redundancy can significantly save storage space. The system then re-divides the complete raw data block into 14 equal-length shards (12 data shards and 2 redundant shards) according to the strategy, and generates new shard identifiers for them. Next, the system obtains a node list marked as "cold data storage pool" from the cluster's node manager. This storage pool is composed of storage hardware with lower cost, larger capacity, but relatively slower I / O performance. The system uses polling to select storage nodes for the 14 new shards from the cold storage pool one by one, and ensures that different shards of the same data block are placed on different physical nodes. Then, the system sends write instructions to the selected cold storage nodes in turn, and writes the content of each newly generated shard to the disk of the target node. After each write operation is successfully completed, the system records the binding mapping relationship between the new identifier of the shard and the identifier of the cold storage node where it is written in real time, such as {"shard_id": "...", "node_id": "cold_node_01"}. When all 14 shards of an original data block are successfully written to the cold storage pool, all these newly generated binding mapping relationships are collected and updated to the system's metadata as a whole, forming the updated storage structure index.

[0141] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can modify or change the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still falls within the protection scope of the present application.

Claims

1. A distributed storage method for vehicle-mounted OBD data, characterized in that, Includes the following steps: Receive raw data blocks uploaded by the vehicle OBD terminal, parse DTC fault codes, vehicle speed, engine speed, gear information and timestamp, assign high criticality level identifiers to data blocks associated with rapid acceleration and deceleration events and DTC fault codes, and assign hot data or cold data identifiers based on the difference between the timestamp and the current time, and establish a data fidelity and criticality score set. Based on the data fidelity and criticality score set, the criticality level identifier, data temperature identifier and fidelity score of each data block are extracted and combined into a standardized multidimensional data feature vector. Then, according to the preset fidelity threshold, criticality level type and temperature type, the multidimensional data feature vector is mapped to the predefined and hierarchical index space address to establish a data hierarchical composite index. Based on the data hierarchical composite index, the data block encoding strategy selection instruction is obtained, the distributed storage system is scheduled to execute the data block encoding strategy selection instruction, the data block is sharded and redundantly encoded, and the encoded shards are written to the distributed storage nodes in different storage pools to generate a distributed storage node layout table. Start a background scheduled task to scan the timestamp information in the data hierarchical composite index, filter the index entries that have been cooled from hot data to cold data, summarize and generate a list of data blocks to be migrated, and establish an updated storage structure index based on the list of data blocks to be migrated and the distributed storage node layout table.

2. The distributed storage method for vehicle OBD data according to claim 1, characterized in that, The steps for obtaining the data fidelity and key score set are as follows: Extract the corresponding vehicle speed value, engine speed value, gear information, sensor self-test status value and data source reputation score from each raw data block. Input the engine speed value and gear information into the transmission ratio function and calculate the theoretical vehicle speed. Perform difference analysis between the theoretical vehicle speed and the vehicle speed value. At the same time, accumulate the abnormal count of the sensor self-test status value. Assign high criticality level identification to the data blocks associated with rapid acceleration and deceleration events and DTC fault codes to generate a set of parameters to be scored. Calculate the fidelity score for each data block based on the set of parameters to be scored; Based on the fidelity score, the criticality identifier of each data block, and the data temperature identifier, a data item identifier set is formed by combining them in sequence, and then grouped according to the index number of the original data block to generate a data fidelity and criticality score set.

3. The distributed storage method for vehicle-mounted OBD data according to claim 1, characterized in that, The steps for obtaining the multidimensional data feature vector are as follows: Based on the index number of each data item in the data fidelity and criticality score set, the criticality identifier, data temperature identifier, and fidelity score of the data item corresponding to the index number are extracted one by one. The criticality identifier is converted through binary encoding, and the data temperature identifier is converted into a standard value through the timeliness weight table to form an initial set of data feature values. Based on the initial set of data feature values, the fidelity threshold, key level type classification table, and temperature type classification table are called respectively. Classification is performed based on the difference between the fidelity score and the fidelity threshold. The key level identifier value is matched with the index position of the key level type classification table, and the data temperature identifier value is matched with the index position of the temperature type classification table. The matching results are combined to form a multidimensional data feature vector.

4. The distributed storage method for vehicle OBD data according to claim 1, characterized in that, The steps for obtaining the hierarchical composite index are as follows: Based on the multidimensional data feature vectors, the corresponding index space addresses are retrieved item by item according to the preset mapping rule table. The index space is located step by step using a step-by-step mapping method. The multidimensional data feature vectors are mapped and located to the addresses in the index space one by one, thus establishing a hierarchical composite index for data.

5. The distributed storage method for vehicle-mounted OBD data according to claim 1, characterized in that, The steps for obtaining the data block encoding strategy selection instruction are as follows: Based on the index space address of each index entry in the data hierarchical composite index, the data items corresponding to the address are retrieved one by one, the key level identifier and data temperature identifier contained in the data item are extracted, and the index matching conditions are formed based on the encoded value of the key level identifier and the classification value of the data temperature identifier, thereby generating a set of encoding strategy query conditions. Based on the set of query conditions for the coding strategy, the preset erasure coding strategy library is called one by one. The applicable key level identifier range and data temperature identifier range of each strategy in the erasure coding strategy library are compared one by one. Erasure coding strategies that meet the query conditions are filtered out, and the strategy index number of the erasure coding strategy that meets the conditions is determined to form an erasure coding strategy index set. Based on the erasure coding policy index set, erasure coding policy entries with corresponding policy index numbers are retrieved one by one from the erasure coding policy library. The encoding rules in the erasure coding policy entries are parsed, and the corresponding erasure coding instructions are output one by one according to the encoding rules to obtain the data block encoding policy selection instructions.

6. The distributed storage method for vehicle OBD data according to claim 1, characterized in that, The steps for obtaining the distributed storage node layout table are as follows: According to the data block encoding strategy, select instructions, perform equal-length splitting of data blocks according to the number of erasure coding fragments, map and bind each fragment identifier to the original data block identifier, and extract the total capacity value, available capacity value, current write rate value, historical write rate standard deviation value and current write task number of each candidate distributed storage node to generate a set of resource status of nodes to be scheduled. Based on the set of resource statuses of the nodes to be scheduled, calculate the node placement score for each storage node; Based on the node placement score of each storage node, all candidate nodes are sorted in descending order of node placement score. The nodes with the highest node placement scores are selected in sequence according to the data shard numbers. Each data shard is assigned to a matching node, and the binding relationship between the data shard and the storage node is recorded to generate a distributed storage node layout table.

7. The distributed storage method for vehicle OBD data according to claim 1, characterized in that, The steps for obtaining the list of data blocks to be migrated are as follows: Scan each index entry according to the data hierarchical composite index, extract the data temperature identifier and corresponding data timestamp value from each index entry, calculate the difference between the data timestamp value and the current system time value, and determine if the data temperature identifier is hot data and the time difference is greater than the preset cooling threshold. Then mark the entry as an index entry to be migrated and generate a list of data blocks to be migrated.

8. The distributed storage method for vehicle-mounted OBD data according to claim 1, characterized in that, The steps for obtaining the updated storage structure index are as follows: Based on the list of data blocks to be migrated, the index numbers of the data blocks in the list of data blocks to be migrated are extracted one by one. The distributed storage node layout table is queried one by one according to the index numbers of the data blocks. The binding relationship between the data shards and storage nodes corresponding to each data block index number is determined. The storage nodes are located one by one through the binding relationship and all data shard contents are read in sequence. The complete data content is reorganized according to the data shard order numbering to generate a complete original data block. Based on the complete original data block, according to the preset cold data encoding strategy, each original data block is re-divided into multiple data shards. Distributed storage nodes with the storage pool category of cold data storage pool are selected one by one to write the data shard content in sequence, and the binding mapping relationship between each data shard and the storage node written is recorded to generate an updated storage structure index.

Images