Internet of Vehicles data storage method and device, equipment and medium

Abstract

The invention discloses an Internet of Vehicles data storage method, apparatus and device, and a medium, and relates to the technical field of data processing, the method comprises the steps of obtaining to-be-processed data according to a preset data access rule in response to a data access request; a plurality of dynamically adjustable data circulation elements are configured in the data access rule; identifying a data source of the to-be-processed data, and converting a data format of the to-be-processed data into a preset format according to the data source; preprocessing the to-be-processed data after format conversion to obtain target data; the target data is data of which the data structure and content both meet business requirements; the target data is classified according to a preset data classification storage rule, the classified target data is stored in the distributed storage system, and multiple pieces of dynamic adjustable service scene information are configured in the data classification storage rule. According to the method and the device, the dynamic adaptability can be improved, and data redundancy and repeated storage conditions are effectively avoided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for storing Internet of Vehicles data. Background Art

[0002] With the rapid development of the Internet of Vehicles (IOV) system, the Internet of Vehicles (IOV), as an integrated network that can realize intelligent traffic management, intelligent dynamic information services and vehicle control, has been increasingly used in the field of transportation systems. Among them, in the Internet of Vehicles environment, vehicles will generate a large amount of Internet of Vehicles data, such as real-time dynamic data, sensor data, user instructions, etc. These data types are rich and diverse in format, and increase exponentially with the growth of the number of vehicles. Therefore, in order to ensure the safety of vehicle use, how to effectively access and store Internet of Vehicles data is crucial.

[0003] At present, the relevant technology uses a fixed format to access data, and all accessed data is uniformly stored in the database through a centralized storage architecture. However, this solution is difficult to cope with the changing needs of different vehicle models, controllers and businesses, resulting in poor dynamic adaptability, and unable to dynamically adjust data storage according to business needs, which easily leads to data redundancy and repeated storage. Summary of the invention

[0004] The purpose of this application is to provide a method, device, equipment and medium for storing Internet of Vehicles data.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for storing Internet of Vehicles data, comprising:

[0007] In response to a data access request, the data to be processed is obtained according to a preset data access rule; the data access rule is configured with a plurality of dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use;

[0008] Identifying a data source of the data to be processed, and converting a data format of the data to be processed into a preset format according to the data source;

[0009] Preprocessing the data to be processed after the format conversion to obtain target data; the target data is data whose data structure and content meet the business requirements;

[0010] According to preset data classification and storage rules, the target data is classified and stored in a distributed storage system, wherein the data classification and storage rules are configured with a plurality of dynamically adjustable business scenario information.

[0011] Optionally, the data access rule further includes: a field mapping rule and a data offset rule; and the preprocessing of the format-converted data to be processed includes:

[0012] Determine the to-be-processed field from the to-be-processed data after the conversion according to the field mapping rule; the to-be-processed field is a field in the to-be-processed data that needs to be mapped to the business data item;

[0013] Performing data cleaning and type conversion processing on the field to be processed to obtain first data;

[0014] If the first data does not conform to the data offset rule, the first data is used as the target data;

[0015] If the first data meets the data offset rule, an offset is obtained; and the first data is adjusted according to the offset to obtain second data as the target data.

[0016] Optionally, the data access rule further includes: a data cleaning rule and a data conversion rule; performing data cleaning and type conversion processing on the field to be processed to obtain the first data includes:

[0017] Perform data cleaning on the to-be-processed field according to the data cleaning rule to obtain cleaned data; the data cleaning process includes at least one of: filtering invalid data and correcting abnormal data;

[0018] The cleaned data is converted according to the data conversion rule to obtain the first data; the conversion process includes at least one item: unit conversion and type conversion.

[0019] Optionally, the data access rule further includes: a conditional trigger rule, after converting the data format of the to-be-processed data into a preset format, the method further includes:

[0020] According to the condition triggering rules, determine whether the preset conditions are met;

[0021] When the preset condition is met, the field mapping rule is triggered according to the preset condition.

[0022] Optionally, the classified target data includes a target field and a time period corresponding to the target field, and the target data is classified according to a preset data classification storage rule, and the classified target data is stored in a distributed storage system, including:

[0023] Classify the target data according to the data source to obtain primary classified data;

[0024] Using a business scenario detection model to detect and process the primary classification data, determine the target fields required for each business scenario in corresponding business analysis and the time period corresponding to the target fields;

[0025] The target field is stored in the distributed storage system according to the business scenario; the distributed storage system is configured with a storage time, and the storage time is determined according to the time period.

[0026] Optionally, the primary classification data is detected and processed using a business scenario detection model to determine the target field required for each business scenario in the corresponding business analysis and the time period corresponding to the target field, including:

[0027] Using a business scenario detection model to perform feature extraction processing on the primary classification data to obtain target features corresponding to each business scenario;

[0028] The target features are predicted and processed to determine the target fields required for each business scenario in the corresponding business analysis and the time period corresponding to the target fields.

[0029] Optionally, the method further comprises:

[0030] Acquire data access configuration information and data classification storage information; the data access configuration information includes at least one of the following: data transmission protocol, data source information, and data format information;

[0031] The data access rules are configured according to the data access configuration information through the data access configuration database, and the data classification storage rules are configured according to the data classification storage information.

[0032] In a second aspect, the present application provides a storage device for Internet of Vehicles data, including:

[0033] An acquisition module, used to respond to a data access request and acquire the data to be processed according to a preset data access rule; the data access rule is configured with a plurality of dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use;

[0034] A format conversion module, used to identify a data source of the data to be processed, and convert the data format of the data to be processed into a preset format according to the data source;

[0035] A preprocessing module is used to preprocess the data to be processed after the format conversion to obtain target data; the target data is data whose data structure and content meet the business requirements;

[0036] The classification storage module is used to classify the target data according to preset data classification storage rules, and store the classified target data in a distributed storage system. The data classification storage rules are configured with multiple dynamically adjustable business scenario information.

[0037] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned methods for storing Internet of Vehicles data.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for storing Internet of Vehicles data.

[0039] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0040] The present application provides a storage method, device, equipment and medium for Internet of Vehicles data, which obtains the data to be processed in accordance with preset data access rules in response to data access requests, wherein the data access rules are configured with multiple dynamically adjustable data circulation elements, identifies the data source of the data to be processed, and converts the data format of the data to be processed into a preset format according to the data source, then pre-processes the data to be processed after the conversion to obtain target data, classifies the target data according to preset data classification storage rules, and stores the classified target data in a distributed storage system, wherein the data classification storage rules are configured with multiple dynamically adjustable business scenario information. Compared with the prior art, in this solution, since multiple dynamically adjustable data flow elements are configured in the data access rules, it is possible to flexibly receive data from different data flow elements according to the access rules after responding to a data access request, and the data flow elements can be dynamically adjusted according to the changing requirements of different vehicle models, controllers or businesses, thereby improving dynamic adaptability; and since multiple dynamically adjustable business scenarios are configured in the data classification storage rules, it is possible to flexibly classify and process the target data according to the business scenarios, and then store the classified data in a distributed storage system. At the same time, the data classification storage rules can be flexibly adjusted according to business needs, which can effectively avoid the occurrence of data redundancy and repeated storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is an application environment diagram of a method for storing Internet of Vehicles data in one embodiment of the present application;

[0043] Figure 2 A schematic diagram of a flow chart of a method for storing Internet of Vehicles data provided in one embodiment of the present application;

[0044] Figure 3 A schematic flow chart of a method for preprocessing the data to be processed after the format conversion to obtain the target data provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the process structure of a method for storing Internet of Vehicles data provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of functional modules of a storage device for Internet of Vehicles data provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] As mentioned in the background technology, the related technology uses predefined data types and formats generated by vehicles, accesses data through fixed interfaces, and stores all accessed data in a unified database. However, this solution is difficult to cope with the changing needs of different vehicle models, controllers, and businesses, resulting in poor dynamic adaptability, and cannot be dynamically adjusted according to business needs, which easily leads to data redundancy or repeated storage.

[0051] Based on the above-mentioned defects, a method for storing Internet of Vehicles data is provided in an embodiment of the present application. Compared with the prior art, in this solution, since a plurality of dynamically adjustable data flow elements are configured in the data access rules, it is possible to flexibly receive data from different data flow elements in accordance with the access rules after responding to a data access request, and the data flow elements can be dynamically adjusted according to the changing requirements of different vehicle models, controllers or businesses, thereby improving dynamic adaptability; and since a plurality of dynamically adjustable business scenarios are configured in the data classification storage rules, it is possible to flexibly classify and process the target data according to the business scenarios, and then store the classified data in a distributed storage system. At the same time, the data classification storage rules can be flexibly adjusted according to business needs, which can effectively avoid the occurrence of data redundancy and repeated storage.

[0052] The storage method of the Internet of Vehicles data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can generate a data access request and send the data access request to the server 104, so that the server 104 responds to the data access request, obtains the data to be processed according to the preset data access rules, and identifies the data source, format conversion and preprocessing operations of the data to be processed to obtain the target data, and then classifies the target data, and sends the classified target data to the data storage system according to the preset data classification storage rules. The data storage system may include: a distributed storage system.

[0053] The above-mentioned preset data access rules and data classification storage rules may be customized by the user in the terminal 102 according to actual needs.

[0054] Optionally, the terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0055] In an exemplary embodiment, Figure 2 As shown, a method for storing vehicle network data is provided. The method is executed by a computer device, and specifically can be executed separately by a computer device such as a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 204. Among them:

[0056] Step 201, in response to a data access request, obtain the data to be processed according to the preset data access rules; the data access rules are configured with multiple dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use.

[0057] It should be noted that the above-mentioned data access request can be sent by each terminal, or by each module in a terminal, for example, it can be sent by the message middleware kafka. The data access request is used to characterize the need to perform a data access operation. The data access request includes the data to be processed. The data to be processed can be data in any format, for example, it can be in text form, picture form or table form. This embodiment does not impose any restrictions on the data format of the data to be processed. The above-mentioned data access rules can be customized according to actual needs, and can flexibly receive data from different data circulation elements without the need to frequently modify the code. It can be configured with multiple dynamically adjustable data circulation elements, which can include, for example: data source information, data transmission protocol, data format information, communication method, field description, etc.

[0058] It is understandable that the above data source refers to the source of data, which can be a database, file, sensor, network service, etc. The database can be, for example, a relational database MySQL, Oracle, a non-relational database MongoDB, etc.; the file can include: CSV file, XML file, JSON file, etc., the sensor data can include: data collected by temperature sensors, pressure sensors, etc.; network services can include data obtained through API interfaces, etc. The above data format refers to the organization and storage method of data, which can include JSON, XML, CSV, etc., and can also include text format (such as TXT), table format (such as CSV-comma separated values), markup language format (such as XML-extensible markup language, HTML-hypertext markup language), binary format (such as PDF), serialization format (such as JSON-JavaScript object representation), etc. Different data formats are suitable for different scenarios. For example, JSON format is widely used in data transmission of Web applications because it is easy to read and parse. The transmission protocol is used to transmit data in the network, for example, it can include HTTP hypertext transfer protocol, HTTPS secure hypertext transfer protocol, FTP file transfer protocol, SMTP mail transfer protocol, MQTT protocol, etc. Different transmission protocols correspond to different data transmission.

[0059] After receiving the data access request, the computer device responds to the data access request and obtains the data to be processed according to the preset data access rules. The data to be processed may include data in different data formats from different data sources, or data transmitted through different data transmission protocols.

[0060] The data flow elements of the data access rules in this embodiment are dynamically adjustable and can be customized according to actual needs, allowing the device to adapt to changes in different data sources and data formats or new data sources and data formats, thereby flexibly receiving data from different data flow elements and improving dynamic adaptability.

[0061] Step 202: Identify the data source of the data to be processed, and convert the data format of the data to be processed into a preset format according to the data source.

[0062] Specifically, after obtaining the data to be processed, the data source of the data to be processed can be identified. Since the data in each data source has one or more corresponding data formats, the data format can be converted into a preset format according to the data access rules. The preset format is the format corresponding to the business requirements.

[0063] Exemplarily, when the data format of the acquired data to be processed is in JSON format, it organizes the data in a key-value pair manner. Since the current data format does not meet the business storage requirements, it needs to be format converted to a preset format, such as converted to CSV format.

[0064] In this step, by identifying the data source of the data to be processed and converting the data format of the data to be processed into a preset format according to the data source, subsequent classified storage can be facilitated, thereby improving data storage efficiency.

[0065] Step 203, pre-processing the format-converted data to be processed to obtain target data; the target data is data whose data structure and content meet the business requirements.

[0066] It should be noted that the target data may include data structure and data content, the data structure includes data format, field requirements, etc., and the data content includes specific data information. The preprocessing of the converted data may include: field mapping, data cleaning, data conversion, data offset adaptation, etc.

[0067] In one embodiment, the above data access rules also include: field mapping rules and data offset rules. Figure 3 As shown, the embodiment of the present application provides a specific implementation method for preprocessing the data to be processed after the conversion format to obtain the target data, and the method includes:

[0068] Step 301 , determining the to-be-processed fields from the to-be-processed data after the format conversion according to the field mapping rule; the to-be-processed fields are fields in the to-be-processed data that need to be mapped to the business data items.

[0069] Step 302: Perform data cleaning and type conversion on the field to be processed to obtain first data.

[0070] Step 303: When the first data does not comply with the data offset rule, the first data is used as the target data.

[0071] Step 304: when the first data meets the data offset rule, obtain the offset, and adjust the first data according to the offset to obtain the second data as the target data.

[0072] It can be understood that the above field mapping rules are used to define which fields in the accessed data to be processed need to be mapped to business data items; since the original values ​​in the acquired data to be processed may not meet business requirements, they need to be dynamically adjusted to meet business requirements; the data offset rule defines the data offset adjustment of the data to be processed to meet business requirements.

[0073] Specifically, after the format of the data to be processed is converted, the fields to be processed are determined from the data to be processed after the format conversion according to the field mapping rules, and then the fields to be processed are cleaned according to the data cleaning rules to obtain the cleaned data, and the data cleaning process includes at least one item: filtering invalid data, correcting abnormal data; and the cleaned data is converted according to the data conversion rules to obtain the first data, and the conversion process includes at least one item: unit conversion, type conversion. Then, according to the data offset rule, it is determined whether the first data complies with the rule. When it does not comply with the data offset rule, the first data is used as the target data; when it complies with the data offset rule, the offset corresponding to the first data is obtained according to the data offset rule, and then the first data is adjusted according to the offset to obtain the second data as the target data. Among them, the data conversion rule is used to define the unit conversion or type conversion of the data, and the above offset is customized for each data according to actual needs, which can be adjusted dynamically.

[0074] Exemplarily, after the format conversion of the data to be processed, the sensorvalue in the format-converted data to be processed can be mapped to the temperature field in the business data item, and then invalid data can be filtered or abnormal data can be corrected according to the data cleaning rules, for example, null values ​​can be removed, and then the timestamps with inconsistent formats can be corrected, etc. For the actual data, the data can be processed by filling default values, using mean or median filling, etc. to obtain the cleaned data, and the cleaned data can be converted into units or types according to the data conversion rules to obtain the first data. Among them, the field type in the data source may be different from the type required in the business system, and automatic type conversion can be achieved through the data conversion rules. The field type of the cleaned data can include strings, numbers, Boolean values, etc. For example, the field timestamp in the cleaned data is a string ("2024-12-07T10:00:00Z"), and the business data item needs to convert it into a timestamp (such as Unix time 1701933600).

[0075] After determining the first data, it can be determined whether it meets the data offset rule. When it meets the data offset rule, the first data is dynamically offset and adapted. For example, the first data includes the temperature value of the sensor. The temperature value of the sensor may be inaccurate and cannot be used for subsequent data analysis. It needs to be corrected. The offset can be obtained first, and then the temperature value of the sensor is added or subtracted by a certain offset to obtain the target data. The target data can be corrected to the actual temperature. For example, when the value of the temperature field temp is the reading of the original sensor, 5 needs to be subtracted to obtain the actual temperature. Therefore, temp_actual=temp-5 is configured in the data offset rule to dynamically implement the offset adaptation of the data, thereby obtaining the target data.

[0076] Optionally, the above data access rules also include: conditional triggering rules. After converting the data format of the data to be processed into a preset format, the above method also includes: judging whether the preset conditions are met according to the conditional triggering rules; when the preset conditions are met, triggering the field mapping rules according to the preset conditions.

[0077] It is understandable that the above-mentioned conditional triggering rules refer to the conditions that trigger the corresponding rules when they are met. Specific mapping logic or processing logic can be triggered according to preset conditions, such as dynamically adjusting the mapping method according to the data source. The preset condition can be, for example, when a certain parameter value is greater than a threshold. When the preset condition is met, the field mapping rule is triggered according to the preset condition. It can be understood that when a certain parameter value is greater than the threshold, the conditional mapping rule is triggered.

[0078] In this embodiment, by setting field mapping rules, data integration can be achieved, which facilitates subsequent data processing and analysis, helps to standardize data from different systems, and improves data relevance and consistency; by setting data cleaning rules, data quality can be improved; by setting data conversion rules, it can adapt to different data processing needs and promote data fusion and sharing; by setting conditional trigger rules, corresponding operations can be automatically triggered according to specific conditions to reduce manual intervention and improve processing efficiency; by setting data offset rules, the flexibility of processing time series data can be improved, adapting to differences in data collection and storage, ensuring the temporal consistency and accuracy of data, and facilitating subsequent data analysis and application.

[0079] Step 204, classify the target data according to the preset data classification storage rules, and store the classified target data in the distributed storage system, wherein the data classification storage rules are configured with multiple dynamically adjustable business scenario information.

[0080] It should be noted that the above data classification storage rules can be customized according to actual needs, for example, they can include classified storage according to different data sources, classified storage according to business scenarios, or classified storage according to data sources and business scenarios. The above distributed storage system can include multiple storage nodes, which are the basic components of the distributed storage system. Each storage node contains a certain storage medium, such as a hard disk, a solid-state drive, etc. The storage node is responsible for storing data blocks or data shards.

[0081] Specifically, after acquiring the target data, the target data is classified and processed, for example, according to the data source or business scenario, to obtain the target data, and then a hash algorithm or a consistent hash algorithm is used to store the data on different storage nodes in the distributed storage system.

[0082] In this embodiment, by storing the classified data in a distributed storage system, compared with a traditional centralized storage system, it is possible to utilize resources of multiple storage nodes and has higher storage capacity, better scalability and stronger fault tolerance.

[0083] The present application provides a storage method for Internet of Vehicles data, which obtains the data to be processed in accordance with preset data access rules in response to a data access request, wherein the data access rules are configured with a plurality of dynamically adjustable data circulation elements, identifies the data source of the data to be processed, and converts the data format of the data to be processed into a preset format according to the data source, and then pre-processes the converted data to obtain target data, classifies the target data according to preset data classification storage rules, and stores the classified target data in a distributed storage system, wherein the data classification storage rules are configured with a plurality of dynamically adjustable business scenario information. Compared with the prior art, in this solution, since multiple dynamically adjustable data flow elements are configured in the data access rules, it is possible to flexibly receive data from different data flow elements according to the access rules after responding to a data access request, and the data flow elements can be dynamically adjusted according to the changing requirements of different vehicle models, controllers or businesses, thereby improving dynamic adaptability; and since multiple dynamically adjustable business scenarios are configured in the data classification storage rules, it is possible to flexibly classify and process the target data according to the business scenarios, and then store the classified data in a distributed storage system. At the same time, the data classification storage rules can be flexibly adjusted according to business needs, which can effectively avoid the occurrence of data redundancy and repeated storage.

[0084] In one embodiment, the process of classifying the target data according to the preset data classification storage rule and storing the classified target data in the distributed storage system includes:

[0085] The target data is classified and processed according to the data source to obtain primary classified data; the primary classified data is detected and processed using a business scenario detection model to determine the target fields and the time period corresponding to the target fields required for each business scenario in the corresponding business analysis; the target fields are stored in a distributed storage system according to the business scenario; the distributed storage system is configured with storage time, which is determined according to the time period.

[0086] In this embodiment, the primary classified data is the data after the target data is classified according to the data source, and the classified target data is the data classified according to the business scenario. The above-mentioned business scenario detection model can be an algorithm model based on machine learning, such as decision trees, random forests, neural networks, etc., or it can be a rule-based expert system. The business scenario detection model needs to be trained and optimized to predict the target field and the corresponding time period. For example, for an e-commerce business scenario, historical order data, user behavior data, etc. can be used for model training to allow the model to learn the target fields and time period characteristics required for different business analyses.

[0087] Specifically, the target data may include multiple different data sources. The target data is classified and processed according to the data sources to obtain primary classified data. For example, the primary classified data includes: vehicle signal data, mobile phone APP logs, IDS security log data, Internet of Vehicles vehicle remote control data, etc., and then further classified according to business scenarios, and then the classified target data is stored in a distributed storage system.

[0088] The above-mentioned classified target data includes target fields and time periods corresponding to the target fields. In the process of determining the target fields and time periods corresponding to the target fields required for each business scenario in the corresponding business analysis, the business scenario detection model may be used to perform feature extraction processing on the classified target data to obtain the target features corresponding to each business scenario; the target features may be predicted to determine the target fields and time periods corresponding to the target fields required for each business scenario in the corresponding business analysis.

[0089] It is understandable that when log data is stored, a large wide table is usually designed to meet the needs of various business data fields. In this way, the table space and data volume will be huge. In actual business scenarios, the field ranges and time periods required for different business scenarios are also different. For example, in specific log analysis scenarios, the required field ranges are usually small and fixed. Therefore, it is necessary to determine the target fields and corresponding time periods required for business analysis based on the business scenario.

[0090] Specifically, the above-mentioned business scenario detection model may include a convolution layer and a fully connected layer. The convolution layer in the business scenario detection model may be used to perform feature extraction processing on the primary classification data to obtain the target features corresponding to each business scenario, and then the target features may be predicted and processed through the fully connected layer to obtain the target fields and the time period corresponding to the target fields required for each business scenario in the corresponding business analysis. The target field may be a field associated with the business scenario, and the time period is used to reflect the characteristics of the storage time, for example, it may be 7 days. After determining the target field and time period for each business scenario, the target field may be stored separately in a distributed storage system as a data slice according to the business scenario. Then, a storage time is set for each target field so that when the data analysis module needs to perform data analysis, data can be extracted and analyzed from a specific data table according to the storage time, thereby improving data processing efficiency.

[0091] In this embodiment, a business scenario detection model is used to perform feature extraction processing on the classified target data, the target features corresponding to each business scenario are obtained, and the target features are predicted, so that the target fields and the time periods corresponding to the target fields required for each business scenario in the corresponding business analysis can be accurately determined, and then stored in a distributed storage system, thereby improving the reading efficiency of high-frequency access data.

[0092] In one embodiment, the present application also provides a specific implementation method for configuring data access rules and data classification storage rules, the method comprising:

[0093] Obtain data access configuration information and data classification storage information; the data access configuration information includes at least one of the following: data transmission protocol, data source information, and data format information; through the data access configuration database, configure data access rules according to the data access configuration information, and configure data classification storage rules according to the data classification storage information. The above-mentioned data classification storage information may include business scenarios.

[0094] Please refer to Figure 4. Figure 4 The schematic diagram of the data dynamic access and classified storage process provided in the embodiment of the present application, the computer device may include a data access configuration database and a distributed storage system, and the data access configuration information is accessed and configured in the data access configuration database, and the classified storage is configured and processed. For example, data access configuration information may be obtained, including data transmission protocol, data source information, data format information or other communication methods, and then the data interface protocol and authentication configuration may be performed on the access adapter, thereby configuring the data access rules, and the data conversion and parsing process may be configured with data format and rules, the data mapping process may be configured with data dictionary and mapping, and the classified storage process may be configured with data usage scenarios and classification rules according to the data classification storage information, thereby sequentially configuring the field mapping rules, data cleaning rules, data conversion rules, conditional triggering rules, data offset rules and data classification storage rules.

[0095] When receiving a data access request, in response to the data access request, data from different data sources is obtained, such as data from kafka, File, MQTT or other data, and connected to the adapter, and data conversion and parsing are performed through pre-configured data cleaning rules and data conversion rules to obtain the first data, and data mapping is performed according to field mapping rules and data offset rules to obtain target data, and then the target data is classified according to the data classification and storage rules, and the classified target data is stored in the distributed storage system. The distributed storage system supports tiered storage of hot and cold data, which can improve the reading efficiency of high-frequency access data, and supports modular design and seamless expansion when adding new controllers or business scenarios.

[0096] In this embodiment, by configuring data access rules, data integrity can be ensured, and the data access rules are dynamically adjustable, so that the system can be compatible with multiple data sources and improve the adaptability of the system; by setting classification storage rules, data can be classified and stored according to certain rules, which is convenient for rapid positioning and retrieval of data, and the classification storage rules can store the data in a suitable storage medium or storage architecture according to the characteristics of the data, thereby realizing the optimal configuration of storage resources. Through dynamic access and classified storage of Internet of Vehicles data, the flexibility of data access management is greatly improved, and the efficiency of Internet of Vehicles data management is further improved.

[0097] Based on the same inventive concept, the embodiment of the present application also provides a storage device for implementing the above-mentioned Internet of Vehicles data. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the one or more embodiments of the storage device for Internet of Vehicles data provided below can refer to the limitations of the storage method for Internet of Vehicles data above, and will not be repeated here.

[0098] In an exemplary embodiment, Figure 5 As shown, a storage device for Internet of Vehicles data is provided, comprising:

[0099] The acquisition module 510 is used to respond to the data access request and acquire the data to be processed according to the preset data access rules; the data access rules are configured with multiple dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use;

[0100] The format conversion module 520 is used to identify the data source of the data to be processed and convert the data format of the data to be processed into a preset format according to the data source;

[0101] The preprocessing module 530 is used to preprocess the data to be processed after the format conversion to obtain target data; the target data is data whose data structure and content meet the business requirements;

[0102] The classification storage module 540 is used to classify the target data according to the preset data classification storage rules, and store the classified target data in the distributed storage system. The data classification storage rules are configured with multiple dynamically adjustable business scenario information.

[0103] As an optional implementation, the preprocessing module 530 is specifically configured to:

[0104] Determine the to-be-processed field from the to-be-processed data after the conversion format according to the field mapping rule; the to-be-processed field is a field in the to-be-processed data that needs to be mapped to the business data item;

[0105] Perform data cleaning and type conversion on the field to be processed to obtain first data;

[0106] When the first data does not comply with the data offset rule, the first data is used as the target data;

[0107] When the first data meets the data offset rule, the offset is obtained; and the first data is adjusted according to the offset to obtain the second data as the target data.

[0108] As an optional implementation, the preprocessing module 530 is further configured to:

[0109] Perform data cleaning on the fields to be processed according to the data cleaning rules to obtain cleaned data; the data cleaning process includes at least one of: filtering invalid data and correcting abnormal data;

[0110] The cleaned data is converted according to the data conversion rules to obtain first data; the conversion process includes at least one of: unit conversion and type conversion.

[0111] As an optional implementation, the above device is also used for:

[0112] According to the condition triggering rules, determine whether the preset conditions are met;

[0113] When the preset conditions are met, the field mapping rules are triggered according to the preset conditions.

[0114] As an optional implementation, the classification storage module 540 is specifically used for:

[0115] Classify and process the target data according to the data source to obtain primary classified data;

[0116] Use the business scenario detection model to detect and process the primary classification data, and determine the target fields and the time period corresponding to the target fields required for each business scenario in the corresponding business analysis;

[0117] The target fields are stored in a distributed storage system according to the business scenario; the distributed storage system is configured with a storage time, which is determined based on a time period.

[0118] As an optional implementation, the classification storage module 540 is further used for:

[0119] Use the business scenario detection model to perform feature extraction on the primary classification data to obtain the target features corresponding to each business scenario;

[0120] Perform prediction processing on the target features to determine the target fields and the time period corresponding to the target fields required for each business scenario in the corresponding business analysis.

[0121] As an optional implementation, the above device is also used for:

[0122] Acquire data access configuration information and data classification storage information; the data access configuration information includes at least one of the following: data transmission protocol, data source information, and data format information;

[0123] The database is configured through data access, data access rules are configured according to the data access configuration information, and data classification storage rules are configured according to the data classification storage information.

[0124] Among them, the storage device for Internet of Vehicles data provided in the embodiment of the present application, since a plurality of dynamically adjustable data flow elements are configured in the data access rules, can flexibly receive data from different data flow elements in accordance with the access rules after responding to a data access request, and can dynamically adjust the data flow elements according to the changing requirements of different vehicle models, controllers or businesses, thereby improving dynamic adaptability; and since a plurality of dynamically adjustable business scenarios are configured in the data classification storage rules, it is possible to flexibly classify and process the target data according to the business scenarios, and then store the classified data in a distributed storage system, and at the same time, the data classification storage rules can be flexibly adjusted according to business needs, which can effectively avoid the occurrence of data redundancy and repeated storage.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for storing vehicle network data is implemented.

[0126] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0128] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0129] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0132] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for storing Internet of Vehicles data, characterized in that: The storage method of the Internet of Vehicles data includes: In response to a data access request, the data to be processed is obtained according to a preset data access rule; the data access rule is configured with a plurality of dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use; Identifying a data source of the data to be processed, and converting a data format of the data to be processed into a preset format according to the data source; Preprocessing the data to be processed after the format conversion to obtain target data; the target data is data whose data structure and content meet the business requirements; According to preset data classification and storage rules, the target data is classified and stored in a distributed storage system, wherein the data classification and storage rules are configured with a plurality of dynamically adjustable business scenario information.

2. The method for storing Internet of Vehicles data according to claim 1, characterized in that: The data access rules also include: field mapping rules and data offset rules; The preprocessing of the format-converted data to be processed comprises: Determine the to-be-processed field from the to-be-processed data after the conversion according to the field mapping rule; the to-be-processed field is a field in the to-be-processed data that needs to be mapped to the business data item; Performing data cleaning and type conversion processing on the field to be processed to obtain first data; If the first data does not conform to the data offset rule, taking the first data as the target data; If the first data meets the data offset rule, an offset is obtained; and the first data is adjusted according to the offset to obtain second data as the target data.

3. The method for storing Internet of Vehicles data according to claim 2, characterized in that: The data access rule also includes: a data cleaning rule and a data conversion rule; performing data cleaning and type conversion processing on the field to be processed to obtain the first data includes: Perform data cleaning on the to-be-processed field according to the data cleaning rule to obtain cleaned data; the data cleaning process includes at least one of: filtering invalid data and correcting abnormal data; The cleaned data is converted according to the data conversion rule to obtain the first data; the conversion process includes at least one item: unit conversion and type conversion.

4. The method for storing Internet of Vehicles data according to claim 2, characterized in that: The data access rule also includes: a conditional trigger rule. After converting the data format of the to-be-processed data into a preset format, the method further includes: According to the condition triggering rules, determine whether the preset conditions are met; When the preset condition is met, the field mapping rule is triggered according to the preset condition.

5. The method for storing Internet of Vehicles data according to claim 1, characterized in that: The classified target data includes a target field and a time period corresponding to the target field, and the target data is classified and processed according to a preset data classification storage rule, and the classified target data is stored in a distributed storage system, including: Classify the target data according to the data source to obtain primary classified data; Using a business scenario detection model to detect and process the primary classification data, determine the target fields required for each business scenario in corresponding business analysis and the time period corresponding to the target fields; The target field is stored in the distributed storage system according to the business scenario; the distributed storage system is configured with a storage time, and the storage time is determined according to the time period.

6. The method for storing Internet of Vehicles data according to claim 5, characterized in that: The primary classification data is detected and processed using a business scenario detection model to determine the target fields required for each business scenario in the corresponding business analysis and the time period corresponding to the target fields, including: Using a business scenario detection model to perform feature extraction processing on the primary classification data to obtain target features corresponding to each business scenario; The target features are predicted and processed to determine the target fields required for each business scenario in the corresponding business analysis and the time period corresponding to the target fields.

7. The method for storing Internet of Vehicles data according to claim 1, characterized in that: The method further comprises: Acquire data access configuration information and data classification storage information; the data access configuration information includes at least one of the following: data transmission protocol, data source information, and data format information; The data access rules are configured according to the data access configuration information through the data access configuration database, and the data classification storage rules are configured according to the data classification storage information.

8. A storage device for Internet of Vehicles data, characterized in that: The storage device for the Internet of Vehicles data includes: An acquisition module, for responding to a data access request and acquiring the data to be processed according to a preset data access rule; the data access rule is configured with a plurality of dynamically adjustable data flow elements; the data flow elements are used to characterize the key components of the flow process from data generation to use; A format conversion module, used to identify a data source of the data to be processed, and convert the data format of the data to be processed into a preset format according to the data source; A preprocessing module is used to preprocess the data to be processed after the format conversion to obtain target data; the target data is data whose data structure and content meet the business requirements; The classification storage module is used to classify the target data according to preset data classification storage rules, and store the classified target data in a distributed storage system. The data classification storage rules are configured with multiple dynamically adjustable business scenario information.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for storing Internet of Vehicles data described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for storing vehicle network data described in any one of claims 1 to 7 are implemented.

Images