Data storage method and system based on field-level associative storage

Abstract

The invention belongs to the technical field of automobile data storage, and provides a data storage method and system based on field-level association storage, and the method comprises the steps: building a storage structure for data to be stored based on fields in a distributed database, and associating the fields; pre-processing the associated storage data, and performing timing data compression and duplicate removal on the pre-processed storage data; and optimizing a query path for the preprocessed storage data based on index establishment and a caching technology. According to the method, the data structure is remarkably simplified by adopting a field-level associated storage mode. According to the method, a lengthy and complex table structure in a traditional data model is disassembled, fields are used as basic units for storage, and an efficient association index is established according to the logic relation between the fields. The data storage efficiency is greatly improved, the data writing and reading speed is remarkably increased, and a solid foundation is laid for subsequent data processing and analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile data storage, and in particular relates to a data storage method and system based on field-level associative storage. Background Art

[0002] With the rapid development of electric vehicle technology and the widespread application of Internet of Vehicles technology, electric vehicles are generating and accumulating massive amounts of data at an unprecedented rate during daily operation. These data not only cover the basic status information of the vehicle, such as battery power, motor working conditions, and body system health, but also record the vehicle's driving trajectory in detail, including key parameters such as location information, speed changes, and mileage. In addition, energy consumption information, as an important part of the operating cost of electric vehicles, is also recorded in detail, including energy consumption differences under different driving modes, energy recovery efficiency, etc. More importantly, the instant feedback of fault alarm data provides the first line of defense for the safe operation of the vehicle, and can promptly detect and warn of potential technical problems.

[0003] The value of these data resources for the all-round management of electric vehicles cannot be underestimated. In terms of vehicle management, they provide fleet operators with comprehensive monitoring methods, which help to achieve remote vehicle dispatch, accurate maintenance plan formulation, and significant improvement in operational efficiency. In the field of fault prediction, through in-depth mining and analysis of historical data, an efficient prediction model can be built to identify components that are about to fail in advance, so as to take preventive measures to avoid operational interruptions and increased maintenance costs caused by sudden failures. In terms of energy consumption optimization, based on the results of big data analysis, driving strategies can be optimized and energy management system parameters can be adjusted to achieve the purpose of energy conservation, emission reduction, and extended driving range.

[0004] However, in the face of such a large and complex data set, existing data storage methods are facing severe challenges. Traditional data architectures are often designed to be complex and difficult to adapt to rapidly changing data types and scales, resulting in inefficient data processing. During the query process, due to unreasonable data organization and slow retrieval speed, the timeliness and accuracy of data analysis are seriously affected. At the same time, the problem of data redundancy is prominent, which not only takes up a lot of storage space, but also increases the difficulty and cost of data management, which is not conducive to efficient large-scale data processing and in-depth analysis. Summary of the invention

[0005] The object of the present invention is to provide a data storage method and system based on field-level associative storage to solve the problems of poor timeliness and accuracy in the prior art, as well as difficulty and high cost of data management.

[0006] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a data storage method based on field-level associative storage, comprising: In a distributed database, the data to be stored is stored in a storage structure based on fields, and the fields are associated; Preprocess the associated stored data, and perform regular data compression and deduplication on the preprocessed stored data; The query path is optimized for the preprocessed stored data based on indexing and caching technology.

[0007] Optionally, in a distributed database, a storage structure is constructed based on fields for the data to be stored, including: The data to be stored is divided into fields for basic vehicle information, real-time driving data, and fault alarm information. Use data modeling tools or database management systems to define the data storage structure, including field names, data types, and constraint rules. Create corresponding data tables and indexes in the distributed database based on the defined storage structure.

[0008] Optionally, basic vehicle information includes vehicle ID, brand, and model; real-time driving data includes speed, location, and power; and fault alarm information includes fault code and alarm time.

[0009] Optionally, associate fields, including: In a distributed database, create an independent data table or collection for each field and associate them through field IDs or keywords.

[0010] Optionally, the associated stored data is preprocessed, including: Data cleaning, filtering and normalization; For real-time data, edge computing technology is used for processing and preliminary analysis, and edge computing nodes are deployed on the vehicle side or roadside equipment to perform real-time data compression and feature extraction processing on the data.

[0011] Optionally, perform periodic data compression and deduplication on the preprocessed stored data, including: Use hash algorithms or data fingerprint technology to uniquely identify data and remove duplicate data; Data compression technology using column storage or data encoding. Column storage stores data of the same field continuously to improve data compression ratio; data encoding technology converts data to reduce storage space occupancy.

[0012] Optionally, the query path is optimized for the preprocessed stored data based on indexing and caching techniques, including: Indexes are created for commonly used query fields, and caching technology is used to cache hot data in memory to reduce disk I / O operations and support fast retrieval, grouping, sorting, and connection operations based on the field level.

[0013] In a second aspect, the present invention provides a data storage system based on field-level associative storage, comprising: A storage structure building module is used to build a storage structure based on fields for the data to be stored in a distributed database and associate the fields; A preprocessing module is used to preprocess the associated stored data, and perform timed data compression and deduplication on the preprocessed stored data; The optimized query module is used to optimize the query path of the pre-processed storage data based on indexing and caching technology.

[0014] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data storage method based on field-level associative storage when executing the computer program.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a data storage method based on field-level associative storage are implemented.

[0016] Compared with the prior art, the present invention has the following technical effects: The present invention achieves a significant simplification of the data structure by adopting a field-level associative storage method. The lengthy and complex table structure in the traditional data model is disassembled, stored with fields as basic units, and an efficient associative index is established based on the logical relationship between the fields. This storage method not only reduces the complexity of the data model, but also greatly improves the efficiency of data storage, significantly improves the data writing and reading speed, and lays a solid foundation for subsequent data processing and analysis.

[0017] In terms of optimizing query performance, an efficient query mechanism is designed. This mechanism fully utilizes the advantages of field-level associative storage and realizes the rapid positioning and retrieval of target data by building an intelligent query path. At the same time, advanced indexing technology and caching strategies are introduced to further shorten the query response time and ensure the efficiency and accuracy of data retrieval. This optimization measure is of great significance for improving the real-time and decision-making support capabilities of the electric vehicle data management system.

[0018] In order to reduce data redundancy, advanced data deduplication and compression technologies are used. During the data writing process, the system will deduplicate new data to avoid the storage of duplicate data. At the same time, an efficient compression algorithm is applied to compress the stored data, thereby effectively reducing data redundancy and saving storage space. This measure not only reduces data storage costs, but also improves the efficiency and reliability of data management.

[0019] In addition, in order to enhance the scalability of the system, distributed database technology is adopted. This technology distributes data storage and management tasks to multiple database nodes, realizing distributed storage and parallel processing of data. This architecture can not only support the storage and management of large-scale data, but also be flexibly expanded according to business needs to meet the growing data processing and analysis needs of the electric vehicle data management system in the future. By adopting distributed database technology, the scalability and stability of the electric vehicle data management system are ensured, providing strong technical support for the sustainable development of the electric vehicle industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention.

[0021] Figure 2 This is a logic block diagram of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 1 , a data storage method based on field-level associative storage, comprising: In a distributed database, the data to be stored is stored in a storage structure based on fields, and the fields are associated; Preprocess the associated stored data, and perform regular data compression and deduplication on the preprocessed stored data; The query path is optimized for the preprocessed stored data based on indexing and caching technology.

[0023] The present invention achieves a significant simplification of the data structure by adopting a field-level associative storage method. The lengthy and complex table structure in the traditional data model is disassembled, stored with fields as basic units, and an efficient associative index is established based on the logical relationship between the fields. This storage method not only reduces the complexity of the data model, but also greatly improves the efficiency of data storage, significantly improves the data writing and reading speed, and lays a solid foundation for subsequent data processing and analysis.

[0024] Example 2, please refer to Figure 2 The present invention discloses a data storage method based on field-level associative storage, which specifically includes: S1: Data storage structure definition: Define the field-level structure of electric vehicle Internet of Vehicles data, including basic vehicle information (such as vehicle ID, brand, model), real-time driving data (such as speed, location, power), fault alarm information (such as fault code, alarm time) and other fields.

[0025] Each field has a clear data type and constraint rules to ensure data consistency and accuracy.

[0026] S2: Field-level associative storage: The electric vehicle Internet of Vehicles data is stored in an associative manner at the field level, that is, the data of each field is stored independently and associated through field IDs or keywords.

[0027] Distributed database technology, such as NoSQL databases (such as MongoDB, Cassandra, etc.), is used to store data on multiple nodes, and high availability and redundant backup of data are achieved through data distribution and replication. Each node is responsible for storing data of some fields and maintaining data consistency and synchronization through internal mechanisms.

[0028] S3: Data preprocessing: Before data storage, the data is preprocessed, including data cleaning, filtering and normalization, to ensure data quality.

[0029] For real-time data, edge computing technology is used for processing and preliminary analysis. Edge computing nodes are deployed on the vehicle side or roadside equipment to process data in real time, such as data compression and feature extraction, to reduce data transmission delays and network bandwidth pressure.

[0030] S4: Query optimization: Provides an efficient data query mechanism that supports fast retrieval, grouping, sorting, and connection operations based on the field level.

[0031] Optimize query performance by building indexes and caching technologies. Establish indexes for commonly used query fields to increase query speed; at the same time, use caching technology to cache hot data in memory to reduce disk I / O operations and further improve query efficiency.

[0032] S5: Data compression and deduplication: Deduplication of duplicate data is performed to save storage space. Hash algorithms or data fingerprint technology are used to uniquely identify data and remove duplicate data.

[0033] Data compression technologies, such as column storage and data encoding, are used to reduce the amount of data during data transmission and storage. Column storage stores data of the same field continuously to improve the data compression ratio; data encoding technology converts data to reduce storage space usage.

[0034] Embodiment 3, S1: Data storage structure definition implementation: Use a data modeling tool or a database management system to define a data storage structure, including field names, data types, constraint rules, etc. According to the defined storage structure, create corresponding data tables and indexes in a distributed database.

[0035] S2: Field-level associative storage implementation: In a distributed database, create an independent data table or collection for each field and associate them through field IDs or keywords. Implement data distribution and replication mechanisms to ensure high availability and redundant backup of data.

[0036] S3: Data preprocessing implementation: Use data processing tools or programming languages ​​(such as Python, Java, etc.) to write data preprocessing scripts, including logic such as data cleaning, filtering, and normalization. Deploy preprocessing scripts on edge computing nodes to process and analyze real-time data.

[0037] S4: Query optimization implementation: Create indexes in distributed databases to improve query speed. Use cache technologies such as Redis and Memcached to cache hot data in memory and reduce disk I / O operations.

[0038] S5: Data compression and deduplication implementation: Use data deduplication tools or algorithms, such as hash tables, Bloom filters, etc. to deduplicate data. Use data compression technologies, such as column storage, data encoding, etc., to reduce the amount of data during data storage and transmission.

[0039] Taking the real-time driving data of electric vehicles as an example, the specific implementation steps are as follows: S1: Define data fields: Define fields including vehicle ID, timestamp, speed, location, power, etc., and clarify the data type and constraint rules of each field.

[0040] S2: Data collection and preprocessing: Collect real-time driving data through vehicle sensors and Internet of Vehicles devices, and perform cleaning, filtering and normalization.

[0041] S3: Field-level associative storage: The processed data is stored in an associative manner at the field level. For example, fields such as vehicle ID, timestamp, speed, location, and power are stored in different data tables and associated through vehicle ID and timestamp.

[0042] S4: Data query and analysis: Provides a field-level query interface to support users to quickly retrieve and analyze data according to their needs. For example, users can query the driving trajectory and energy consumption information of a specific vehicle by vehicle ID and time range.

[0043] S5: Data compression and deduplication: Regularly deduplicate stored data and use data compression technology to reduce storage space usage.

[0044] In yet another embodiment of the present invention, a data storage system based on field-level associative storage is provided, which can be used to implement the above-mentioned data storage method based on field-level associative storage. Specifically, the system includes: A storage structure building module is used to build a storage structure based on fields for the data to be stored in a distributed database and associate the fields; A preprocessing module is used to preprocess the associated stored data, and perform timed data compression and deduplication on the preprocessed stored data; The optimized query module is used to optimize the query path of the pre-processed storage data based on indexing and caching technology.

[0045] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0046] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or a corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a data storage method based on field-level associative storage.

[0047] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of a data storage method based on field-level associative storage in the above embodiment.

[0048] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0050] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0051] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A data storage method based on field-level associative storage, characterized in that: include: In a distributed database, the data to be stored is stored in a storage structure based on fields, and the fields are associated; Preprocess the associated stored data, and perform regular data compression and deduplication on the preprocessed stored data; The query path is optimized for the preprocessed stored data based on indexing and caching technology.

2. A data storage method based on field-level associative storage according to claim 1, characterized in that: In a distributed database, the data to be stored is stored in a structure based on fields, including: The data to be stored is divided into fields for basic vehicle information, real-time driving data, and fault alarm information. Use data modeling tools or database management systems to define the data storage structure, including field names, data types, and constraint rules. Create corresponding data tables and indexes in the distributed database based on the defined storage structure.

3. A data storage method based on field-level associative storage according to claim 2, characterized in that: Basic vehicle information includes vehicle ID, brand, and model; real-time driving data includes speed, location, and power; and fault alarm information includes fault code and alarm time.

4. The data storage method based on field-level associative storage according to claim 1, characterized in that: Associate fields, including: In a distributed database, create an independent data table or collection for each field and associate them through field IDs or keywords.

5. The data storage method based on field-level associative storage according to claim 1, characterized in that: Preprocess the associated stored data, including: Data cleaning, filtering and normalization; For real-time data, edge computing technology is used for processing and preliminary analysis, and edge computing nodes are deployed on the vehicle side or roadside equipment to perform real-time data compression and feature extraction processing on the data.

6. The data storage method based on field-level associative storage according to claim 1, characterized in that: Perform regular data compression and deduplication on the preprocessed stored data, including: Use hash algorithms or data fingerprint technology to uniquely identify data and remove duplicate data; Data compression technology using column storage or data encoding. Column storage stores data of the same field continuously to improve the data compression ratio; data encoding technology converts data to reduce storage space occupancy.

7. The data storage method based on field-level associative storage according to claim 1, characterized in that: Optimize the query path of preprocessed stored data based on indexing and caching technology, including: Indexes are created for commonly used query fields, and caching technology is used to cache hot data in memory to reduce disk I / O operations and support fast retrieval, grouping, sorting, and connection operations based on the field level.

8. A data storage system based on field-level associative storage, characterized in that: include: A storage structure building module is used to build a storage structure based on fields for the data to be stored in a distributed database and associate the fields; A preprocessing module is used to preprocess the associated stored data, and perform timed data compression and deduplication on the preprocessed stored data; The optimized query module is used to optimize the query path of the pre-processed storage data based on indexing and caching technology.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data storage method based on field-level associative storage as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a data storage method based on field-level associative storage as described in any one of claims 1 to 7 are implemented.

Images