Data management method, database system, computer device, storage medium, program product and mobile platform

Abstract

The invention discloses a data management method, a database system, computer equipment, a storage medium, a program product and a mobile platform, which are used for embedded equipment, and the method comprises the following steps: compiling according to a frame data structure of a sensor and a user code in advance in a compiling period to obtain a database aggregation module; and the database aggregation module is adopted to perform data aggregation on the acquired sensor data and upload the data to the cloud. According to the application, compiling is carried out according to the frame data structure of the sensor and the user code in the compiling period to obtain the data aggregation module, and the data aggregation module is adopted to carry out data aggregation on the sensor data and upload the sensor data to the cloud during data acquisition. The data aggregation is transferred to the compiling period, and the pressure of a CPU and a memory in the running period is relieved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data management method, database system, computer equipment, storage medium, program product, and mobile platform. Background Technology

[0002] Data acquisition and storage are essential capabilities for every embedded device, especially in autonomous driving scenarios. Typically, an ECU (Electronic Control Unit) is connected to multiple sensor devices and control units to collect, calculate, and store large amounts of data in real time. CPU and memory resources on embedded devices are extremely sensitive. When the number of modules is large, each module has strict CPU utilization and memory usage thresholds to ensure that other functions can work in a coordinated manner. This places even stricter demands on data storage resources.

[0003] For data storage, the industry commonly uses traditional relational databases such as MySQL and SQLite as the underlying storage. These databases are characterized by row-based data storage and primarily address relationships and transactions between data, making them more suitable for OLTP (online transaction processing) scenarios. With the development of the internet, a large amount of data exhibits time-series characteristics, leading to the emergence of column-based databases specifically designed for time-series data, such as TDEngine and InfluxDB. These non-relational databases are generally suitable for OLAP (online analytical processing) scenarios.

[0004] In resource-constrained embedded devices, there is a significant amount of data acquisition work. The acquired data typically requires data insertion, processing, and disk storage. In security-related scenarios (e.g., autonomous driving), substantial computation is also required during data processing. Embedded devices are characterized by extremely sensitive CPU and memory resources, and the data acquired by them is time-series; therefore, the common practice is to use general-purpose time-series databases. Furthermore, data retrieval is generally not required on embedded devices.

[0005] Embedded devices typically contain data from multiple sensors. This sensor data is acquired on the device and then inserted into a time-series database. The time-series database performs aggregation and transformation operations on the data over a period of time. The industry standard practice is to perform row and column transformations on the sensor data according to the time sequence to achieve a better compression ratio and save storage space.

[0006] Insertion operations in time-series databases can use SQL statements as database table operation statements (e.g., create, insert, modify, and query). The SQL statement is first parsed into the corresponding C++ function, and then the corresponding function call is executed to complete the runtime data insertion function.

[0007] Database operations are performed using SQL statements, and the SQL statement parsing process is a runtime event. Describing database operation statements as strings involves statement parsing and conversion, which consumes significant CPU resources and impacts the performance of embedded devices. Summary of the Invention

[0008] This application provides a data management method, a database system, a computer device, a storage medium, a program product, and a mobile platform to at least solve one of the above-mentioned technical problems.

[0009] In a first aspect, embodiments of this application provide a data management method for an embedded device, the method comprising: compiling a database aggregation module in advance based on the sensor frame data structure and user code during compilation; and using the database aggregation module to aggregate the collected sensor data and upload it to the cloud.

[0010] In some embodiments, the database aggregation module includes frame data structure description information and table storage structure; the database aggregation module is pre-compiled based on the sensor's frame data structure and user code during compilation, including: the frame data structure for acquiring sensor data; Generate frame data structure description information based on the memory layout of the frame data structure; A table storage structure is constructed based on the frame data structure description information, and the table storage structure includes a compression processing module for compressing different time-series data.

[0011] In some embodiments, the frame data structure description information includes type information of various time-series data contained in the sensor data; The table storage structure is constructed based on the frame data structure description information, including configuring multiple compression processing modules according to the type information of various time-series data.

[0012] In some embodiments, the database aggregation module is used to aggregate the collected sensor data and upload it to the cloud, including: From the database aggregation module, obtain the target frame data structure description information and target table storage structure corresponding to the collected sensor data; Based on the target frame data structure description information, the time-series data in the collected sensor data is read in a columnar manner, and the read field data is written into the compression processing module of the target table storage structure; When the preset conditions are met, the target table storage structure for writing data will be uploaded to the cloud or processed for disk write and then uploaded to the cloud.

[0013] In some embodiments, the preset conditions can be configured as follows: the memory usage in the compression processing module reaches a set memory threshold, or the number of frames of the collected sensor data reaches a set frame number threshold, or an external event trigger signal is received.

[0014] In some embodiments, when preset conditions are met, the target table storage structure for which data is written is uploaded to the cloud or is written to disk and then uploaded to the cloud, including: During the mass production phase, the target table storage structure containing multiple written data is stored as a single database file, and the single database file is uploaded to the cloud. During the data acquisition phase, the target table storage structure containing the data is stored as multiple corresponding database files, and the multiple database files are uploaded to the cloud.

[0015] In some embodiments, the database aggregation module is used to aggregate the collected sensor data, which further includes: When the collected sensor data is array data, and the elements in the array data are time-series data, single-frame aggregation is performed on the array data; When the collected sensor data is array data, and the corresponding element data between multiple array data corresponding to multiple frames of sensor data is time-series data, the array data is aggregated across multiple frames.

[0016] In some embodiments, the method further includes: when data retrieval is required, performing data retrieval based on the frame data structure description information in the database aggregation module.

[0017] Secondly, embodiments of this application provide a database system including a device and a cloud, wherein the device is configured in write-only mode to execute the method described in any embodiment of this application; and the cloud is configured in read-only and / or read-write mode.

[0018] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the method described in any embodiment of this application.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program / instructions thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the method described in any embodiment of this application.

[0021] Sixthly, embodiments of this application provide a mobile platform, characterized in that it is equipped with the computer device described in any embodiment of this application.

[0022] This application compiles a data aggregation module based on the sensor's frame data structure and user code during the compilation phase. This module is then used to aggregate sensor data and upload it to the cloud during data acquisition. This shifts data aggregation to the compilation phase, reducing runtime CPU and memory pressure. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating an embodiment of the data management method of this application; Figure 2 This is a flowchart illustrating another embodiment of the data management method of this application; Figure 3 This is a schematic diagram of an embodiment of the table storage structure in this application; Figure 4 This is a schematic diagram of an embodiment of the table storage structure in this application; Figure 5 This is a flowchart illustrating another embodiment of the data management method of this application; Figure 6 This is a schematic flowchart illustrating a data management method according to an embodiment of this application, from table creation and database creation to data insertion. Figure 7 This is a flowchart illustrating a data management method from table creation and database creation to data insertion, according to another embodiment of this application. Figure 8 This is a schematic diagram of another embodiment of the data management method of this application; Figure 9 This is a schematic diagram of the single-frame aggregation structure in this application; Figure 10 This is a schematic diagram of the multi-frame aggregation structure in this application; Figure 11This is a schematic diagram of another embodiment of the data management method of this application; Figure 12 This is a schematic diagram of one embodiment of multi-table file storage and single-table multi-file storage in this application; Figure 13 This is a schematic diagram of an embodiment of the database system of this application; Figure 14 This is a schematic diagram of the structure of an embodiment of the computer device of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0026] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0027] In related technologies, embedded devices have extremely limited memory and CPU resources. Since columnar storage of frame data requires columnar access and row-column transformation of fields, such processing, if performed during program runtime, would consume significant CPU resources. However, programming languages ​​like C / C++ possess both compile-time and runtime characteristics. One of the innovations of this application is elevating columnar access and row-column transformation to compile-time. Compile-time refers to the process of compiling source code into executable code, while runtime refers to the actual execution of the executable code on the computer. In terms of performance, C++ is the preferred mainstream programming language for embedded devices. While existing databases in the industry provide C++ interfaces, they only offer runtime data operations. This application will fully utilize the compile-time capabilities of C++, significantly improving data insertion performance through the description of compile-time frame data structures.

[0028] like Figure 1 As shown, an embodiment of this application provides a data management method for an embedded device, the method comprising: S10. During the compilation phase, the database aggregation module is pre-compiled based on the sensor's frame data structure and user code.

[0029] S20. The collected sensor data is aggregated using the database aggregation module and uploaded to the cloud.

[0030] In this embodiment, a data aggregation module is compiled during the compilation phase based on the sensor's frame data structure and user code. This module is then used to aggregate sensor data and upload it to the cloud during data acquisition. This shifts data aggregation to the compilation phase, reducing CPU and memory pressure during runtime.

[0031] In some embodiments, the database aggregation module includes frame data structure description information and table storage structure. For example... Figure 2 The diagram shown is a flowchart of another embodiment of the data management method of this application. In this embodiment, a database aggregation module is pre-compiled based on the sensor's frame data structure and user code during compilation, including: S11. Obtain the frame data structure of the sensor data. The frame data structure includes information such as the data type definitions of the fields in the frame data. For example, using a C language structure as the frame data structure description, such as a frame data structure describing coordinates, its C language definition might be as follows: S12. Generate frame data structure description information based on the memory layout of the frame data structure.

[0032] For example, a frame data structure has a specific memory layout, and the description of this memory layout is called the frame data structure description information. This information describes the order, name, type, and offset of each field in the frame data structure. The description information can take various forms; one possible description information can be represented using C++ template pseudocode as follows: Here, Offsets corresponds to the offset of each field in the memory layout, and the last 24 represents the size occupied by the data structure; Types sequentially indicates the type order of the field members in the memory layout. The process of generating the description information is a fixed procedure that can be automated using existing related software tools, and this application does not limit it in this regard.

[0033] Since different protocols (e.g., Protobuf, JSON, etc.) have different memory layouts, but the description of the memory layout can be unified, this description information can describe the memory layout of multiple protocol languages, masking the differences between different protocols.

[0034] S13. Construct a table storage structure based on the frame data structure description information. The table storage structure includes a compression processing module for compressing different time-series data.

[0035] For example, the frame data structure description information includes the type information of various time-series data contained in the sensor data; constructing a table storage structure based on the frame data structure description information includes: configuring corresponding multiple compression processing modules according to the type information of various time-series data.

[0036] For example, the type information of various time series data includes, but is not limited to, longitude, latitude, and elevation. Correspondingly, three compression processing modules are configured for the three types of time series data, where the compression processing modules can employ encoders or streaming compressors.

[0037] In some embodiments, after obtaining the frame data structure description information, the columnar field accessor configured by the embedded device can be used to access the fields. For example, a Coord data instance is as follows: For its binary buffer representation, a possible pseudocode implementation of the columnar field accessor for accessing longitude is as follows, to achieve row-column transformation, where Description represents the Coord memory layout description mentioned above: For more complex and deeply nested protocol structures (such as the Protobuf protocol mentioned in the previous embodiments), data access can be performed through consecutive strides, for example: All of the above operations occur during the compile-time phase when the C++ compiler performs the build.

[0038] With the frame data structure description information describing the memory layout of the frame data at compile time, the table creation process of the database table storage structure can be completed at compile time; and with the columnar field accessor at compile time, columnar field access can be realized at compile time.

[0039] Based on the above frame data structure description information and compile-time information such as columnar field accessors, a possible pseudocode for database creation and table creation operations is as follows: In this context, italics indicate names or parameters that need to be specified. The Description is passed as a compile-time parameter to CREATE_TABLE, and the compile-time interface uses this parameter to create the table storage structure. Optionally, Description can be replaced with Coord, and the matching can be performed using C++ type lookup rules.

[0040] like Figure 3 The diagram shown is a schematic representation of an embodiment of the table storage structure in this application. Figure 3 As shown, the frame data structure description information is generated based on the sensor's frame data structure, and then further combined with user code (such as the pseudocode for database creation and table creation operations in the aforementioned embodiment) and input into the compiler to obtain the database aggregation module.

[0041] like Figure 4 The diagram shown is a schematic representation of an embodiment of the table storage structure in this application. This embodiment uses a frame data structure describing coordinates as an example. The frame data structure description information includes three types of time-series data: longitude, latitude, and elevation. Correspondingly, encoders are configured for each of the three types of time-series data to perform encoding and compression processing. The values ​​of each field are read using compile-time columnar field access and placed into the encoder of their respective fields, thus completing the row-column conversion and simultaneously performing the encoding and storage action. The table storage structure resides in memory, and encoding and storage can reduce memory space usage to some extent.

[0042] like Figure 5 The diagram shown is a flowchart illustrating another embodiment of the data management method of this application. In this embodiment, the database aggregation module is used to aggregate the collected sensor data and upload it to the cloud, including: S21. Obtain the target frame data structure description information and target table storage structure corresponding to the collected sensor data from the database aggregation module.

[0043] For example, the database aggregation module includes frame data structure description information and table storage structure for various sensor data. Sensors include, for example, positioning sensors, vision sensors, and millimeter-wave radar. Taking a positioning sensor as an example, the corresponding sensor data can include location data including longitude, latitude, and altitude. The corresponding frame data structure description information and table storage structure can be constructed in the database aggregation module; the specific construction method can be found in the aforementioned embodiments and will not be repeated here.

[0044] S22. Based on the target frame data structure description information, perform columnar reading of the time-series data in the acquired sensor data (e.g., columnar reading via a columnar field accessor), and write the read field data into the compression processing module of the target table storage structure. The compression processing module can employ an encoder or a streaming compressor. For example, the database aggregation module may also be configured with a field columnar accessor for columnar reading of the time-series data in the acquired sensor data.

[0045] by Figure 4 Taking the frame data structure describing coordinates as an example, the three types of time series data—longitude, latitude, and altitude—are read in a columnar manner, and the read field data is written into the corresponding encoder.

[0046] S23. When preset conditions are met, the target table storage structure for writing data is uploaded to the cloud or processed for disk write and then uploaded to the cloud. The preset conditions can be configured as follows: the memory usage in the compression processing module reaches a set memory threshold, or the number of frames of the collected sensor data reaches a set frame count threshold, or an external event trigger signal is received (such as a signal indicating that the user actively requests data). Disk write refers to writing the target storage structure for writing data into the memory of the embedded device.

[0047] In some embodiments, when preset conditions are met, the target table storage structure (data stored in memory) can be directly uploaded to the cloud, or the target table storage structure can be written to disk and then uploaded to the cloud.

[0048] In this embodiment, the target frame data structure description information and target table storage structure corresponding to the collected sensor data are obtained from the database aggregation module. Based on the target frame data structure description information, the time-series data in the collected sensor data is read columnarly, and the read field data is written to the compression processing module of the target table storage structure, achieving row-column conversion. The data aggregation module is compiled during compilation based on the sensor's frame data structure and user code. During data acquisition, this data aggregation module aggregates the sensor data and uploads it to the cloud. This shifts data aggregation to the compilation phase, reducing runtime CPU and memory pressure.

[0049] This application also provides an example of the entire data management process, from database creation and table creation to data insertion. For example... Figure 6 The diagram shown is a flowchart illustrating a data management method according to an embodiment of this application, from database creation and table creation to data insertion.

[0050] like Figure 6As shown, the process includes creating a database (CREATE_DATABASE), creating a table (CREATE_TABLE), inserting data (INSERT_INTO), columnar access and row / column conversion, filling the table storage structure, and compressing and writing the corresponding database to disk after the conditions are met.

[0051] The data insertion process involves columnar access, which allows the columnar reading of field members of the frame data structure (e.g., longitude and latitude fields). The data is then placed into the encoder of each field member. When specific conditions are met, the data in the encoder is compressed, and finally, it is stored on disk. Since frame data occurs at a specific frequency, this process repeats continuously. The triggering condition can be set to a specified memory usage in the encoder or a specified number of data frames.

[0052] like Figure 7 The diagram shown is a flowchart illustrating a data management method from database creation and table creation to data insertion, according to another embodiment of this application. Figure 6 The difference in the Chinese embodiment is that the encoder is replaced with a streaming compressor, thereby eliminating the encoding step.

[0053] In some embodiments, frame data exhibits temporal characteristics due to its unique frequency of occurrence. Temporal data refers to data representing the same attribute that changes linearly without sudden jumps. For example, the latitude and longitude information of a vehicle during its movement constitutes a frame, and this information changes over time. However, only data with the same semantic meaning (e.g., longitude or latitude) exhibits temporal characteristics; longitude and latitude each exhibit temporal characteristics separately. Therefore, storing the longitude and latitude of multiple frames of data separately is the process of columnar storage. The transformation from "row storage - multiple frames, multiple columns (latitude and longitude)" data to "column storage - multiple columns, multiple frames" data format is called row-column conversion. Because of the temporal characteristics of the converted data, related technologies have developed a series of encoding algorithms (such as DELTA, RLE, etc.), which help reduce the space occupied by temporal data.

[0054] In some embodiments, in order to support columnar field access to nested structures or data structures, the process from columnar access to instantiated table structure storage is a compile-time row-column transformation. The pseudocode for this part can be designed as follows: Here, `storage` is the (encoded or stream-compressed) storage result of the basic type `Fundamental`. `Stridable` is the representation after each `Stride` of the `Description Accessor`. This pseudocode fragment describes the row and column transformation logic for the basic type.

[0055] For Structural objects, the pseudocode algorithm for row and column transformation is as follows: Among them, storage <currentfield>The result is a Convert of type CurrentField, used to recursively convert the value of this field.

[0056] The algorithm for array type conversion is as follows: For variable-length arrays (referred to as flexible arrays in C), since the length of the data is not known until runtime, the row and column transformation operations must be performed at runtime. In this case, the storage must be able to handle data insertion at runtime. Handling variable-length structures involves runtime data. The variable-length length (Num) and underlying type (Type) can be obtained through `Stridable`. The flexible array is then dimensionality-reduced before proceeding internally. Row and column transformations and encoding compression are performed on the internal data. Due to its recursive process, the above algorithm can be repeated. The above code implementation is based on recursion. For one-dimensional variable-length arrays, an optimized algorithm can be designed to omit recursion, reducing compilation complexity.

[0057] like Figure 8 The diagram shown is a schematic representation of another embodiment of the data management method of this application. In this embodiment, the database aggregation module is used to aggregate the collected sensor data, and the method further includes: S24. When the collected sensor data is array data and the elements in the array data are time-series data, perform single-frame aggregation on the array data.

[0058] S25. When the collected sensor data is array data, and the corresponding element data between multiple array data corresponding to multiple frames of sensor data is time-series data, multi-frame aggregation is performed on the array data.

[0059] For example, since the row-column transformation process occurs in a multi-frame data aggregation scenario, when arrays are involved, if the data within the array exhibits temporal characteristics, the algorithm for row-column transformation of array-formatted data can be further divided into aggregation of array data within a single frame and aggregation of array data with the same index across multiple frames, referred to simply as single-frame aggregation and multi-frame aggregation. Single-frame aggregation and multi-frame aggregation save memory usage and storage space.

[0060] For example, for the following frame structure, when multiple frames of data arrive, the final results of single-frame aggregation and multi-frame aggregation are drastically different. Take two frames of this data structure as an example: like Figure 9 The diagram shown is a schematic diagram of the single-frame aggregation structure in this application. In the case of single-frame aggregation, the storage arrangement after aggregation is as follows: the data points [0] to [3] in single-frame data points_0 are continuous, and the data points [0] to [3] in single-frame data points_1 are continuous. Intra-frame aggregation is performed during aggregation, that is, single-frame aggregation.

[0061] like Figure 10 The diagram shown is a schematic diagram of the structure of multi-frame aggregation in this application. In the case of multi-frame aggregation, the storage arrangement after aggregation is as follows: the data points [0] to [3] in single-frame data points_0 are not continuous, the data points [0] to [3] in single-frame data points_1 are not continuous, but the data in the same position in single-frame data points_0 and single-frame data points_1 are continuous. Therefore, the 0th data point[0] of single frame data points_0 and the 0th data point[0] of single frame data points_1 are aggregated together; the 1st data point[1] of single frame data points_0 and the 1st data point[1] of single frame data points_1 are aggregated together; the 2nd data point[2] of single frame data points_0 and the 2nd data point[2] of single frame data points_1 are aggregated together; the 3rd data point[3] of single frame data points_0 and the 3rd data point[3] of single frame data points_1 are aggregated together, that is, multi-frame aggregation.

[0062] The main difference between single-frame aggregation and multi-frame aggregation lies in the data characteristics. Single-frame aggregation is suitable when the array within a single frame of data is continuous, such as points represented on a line. The continuity of the line provides an encoding condition for improving the compression ratio. Multi-frame aggregation is suitable when multiple points exhibit continuity across multiple frames of data (e.g., longitude and latitude data in coordinate frames). Based on the different data characteristics of the array and the continuity of these data characteristics, this application embodiment rationally selects either single-frame aggregation or multi-frame aggregation strategies, saving memory usage and storage space.

[0063] like Figure 11 The diagram shown is a schematic representation of another embodiment of the data management method of this application. In this embodiment, when a preset condition is met, the target table storage structure for writing data is uploaded to the cloud or processed for disk persistence and then uploaded to the cloud, including: S231. During the mass production stage, the target table storage structure containing multiple written data is stored as a single database file, and the single database file is uploaded to the cloud.

[0064] For example, in the field of intelligent driving, depending on the actual scenario, the data in the mass production stage is small, so multiple table storage structures are usually stored in a single file. That is, when the disk write trigger condition is met, all tables can be written to disk at the same time according to the business situation. Storing multiple tables in a single file is called multi-table file storage.

[0065] S232. During the data acquisition phase, the target table storage structure containing the multiple data entries is stored as multiple corresponding database files, and the multiple database files are uploaded to the cloud.

[0066] For example, in the field of intelligent driving, during the data acquisition phase, due to the huge amount of data, a single table usually corresponds to multiple data files, which is called single-table multi-file storage.

[0067] like Figure 12 The diagram shown is a schematic representation of one embodiment of multi-table file storage and single-table multi-file storage in this application. In multi-table file storage, tables 1 to 3 are simultaneously stored as a single database file to construct the database; in single-table multi-file storage, tables 1 to 3 are stored in different data files.

[0068] Based on different scenarios (such as the mass production stage and the data acquisition stage in the field of intelligent driving), this application's embodiments divide the database into two storage engines: single-table multi-file storage and multi-table multi-file storage, which reduces the use of locks and improves CPU execution efficiency.

[0069] Furthermore, in some embodiments, the disk write-to-disk method for single-table multi-file storage still requires locking during the disk write phase, depending on whether the disk write is performed via file memory mapping or via write operation. If the size of the encoded or compressed data can be predicted in advance, the disk write operation can be transformed from locking into an atomic operation.

[0070] The disk-based storage method for multiple form files is based on what was mentioned earlier. On the device side, there are usually no situations where multiple threads write to a single table. Typically, one thread inserts into one or more tables. In this case, the table can be regarded as thread-local (thread variable), which physically achieves isolation, so there is no need to add locks.

[0071] In some embodiments, the data management method further includes: when data retrieval is required, performing data retrieval based on the frame data structure description information in the database aggregation module.

[0072] Taking in-vehicle applications as an example, although data retrieval is not usually required, when it does occur, the database interface using the data management method of this application can perform data retrieval based on the frame data structure description information at compile time. For example, retrieval code such as the following can be implemented. Because the statement arrangement of this part of the code also occurs at compile time, the retrieval efficiency on embedded devices is improved.

[0073] Databases that can operate on embedded devices in related technologies possess a certain degree of versatility, fulfilling the functions of database insertion, deletion, modification, and querying. However, these databases parse SQL statements at runtime, converting them into program execution logic before completing the corresponding operations. Simultaneously performing data insertion, calculation, and querying on computationally intensive embedded devices consumes significant CPU resources, inevitably leading to CPU contention. To address this issue, this application further proposes a database system comprising a device and a cloud. The device is configured in write-only mode for executing the methods described in any embodiment of this application; the cloud is configured in read-only and / or read-write mode.

[0074] For example, the device end can be a vehicle, and the cloud end can be a cloud server. The vehicle is equipped with one or more embedded devices for collecting and storing vehicle data. As described in the data management method of the preceding embodiments, the database aggregation module ultimately aggregates the collected sensor data and uploads it to the cloud.

[0075] In the cloud, the database can achieve runtime SQL statement parsing capabilities based on compile-time statements. This part of the interface is consistent with the database implementation in the industry, so it will not be described in detail here.

[0076] like Figure 13 The diagram shown is a schematic representation of an embodiment of the database system of this application. It includes a device (configured with a data aggregation module), a cloud (which includes a database retrieval module and a database deletion / modification module), and a front-end (which includes a data display module). In this embodiment, data collection and aggregation are completed through the device segment, and then the aggregated data is uploaded to the cloud via a network or hard drive. The device segment is configured in read-only mode, and the cloud is configured in read-only and / or read-write mode.

[0077] The device only produces data, so it only performs the data aggregation part. Here, "aggregation" refers to the need to perform columnar access, row and column transformation, and encoding compression to improve compression efficiency.

[0078] The cloud is the final storage location for data. Devices typically transmit data via networks or hard drives, while the cloud handles data maintenance and retrieval.

[0079] Records of necessary data such as table structure metadata and table names used for database retrieval in the cloud. This information only needs to be stored in a metadata file and used for data selection and location during data retrieval.

[0080] Even if there are no resource limitations in the cloud, data deletion and modification are rare occurrences; therefore, the solution described in this application separates retrieval and deletion / modification. Optionally, data deletion / modification and retrieval can be combined into a single cloud module. This application does not limit this approach.

[0081] The front end communicates with the cloud. For example, the cloud uses data retrieval functions to select data and return it to the front end for display.

[0082] In some embodiments, when performing data deletion or modification operations in the cloud, the possible methods include, but are not limited to, the following: 1. Lazy deletion is achieved by using field tags so that the actual deletion of data can be completed in a later stage.

[0083] 2. When modifications are made, if the size of the modified column data is smaller than the original data size, it can be modified directly on the spot; if it is larger than the original data size, the data file needs to be recreated.

[0084] 3. Alternatively, the data file can be partitioned, with each partition having its own identifier. When a data column exceeds the existing data, the current partition can be discarded, and a new partition can be created at the end of the file, modifying the corresponding field. Note that when reading the data, the data index based on the partitions may need to be sorted once.

[0085] Based on the above embodiments, the database system of this application divides the database into three independent modes according to the specific business scenarios of data addition, data query and data deletion and modification, which physically decouples the complexity of data operation and improves resource utilization efficiency.

[0086] In some embodiments, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any embodiment of this application.

[0087] In some embodiments, this application also provides a computer-readable storage medium storing a computer program / instructions thereon, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0088] In some embodiments, this application also provides a computer program product, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0089] In some embodiments, this application also provides a mobile platform, characterized in that it is equipped with the computer device described in any embodiment of this application.

[0090] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0091] Figure 14 This is a schematic diagram of the hardware structure of a computer device for executing a data management method according to another embodiment of this application, as shown below. Figure 14 As shown, the device includes: One or more processors 1210 and memory 1220, Figure 14 Take the 1210 processor as an example.

[0092] The device for performing the data management method may further include an input device 1230 and an output device 1240.

[0093] The processor 1210, memory 1220, input device 1230, and output device 1240 can be connected via a bus or other means. Figure 14 Taking the example of a connection between China and Israel via a bus.

[0094] The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the data management method in the embodiments of this application. The processor 1210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1220, thereby implementing the data management method of the above-described method embodiments.

[0095] The memory 1220 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the data management device. Furthermore, the memory 1220 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include memory remotely located relative to the processor 1210, and these remote memories may be connected to the data management device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] Input device 1230 can receive input digital or character information and generate signals related to user settings and function control of the data management device. Output device 1240 may include display devices such as a display screen.

[0097] The one or more modules are stored in the memory 1220, and when executed by the one or more processors 1210, they perform the data management method in any of the above method embodiments.

[0098] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0099] The computer device in this application embodiment exists in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0100] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0101] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0102] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0103] (5) Other electronic devices with data interaction functions.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / currentfield>

Claims

1. A data management method for an embedded device, the method comprising: The database aggregation module is pre-compiled based on the sensor's frame data structure and user code during the compilation phase. The database aggregation module is used to aggregate the collected sensor data and upload it to the cloud.

2. The method according to claim 1, characterized in that, The database aggregation module includes frame data structure description information and table storage structure; The database aggregation module is pre-compiled during compilation based on the sensor's frame data structure and user code, including: Frame data structure for acquiring sensor data; Generate frame data structure description information based on the memory layout of the frame data structure; A table storage structure is constructed based on the frame data structure description information, and the table storage structure includes a compression processing module for compressing different time-series data.

3. The method according to claim 2, characterized in that, The frame data structure description information includes the type information of various time-series data contained in the sensor data; The table storage structure is constructed based on the frame data structure description information, including configuring multiple compression processing modules according to the type information of various time-series data.

4. The method according to claim 3, characterized in that, The database aggregation module is used to aggregate the collected sensor data and upload it to the cloud, including: From the database aggregation module, obtain the target frame data structure description information and target table storage structure corresponding to the collected sensor data; Based on the target frame data structure description information, the time-series data in the collected sensor data is read in a columnar manner, and the read field data is written into the compression processing module of the target table storage structure; When the preset conditions are met, the target table storage structure for writing data will be uploaded to the cloud or processed for disk write and then uploaded to the cloud.

5. The method according to claim 4, characterized in that, When preset conditions are met, the target table storage structure for writing data is uploaded to the cloud or processed for disk persistence and then uploaded to the cloud, including: During the mass production phase, the target table storage structure containing multiple written data is stored as a single database file, and the single database file is uploaded to the cloud. During the data acquisition phase, the target table storage structure containing the data is stored as multiple corresponding database files, and the multiple database files are uploaded to the cloud.

6. The method according to any one of claims 1-5, characterized in that, The database aggregation module is used to aggregate the collected sensor data, and the method further includes: When the collected sensor data is array data, and the elements in the array data are time-series data, single-frame aggregation is performed on the array data; When the collected sensor data is array data, and the corresponding element data between multiple array data corresponding to multiple frames of sensor data is time-series data, the array data is aggregated across multiple frames.

7. The method according to any one of claims 1-5, characterized in that, Also includes: When data retrieval is required, the data is retrieved based on the frame data structure description information in the database aggregation module.

8. A database system, characterized in that, The system includes a device and a cloud, wherein the device is configured in write-only mode for executing the method of any one of claims 1-7; and the cloud is configured in read-only and / or read-write mode.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

12. A mobile platform, characterized in that, The computer device of claim 9 is installed.

Images