A sensor data organization management method and system

Abstract

The application provides a sensor data organization management method and system, comprising: when sensor data collected by a sensor carried in a unmanned device needs to be written into the unmanned device, writing the obtained sensor data into a sensor data file; when the unmanned device needs to access the sensor data, reading the sensor data file based on a retrieval condition set by an application program, and returning the read data to the application program. The data organization management is based on each original data of the sensor, the attributes corresponding to the original data and a header file. The application realizes the organization and management of the sensor data by designing a data organization method without modifying a file system and without installing additional software, supports the quick retrieval and positioning of the data, and does not increase the running cost of the system too much.

Description

Technical Field

[0001] This invention relates to the field of sensor technology, and more specifically to a sensor data organization and management method and system. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, concepts such as the Internet of Things (IoT), smart manufacturing, and autonomous driving have begun to enter people's field of vision. Generally speaking, whether it's the IoT or autonomous driving, the realization of these technologies requires a large amount of hardware and technical support, among which sensors are the most crucial. In autonomous driving technology, the onboard computer uses various sensors to measure and control various information such as temperature, pressure, position, rotational speed, acceleration, and vibration in real time and accurately. The key to measuring the level of the control system lies in the number and quality of the sensors. Currently, an ordinary family car has approximately 200 sensors, while an autonomous vehicle has more than 400. With the development of technology, the number of sensors required for autonomous driving will continue to increase, and the required level of technology will become increasingly demanding.

[0003] As the entry point for information collection, sensors and their technologies are the "eyes," "ears," and "skin" of the Internet of Things, smart industry, smart devices, and autonomous driving. They are responsible for transmitting the data and information that they "see," "hear," and "feel" to the "brain" for decision-making and system control. Therefore, the organization and management of sensor data will have a significant impact on the entire system application, especially in the field of unmanned system applications such as "human-out-of-the-loop."

[0004] Traditional sensor data management is mostly based on databases, with some relying on file systems. For database-based sensor data organization and management, applications format and store data for each sensor by creating tables for easy searching and location. This approach requires database software support and suffers from efficiency degradation when performing multi-table searches and data fusion. Data transfer between different databases involves format conversion, import / export operations, and most databases only support one data type, such as structured data or key / value data, offering insufficient support for mixed-structure data. File system-based data organization and management does not require additional database support but lacks indexing support and cannot perform multi-file joint searches. For small drones and unmanned vehicles carrying multiple sensors, power and space limitations prevent the installation of conventional industrial control computers, servers, and PCs; they typically use small development boards. However, the processing power of these development boards is far weaker than that of PCs, making it impractical to install and run large databases or multiple databases on them. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings, this invention proposes a sensor data organization and management method, comprising:

[0006] When the sensors on the unmanned device collect data that needs to be written back to the unmanned device: the acquired sensor data is written to the sensor data file;

[0007] When an unmanned device needs to access data from its onboard sensors: based on the search criteria set in the application, the device reads the data file corresponding to the sensor according to the search bar, and returns the read data to the application.

[0008] Preferably, writing the acquired sensor data into a sensor data file includes:

[0009] Step SW1: Access the data file corresponding to the sensor based on the data interface;

[0010] Step SW2: Based on the raw sensor data, various attribute data, and index information, call the data writing interface;

[0011] Step SW3: Locate the original address space array corresponding to each type of data in the header of the data file and the length of data already written for each data type. Calculate the data blocks to be allocated for writing each type of data this time, as well as the address space data blocks to be allocated for managing these data blocks. Then, allocate these data blocks and address space data blocks to the corresponding address space metadata area for management.

[0012] Step SW4: Sequentially select one data type from the data types and perform the write operation until all data types have been written.

[0013] Step SW5: Continue executing the process from Step SW2 to Step SW4 until the sensor is turned off or no more data needs to be written.

[0014] Preferably, accessing the data file corresponding to the sensor based on the data interface includes:

[0015] Determine if the data file corresponding to the sensor exists;

[0016] When it exists, access the data file corresponding to the sensor, and write the attribute information to the header of the data file;

[0017] If the sensor does not exist, create a data file corresponding to the sensor, set the logical block size of the file, and write the sensor attribute information into the header of the created data file.

[0018] Preferably, the write operation includes:

[0019] Find the location of the original array in the address space corresponding to this type of data in the file header and the length of data of this type that has been written. Then, according to the 4-level addressing scheme, parse the location of the data to be written in the address space of this type of data.

[0020] Based on the length of the data to be written of the data type and the distribution of the data blocks allocated for this write, write the data of this type into the sensor data file;

[0021] Update the length information of the data type described in the data file header to complete this write operation for this data type.

[0022] Preferably, the 4-level addressing scheme includes:

[0023] The first level uses direct addressing, while the second, third, and fourth levels all use indirect addressing.

[0024] Preferably, each code in the first level of encoding corresponds to a specific data block;

[0025] The second, third, and fourth levels of encoding are all used to point to a data block address;

[0026] The data block pointed to by the second-level code is used to store multiple first-level codes;

[0027] The data block pointed to by the third-level code is used for multiple second-level codes in memory;

[0028] The data block pointed to by the fourth-level code is used to store multiple third-level codes.

[0029] Preferably, the index information includes time index information and spatial index;

[0030] The data type includes at least one or more of the following: raw data and attributes.

[0031] Preferably, the step of reading the data file corresponding to the sensor based on the search bar includes:

[0032] Open the data file corresponding to the sensor based on the outer data interface;

[0033] Based on the data interface, the sensor data file is accessed to obtain list information;

[0034] The sensor data file is read based on the list information.

[0035] Preferably, the step of opening the data file corresponding to the sensor based on the outer data interface further includes:

[0036] Reading the sensor data file will stop if the data file corresponding to the sensor does not exist.

[0037] Preferably, accessing the sensor data file to obtain list information includes:

[0038] Determine whether to submit search criteria:

[0039] When search criteria are submitted, the sensor data file is searched by type according to the search criteria to obtain the first list information;

[0040] When no retrieval information is submitted: the second list information is determined by calculating the entire sensor data file based on the reading range and the address space of the original data;

[0041] The first list information includes: address, length; the second list information includes: offset, length;

[0042] The search criteria include attribute information and index information.

[0043] Preferably, the step of searching the sensor data file by type according to the search criteria to obtain the first list information includes:

[0044] Based on the search criteria for each type, the corresponding address space metadata is found from the type information in the header of the data file. Then, each address space is searched to obtain a list of the locations of the original data that meet the search criteria.

[0045] Based on the relationship between the search criteria, logical calculations are performed on the lists returned for each type of data to obtain the address and length of the original data that meets all the search criteria in the entire sensor data file, which is used as the first list information.

[0046] Preferably, the step of calculating and determining the second list information for the entire sensor data file based on the reading range and the address space of the original data includes:

[0047] Address conversion is performed based on the reading range and the address space of the raw data. The offset and length of the required data in the entire sensor data are calculated as the second list information.

[0048] Preferably, the data file corresponding to the sensor is constructed based on the original data of the sensor, the attributes corresponding to the original data, and the header file.

[0049] Preferably, the data file includes: a header file, multiple raw data and address space data, attribute address space data and attribute data corresponding to each raw data;

[0050] The header file includes: the size of the logic block, the number of sensor attributes, the raw data address space data corresponding to each raw data, the raw data length, the attribute name, and the attribute address space metadata;

[0051] The size of the logic block can be dynamically set or fixed.

[0052] Based on the same inventive concept, this invention also provides a sensor data organization and management system, including: a data management layer, a sensor device layer, a user application layer, and a file system layer; the sensor device layer includes various sensor devices; the file system layer is used to store data files corresponding to various sensors; the application layer includes various applications.

[0053] The data management layer includes: an outer data interface, a read module, and a write module;

[0054] The outer layer number interface is used to obtain sensor data from the sensor device layer through the user application layer;

[0055] The reading module is invoked when the unmanned device needs to access the sensor data it carries. The reading module is used to read the data file corresponding to the sensor based on the search conditions set by the application and according to the search bar, and return the read data to the application.

[0056] The writing module is invoked when the sensors on the unmanned device collect data that needs to be written to the unmanned device. The application calls the writing module to write the acquired sensor data into the sensor data file.

[0057] The beneficial effects of this invention are as follows:

[0058] This invention provides a sensor data organization and management method and system. Based on sensor data, it enables the organization and management of various sensor data using a small development board without requiring any modification to the file system. It is also independent of a specific file system and provides a certain data retrieval capability for sensor data without the need for a large database. Furthermore, it can accelerate the location of the required data without traversing the sensor data.

[0059] The technical solution provided by this invention can quickly and effectively access data when the single-frame sensor data is large, and the effect is more outstanding.

[0060] The technical solution provided by this invention can efficiently organize three different types of data—metadata, index information, and raw data—into the same file;

[0061] The technical solution provided by this invention effectively manages the address space of different types of data within a file without causing chaos, effectively solving the problem of sensor data storage and management based on a file system.

[0062] The technical solution provided by this invention, through a data organization and management method based on logic blocks, allows users to flexibly adjust and configure logic blocks according to the size of the data from the sensors being processed, thereby increasing data storage and retrieval efficiency.

[0063] The present invention provides an unmanned device that can organize and manage sensor data using a data organization method on a portable small development board, supporting rapid data retrieval and positioning, without requiring additional database support and without increasing the system's operating overhead. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of a sensor data organization and management method provided by the present invention;

[0065] Figure 2 This is a schematic diagram of the sensor data storage hierarchy provided by the present invention;

[0066] Figure 3 This is a schematic diagram of the sensor data reading hierarchy provided by the present invention;

[0067] Figure 4 This is a schematic diagram of the operation process for writing sensor data to a file provided by the present invention;

[0068] Figure 5 This is a schematic diagram of the operation process for reading sensor data from a file provided by the present invention;

[0069] Figure 6 This is a schematic diagram of the file structure provided by the present invention;

[0070] Figure 7 This is a schematic diagram of the file header structure provided by the present invention;

[0071] Figure 8 It is the total length of the code required when encoding the address space in one dimension;

[0072] Figure 9 It is the total length of the code required when encoding the address space using equal-length blocks;

[0073] Figure 10 This is the total length of the encoding required when encoding the address space using a 4-level addressing scheme.

[0074] Figure 11 This is the data structure corresponding to the file header provided by this invention;

[0075] Figure 12 This is a schematic diagram of the sensor data file structure after the initial writing of 19402 bytes of content into the attributeless sensor data file provided by the invention.

[0076] Figure 13 This is a schematic diagram of the sensor data file structure after 98,000 bytes of content have been written into the attributeless sensor data file provided by the invention;

[0077] Figure 14 This is a schematic diagram of the file structure of the attributeless sensor data file provided by the invention after allocating one first-level indirect addressing data block on a 98,000-byte basis;

[0078] Figure 15 This is a schematic diagram showing the file numbering after the address space of the attributeless sensor data is written once and then switched from first-level indirect addressing to second-level indirect addressing.

[0079] Figure 16 This is a schematic diagram of the attribute-less sensor data file structure and address space mapping provided by the present invention;

[0080] Figure 17 This is a schematic diagram of the file structure after the sensor with three attributes is first written with 19402 bytes of raw data and each attribute, as provided by the invention. Detailed Implementation

[0081] This invention designs and implements a sensor data organization and management method, system, and unmanned device. It primarily addresses the problem of unified storage and retrieval of sensor data and its index data collected by unmanned systems during task execution. The key technical points of this invention include: a unified data abstraction model, a multi-level metadata organization model, and a file organization model. Using this solution, sensor data can be conveniently stored, read, and retrieved without the use of external information.

[0082] This invention presents a unified data organization model that efficiently organizes three different types of data—metadata, index information, and raw data—into a single file. It proposes a hierarchical index-based unified address translation model, effectively managing the address spaces of different data types within a file without causing confusion. The invention's logical block-based data organization and management method allows users to flexibly adjust and configure logical blocks according to the size of the sensor data being processed, thereby increasing data storage and retrieval efficiency. It also provides an efficient data access interface, facilitating user management and organization of sensor data. This invention can uniformly organize sensor-related raw data and attribute data into a single file. It proposes a four-level addressing scheme, which provides an independent address space for each type of data. Finally, it provides an interface for the aforementioned data organization methods, offering the ability to read, write, retrieve, and manage sensor-related data.

[0083] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Example 1

[0085] This invention provides a sensor data organization and management method, such as... Figure 1 As shown, it includes:

[0086] S1. When the sensors on the unmanned device collect data that needs to be written to the unmanned device: write the acquired sensor data to the sensor data file.

[0087] like Figure 2 As shown, the sensor data organization and management method and system of the present invention occupy a hierarchical position in the sensor data storage process. This hierarchy is above the specific file system and below the user application.

[0088] S2. When the unmanned device needs to access the sensor data it carries: based on the search conditions set by the application, and according to the search bar, the data file corresponding to the sensor is read, and the read data is returned to the application.

[0089] like Figure 3The diagram illustrates the hierarchical relationship of this invention during sensor data reading. This hierarchy is above the specific file system and below the user application. Based on the search criteria provided by the user application, this invention reads the sensor data file, performs a search based on those criteria, then reads the data matching the search criteria based on the search results and returns it to the upper-level user application.

[0090] like Figure 4 As shown, the data writing in step S1 specifically includes the following steps:

[0091] Step SW1: The user calls the outer data interface provided by this invention to open a sensor data file. If the file does not exist, an empty sensor data file is created.

[0092] Step SW2: After the file is successfully opened, the user can set (add) the attribute information corresponding to the sensor as needed, such as time index, spatial index, etc. This attribute information will be written to the file header. For an empty file, the user can set the logical block size.

[0093] Step SW3: Users can call the data writing interface provided by this invention. This interface requires users to provide raw sensor data, various attribute data, and index information, such as time index information and spatial index information.

[0094] Step SW4: Based on the data types submitted by the user (raw data, time index, spatial index, attribute 1, etc.), find the location of the original address space array corresponding to each type of data in the file header and the length of data already written for each data type. Calculate the data blocks that need to be allocated for writing each type of data this time, as well as the address space data blocks that need to be allocated to manage these data blocks. Then, allocate these data blocks and address space data blocks to the corresponding address space metadata area for management.

[0095] Step SW5: For each data type (raw data, time index, spatial index, attribute 1, etc.), find the location of the original array in the address space corresponding to the data type in the file header and the length of data already written for that data type. Then, according to the three-way addressing scheme, parse the location of the data to be written in the address space of that data type.

[0096] Step SW6: Based on the length of the data to be written and the distribution of the data blocks allocated for this write, write the data of this type to the sensor data file in different ways;

[0097] Step SW7: Update the length information of this type of data in the file header to complete this write operation of this type of data;

[0098] Step SW8: For other types of data, perform the operations of steps SW5-SW7 above until all types of data submitted by the user this time have been written.

[0099] Step SW9: The user continues to acquire and store sensor data, which requires executing the above steps SW2-SW8 until the sensor is turned off or data storage is no longer needed. At this point, the user calls the file closing operation provided by this invention to close the file and complete the entire sensor data writing process.

[0100] like Figure 5 As shown, the data reading operation in step S2 specifically includes the following steps:

[0101] Step SR1: The user calls the outer data interface provided by this invention to open the specified sensor data file. If the file does not exist, a prompt code is returned to the user.

[0102] Step SR2: After the file is successfully opened, the user calls the data reading interface and can set the search conditions for various types of data as needed, such as attribute information, such as time index, spatial index, etc.

[0103] Step SR3: If the user does not submit search criteria, proceed to step SR5; otherwise, based on the search criteria information submitted by the user, find the corresponding address space metadata according to the type information in the file header, and then search in each address space to obtain a list of the locations of the original data that meet the search criteria (the locations in the entire sensor data file).

[0104] Step SR4: Based on the relationships (intersection, union) between the search conditions provided by the user, perform logical calculations on the lists returned for each type of data to obtain the final list of <address, length> of the original data that meets all search conditions in the entire sensor data file, and return it to the user. The entire search process ends, and the process jumps to step SR6.

[0105] Step SR5: Perform address conversion based on the user-specified reading range and the address space of the raw data, calculate the offset and length of the required data in the entire sensor data, and obtain a list consisting of <offset, length>.

[0106] Step SR6: If the <offset, length> list is empty, the reading process ends directly; otherwise, based on the obtained <offset, length> list, read the sensor data file item by item, return the results to the user, and the reading process is complete.

[0107] The logical blocks or data blocks (lbs) involved in the data reading and writing process are the smallest units for organizing data in this invention. Users can flexibly set the logical block size for each type of sensor data file. Compared with traditional file systems, this invention provides a more flexible array organization unit. It is worth noting that once the logical block size of each type of sensor data file is set, it cannot be changed.

[0108] Preferably, the logical block is the smallest unit for organizing and allocating data within the sensor data file, based on the size of the sensor data. Its size (LB_SIZE) is generally 2n times that of the file system block.

[0109] This invention implements a sensor data organization and management method based on a sensor data organization model, such as... Figure 6 As shown, the sensor data organization model of the present invention organizes sensor data files by organizing the original sensor data and the attribute information corresponding to the original sensor data into the same sensor data file.

[0110] The sensor data organization model divides the data into three categories:

[0111] The first category is raw related data, which includes raw data, attribute information corresponding to the raw data, index information, etc.

[0112] The second category is address space data: data that manages the above data. Address space data is divided into four levels (see the description below).

[0113] The third category is the file header: the structure of the file header is as follows... Figure 7 As shown, it contains three main fields.

[0114] (1) The number of attributes indicates the number of attributes corresponding to the sensor;

[0115] (2) The raw data address space metadata area and raw data length are used to manage the data blocks occupied by the sensor raw data address space and to represent the length of the sensor raw data that has been stored.

[0116] (3) Attribute name, attribute address space metadata area, and attribute data length respectively represent the name of a certain attribute corresponding to the sensor data, the metadata of the address space occupied by the data managing the attribute, and the length of the data of the attribute that has been written to the file.

[0117] Existing technologies often employ conventional unified linear encoding. However, for applications using multiple types of data within the same file, a mechanism is needed to correctly separate these data types, requiring an independent address space for each. Using the same linear encoding in existing technologies leads to address space chaos, making it impossible to distinguish data types. Therefore, this invention introduces an independent addressing scheme, constructing a separate linear address for each data type, while ensuring the entire file has a unique linear address managed by the file system. This solves the problem of independent address space calculation for multiple data types within the same linear address space (within the same sensor data file). In other words, the address space of the sensor data file is managed by the file system, and the "XX address space metadata area" in the file header manages the address space data for data of type XX, while the address space data of XX itself manages data of type XX. The attribute data here includes the generation time, sensor pose, data size, etc., corresponding to a frame of sensor data.

[0118] This invention employs a hierarchical addressing model to reduce the file space occupied by the address space and accelerate the actual needs of address space calculation. Objects at each level in the addressing scheme manage different ranges of address space sequentially, and the managed address space range expands exponentially with increasing level. Higher-level objects manage address space by nesting lower-level objects. During addressing, the required addressing level is first determined based on the address range managed by each level of object. Then, within that level's address space, the data block containing the queried address is obtained by iterating through lower-level addressing schemes. Finally, the offset within the block is calculated to obtain the final address information, which is then used for write, retrieval, and read operations.

[0119] Generally speaking, there are many schemes or strategies for encoding each address in a linear address space. These encodings need to be stored in a limited area of ​​the file's metadata area, and this area is generally not large, such as hundreds of bytes or less.

[0120] The simplest approach is to treat the address space as a one-dimensional array and encode it using one-dimensional encoding, where the array index uniquely identifies a given address. However, this method is inefficient and has a long encoding length; as the address space grows, the encoding length also increases, making it generally unsuitable for storage in the metadata area. The total encoding length required for this method when encoding the address space is as follows: Figure 8 As shown.

[0121] Alternatively, equal-length blocks can be used, such as dividing every 10,000 addresses into one block. This method can significantly reduce the encoding length and speed up the query. For example, for address 108234, we can quickly determine that it is in the 109th block. After finding the 109th block, the 8234th position within that block is the one we are looking for. This method is a simple two-level addressing scheme. The first level determines the block to which the address belongs, and the second level determines the position within the block. However, the space occupied by this encoding method in the metadata area is still uncertain (it is not fundamentally different from one-dimensional linear encoding), and the block size is not flexible, as the data size of different types of sensors may vary greatly. The total encoding length required for this encoding method when encoding the address space is as follows: Figure 9 As shown.

[0122] In a data file, each data type is provided with an isolated, independent address space, and each data type's address space can be managed by a four-level addressing scheme; this invention specifically adopts a four-level addressing model. The first level uses direct addressing, while the second, third, and fourth levels all use indirect addressing. Each code in the first level corresponds to a specific data block; the codes in the second, third, and fourth levels all point to a data block address; the data block address pointed to by the second-level code is used to store multiple first-level codes; and so on, the data block address pointed to by the third-level code is used to store multiple second-level codes; the data block address pointed to by the fourth-level code is used to store multiple third-level codes. Clearly, this nested approach represents a wider range of address space because the representable space increases exponentially with each additional level. Therefore, a small amount of information can be placed in the metadata area to represent a large address space. The total length of the code required for encoding the address space using this method is as follows: Figure 10 As shown.

[0123] The specific structure of the four-level addressing model used in this invention is as follows: Figure 11 As shown, this can be specifically represented as:

[0124] (1) The lowest level records the block number (lb_ID) of the data block where the above data is located. Each data block number is represented by 4 bytes. One data block can record LB_SIZE / 4 data block addresses that store the above data.

[0125] (2) The second-lowest level records the data block number where the lowest level address space data is located. A second-lowest level address space data block can also record LB_SIZE / 4 lowest level address space data blocks.

[0126] (3) The highest level address space data block records LB_SIZE / 4 second-highest level address space data blocks;

[0127] (4) With LB_SIZE=8K and only one top-level data block, this 4-level address space can manage (8096 / 4) 3 logic blocks, which is 64TB of single sensor data.

[0128] It can be seen that the scheme for solving the lowest level of address space data is the "direct addressing" scheme, the next lowest level is called "first-level indirect addressing", the next highest level is called "second-level indirect addressing", and the highest level is called "third-level indirect addressing". Obviously, the higher the addressing level used, the greater the overhead of solving the address space, while the lower the addressing level used, the smaller the overhead of solving the address space.

[0129] Integer data structures are used to manage the address space of various types of data. The "address space metadata area" in the file header is represented as intiblock

[15] . Among them: iblock[0]-iblock

[11] are used to manage direct addressing, iblock

[12] is used to manage first-level indirect addressing, iblock

[13] is used to manage second-level indirect addressing, and iblock

[14] is used to manage third-level indirect addressing. Note that in subsequent use, iblock

[15] and addrSpaceMetaInfo

[15] and rawDataAddrSpaceMetaInfo

[15] in the file header structure are different representations of the same object.

[0130] For the iblock

[15] address space management structure, under the current settings, the maximum address space it manages can reach (12+LB_SIZE / 4+(LB_SIZE / 4)2+(LB_SIZE / 4)3)*LB_SIZE bytes. The address space range for direct addressing management is [0, 12 * LB_SIZE), the address space range for Level 1 indirect addressing management is [12 * LB_SIZE, (12 + LB_SIZE / 4) * LB_SIZE), the address space range for Level 2 indirect addressing management is [(12 + LB_SIZE / 4) * LB_SIZE, (12 + LB_SIZE / 4 + (LB_SIZE / 4)2) * LB_SIZE), and the address space range for Level 3 indirect addressing management is [(12 + LB_SIZE / 4 + (LB_SIZE / 4)2) * LB_SIZE, (12 + LB_SIZE / 4 + (LB_SIZE / 4)2 + (LB_SIZE / 4)3) * LB_SIZE]. Simply put, given an address `addr`, by comparing it with the above four ranges, it is possible to quickly determine which level of addressing management is responsible for that address.

[0131] Taking LB_SIZE=8KB as an example, if we want to find the actual offset of address (offset) 19402 in a certain address space in the entire sensor data file (each data type has an independent address space, but all data, address space and header file are stored in the same file, and they share the unique address space of the file, which is managed by the file system), we first need to determine which level of addressing is used to solve it. Since 2*8192<19402<3*8192, direct addressing is used here, which is managed by iblock[2]. The content of Iblock[2] is a logical block number, denoted as w, which represents the wth logical block of the sensor data file starting from the beginning position. That is to say, the 19402nd byte (position) in the above address space corresponds to a certain position in the wth logical block of the sensor data file. Specifically, the position is 19402%8192=3018 (because all data and content are organized and managed according to logical blocks).

[0132] To find the actual offset of address (offset) 109721 in the entire sensor data file, we first need to determine which level of addressing manages 109721 (12*8192<109721<(12+8192 / 4)*8192, which is solved by first-level indirect addressing), corresponding to iblock

[12] . Assume that the logical block number stored in iblock

[12] is x, which corresponds to the xth logical block starting from the beginning of the sensor data file. This logical block contains a series of logical block numbers, which can be regarded as an array of length LB_SIZE / 4, and the array content is the logical block number. To determine the actual position of 109721 in the entire sensor data file, logical block x alone is not enough; we also need to know which record in x manages this actual position. Let's first consider x as an array. Then x[0] represents the first logical block number it records. It manages the address range [8192*12, 8192*(12+1)). It is not difficult to deduce that 109721 is managed by x[1]. The value y of x[1] represents that the logical block is the yth logical block starting from the beginning of the entire sensor data file. Then the real position of 109721 in the entire file is y*8192+109721%8192=y*8192+3225. That is, the real position of the 109721st byte in a certain address space is y*8192+3225 in the entire sensor data file. It can also be understood as the 3225th position in the yth logical block of the entire sensor data file.

[0133] For two-level indirect addressing, to find the position p in a certain address space and its position in the entire sensor data file, first obtain the logical block number z of the iblock

[13] record; then calculate which record in z manages the position to be found, assuming the corresponding logical block number is s; then calculate which record in s manages the position to be found, assuming the corresponding logical block number is t, then the position p in the entire sensor data file is t*LB_SIZE+p%LB_SIZE.

[0134] For level 3 indirect addressing, the process is similar to the above. The offset must be found layer by layer.

[0135] Finally, the position in a certain address space is determined to correspond to its position in the entire sensor data file.

[0136] In fact, both LB_SIZE and the number of bytes occupied by the representation of each logical block can be adjusted, which allows the solution provided by this invention to flexibly adjust the relevant parameters according to actual needs, so as to reduce the file space occupied by address space data.

[0137] Example 2:

[0138] The present invention provides a sensor data organization and management method using the process of writing sensor data with only raw data and no attributes as an example. The data writing operation in this embodiment specifically includes the following steps:

[0139] Step SI_W1: The user opens the visualImage.dat file and specifies the logical block size LB_SIZE as 8KB if the file does not exist. This file is used to store the raw data of the visible light sensor.

[0140] Step SI_W2: Since the file does not exist, this invention creates an empty visualImage.dat file, returns information indicating that the attribute is empty to the user, and caches the contents of the file header, only the file header (which also occupies an 8K data block). (See comparison...) Figure 7 The given file header structure confirms that the logical block size of this file is represented by 0x0000000D (four bytes from 0x00 to 0x03, corresponding to 13 in decimal), and the specific value is 2^13 = 8192, which is 8KB; the number of attributes recorded afterward is 0 (four bytes from 0x04 to 0x07, corresponding to 0 in decimal); the contents of the original data address space metadata area iblock

[15] recorded in the following 60 bytes (0x08-0x43) are all 0, indicating that there is no data at this time; 0x44-0x4B represents the size of the original data currently recorded, which is 0 bytes initially; 0x4C-0x2000 belongs to the padding part, which is an invalid data area in order to fill 8KB;

[0141] Step SI_W3: The user opens the visible light sensor device, such as / dev / video0, and calls the driver corresponding to the visible light device (readData() function) to read data from the device frame by frame, assuming it is 19402 bytes;

[0142] Step SI_W4: Calculate how many data blocks and how many address space blocks are needed to store 19402 bytes of raw data; since the file currently does not have any data blocks, it is necessary to allocate them. Each data block is used to store the original data itself. Since there is no address space data block at present, if the original data is written to these 3 data blocks in the future, the sensor data file will only have 3 data blocks. It can manage the address space of [0, 19402) by direct addressing. Therefore, it is not necessary to apply for logical blocks to store address space data. It is only necessary to assign the IDs of the 3 data blocks to iblock[0], iblock[1] and iblock[2] respectively. Since the logical block number and address or offset are calculated from 0, and the file header occupies logical block 0, the block numbers of the 3 newly allocated data blocks are 1, 2 and 3 respectively. They are placed in the corresponding positions in the file header in the cache.

[0143] Step SI_W5: Write the 19402 bytes read from the sensor to the disk. Since the three allocated data blocks are contiguous and there are no incomplete data blocks left over from the previous write, the 19402 bytes can be written to logic blocks 1, 2, and 3 in one go. After successful writing, update the header content. The sensor data file structure at this point is as follows: Figure 12 As shown, the shaded lines represent the data blocks occupied by the original data. It is clear that data blocks 1 and 2 are already full, while data block 3 still has some area available for the next write. If the write fails, a rollback operation is performed, the three newly allocated data blocks are cleared, and the changes to the header are undone.

[0144] Step SI_W6: After multiple reads and writes, assuming the current data volume is 98,000 bytes, the user continues to read 19,402 bytes and prepares to write it to the sensor data file. At this point, the file structure is as follows: Figure 13 As shown, logic block 12 still has 8192 - 98000% 8192 = 304 bytes remaining for subsequent writing;

[0145] Step SI_W7: Calculate the number of data blocks and address space data blocks required for this 19042-byte write; first, calculate the number of data blocks. Next, we calculate the number of address space data blocks. Since direct addressing has been used previously, no address space data blocks have been allocated. However, after writing 19402 bytes, the total length of the original data is 98000 + 19402 = 117402 bytes, requiring the use of first-level indirect addressing. Therefore, one data block needs to be allocated for first-level indirect addressing to store the address space data. Thus, a total of four logical blocks need to be requested before this write operation. To ensure data continuity, we allocate address space data blocks first, and then allocate blocks to store the original data. The data block, specifically logic block 13, is allocated to first-level indirect addressing, while logic blocks 14, 15, and 16 are allocated to the raw data. These three data blocks are managed by first-level indirect addressing. During the data writing process, because logic block 12, left over from the previous write, still has 304 bytes available for writing, and it is not contiguous with the three data blocks allocated this time, the write is divided into two parts: the first writes 304 bytes to logic block 12, and the second writes 19098 bytes to logic blocks 14, 15, and 16. The sensor data file structure is as follows: Figure 14 As shown, logic block 13, as an address space data block, records the block numbers of the three logic blocks allocated above.

[0146] Step SI_W8: Continue writing the next 19402 bytes. The method of calculating data blocks and address space data blocks is the same as before. This time, only 3 data blocks (logical blocks 17, 18, and 19) need to be allocated. The allocated data blocks are contiguous with the 3 data blocks allocated last time. This write can be completed in one go, writing directly from logical block 16 to logical block 19, and then updating the content in logical block 13 by appending 17, 18, and 19. From this, we can see the advantage of this invention by allocating address space data blocks before allocating data blocks. This can reduce the number of read and write operations and improve efficiency.

[0147] Step SI_W9: Figure 15This diagram illustrates the process of switching the address space from Level 1 indirect addressing to Level 2 indirect addressing due to the writing of an extremely long data frame. Before the write, logical block 13 is the address space data block corresponding to Level 1 addressing, and logical blocks 1-12, 14, and 15 (a total of 14 data blocks) store the original data. Now, a very long frame of approximately 4302 * 8192 bytes is to be written (assuming the file system supports such a long data write). Therefore, 4302 data blocks need to be allocated. However, when switching from Level 1 indirect addressing to Level 2 indirect addressing, the number of Level 1 indirect address space data blocks required is 0, while the number of Level 2 indirect addressing Level 1 blocks is 1, and the number of Level 2 blocks is 2. A total of 4305 logical blocks need to be requested. According to the allocation rules mentioned above, three address space data blocks are allocated first, with one first-level block first and two second-level blocks second. The block numbers of the second-level blocks (17 and 18) are written into the first-level block (16). Then, the block numbers of the following 4302 data blocks are filled into the first-level indirect address space data block (block numbers 19-2064 are filled into logical block 13) and the second-level blocks managed by the second-level indirect address space data block (block numbers 2065-4112 are filled into logical block 17, and block numbers 4113-4320 are filled into logical block 18). After the data is successfully written, the relevant fields in the header are updated.

[0148] Step SI_W10: When the user no longer needs to write data or needs to close the file, the mechanism provided by this invention will persist the above header to disk and then call the file system's file closing operation to normally close the data stream.

[0149] Taking the sensor data reading process with only raw data and no attributes as an example, if the user wants to read length bytes from position 'a' of the raw sensor data, the data reading process of this invention is as follows:

[0150] Step SI_R1: The user opens the visualImage.dat file. If the file does not exist, a failure code is returned.

[0151] Step SI_R2: Read and parse the file header to obtain the header structure and cache it in memory;

[0152] Step SI_R3: Obtain the raw data address space metadata rawDataAddrSpaceMetaInfo

[15] and the total length of the raw data rawDataSize from the header structure;

[0153] Step SI_R4: Determine the relative relationship between the starting position a, the ending position a+length-1 and rawDataSize. If a>rawDataSize, return directly; if a+length-1>rawDataSize, change the ending position to rawDataSize.

[0154] Step SI_R5: According to the aforementioned addressing method, calculate the real address of the location to be read, a, in the visualImage.dat file and the list of logical blocks containing the subsequent length bytes from rawDataAddrSpaceMetaInfo

[15] , assuming it is {34,67,68,75,76,92};

[0155] Step SI_R6: Based on the distribution of lbList, optimize the read operation to reduce the number of file read operations. Taking the above example, logical blocks 67 and 68 are consecutive and can be merged into one read operation. Logical blocks 75 and 76 are consecutive and can be merged into one read operation. The above 6 logical blocks can be acquired by 4 read operations.

[0156] Step SI_R7: Return the read data to the user. If the user wants to continue reading data, repeat the above SI_R2-6 process (the reason for starting from SI_R2 is to prevent the user from writing new data while reading) until the user no longer needs to read data.

[0157] Step SI_R8: The user invokes the file close operation, ending the data reading process, the file is closed normally, and the corresponding buffer is released.

[0158] Example 3:

[0159] Based on the above embodiments, it can be seen that the present invention does not speed up the data reading process and also wastes storage space. This is because the main problem solved by the present invention is the organization, storage, and retrieval of sensor data with attribute information. The following uses the reading and writing process of sensor data with attributes as an example to illustrate how the present invention organizes and manages data, for comparison with the above embodiments.

[0160] The present invention provides a sensor data organization and management method, taking the writing process of attributed sensor data as an example. The data writing process in this embodiment is as follows:

[0161] Step SII_W1: The user opens the visualImage.dat file, which is used to store the raw data of the visible light sensor and its corresponding time index, spatial index and data object data;

[0162] Step SII_W2: Since the file does not exist, this invention creates an empty visualImage.dat file and returns the information that the file is empty to the user;

[0163] Step SII_W3: The user sets the attribute information that needs to be configured in this file; the attribute information includes: temporal index, spatial index, and data object;

[0164] The data structure includes: a time index (time stamp, millisecond precision, specifically the number of milliseconds since January 1, 1970); a spatial index (longitude, latitude, and altitude when the UAV or unmanned vehicle collected the data); a data object (object identified by an image recognition program, such as "car" or "license plate"); bytes 0x00-0x03 (4 bytes) indicating a logical block size of 2^13 = 8192 bytes; bytes 0x04-0x07 (4 bytes) indicating the current number of configured attributes for the sensor's raw data is 3; bytes 0x08-0x43 (60 bytes) indicating the address space metadata area of ​​the sensor's raw data is empty (because no raw data has been stored in the file during initialization); bytes 0x44-0x4B (8 bytes) indicating the length of the sensor's raw data in the current file, or the length of the sensor's raw data address space that has been used, initialized to 0; and bytes 0x4C-0x8B (64 bytes) indicating the "temporal" value corresponding to the time index. The "index" attribute name; the 60 bytes from 0x8C to 0xC7 represent the address space metadata area corresponding to the time index attribute, initialized to 0; the 8 bytes from 0xC8 to 0xCF represent the total length of data already written to disk for the time index attribute, or the length of the address space corresponding to this attribute that has been used, initialized to 0; the following 132 bytes from 0xD0 to 0x163 represent the content of the attrMetaData structure corresponding to the space index attribute; the 132 bytes from 0x164 to 0x1E7 represent the content of the attrMetaData structure corresponding to the data object attribute; the 0s between 0x1E8 and 0x1FFF are used for padding;

[0165] Step SII_W4: The user opens the visible light sensor device, such as / dev / video0, and calls the corresponding driver (readData() function) to read data frame by frame from the device, assuming 19402 bytes; at the same time, the time of acquiring the frame data is obtained (assuming 1626338158729, corresponding to hexadecimal 0x17A A94E E889), and the spatial information of the frame is obtained, assuming (longitude 115.923, latitude 40.124, altitude 20.04m, for ease of calculation, longitude, latitude, and altitude are all magnified by 1000 times, corresponding to hexadecimal 0x1 C4D3, 0x9CBC, and 0x4E48 respectively); assuming the user's image detection program finds a red car in the image of this frame, represented as "type:car" and "color:red";

[0166] Step SII_W5: The user encapsulates the above time index, spatial index, and feature index into corresponding structures, defining three data structures: struct spaceIndex, struct timeIndex, and struct featureMapIndex, as follows: Figure 16 As shown; then, the writeData(char*rawData,struct timeIndex ti,struct spaceIndex si,struct featureMapIndex ami) interface provided by this invention is called to write the raw data and its associated attribute information to the sensor data file;

[0167] Step SII_W6: Calculate the number of logical blocks required for this write operation; specifically:

[0168] (1) For raw data, storing the raw data itself requires There are three logical blocks. Since there are currently no address space data blocks, if the raw data is written to these three data blocks in the future, the sensor data file will only have three data blocks. The address space of [0, 19402) can be managed directly by addressing. Therefore, it is not necessary to apply for logical blocks to store address space data. It is only necessary to assign the IDs of the three data blocks applied for to the corresponding positions of the raw data to iblock[0], iblock[1], and iblock[2] (that is, the corresponding positions of the rawDataAddrSpaceMetaInfo

[15] array). Therefore, a total of three logical blocks have been allocated for the raw data this time.

[0169] (2) For time index data, the structure itself occupies a length of sizeof(struct timeIndex), which is fixed. Let’s assume it is 24 bytes here (considering the alignment of the structure). Since this is the first time it is written, one logical block needs to be allocated for the time index data. Similarly, the management of this logical block itself only uses direct addressing, so no address space data block needs to be allocated. Therefore, one logical block is allocated for the time index attribute this time.

[0170] (3) Similarly, the spatial index attribute is allocated a total of 1 data block, and the data object attribute is also allocated a total of 1 data block.

[0171] (4) After calculation, a total of 6 data blocks need to be requested for this write. The corresponding block numbers are: 1, 2, and 3 for the original data, 4 for the time index, 5 for the spatial index, and 6 for the data object.

[0172] Step SII_W7: After block allocation is complete, first, assign the block numbers of the above 6 data blocks to their corresponding positions in the header file. Then, write the original data and its corresponding attribute data into the sensor data file. The file structure is as follows: Figure 17As shown. The contents of the time index data block are as follows: 8 bytes from 0x8000 to 0x8007 represent the timestamp; 8 bytes from 0x8008 to 0x800F represent the starting position of the raw data being written in its corresponding address space, which is 0 because it is the first write; 4 bytes from 0x8010 to 0x8013 represent the length of the raw sensor data being written; and 4 bytes from 0x8014 to 0x8017 are used for structure alignment and are padded with 0. The contents of the spatial index data block are as follows: 4 bytes from 0xA000 to 0xA003 represent longitude values ​​magnified by 1000 times; 4 bytes from 0xA004 to 0xA007 represent latitude values ​​magnified by 1000 times; 4 bytes from 0xA008 to 0xA00B represent altitude values ​​(in meters) magnified by 1000 times; 4 bytes from 0xA00C to 0xA00F represent the length of the original sensor data written this time; and 8 bytes from 0xA010 to 0xA017 represent the starting position of the original data written this time in its address space. The data index block content is as follows: Bytes 0xC000-0xC003 represent the number of features of the data object corresponding to the original data being written; bytes 0xC004-0xC007 represent the length of the first feature's name (i.e., the length of "type"); bytes 0xC008-0xC00B represent the length of the second feature's name (i.e., the length of "color"); and bytes 0xC00C-0xC00F represent the length of the value corresponding to the first feature (i.e., the length of "car"). The length of the second feature is represented by the following bytes: 0xC010-0xC013 (4 bytes) represents the length of the value corresponding to the second feature (i.e., the length of "red"); 0xC014-0xC017 (4 bytes) represents the length of the original data written; 0xC018-0xC01F (8 bytes) represents the starting position of the original data written in its address space; and 0xC020-0xC02E (15 bytes) represents the hexadecimal representation of each character of "type", "car", "color", and "red".

[0173] Step SII_W8: The subsequent data writing and logical block allocation are similar to the process in "Example 2". First, calculate the number of data blocks needed to store each type of data, and then calculate the number of address space data blocks needed to manage these data blocks. The logical blocks allocated for each type of data are arranged with the address space data blocks distributed consecutively first, followed by the data blocks, and then their corresponding address space data blocks, to improve read and write efficiency. Then, for each type of data, according to its address space mapping position in the entire sensor data file, the corresponding data is written to the file (first the original sensor data, then the attribute data). If the writing is successful, the length of the data already written to disk for that type of data is updated in the file header; if unsuccessful, the space occupied by that type of data needs to be rolled back, and the allocated logical blocks are released (if the original data writing fails, subsequent attribute data is not written to the file).

[0174] Step SII_W9: When the user no longer needs to write data or needs to close the file, the mechanism provided by this invention will persist the above header to disk and then call the file system's file closing operation to normally close the data stream.

[0175] Taking the above-mentioned data reading process of sensor data with attributes on the disk as an example, if the user wants to read the original sensor data from time 1 to time 2, the data reading process of the present invention is as follows: Step SII_R1: The user opens the visualImage.dat file. If the file does not exist, a failure code is returned.

[0176] Step SII_R2: Read and parse the file header to obtain the header structure and cache it in memory; Step SII_R3: Obtain the address space metadata of each attribute from the header structure. The attrMetaData that manages the time index data is assumed to be tamd. Obtain its corresponding address space metadata, denoted as tamd.addrSpaceMetaInfo, and the total length of the time index data, tamd.attrDataSize.

[0177] Step SII_R4: Following the sequential reading method in "Example 2", sequentially read the time index data managed by tamd.addrSpaceMetaInfo. Each time, read out the object struct timeIndex ti of the time index data structure, and then determine the relationship between ti.timestamp and time1, time2, until a ti.timestamp≥time1 is found, which means that the original data corresponding to this index meets the query requirements. Put the key-value pair composed of ti.offset and ti.length into the list to be returned, results (results.push_back(std::make_pair(ti,offset,ti.length))).

[0178] Step SII_R5: Continue reading the time index data to obtain the time index data structure object tix. If time2≥tix and timestamp≥time1, it means that the original data corresponding to the index meets the query requirements. Put the key-value pair composed of tix.offset and tix.length into the list to be returned, results (results.push_back(std::make_pair(tix,offset,tix.length))).

[0179] Step SII_R6: Iterate through the results in the above results. For each tix.offset, perform address conversion using the aforementioned addressing scheme to obtain its true position in the entire sensor data file. Then, use the read() interface provided by the file system to read the data of length tix.length starting from position in the file and return this data to the user.

[0180] Taking the aforementioned sensor data with attributes dropped onto the disk as an example, if a user wants to read the raw sensor data with a time range of time1 to time2 and a spatial range of (x1, y1, z1) to (x2, y2, z2), the data reading process of this invention that satisfies multiple attribute information is as follows:

[0181] Step SIII_R1: The user opens the visualImage.dat file. If the file does not exist, a failure code is returned.

[0182] Step SIII_R2: Read and parse the file header to obtain the header structure and cache it in memory;

[0183] Step SIII_R3: Obtain the address space metadata of each attribute from the header structure. Assuming that the attrMetaData managing the time index data is tamd, obtain its corresponding address space metadata, denoted as tamd.addrSpaceMetaInfo, and the total length of the time index data, tamd.attrDataSize; assuming that the attrMetaData managing the space index data is samd, obtain its corresponding address space metadata, denoted as samd.addrSpaceMetaInfo, and the total length of the time index data, samd.attrDataSize.

[0184] Step SIII_R4: Read the time index data managed by tamd.addrSpaceMetaInfo sequentially. Read the time index data structure object struct timeIndex ti one time at a time. Then, determine the relationship between ti.timestamp and time1, time2, until a ti.timestamp ≥ time1 is found. This means that the original data corresponding to this index meets the query requirements. Put the key-value pair composed of ti.offset and ti.length into the list to be returned, results1 (results1.push_back(std::make_pair(ti,offset,ti.length))).

[0185] Preferably, the sequential reading method is the same as that in "Example 2";

[0186] Step SIII_R5: Continue reading the time index data to obtain the time index data structure object tix. If time2≥tix.timestamp≥time1, it means that the original data corresponding to the index meets the query requirements. Put the key-value pair composed of tix.offset and tix.length into the list to be returned, results (results.push_back(std::make_pair(tix,offset,tix.length))).

[0187] Step SIII_R6: Read the time index data managed by samd.addrSpaceMetaInfo sequentially. Read one time index data structure object struct spaceIndex si at a time. Then, determine the relationship between its longitude, latitude, height and (x1, y1, z1) and (x2, y2, z2). If (x2, y2, z2) ≥ si.(longitude, latitude, height) ≥ (x1, y1, z1), it means that the original data corresponding to the index meets the query requirements. Put the key-value pair composed of si.offset and si.length into the list to be returned, results2 (results2.push_back(std::make_pair(si,offset,si.length))).

[0188] Preferably, the sequential reading method is the same as that in "Example 2";

[0189] Step SIII_R7: Based on the search conditions set by the user (the two conditions are in an AND relationship), calculate the intersection of the returned results1 and results2, which is results = results1 ∩ results2, and is the relevant information of the original data that meets the conditions.

[0190] Step SIII_R8: Iterate through the results in the above results. For each result.offset, perform address translation using the aforementioned addressing scheme to obtain its true position in the entire sensor data file. Then, use the read() interface provided by the file system to read the data of length result.length starting from position in the file and return this data to the user.

[0191] Compared to Example 2, Example 3 provides a data filtering function, offering simple data retrieval capabilities. In most cases, the length of a single frame of data output by the sensor is much greater than the length of its corresponding attribute data. The time required to traverse all the raw data to find data that meets the user's conditions is much greater than the time required to read the relevant attribute data. In Example 2, only the relevant attribute data is read, and then filtered according to the user-submitted search conditions to obtain the location information (the location and length of the corresponding raw data) that meets the user's conditions. Then, only this small amount of location information needs to be used to read the address space of the raw data. Obviously, this method will greatly reduce the data reading time on average, thereby improving reading efficiency.

[0192] Unmanned equipment includes robots, drones, unmanned underwater vehicles, unmanned boats, and self-driving cars.

[0193] Example 4

[0194] Based on the same inventive concept, the present invention also provides a sensor data organization and management system, comprising: a data management layer, a sensor device layer, a user application layer, and a file system layer; the sensor device layer includes various sensor devices; the file system layer is used to store data files corresponding to various sensors; and the application layer includes various applications.

[0195] The data management layer includes: an outer data interface, a read module, and a write module;

[0196] The outer layer number interface is used to obtain sensor data from the sensor device layer through the user application layer;

[0197] The reading module is invoked when the unmanned device needs to access the data of the sensors it carries. The reading module is used to read the data files corresponding to the sensors based on the search conditions set by the application and according to the search bar, and return the read data to the application.

[0198] The writing module is invoked when the sensors on the unmanned device collect data that needs to be written to the unmanned device. The application calls the writing module to write the acquired sensor data into the sensor data file.

[0199] The sensor data organization and management system in this embodiment is designed to implement the sensor data organization and management method of the present invention. The functions implemented by each module can be referred to the foregoing embodiments, and will not be repeated here.

[0200] The terms "data processing apparatus," "data processing system," "user equipment," or "computing device" encompass all kinds of apparatus, devices, and machines for processing data, including, for example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement various computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0201] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including assembly or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.

[0202] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for organizing and managing sensor data, characterized in that: The method includes: When the sensors on the unmanned device collect data that needs to be written back to the unmanned device: the acquired sensor data is written to the sensor data file; When an unmanned device needs to access the data from its onboard sensors: based on the search criteria set by the application, the device reads the data file corresponding to the sensor according to the search criteria and returns the read data to the application. The step of writing the acquired sensor data into a sensor data file includes: Step SW1: Access the data file corresponding to the sensor based on the data interface; Step SW2: Based on the raw sensor data, various attribute data, and index information, call the data writing interface; Step SW3: Locate the original address space array corresponding to each type of data in the header of the data file and the length of data already written for each data type. Calculate the data blocks to be allocated for writing each type of data this time, as well as the address space data blocks to be allocated for managing these data blocks. Then, allocate these data blocks and address space data blocks to the corresponding address space metadata area for management. Step SW4: Sequentially select one data type from the data types and perform the write operation until all data types have been written. Step SW5: Continue executing the above steps SW2-SW4 until the sensor is turned off or no more data needs to be written; The write operation includes: Find the location of the original array in the address space corresponding to this type of data in the file header and the length of data of this type that has been written. Then, according to the 4-level addressing scheme, parse the location of the data to be written in the address space of this type of data. Based on the length of the data to be written of the data type and the distribution of the data blocks allocated for this write, write the data of this type into the sensor data file; Update the length information of the data type described in the data file header to complete the current write operation of this data type. The index information includes time index information and spatial index; The data types include: raw data and attributes; The data file includes: a header file, multiple raw data and address space data, attribute address space data and attribute data corresponding to each raw data; The header file includes: the size of the logic block, the number of sensor attributes, the raw data address space data corresponding to each raw data, the raw data length, the attribute name, and the attribute address space metadata; The size of the logic block can be dynamically set or fixed. The data file corresponding to the sensor is constructed based on the original data of the sensor, the attributes corresponding to the original data, and the header file. The four-level addressing scheme includes: The first level uses direct addressing, while the second, third, and fourth levels all use indirect addressing.

2. The method as described in claim 1, characterized in that, The method of accessing the data file corresponding to the sensor based on the data interface includes: Determine if the data file corresponding to the sensor exists; When it exists, access the data file corresponding to the sensor and write the header information corresponding to the data file to the header of the data file; If the sensor does not exist, create the data file corresponding to the sensor, set the logical block size of the file, and write the header information of the created data file into the header of the created data file.

3. The method as described in claim 1, characterized in that, In the first level of encoding, each code corresponds to a specific data block; The second, third, and fourth levels of encoding are all used to point to a data block address; The data block pointed to by the second-level code is used to store multiple first-level codes; The data block pointed to by the third-level code is used to store multiple second-level codes; The data block pointed to by the fourth-level code is used to store multiple third-level codes.

4. The method as described in claim 1, characterized in that, The step of reading the data file corresponding to the sensor according to the search criteria includes: Open the data file corresponding to the sensor based on the outer data interface; Based on the data interface, the sensor data file is accessed to obtain list information; The sensor data file is read based on the list information.

5. The method as described in claim 4, characterized in that, The step of opening the data file corresponding to the sensor based on the outer data interface also includes: Reading the sensor data file will stop if the data file corresponding to the sensor does not exist.

6. The method as described in claim 4, characterized in that, The access to the sensor data file to obtain list information includes: Determine whether to submit search criteria: When search criteria are submitted, the sensor data file is searched by type according to the search criteria to obtain the first list information; When no retrieval information is submitted: the second list information is determined by calculating the entire sensor data file based on the reading range and the address space of the original data; The first list information includes: address, length; the second list information includes: offset, length; The search criteria include attribute information and index information.

7. The method as described in claim 6, characterized in that, The step of searching the sensor data file by type according to the search criteria to obtain the first list information includes: Based on the search criteria for each type, the corresponding address space metadata is found from the type information in the header of the data file. Then, each address space is searched to obtain a list of the locations of the original data that meet the search criteria. Based on the relationship between the search criteria, logical calculations are performed on the lists returned for each type of data to obtain the address and length of the original data that meets all the search criteria in the entire sensor data file, which is used as the first list information.

8. The method as described in claim 6, characterized in that, The step of calculating and determining the second list information based on the reading range and the address space of the original data for the entire sensor data file includes: Address conversion is performed based on the reading range and the address space of the raw data. The offset and length of the required data in the entire sensor data are calculated as the second list information.

9. A sensor data organization and management system, characterized in that: The system includes: a data management layer, a sensor device layer, a user application layer, and a file system layer; the sensor device layer includes various sensor devices; the file system layer is used to store data files corresponding to various sensors; the user application layer includes various application programs; The data management layer includes: an outer data interface, a read module, and a write module; The outer data interface is used to obtain sensor data from the sensor device layer through the user application layer; The reading module is invoked when the unmanned device needs to access the sensor data it carries. The reading module is used to read the data file corresponding to the sensor based on the search conditions set by the application and return the read data to the application. The writing module is invoked when the sensors on the unmanned device collect data that needs to be written to the unmanned device. The application calls the writing module to write the acquired sensor data into the sensor data file. The step of writing the acquired sensor data into a sensor data file includes: Step SW1: Access the data file corresponding to the sensor based on the data interface; Step SW2: Based on the raw sensor data, various attribute data, and index information, call the data writing interface; Step SW3: Locate the original address space array corresponding to each type of data in the header of the data file and the length of data already written for each data type. Calculate the data blocks to be allocated for writing each type of data this time, as well as the address space data blocks to be allocated for managing these data blocks. Then, allocate these data blocks and address space data blocks to the corresponding address space metadata area for management. Step SW4: Sequentially select one data type from the data types and perform the write operation until all data types have been written. Step SW5: Continue executing the above steps SW2-SW4 until the sensor is turned off or no more data needs to be written; The write operation includes: Find the location of the original array in the address space corresponding to this type of data in the file header and the length of data of this type that has been written. Then, according to the 4-level addressing scheme, parse the location of the data to be written in the address space of this type of data. Based on the length of the data to be written of the data type and the distribution of the data blocks allocated for this write, write the data of this type into the sensor data file; Update the length information of the data type described in the data file header to complete the current write operation of this data type. The index information includes time index information and spatial index; The data types include: raw data and attributes; The data file includes: a header file, multiple raw data and address space data, attribute address space data and attribute data corresponding to each raw data; The header file includes: the size of the logic block, the number of sensor attributes, the raw data address space data corresponding to each raw data, the raw data length, the attribute name, and the attribute address space metadata; The size of the logic block can be dynamically set or fixed. The data file corresponding to the sensor is constructed based on the original data of the sensor, the attributes corresponding to the original data, and the header file.

Images