Multi-dimensional data indexing method for hard real-time task scheduling and related equipment
By combining spatial segmentation and Trie structure methods, efficient indexing of multidimensional data is solved, the problem of untimely task scheduling in the medium and high-dimensional data environments in the existing technology is solved, the requirement of hard real-time task scheduling is realized, and the use of storage space is optimized.
Patent Information
- Application Number
- CN202411729057.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-04-29
AI Technical Summary
The prior art is difficult to achieve efficient task scheduling in high-dimensional data environments, resulting in untimely data processing and unable to meet hard real-time requirements.
The multidimensional data is indexed using a method combining spatial segmentation and Trie structure, spatial segmentation is performed through grid structures, and data indexing is achieved using customized Trie tree structures to ensure that the index structure does not degenerate during insertion and deletion operations.
It realizes efficient indexing and management of high-dimensional data, ensures timely processing of high-priority data, meets the needs of hard real-time task scheduling, and reduces the storage space required for index building.
Smart Images
Figure CN119903092A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer real-time task scheduling, and in particular to a multidimensional data indexing method and related equipment for hard real-time task scheduling. Background Art
[0002] Task scheduling and migration in computers are generally implemented by task scheduling systems. In view of the large-scale, linear data processing load and hard real-time data processing requirements in embedded applications, the task scheduling system needs to be redesigned. Traditional task scheduling principles are based on factors such as overall system performance and power consumption, and use resource margin, time information and task priority as scheduling principles. When the task scheduling process is slow, data processing may not be timely, exceeding the time limit for data processing in the application scenario, resulting in catastrophic consequences.
[0003] The existing technology adopts a data-driven task scheduling system design to meet the hard real-time guarantee requirements of task scheduling. When scheduling tasks, the system first traverses the data objects to be processed to determine the data priority order, and then indexes the data objects in order from high to low data priority, associates the corresponding task list and other task-related data block information, and finally executes the corresponding tasks in order from high to low data object priority.
[0004] In the above task scheduling process, it is necessary to frequently retrieve a large number of data objects to be managed. The core is to achieve caching and efficient retrieval of a large number of data objects. The relationship between data objects and tasks is generally maintained by a database. Database technology is one of the core technologies of information systems, and it mainly studies how to organize and store data, and how to efficiently obtain and process data. Among them, the index structure and method of data directly affect the efficiency of data acquisition.
[0005] Indexing efficiency is closely related to the distribution of the data itself. When the dimension of the data to be processed is low, basic data structures such as linked lists, arrays, and hash tables can be used for management. When the data dimension increases to a certain threshold (for example, 6 dimensions or above), how to efficiently index and manage the database becomes the main issue affecting real-time performance.
[0006] For the management of multidimensional data, the current mainstream technology is based on multidimensional index structures such as R-tree, SR-tree, KD-tree, and KDB-tree. The basic principle of these index structures is data partitioning, that is, according to the data distribution, the entire data space is partitioned into multiple small blocks, and then organized into a coarse-to-fine tree structure for retrieval. The advantage of data partitioning is that it theoretically has the conditions for constructing a balanced tree, so under the optimized conditions, the depth of the tree is shallow and the retrieval performance is good. However, since the construction of a balanced tree requires global data distribution information, the complexity of the data insertion and deletion algorithm is extremely high and cannot be used in real application scenarios. The insertion and deletion algorithm of real applications uses dynamic programming optimization, which cannot guarantee the global optimal solution, so it may cause the index structure to degenerate. The degradation of the index structure makes the worst performance of the system unpredictable and the upper bound of the system resource requirements unestimated, so it cannot be applied to embedded applications with strong real-time requirements. At the same time, the insertion and deletion algorithm based on dynamic programming has a large amount of calculation and is not friendly to embedded application scenarios. Summary of the invention
[0007] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a multidimensional data indexing method and related equipment for hard real-time task scheduling, so as to solve the technical problem of how to improve the real-time requirements of task scheduling in the prior art.
[0008] The present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a multidimensional data indexing method for hard real-time task scheduling, comprising: Receive the task list and data queue in the application request; Prioritize data objects in the data queue; The sorted data objects are indexed in sequence, and the indexed data objects are associated with the received task list to complete the multi-dimensional data indexing work.
[0009] Preferably, in the step of receiving the task list and data queue in the application request, the data objects of the data queue include but are not limited to multi-dimensional semantic information of sensor type, priority, time range, spatial position and target color value.
[0010] Preferably, in the step of prioritizing the data objects in the data queue, the priorities are arranged from high to low in the order of data object x_1, data object x_2 to data object x_m.
[0011] Preferably, in the step of sequentially indexing the sorted data objects, a method combining space partitioning with a Trie structure is used to index the sorted data objects.
[0012] Furthermore, the space is divided into two parts using a grid structure. The Trie structure performs data indexing based on a customized Trie tree structure, and does not cause structural degradation during the process of adding new data objects or deleting existing data objects, so as to ensure that the upper bound of the indexing operation time is predictable. Under the Trie structure, the worst time complexity of indexing, inserting, and deleting operations is O(N), where N is the dimension size of the data and is equal to the level of the Trie structure.
[0013] Furthermore, when using the Trie structure for indexing, each level represents a dimensional direction of the space. Each dense grid space cavity is an end node of a Trie structure index and points to the corresponding task list. The Trie structure is compressed through a compression algorithm to reduce the storage space required to construct the Trie.
[0014] Furthermore, in the step of associating the indexed data object with the received task list, each dense lattice space cavity is individually associated with a task list. When multiple dense lattice space cavities correspond to the same task list, adjacent dense lattice space cavities of the same task list are merged, and the mapping relationship of the Trie structure is adjusted accordingly.
[0015] In a second aspect, the present invention further provides a multidimensional data indexing system for hard real-time task scheduling, comprising: A data receiving module, used for receiving a task list and a data queue in an application request; A data sorting module is used to prioritize data objects in a data queue; The data indexing module is used to index the sorted data objects in sequence, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
[0016] In a third aspect, the present invention also provides a mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the multidimensional data indexing method for hard real-time task scheduling as described above are implemented.
[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the multidimensional data indexing method for hard real-time task scheduling as described above.
[0018] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention provides a multidimensional data indexing method for hard real-time task scheduling, which prioritizes data objects in the data queue by receiving a task list and a data queue in an application request; and sequentially indexes the sorted data objects and associates them with the task list. The present invention can ensure that high-priority data is processed in a timely manner, and meets the hard real-time requirements of the task scheduling process in data processing applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flow chart of a multidimensional data indexing method for hard real-time task scheduling in an embodiment of the present invention; Figure 2 It is a framework flow chart of task scheduling and data indexing in an embodiment of the present invention; Figure 3 This is an example diagram of a 2-dimensional Trie index based on lattice vector quantization in an embodiment of the present invention; Figure 4 A diagram of the data-task index association process in an embodiment of the present invention; Figure 5 This is an example diagram of a compressed Trie index in an embodiment of the present invention; Figure 6 Schematic diagram of a data set showing the relationship between data and task lists in an embodiment of the present invention; Figure 7 A schematic diagram of a compressed and merged Trie index structure in an embodiment of the present invention; Figure 8 It is a schematic diagram of the structure of a multidimensional data indexing system for hard real-time task scheduling in an embodiment of the present invention; In the figure: 1. Data receiving module; 2. Data sorting module; 3. Data indexing module. DETAILED DESCRIPTION
[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0021] The purpose of the present invention is to provide a multidimensional data indexing method and related equipment for hard real-time task scheduling, so as to solve the technical problem of how to improve the real-time requirement of task scheduling in the prior art.
[0022] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1 and Figure 2The present invention provides a multidimensional data indexing method for hard real-time task scheduling, comprising: Step 1, receiving the task list and data queue in the application request; Specifically, in the step of receiving the task list and data queue in the application request, in the multi-sensor data fusion processing application, the data objects of the data queue include but are not limited to multi-dimensional semantic information of sensor type, priority, time range, spatial position and target color value. Among them, the dimension of the data object in the present invention is limited to within 20 dimensions.
[0023] Step 2, prioritize the data objects in the data queue; Specifically, in the step of prioritizing the data objects in the data queue, the priorities are arranged from high to low in the order of data object x_1, data object x_2 to data object x_m.
[0024] Step 3, index the sorted data objects in turn, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
[0025] Specifically, in the step of sequentially indexing the sorted data objects, a method combining space partitioning and Trie structure is used to index the sorted data objects.
[0026] The space partitioning adopts the grid structure for space partitioning. Since the distribution of actual data objects in each dimension is uncertain, the present invention does not limit the form of the grid structure. Taking the Trie structure under the two-dimensional Z grid and the dense grid as an example, the index of the data space is as follows: Figure 3 Compared with the Z grid, the dense grid has higher retrieval efficiency in applications that use Euclidean distance as the retrieval metric, as shown in Table 1. In mobile communication applications, the two-dimensional space uses a hexagonal A2 dense grid space cavity. Compared with the square Z grid, fewer base stations can be used to cover the same geographical area. At the same time, each two-dimensional square grid has 8 adjacent subspaces, while the hexagonal grid has only 6 adjacent subspaces. The number of subspaces covered by the same range of retrieval is reduced, and the retrieval speed is improved.
[0027] Table 1 Comparison of neighbor numbers between dense grid and Z grid
[0028] When using the Trie structure for indexing, each level represents a dimension of space. Each leaf node at the end of the Trie structure represents a dense grid space cavity and points to the corresponding task list, such as Figure 4 shown.
[0029] Specifically, the system initialization process divides the data space, constructs the Trie structure and associates the task list. Since there are a large number of dense grid space cavities without data in the actual data space, there must be nodes without corresponding data in the corresponding Trie structure. Therefore, the Trie structure can be compressed through a compression algorithm to reduce the storage space required to construct the Trie. Figure 5 The figure shows an example of compression of a Trie index. On the other hand, each dense grid can theoretically represent a task list separately, but in practice, multiple dense grids in a certain dimension may correspond to the same task list. Therefore, when the actual storage space is limited, adjacent dense grids of the same task list can be merged and the mapping relationship of the Trie structure can be adjusted accordingly.
[0030] When the task scheduling system of the present invention works normally, the data objects first enter the task scheduler and are sorted. The scheduler finds the corresponding grid for each data object through the Trie index, thereby obtaining the corresponding task list.
[0031] The implementation of the Trie structure can be implemented on the CPU platform based on basic data structures such as arrays, linked lists, hash tables, etc., or through dedicated hardware such as FPGA, memory, etc. When it is necessary to add, delete, or merge Trie nodes, the system runs an update program.
[0032] Example 1 This embodiment uses the multi-sensor data fusion processing in a certain industry application as an example to illustrate the multi-dimensional data indexing method process for hard real-time task scheduling provided by the present invention. In this scenario, the amount of data to be processed collected by the sensor at different times is constantly changing, and the range of change is also very large. In addition, the sensor data may contain information in multiple dimensions, and the task scheduling system needs to support the rapid indexing of multi-dimensional data.
[0033] This embodiment assumes that the data object is three-dimensional, namely: sensor type, data object side length, data arrival time period. There are 4 types of sensor types, the data format is a two-dimensional square image input, the side length varies in the range of 0-1040, and the data arrival time period is in seconds, varying in the range of 0-4.
[0034] according to Figure 6 The data objects shown above are all possible data objects faced by the task scheduling system. The Z grid or dense grid can be used to partition the data space, and the data set in the above figure can be constructed and compressed as follows: Figure 7 The minimized Trie structure shown in Figure 1. D2 and D5 both correspond to task G3, and their dense grid space cavities are adjacent, so they are merged into a Trie structure leaf node.
[0035] After the data objects enter the task scheduler in sequence, the data list is first sorted by data block priority. After sorting, each data object is traversed and the corresponding task list is obtained by traversing the Trie structure, such as Figure 7 shown.
[0036] In the design of the task scheduling system for which the present invention is directed, the processing of key data generally adopts fixed task steps, and the task execution time is determinable. The only uncertainty is the time consumed by the task scheduling process. The data indexing scheme of the present invention limits the indexing time of data objects to a certain range, thus meeting the hard real-time requirements of task scheduling.
[0037] The present invention adopts lattice vector quantization technology to realize spatial segmentation of multidimensional data. Lattice vector quantization is a method of using a mathematical tool such as a lattice to realize spatial segmentation. A lattice is a geometric structure that can fill a space without overlapping. For example, a square can fill a two-dimensional plane, and a cube can fill a three-dimensional space, both of which are relatively simple lattice structures. For a data point vector in a multidimensional space, lattice vector quantization is essentially to find the lattice point closest to the vector in a designed geometric model. The vector data after lattice vector quantization is the centroid point of a dense lattice space cavity, which can support the rapid retrieval of a dense lattice space cavity and an adjacent dense lattice space cavity.
[0038] The present invention adopts the principle of combining space partitioning and Trie structure. First, the dense grid space cavity itself exists objectively and is not affected by data distribution. After the space is mechanically divided into multiple independent dense grid space cavities, the dense grid space cavity can be used to manage data, which alleviates the impact of data distribution on the performance of the index structure. Secondly, when the data distribution in the space is relatively sparse, there will be a large number of dense grid space cavities without data in the space after partitioning, resulting in waste of resources. The biggest advantage of the Trie structure is that the structure only needs to represent nodes containing data. Therefore, on the basis of space partitioning, the classic Trie structure can be adjusted, and each layer of the Trie represents a dimensional direction of the space. The size of the Trie node alphabet of the layer is consistent with the number of dense grid space cavities of the corresponding dimension. The number of Trie structure layers is the same as the number of spatial dimensions, and the leaf node represents the dense grid space cavity where the data is located. The adjusted Trie structure has the ability to manage multidimensional data. When indexing, the target data is quickly found by traversing the nodes downward step by step.
[0039] In summary, the present invention provides a multidimensional data indexing method for hard real-time task scheduling, which prioritizes data objects in the data queue by receiving a task list and a data queue in an application request; indexes the sorted data objects in turn, associates the indexed data objects with the received task list, and completes the multidimensional data indexing work. Through hard real-time task scheduling, the present invention can ensure that high-priority tasks are processed in a timely manner, reduce task waiting time, and improve the efficiency of overall task processing.
[0040] Example 2 according to Figure 8 As shown, this embodiment provides a multidimensional data indexing system for hard real-time task scheduling, including: Data receiving module 1, used for receiving the task list and data queue in the application request; Data sorting module 2, used to prioritize data objects in the data queue; The data indexing module 3 is used to index the sorted data objects in sequence, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
[0041] Example 3 The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, such as a multidimensional data indexing program for hard real-time task scheduling.
[0042] When the processor executes the computer program, the steps of implementing the multidimensional data indexing method for hard real-time task scheduling are as follows, for example: Receive the task list and data queue in the application request; Prioritize data objects in the data queue; The sorted data objects are indexed in sequence, and the indexed data objects are associated with the received task list to complete the multi-dimensional data indexing work.
[0043] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized, for example: Data receiving module 1, used for receiving the task list and data queue in the application request; Data sorting module 2, used to prioritize data objects in the data queue; The data indexing module 3 is used to index the sorted data objects in sequence, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
[0044] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the mobile terminal.
[0045] For example, the computer program may be divided into a data receiving module 1, a data sorting module 2, and a data indexing module 3; The specific functions of each module are as follows: Data receiving module 1, used for receiving the task list and data queue in the application request; Data sorting module 2, used to prioritize data objects in the data queue; The data indexing module 3 is used to index the sorted data objects in sequence, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
[0046] The mobile terminal may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The mobile terminal may include, but is not limited to, a processor and a memory.
[0047] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the mobile terminal, and uses various interfaces and lines to connect various parts of the entire mobile terminal.
[0048] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the mobile terminal by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.
[0049] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0050] Example 4 The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multidimensional data indexing method for hard real-time task scheduling are implemented.
[0051] If the module / unit integrated in the mobile terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0052] Based on this understanding, the present invention implements all or part of the processes in the above method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned aggregate reinforcement learning resource scheduling method can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.
[0053] The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0054] It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.
[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A multidimensional data indexing method for hard real-time task scheduling, characterized in that: include: Receive the task list and data queue in the application request; Prioritize data objects in the data queue; The sorted data objects are indexed in sequence, and the indexed data objects are associated with the received task list to complete the multi-dimensional data indexing work.
2. A multidimensional data indexing method for hard real-time task scheduling according to claim 1, characterized in that: In the step of receiving the task list and data queue in the application request, the data objects of the data queue include but are not limited to multi-dimensional semantic information of sensor type, priority, time range, spatial position and target color value.
3. The multidimensional data indexing method for hard real-time task scheduling according to claim 1, characterized in that: In the step of prioritizing the data objects in the data queue, the priorities are arranged from high to low in the order of data object x_1, data object x_2 to data object x_m.
4. The multidimensional data indexing method for hard real-time task scheduling according to claim 1, characterized in that: In the step of sequentially indexing the sorted data objects, a method combining space partitioning and Trie structure is used to index the sorted data objects.
5. The multidimensional data indexing method for hard real-time task scheduling according to claim 4, characterized in that: The space segmentation adopts a grid structure for space segmentation, and the Trie structure performs data indexing based on a customized Trie tree structure, and no structural degradation occurs during the process of adding new data objects and deleting existing data objects, so as to ensure that the upper bound of the indexing operation time is predictable; under the Trie structure, the worst time complexity of indexing, inserting, and deleting operations is O(N), where N is the dimension size of the data and is equal to the level of the Trie structure.
6. The multidimensional data indexing method for hard real-time task scheduling according to claim 5, characterized in that: When using the Trie structure for indexing, each level represents a dimensional direction of the space. Each dense grid space cavity is an end node of the Trie structure index and points to the corresponding task list. The Trie structure is compressed through a compression algorithm to reduce the storage space required to build the Trie.
7. The multidimensional data indexing method for hard real-time task scheduling according to claim 6, characterized in that: In the step of associating the indexed data object with the received task list, each dense lattice space cavity is individually associated with a task list. When multiple dense lattice space cavities correspond to the same task list, adjacent dense lattice space cavities of the same task list are merged, and the mapping relationship of the Trie structure is adjusted accordingly.
8. A multidimensional data indexing system for hard real-time task scheduling, characterized in that: include: A data receiving module, used for receiving a task list and a data queue in an application request; A data sorting module is used to prioritize data objects in a data queue; The data indexing module is used to index the sorted data objects in sequence, associate the indexed data objects with the received task list, and complete the multi-dimensional data indexing work.
9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multidimensional data indexing method for hard real-time task scheduling as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multidimensional data indexing method for hard real-time task scheduling as described in any one of claims 1 to 7 are implemented.