Multi-modal knowledge graph storage optimization method and system in airborne environment

Abstract

The invention discloses a multi-modal knowledge graph storage optimization method and system in an airborne environment, and the method comprises the steps: obtaining and preprocessing multi-modal graph data of a helicopter cockpit, splitting the data into adaptive sub-graphs, and carrying out the grouping and sorting of the sub-graphs, so as to obtain an edge data set with an optimized storage sequence; memory allocation and address mapping of the adaptive sub-graph are completed based on unified virtual addressing, a direct memory access technology is deployed, and a memory direct operation link is established through a PCIe channel; in combination with the optimized edge data set, starting a memory merging prefetching function of the graphics processor to load unoperated data of the adaptive sub-graph; the high-dimensional vertexes of the sub-graphs are segmented to generate virtual vertexes with balanced dimensions, and the virtual vertexes are incorporated into a parallel operation process of a graphics processor to replace the original high-dimensional vertexes to participate in calculation. The problems that in the prior art, airborne helicopter cockpit multi-modal scene adaptation is not carried out, multi-modal data processing is split, and the real-time performance and accuracy are low are solved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a method and system for optimizing the storage of multimodal knowledge graphs in an airborne environment. Background Technology

[0002] Currently, most triplet storage solutions are centralized, with data management concentrated in one location. This centralized triplet storage solution supports more triplets within a single storage unit, but its drawbacks include a lack of flexibility and insufficient application support. Therefore, it's necessary to consider how to partition the storage based on the characteristics of the data and applications.

[0003] The diverse query formats of various knowledge types in multimodal knowledge graphs mean that traditional triplet storage cannot meet the performance requirements of querying in different scenarios, and it lacks index design for complex knowledge describing time sequences, processes, events, etc. In centralized databases, the execution overhead of queries mainly includes: Total Cost = I / O Cost + CPU Cost + Memory Cost. Therefore, it is necessary to specifically adjust the performance parameters of the knowledge graph storage system and adopt strategies such as model optimization to balance the execution efficiency of these three costs, thereby reducing the resource consumption of knowledge queries and improving execution efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for optimizing the storage of multimodal knowledge graphs in an airborne environment, in order to solve the problems in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for optimizing the storage of multimodal knowledge graphs in an airborne environment, the method comprising: Acquire multimodal graph data in a helicopter cockpit scene, and preprocess the graph data; The graph data is split into adaptive subgraphs, and each adaptive subgraph is grouped and sorted to obtain an edge data set with optimized storage order; Based on the unified virtual addressing method, memory allocation and address mapping are performed on the adaptation subgraph; Deploy direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping; Combined with the edge data set with the optimized storage order, the memory merging prefetch function of the graphics processor is enabled to preload the unprocessed portion of the adapted subgraph data; The high-dimensional vertices in the adaptation subgraph are segmented to generate multiple virtual vertices with balanced dimensions. The virtual vertices are incorporated into the parallel computing process of the graphics processor, replacing the high-dimensional vertices in the computation.

[0006] Furthermore, in one implementation, the step of acquiring multimodal graph data in a helicopter cockpit scenario and preprocessing the graph data includes: Acquire multimodal graph data of a helicopter cockpit scene, the multimodal graph data including spatial vertices and non-spatial vertices; bind coordinates to each of the spatial vertices.

[0007] Specifically, in this embodiment, the spatial vertices include terrain, obstacles, navigation points, etc.; the non-spatial vertices include engine status, alarm events, etc.; each spatial vertex is bound to coordinates, for example, terrain vertices are bound to three-dimensional coordinates including latitude, longitude and altitude, and obstacle vertices are bound to their spatial position coordinates.

[0008] Furthermore, in one implementation, the step of splitting the graph data into adapting subgraphs, and grouping and sorting each adapting subgraph to obtain an edge data set with optimized storage order includes: The graph data is split based on a geometric partitioning algorithm to obtain multiple initial subgraphs; According to a preset threshold, the initial subgraph is integrated into several subgraphs of similar size, namely the adapted subgraph; Based on the source vertices in the adaptation subgraph, all edges are grouped and sorted to obtain the edge data set with the optimized storage order.

[0009] Specifically, in this embodiment, all vertices can be sorted according to the coordinate direction that is strongly related to the flight scene; the half of the vertices with smaller coordinates can be assigned to the first subgraph, and the other half can be assigned to the second subgraph; the first and second subgraphs can be repeatedly divided by recursive coordinate bisection to obtain multiple initial subgraphs.

[0010] Furthermore, in one implementation, the step of allocating memory and mapping addresses to the adaptation subgraph based on the unified virtual addressing method includes: Based on the total capacity of the subgraph and the size of a single subgraph, a unified virtual addressing mode is enabled to map the host memory of the central processing unit and the memory of the graphics processing unit to the same virtual address space. According to the priority of the subgraph, each subgraph is mapped to a contiguous virtual address space within the corresponding partition.

[0011] Furthermore, in one implementation, the deployment of direct memory access technology, establishing a direct memory operation link through a PCIe channel, includes: Based on a unified virtual addressing address space, a direct memory access link for PCIe channels is established to transfer the subgraph from the central processing unit host memory to the graphics processing unit memory. Transmission tasks are scheduled according to subgraph priority.

[0012] Furthermore, in one implementation, the step of segmenting the high-dimensional vertices in the adaptation subgraph to generate multiple dimensionally balanced virtual vertices includes: Based on the edge data set with the optimized storage order, the number of edges associated with vertices in the adaptation subgraph is counted, i.e., dimension statistics are performed. Based on the dimensionality statistics, a standard for vertex dimension limit values ​​is set according to the power-law characteristics of graph data. Based on the vertex dimension limit value standard, high-dimensional vertices in the adaptation subgraph that exceed the vertex dimension limit value standard are segmented to generate multiple virtual vertices with balanced dimensions.

[0013] Furthermore, in one implementation, the step of integrating the initial subgraph into several subgraphs of approximately the same size, i.e., the adapted subgraph, according to a preset threshold, includes: Set the preset threshold to 80% of the single memory block capacity of the graphics processor.

[0014] In a second aspect, the present invention provides a multimodal knowledge graph storage optimization system for an airborne environment, the system being used to implement the method described in the first aspect, the system comprising: The preprocessing module is used to acquire multimodal graph data in the helicopter cockpit scene and preprocess the graph data. The splitting module is used to split the graph data into adaptive subgraphs, and to group and sort each of the adaptive subgraphs to obtain an edge data set with optimized storage order; The mapping module is used to perform memory allocation and address mapping on the adaptation subgraph based on the unified virtual addressing method. The deployment module is used to deploy direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping. The loading module is used to combine the edge data set with the optimized storage order, enable the memory merging prefetch function of the graphics processor, and preload the unprocessed part of the adapted subgraph data. The segmentation module is used to segment the high-dimensional vertices in the adaptation subgraph and generate virtual vertices with balanced dimensions. The implementation module is used to incorporate the virtual vertices into the parallel computing process of the graphics processor, replacing the high-dimensional vertices in the computation.

[0015] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.

[0016] Fourthly, the present invention provides a non-transitory computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0017] This invention provides a method and system for optimizing the storage of multimodal knowledge graphs in an airborne environment. The method includes: acquiring multimodal graph data in a helicopter cockpit scenario and preprocessing the graph data; splitting the graph data into adaptation subgraphs, grouping and sorting each adaptation subgraph to obtain an edge data set with optimized storage order; allocating memory and mapping addresses for the adaptation subgraphs based on a unified virtual addressing method; deploying direct memory access technology, establishing a direct memory operation link through a PCIe channel based on the adaptation subgraphs after memory allocation and address mapping; enabling the memory merging and prefetching function of the graphics processor (GPU) to preload the unprocessed parts of the adaptation subgraph data, combined with the edge data set with optimized storage order; segmenting high-dimensional vertices in the adaptation subgraphs to generate virtual vertices with balanced dimensions; and incorporating the virtual vertices into the parallel computing process of the GPU to replace the high-dimensional vertices in the computation.

[0018] Multimodal, large-scale graph data in the helicopter cockpit environment is the core data foundation for ensuring the safe and efficient execution of complex missions by modern helicopters. It organizes all flight-related information through a graph structure, supporting situational awareness, intelligent decision-making, and human-machine collaboration. Processing this data faces challenges such as its massive scale, diverse modalities, and high real-time requirements. It necessitates the use of techniques such as subgraph segmentation, memory optimization, and virtual nodes to ensure efficient processing with limited hardware resources, ultimately improving the helicopter's survivability and mission performance in complex environments. Understanding the organization and processing methods of this multimodal graph data is of great significance for helicopter avionics system design, intelligent cockpit development, and flight safety assurance. Attached Figure Description

[0019] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for optimizing the storage of multimodal knowledge graphs in an airborne environment, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a geometric partitioning algorithm provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an airborne multimodal knowledge graph storage optimization system provided in an embodiment of the present invention; Among them, 10 is the preprocessing module, 20 is the splitting module, 30 is the mapping module, 40 is the deployment module, 50 is the loading module, 60 is the segmentation module, and 70 is the implementation module. Detailed Implementation

[0021] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Figure 1 A flowchart illustrating a visual knowledge system modeling method provided by this invention is shown below. Figure 1 As shown in the figure, an embodiment of the present invention provides a method for optimizing the storage of multimodal knowledge graphs in an airborne environment, comprising: S101, acquire multimodal graph data in the helicopter cockpit scene, and preprocess the graph data; specifically, in this embodiment, the multimodal graph data in the helicopter cockpit scene is essentially an airborne scene knowledge graph; wherein the knowledge graph entities correspond to the vertices of the graph data, and the relations correspond to the edges of the graph data; S102, the graph data is split into adaptation subgraphs, and each adaptation subgraph is grouped and sorted to obtain an edge data set with optimized storage order; S103, based on the unified virtual addressing method, memory allocation and address mapping are performed on the adaptation subgraph; S104 deploys direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping. S105, in conjunction with the edge data set with optimized storage order, the memory merging and prefetching function of the graphics processor is enabled to preload the unprocessed portion of the adapted subgraph data; in this embodiment, the unprocessed portion of the subgraph data refers to the unaccessed data required for subsequent operations. S106, segment the high-dimensional vertices in the adaptation subgraph to generate multiple virtual vertices with balanced dimensions; S107, the virtual vertex is incorporated into the parallel computing process of the graphics processor, replacing the high-dimensional vertex in the computing.

[0023] The multimodal knowledge graph storage optimization method provided in this embodiment solves the problems of existing technologies, such as lack of adaptation to multimodal scenarios in airborne helicopter cockpits, fragmented multimodal data processing, and low real-time performance and accuracy.

[0024] Generally, large-scale graph data refers to unstructured graph data with 10 million or more vertices and 100 million or more edges, where the size of a single dataset exceeds the physical memory of a single Graphics Processing Unit (GPU). GPU physical memory typically ranges from 16GB to 48GB. The core characteristic of graph data is the presence of vertex-edge relationships, and the distribution of the number of edges associated with a vertex follows a power-law property. In this embodiment, multimodal large-scale graph data in a helicopter cockpit environment refers to an ultra-large-scale dataset that integrates various types of data, such as flight parameters, sensor information, environmental perception data, and human-machine interaction commands, organized through a graph structure. This data cannot be fully loaded into a single GPU memory for processing and requires special storage and computation optimization strategies.

[0025] Specifically, the multimodal data in the helicopter cockpit scenario mainly includes: flight status parameters, power system parameters, navigation and geographic information, environmental perception data, human-machine interaction data, system status and alarms, etc. When multimodal data from a helicopter cockpit is organized using a graph structure, it has the following characteristics: Vertex types include: Entity vertices: Represent helicopter system components such as engines, rotors, avionics, etc.; geographical entities such as airports, navigation points, obstacles, etc.; and mission elements such as targets, threats, etc. State apex: Represents the flight state at a specific moment, such as "altitude 200m + airspeed 120 knots + hovering state", system state, such as "engine temperature too high", etc. Event apex: Represents a critical operation or event such as "emergency landing procedure initiated"; Edge types include: Temporal edges: connect the states of the same entity at different points in time, reflecting the evolution of the state. Causal edge: indicates the causal relationship between events, such as "engine failure, alarm trigger, pilot operation"; Spatial edge: Represents the spatial relationship between geographic entities, such as "Obstacle A is 500m away from the flight path"; Dependency edges: represent functional dependencies between system components, such as "hydraulic system failure, rotor control limitation"; Command edge: Indicates the flow of commands in human-computer interaction, such as "pilot voice command, system execution".

[0026] Multimodal, large-scale graph data in the helicopter cockpit environment is the core data foundation for ensuring the safe and efficient execution of complex missions by modern helicopters. It organizes all flight-related information through a graph structure, supporting situational awareness, intelligent decision-making, and human-machine collaboration. Processing this data faces challenges such as its massive scale, diverse modalities, and high real-time requirements. It necessitates the use of techniques such as subgraph segmentation, memory optimization, and virtual nodes to ensure efficient processing with limited hardware resources. This allows the knowledge graph to effectively support critical tasks such as fault diagnosis, situational awareness, and path planning in the airborne environment, ultimately improving the helicopter's survivability and mission performance in complex environments. Understanding the organization and processing methods of this multimodal graph data is of great significance for helicopter avionics system design, intelligent cockpit development, and flight safety assurance.

[0027] In the multimodal knowledge graph storage optimization method provided by this embodiment of the invention, step S101 involves acquiring multimodal graph data in a helicopter cockpit scenario and preprocessing the graph data, including: S1011, acquire multimodal graph data of the helicopter cockpit scene, the multimodal graph data includes spatial vertices and non-spatial vertices; bind coordinates to each of the spatial vertices.

[0028] Specifically, in this embodiment, the spatial vertices include terrain, obstacles, navigation points, etc.; the non-spatial vertices include engine status, alarm events, etc.; each spatial vertex is bound to coordinates, for example, terrain vertices are bound to three-dimensional coordinates including latitude, longitude and altitude, and obstacle vertices are bound to their spatial position coordinates.

[0029] In this embodiment, the preprocessing stage can lay a precise, efficient, and stable data foundation for subsequent subgraph splitting, storage optimization, and GPU parallel computing; Binding spatial vertices to coordinates in helicopter cockpit multimodal graph data provides core support for subsequent subgraph splitting, spatial correlation analysis, and efficient GPU computing. First, providing bound two-dimensional / three-dimensional coordinates is a key prerequisite for subsequent geometric algorithms such as recursive coordinate bisection to split and adapt the subgraph. Based on the coordinates, the graph data can be divided into spatial regions to ensure that the split subgraphs have spatial continuity, avoid the vertices across spatial regions being cut off, and ensure the integrity of the association of spatial data such as terrain, obstacles, and navigation points, which meets the spatial situational awareness requirements of the helicopter cockpit. Secondly, coordinate information can clearly define the core association attributes between spatial vertices, such as position, distance, and relative orientation, and can be directly used to construct key edge data such as "spatial distance edge" and "threat range edge". For example, by calculating the distance between obstacles and helicopter flight paths using coordinates, data support can be provided for flight safety early warning, and the accuracy of fault diagnosis and situation assessment can be improved. Furthermore, the subgraph based on coordinate partitioning has highly clustered vertices and edges in space, and the corresponding data storage has good locality. When the GPU is computing, it can achieve continuous access to data based on spatial coordinates, reduce the frequency of random memory reads, improve cache hit rate, and thus speed up the computing speed of tasks such as spatial situation analysis and path planning, meeting the real-time requirements of airborne environments.

[0030] In a method for optimizing the storage of multimodal knowledge graphs in an airborne environment provided by an embodiment of the present invention, step S102 involves splitting the graph data into adapting subgraphs, grouping and sorting each adapting subgraph to obtain a set of edge data with optimized storage order, including: S1021, The graph data is split based on a geometric partitioning algorithm to obtain multiple initial subgraphs; S1022, according to a preset threshold, the initial subgraph is integrated into several subgraphs of similar size, namely the adapted subgraph; S1023, group and sort all edges according to the source vertices in the adaptation subgraph to obtain the edge data set with optimized storage order.

[0031] Specifically, in this embodiment, all vertices can be sorted according to the coordinate direction that is strongly related to the flight scene; the half of the vertices with smaller coordinates can be assigned to the first subgraph, and the other half can be assigned to the second subgraph; the first and second subgraphs can be repeatedly divided by recursive coordinate bisection to obtain multiple initial subgraphs. Specifically, such as Figure 2 As shown, this embodiment of the invention employs a geometric partitioning algorithm, utilizing vertex coordinate information to partition the graph. This is based on the following assumptions: Besides the vertex set V = (v1, v2, ..., v...),... n In addition to the vertices themselves, there exist two-dimensional or three-dimensional coordinates for the vertices. For any vi ∈ V, there exists a two-dimensional coordinate v. i = (x i y i (or three-dimensional coordinates v) i = (x i y i , z i Then, select any coordinate direction (e.g., the y-direction) as the reference, and sort all vertices according to their y-coordinates. Assign half of the vertices with smaller y-coordinates to P1, and the other vertices to P2. Then, use the recursive coordinate bisection (RCB) method to partition the graph in multiple ways.

[0032] In this embodiment, by splitting the data in S1022, several subgraphs of similar size are obtained, enabling large-scale graph data to adapt to GPU memory capacity and avoiding the problem of being unable to load due to excessive data. The subgraphs generated in this step provide independent data units for subsequent edge sorting and provide a clear data capacity reference for memory allocation, that is, ensuring that the allocated memory space matches the size of the subgraph. By matching the storage order of the edges with the GPU memory access mode in S1023, the foundation for efficient access is laid for subsequent operations. The sorting result directly provides "continuous access data characteristics" for the subsequent memory merging prefetching function, allowing the prefetching module to accurately identify the data blocks that need to be loaded later and improve prefetching efficiency.

[0033] In an airborne environment multimodal knowledge graph storage optimization method provided by this invention, step S103, based on a unified virtual addressing method, performs memory allocation and address mapping on the adaptation subgraph, including: S1031, based on the total capacity of the subgraph and the size of a single subgraph, a unified virtual addressing mode is enabled to map the host memory of the central processing unit and the memory of the graphics processing unit to the same virtual address space. S1032, according to the priority of the subgraph, map each subgraph to a contiguous virtual address space within the corresponding partition.

[0034] Specifically, in this embodiment, in the helicopter cockpit airborne system, the central processing unit (CPU) host memory is used to store multimodal graph data not loaded into the GPU (such as historical mission subgraphs, non-real-time environmental data), core program instructions of the avionics system, temporary flight control parameters, etc.; specifically, address partitions can be divided according to high priority such as fault alarm subgraphs and flight status subgraphs. In this embodiment, the CPU host memory, i.e. the main memory of the helicopter avionics system, is directly accessible to the CPU, while the GPU can indirectly access it through Unified Virtual Addressing (UVA) and Direct Memory Access (DMA). This step provides core support for the efficient storage and scheduling of multimodal graph data in airborne environments through a hierarchical mapping design with unified virtual addressing; First, by breaking down physical memory isolation in this step, unified address management is achieved. The S1031 maps the CPU host memory and GPU memory to the same virtual address space, eliminating the need for the CPU to participate in complex address translation. The GPU can directly access the adaptation subgraph data in both types of memory through virtual addresses. This eliminates the address barrier for data transfer from CPU memory to GPU memory in traditional architectures, greatly simplifies the addressing process for subsequent DMA transfers, shortens data transfer preparation time, and meets the real-time requirements of airborne scenarios. Secondly, priority-based partitioning and mapping ensures exclusive access to core task resources. The S1032 is based on subgraph priority. For example, if the fault alarm subgraph has a higher priority than the flight status subgraph, virtual address partitions are divided. High-priority subgraphs can be mapped to the GPU high-bandwidth memory partition, while low-priority subgraphs are mapped to the CPU host memory partition. This design avoids core tasks and non-core tasks competing for GPU high-speed memory resources, ensuring the data access rate of critical tasks such as fault diagnosis and flight situation awareness. At the same time, it utilizes the large capacity advantage of the CPU host memory to store low-priority data, improving the overall memory utilization rate. Furthermore, by implementing continuous subgraph addressing and adapting to the GPU memory prefetching characteristics, each subgraph is mapped to a continuous virtual address space. Combined with the edge data sorting optimization in the previous steps, the GPU memory merging prefetching function can accurately predict the address range of subsequent unaccessed subgraph data based on the current computation address, thereby achieving parallel data prefetching and computation and further reducing the waiting latency of GPU computation.

[0035] In the airborne environment multimodal knowledge graph storage optimization method provided by this embodiment of the invention, step S104 involves deploying direct memory access technology and establishing a direct memory operation link through a PCIe channel, including: S1041, Based on the unified virtual addressing address space, a direct memory access link for the PCIe channel is established to transfer the subgraph from the central processing unit host memory to the graphics processing unit memory; S1042, schedule transmission tasks according to subgraph priority.

[0036] This step addresses the pain points of efficiency and security in multimodal graph data transmission in airborne environments through a collaborative design of DMA technology, PCIe link, and priority scheduling. First, this step eliminates the reliance on CPU relay, significantly improving data transmission efficiency. The S1041 establishes a DMA link based on a unified virtual addressing address space, allowing data to be directly transferred from CPU host memory to GPU memory via the PCIe channel, without CPU involvement in data transfer. This reduces CPU resource consumption (the CPU can simultaneously handle other core tasks of the avionics system), and the high bandwidth of the PCIe channel enables high-speed transmission of large-capacity adaptive subgraphs. Compared to the traditional CPU relay mode, transmission latency is reduced by more than 50%, meeting the real-time requirements of airborne scenarios. Secondly, this step prioritizes data transmission to ensure the timeliness of core tasks. The S1042 priority scheduling mechanism can prioritize the transmission of high-priority subgraph data such as fault alarms and flight status, ensuring that such core data enters the GPU memory for computation as soon as possible, and preventing low-priority subgraphs (such as historical task data) from occupying transmission bandwidth. This ensures the response speed of critical tasks such as fault diagnosis and situational awareness from the transmission level, and improves the operational safety and reliability of the airborne system. Furthermore, parallel processing of data transfer and computation is achieved, improving GPU resource utilization. The independent transfer characteristics of the DMA link allow the GPU to process the current subgraph data while the DMA background simultaneously transfers the next high-priority subgraph data. Combined with the contiguous memory address mapping of the preceding steps, the transferred data can be directly captured by the GPU memory prefetch function, achieving seamless "computation-transfer" integration, significantly improving the GPU's parallel utilization, and avoiding resource idleness caused by waiting for data.

[0037] In an airborne environment multimodal knowledge graph storage optimization method provided by this invention, step S106, segmenting the high-dimensional vertices of the adaptation subgraph to generate multiple dimensionally balanced virtual vertices, includes: S1061, Based on the edge data set with the optimized storage order, perform vertex-related edge count statistics, i.e. dimension statistics, for the vertices in the adaptation subgraph; S1062, Combining dimensional statistics results, set vertex dimension limit value standards based on the power-law characteristics of graph data; S1063, based on the vertex dimension limit value standard, segment the high-dimensional vertices in the adaptation subgraph whose dimensions exceed the vertex dimension limit value standard, and generate multiple virtual vertices with balanced dimensions.

[0038] Specifically, in this embodiment, the power-law characteristic of graph data originates from the statistical regularity of real graph data, defined as the distribution characteristic where a few vertices have a large number of associated edges (i.e., high-dimensional vertices), while most vertices have very few associated edges. The boundary value standard for this step needs to be combined with the subgraph size and memory access efficiency to ensure that the segmented virtual vertices will not cause memory fragmentation due to being too small, or exceed the pre-allocated memory space due to being too large. Specifically, in this embodiment, the vertex dimension boundary value standard is usually set according to the quantile of the number of associated edges of a vertex, such as the 95th quantile, which means that only 5% of vertices have more than this value, providing a clear quantitative basis for vertex segmentation. In the preceding steps, since unified virtual addressing memory allocation has been performed, the adapted subgraph data has been mapped to GPU memory. The computation for identifying high-dimensional vertices can be executed in parallel on the GPU, avoiding CPU latency. Simultaneously, the direct memory access data transfer in the preceding steps has loaded the adapted subgraph data into GPU memory, allowing direct data access during the identification process and ensuring real-time performance. Therefore, through this step, based on the optimized storage data set of edge data, the sorted edges are grouped according to the source vertices, enabling rapid counting of the number of associated edges for each vertex without traversing messy edge data, significantly improving the efficiency of high-dimensional vertex identification. Furthermore, this step eliminates access irregularities caused by differences in vertex dimensions, and based on efficient data transfer capabilities, ensures that the segmented virtual vertex data can be quickly loaded into GPU memory, preparing for parallel computation.

[0039] In the multimodal knowledge graph storage optimization method provided by this embodiment of the invention, step S1022 involves integrating the initial subgraph into several subgraphs of approximately similar size, i.e., the adapted subgraph, according to a preset threshold, including: S10221, Set the preset threshold to 80% of the capacity of a single memory block of the graphics processor.

[0040] Setting the preset threshold to 80% of the GPU's single memory block capacity to fit the subgraph can accurately balance memory utilization and system stability in airborne scenarios. First, to avoid the risk of GPU memory overflow and ensure operational stability, a threshold of 80% is reserved with 20% of GPU memory redundancy space, which can accommodate temporary data generated during subgraph operations (such as intermediate results of virtual vertex splitting and edge association calculation cache), avoiding the interruption of operations and system crashes caused by subgraph data filling up GPU memory, which meets the high reliability requirements of helicopter cockpit systems. Secondly, maximizing GPU memory utilization and reducing data redundancy transmission thresholds avoids both excessively small subgraph size (leading to low GPU memory utilization and too many subgraphs) and exceeding the subgraph size limit, ensuring that a single adapted subgraph can be loaded into GPU memory at once to participate in the operation, without having to split the subgraph into multiple transmissions, greatly reducing the frequency and time of DMA data transmission and improving computational efficiency. Finally, adapting to the parallel computing characteristics of GPUs improves task response speed. Approximately sized subgraphs can evenly distribute the computing load of GPU cores, avoiding the imbalance of "large subgraphs occupying too many cores and small subgraphs having idle cores". At the same time, the fixed-ratio threshold design standardizes the subgraph size, which makes it easier for the GPU memory prefetch function to predict data addresses, further shortening the computing wait time and meeting the real-time requirements of onboard tasks.

[0041] like Figure 3 As shown, this embodiment, based on providing a method for optimizing the storage of multimodal knowledge graphs in an airborne environment, also provides a system for optimizing the storage of multimodal knowledge graphs in an airborne environment to implement any of the methods described in this embodiment. The system includes: Preprocessing module 10 is used to acquire multimodal graph data in a helicopter cockpit scene and preprocess the graph data; The splitting module 20 is used to split the graph data into adaptive subgraphs, group and sort each of the adaptive subgraphs to obtain an edge data set with optimized storage order; Mapping module 30 is used to perform memory allocation and address mapping on the adaptation subgraph based on the unified virtual addressing method; Deployment module 40 is used to deploy direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping. The loading module 50 is used to combine the edge data set with the optimized storage order, enable the memory merging prefetch function of the graphics processor, and preload the unprocessed part of the adaptation subgraph data. The segmentation module 60 is used to segment the high-dimensional vertices in the adaptation subgraph and generate virtual vertices with multiple dimensions balanced. The implementation module 70 is used to incorporate the virtual vertex into the parallel computing process of the graphics processor, replacing the high-dimensional vertex in the computing.

[0042] Multimodal, large-scale graph data in the helicopter cockpit environment is the core data foundation for ensuring the safe and efficient execution of complex missions by modern helicopters. It organizes all flight-related information through a graph structure, supporting situational awareness, intelligent decision-making, and human-machine collaboration. Processing this data faces challenges such as its massive scale, diverse modalities, and high real-time requirements. It necessitates the use of techniques such as subgraph segmentation, memory optimization, and virtual nodes to ensure efficient processing with limited hardware resources, ultimately improving the helicopter's survivability and mission performance in complex environments. Understanding the organization and processing methods of this multimodal graph data is of great significance for helicopter avionics system design, intelligent cockpit development, and flight safety assurance.

[0043] Furthermore, based on the methods and systems provided in the embodiments of the present invention, the present invention also provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the methods described in the various embodiments.

[0044] Furthermore, based on the above methods, systems, and electronic devices, this invention also provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described method embodiments.

[0045] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0046] Based on the above embodiments, the present invention provides a method and system for optimizing the storage of multimodal knowledge graphs in an airborne environment. The method includes: acquiring multimodal graph data in a helicopter cockpit scenario and preprocessing the graph data; splitting the graph data into adaptation subgraphs, grouping and sorting each adaptation subgraph to obtain an edge data set with optimized storage order; allocating memory and mapping addresses for the adaptation subgraphs based on a unified virtual addressing method; deploying direct memory access technology, establishing a direct memory operation link through a PCIe channel based on the adaptation subgraphs after memory allocation and address mapping; enabling the memory merging and prefetching function of the graphics processor to preload the unprocessed parts of the adaptation subgraph data, combined with the edge data set with optimized storage order; segmenting high-dimensional vertices in the adaptation subgraphs to generate virtual vertices with balanced dimensions; and incorporating the virtual vertices into the parallel computing process of the graphics processor to replace the high-dimensional vertices in the computation.

[0047] Multimodal, large-scale graph data in the helicopter cockpit environment is the core data foundation for ensuring the safe and efficient execution of complex missions by modern helicopters. It organizes all flight-related information through a graph structure, supporting situational awareness, intelligent decision-making, and human-machine collaboration. Processing this data faces challenges such as its massive scale, diverse modalities, and high real-time requirements. It necessitates the use of techniques such as subgraph segmentation, memory optimization, and virtual nodes to ensure efficient processing with limited hardware resources, ultimately improving the helicopter's survivability and mission performance in complex environments. Understanding the organization and processing methods of this multimodal graph data is of great significance for helicopter avionics system design, intelligent cockpit development, and flight safety assurance.

[0048] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0049] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0050] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for optimizing the storage of multimodal knowledge graphs in an airborne environment, characterized in that, The method includes: Acquire multimodal graph data in a helicopter cockpit scene, and preprocess the graph data; The graph data is split into adaptive subgraphs, and each adaptive subgraph is grouped and sorted to obtain an edge data set with optimized storage order; Based on the unified virtual addressing method, memory allocation and address mapping are performed on the adaptation subgraph; Deploy direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping; Combined with the edge data set with the optimized storage order, the memory merging prefetch function of the graphics processor is enabled to preload the unprocessed portion of the adapted subgraph data; The high-dimensional vertices in the adaptation subgraph are segmented to generate multiple virtual vertices with balanced dimensions. The virtual vertices are incorporated into the parallel computing process of the graphics processor, replacing the high-dimensional vertices in the computation.

2. The method for optimizing multimodal knowledge graph storage in an airborne environment according to claim 1, characterized in that, The acquisition of multimodal graph data in the helicopter cockpit scene, and the preprocessing of the graph data, include: Acquire multimodal graph data of a helicopter cockpit scene, the multimodal graph data including spatial vertices and non-spatial vertices; bind coordinates to each of the spatial vertices.

3. The method for optimizing the storage of multimodal knowledge graphs in an airborne environment according to claim 1, characterized in that, The step of splitting the graph data into adaptable subgraphs, grouping and sorting each adaptable subgraph to obtain an edge data set with optimized storage order includes: The graph data is split based on a geometric partitioning algorithm to obtain multiple initial subgraphs; According to a preset threshold, the initial subgraph is integrated into several subgraphs of similar size, namely the adapted subgraph; Based on the source vertices in the adaptation subgraph, all edges are grouped and sorted to obtain the edge data set with the optimized storage order.

4. The method for optimizing the storage of multimodal knowledge graphs in an airborne environment according to claim 1, characterized in that, The method based on unified virtual addressing, which performs memory allocation and address mapping on the adaptation subgraph, includes: Based on the total capacity of the subgraph and the size of a single subgraph, a unified virtual addressing mode is enabled to map the host memory of the central processing unit and the memory of the graphics processing unit to the same virtual address space. According to the priority of the subgraph, each subgraph is mapped to a contiguous virtual address space within the corresponding partition.

5. The method for optimizing the storage of multimodal knowledge graphs in an airborne environment according to claim 4, characterized in that, The deployment of Direct Memory Access technology establishes a direct memory operation link through the PCIe channel, including: Based on a unified virtual addressing address space, a direct memory access link for PCIe channels is established to transfer the subgraph from the central processing unit host memory to the graphics processing unit memory. Transmission tasks are scheduled according to subgraph priority.

6. The method for optimizing the storage of multimodal knowledge graphs in an airborne environment according to claim 4, characterized in that, The step of segmenting the high-dimensional vertices in the adaptation subgraph to generate multiple dimensionally balanced virtual vertices includes: Based on the edge data set with the optimized storage order, the number of vertex-related edges is counted for vertices in the adaptation subgraph, i.e., dimension counting. Based on the dimensionality statistics, a standard for vertex dimension limit values ​​is set according to the power-law characteristics of graph data. Based on the vertex dimension limit value standard, high-dimensional vertices in the adaptation subgraph that exceed the vertex dimension limit value standard are segmented to generate multiple virtual vertices with balanced dimensions.

7. The method for optimizing the storage of multimodal knowledge graphs in an airborne environment according to claim 3, characterized in that, The process of integrating the initial subgraph into several subgraphs of approximately the same size, i.e., the adapted subgraphs, according to a preset threshold, includes: Set the preset threshold to 80% of the single memory block capacity of the graphics processor.

8. A multimodal knowledge graph storage optimization system for airborne environments, characterized in that, The system is used to implement the method as described in any one of claims 1-7, the system comprising: The preprocessing module is used to acquire multimodal graph data in the helicopter cockpit scene and preprocess the graph data. The splitting module is used to split the graph data into adaptive subgraphs, and to group and sort each of the adaptive subgraphs to obtain an edge data set with optimized storage order; The mapping module is used to perform memory allocation and address mapping on the adaptation subgraph based on the unified virtual addressing method. The deployment module is used to deploy direct memory access technology, which establishes a direct memory operation link through the PCIe channel based on the adaptation subgraph after memory allocation and address mapping. The loading module is used to combine the edge data set with the optimized storage order, enable the memory merging prefetch function of the graphics processor, and preload the unprocessed part of the adapted subgraph data. The segmentation module is used to segment the high-dimensional vertices in the adaptation subgraph and generate virtual vertices with balanced dimensions. The implementation module is used to incorporate the virtual vertices into the parallel computing process of the graphics processor, replacing the high-dimensional vertices in the computation.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images