Data management method and system based on graph database, and storage medium

Abstract

The invention discloses a data management method based on a graph database, a storage medium and a system, and the method comprises the steps: 1, carrying out the data management through employing the graph database, and storing each record of structured data as a node; file data are stored by means of a file management system, and paths of the file data are stored in corresponding nodes in a graph database; step 2, storing nodes and child nodes by using a directed relationship of a graph database; 3, designing a data storage mode, and storing a descendant node template of the current node; and 4, designing a data exporting function, and selecting whether to establish a multi-hop connection between the node and the descendant node according to the option. According to the method and the device, the structured data and the file data are efficiently managed, and the relevance, the query efficiency and the management convenience of the data are improved.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and specifically to a data management method, storage medium, and system based on graph databases. Background Technology

[0002] In certain application scenarios, a large amount of data is generated, encompassing structured, semi-structured, and unstructured data. Taking UUV competitions as an example, the structured data, such as trajectory data and command data, records the movement trajectory information of the UUVs during the competition; the semi-structured data includes scene JSON files and configuration XML files generated during the competition; and the unstructured data includes supporting materials, videos, and images submitted by the teams to help prove the competition process and results.

[0003] Traditional data management methods struggle to effectively handle this type of complex data. Specifically, they typically use relational databases to manage structured data, while relying on file management systems to store semi-structured and unstructured data as files. This separate management model introduces several drawbacks, primarily including:

[0004] (1) Because the file data lacks effective tags and only uses the file name as a unique identifier, it often takes a lot of effort to find specific data;

[0005] (2) Due to the lack of data templates, different people store data in various formats, which makes it difficult to manage and export the required data;

[0006] (3) There is a lack of unified software management for both unstructured and structured data. Taking the acquisition of trajectory data (structured data) of a certain game as an example, users have to first query the start and end times of the game in a document (unstructured data), and then enter the corresponding database query statement to filter the data within the corresponding time period based on this time information. The entire operation process is cumbersome and complex, and requires data management personnel to master database-related knowledge, which greatly reduces the efficiency and convenience of data management. Moreover, over time, data management personnel may forget the relationships between data, which will hinder data management.

[0007] Therefore, a more effective data management method and software are needed to comprehensively manage structured and unstructured data, improve the efficiency and convenience of data management, and effectively reduce management costs. Summary of the Invention

[0008] The purpose of this invention is to provide a data management method, storage medium, and system based on graph databases to achieve efficient management of structured and file data, improving data correlation, query efficiency, and management convenience. This method and software are applicable not only to UUV competition data management but also to other types of data management. Its specific functions include:

[0009] 1) Supports adding, deleting, querying, and modifying data nodes and data node attributes;

[0010] 2) A unified database is used to manage both structured and file data, eliminating the need to repeatedly switch between structured and file data when querying data;

[0011] 3) Supports real-time and offline access to structured data;

[0012] 4) Supports defining data templates to unify data storage formats;

[0013] 5) Supports exporting data to a specified folder according to personalized export settings.

[0014] This invention will achieve the above-mentioned functions by taking the following measures to improve the efficiency of data management:

[0015] 1) A graph database is used for data management, storing each record of structured data as a node; file data is stored using a file management system, and its path is stored in the corresponding node in the graph database; nodes, node attributes, and relationships between nodes can be freely defined. A data storage model is designed to store the templates of descendant nodes of the current node. Multi-hop relationships between nodes are established based on the export settings.

[0016] 2) Develop software to provide a visual user interface. Design the page to manage data, read data from the graph database, and display the data in a tree structure. For the currently selected node, design functions for adding, deleting, querying, and modifying attributes; defining descendant data templates; adding descendant nodes according to the templates; and adding child nodes.

[0017] Specifically, the technical solution of the present invention is as follows:

[0018] A data management method based on graph database includes the following steps:

[0019] Step 1: Use a graph database for data management, storing each record of structured data as a node; file data is stored using a file management system, and its path is stored in the corresponding node of the graph database.

[0020] Step 2: Use the directed relation of the graph database to store nodes and child nodes;

[0021] Step 3: Design the data storage pattern to store the templates of the descendant nodes of the current node;

[0022] Step 4: Design the data export function, and select whether to establish multi-hop connections between nodes and their descendants based on the options.

[0023] Furthermore, the nodes and their attributes can be freely defined. The nodes include various types of business data nodes, unstructured data type nodes, and structured data type nodes; the node attributes include the attributes that various types of business data need to store.

[0024] Furthermore, the client uses Vue + Electron, the backend uses Spring Boot, and the database uses Neo4j graph database.

[0025] Furthermore, in step 2, the create statement is used to create a relationship between node s and node n, and node n becomes a descendant node of node s.

[0026] Furthermore, in step 3, firstly, for each node, a set of data nodes is created as templates for its descendant nodes; then, a relational "template" is created, pointing from the current node to the head node of the descendant node, so that the template can be called when storing new data, thereby ensuring the uniformity and convenient management of the descendant node directory structure.

[0027] Furthermore, in step 4, a personalized export function was developed to meet personalized export requirements, and multi-hop connections between data were established based on the personalized export configuration.

[0028] A graph database-based data management system is provided to implement the graph database-based data management method as described in any one of claims 1-6. The system adopts a client / server architecture to facilitate interaction with a file management system, and includes the following modules:

[0029] The menu bar module includes a query button and a settings button; the query button can jump to the data query page, and the settings button is used to open the settings page, set the login account and password, and the root directory for file storage.

[0030] The main page module displays data in a tree structure and uses lazy loading to prevent software crashes caused by excessive data volume. The tree nodes display the node name and other attribute key-value pairs, with each pair separated by a semicolon.

[0031] The information bar module allows users to view all attribute keys of the currently selected node by clicking on the tree node. Users can add, modify, and delete attribute keys. If the current node is a "file" and its attributes include "file path," "open file" and "open folder" buttons are added after the attribute. The information bar also provides operation function buttons for the current node, including "add attribute," "upload file," "add child node," "set template," "delete current node," and "access data."

[0032] In the file upload module, clicking the upload file button and selecting the corresponding file will send it to the backend. The backend will automatically store the file in the root directory and add the attribute "file path" to the current node, with the attribute value being the corresponding file path.

[0033] In the template setting module, after clicking the "Set Template" button, you will be redirected to the template settings page. You can add child nodes by clicking with the mouse and set the child node template for the current node.

[0034] Add a child node function module, which allows users to choose to add child nodes by template or customize them;

[0035] The data access module is divided into "real-time data access" and "offline data access". The "real-time data access" is configured with UDP protocol, visual configuration of the fields corresponding to the bit length, backend opening of the listening module, listening for UDP data, and parsing the received data according to the configuration and storing it in the graph database. The "offline data access" requires uploading a data file, and the backend parses the data file and imports it into the graph database.

[0036] In the data query module, the backend obtains the template of the current node and sends it to the frontend to display the data in a tree structure. Clicking on any tree node allows you to set filter conditions and choose whether to export it.

[0037] A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the steps of the graph database-based data management method.

[0038] The beneficial effects of this invention include:

[0039] 1) Efficient data relationship management: By storing and managing structured and unstructured data through graph databases, the relationships between data can be clearly represented, improving data query efficiency and management convenience;

[0040] 2) Data template management function: Users can customize child node templates as needed, unify the child node storage structure, and call templates to customize the filtering range and choose whether to export, so as to meet different data export needs;

[0041] 3) Real-time data access: It uses the UDP protocol to access data sent by external devices, which can quickly obtain structured data and ensure the timeliness and accuracy of the data;

[0042] 4) Cross-platform application: Based on the software architecture of Vue, Electron and Spring Boot, it implements a cross-platform desktop application, which is convenient for users to use on different operating systems. Attached Figure Description

[0043] Figure 1. Schematic diagram of Neo4J data storage case.

[0044] Figure 2. Schematic diagram of Neo4J personalized data search case.

[0045] Figure 3. Overall architecture diagram of the data management system based on graph database.

[0046] Figure 4. Node loading flowchart.

[0047] Figure 5 File upload flowchart.

[0048] Figure 6 Data access flowchart.

[0049] Figure 7. Flowchart for adding child nodes.

[0050] Figure 8. Flowchart of query and export data. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings.

[0053] The data management method of the present invention includes the following steps:

[0054] Step 1: Use Neo4j graph data management.

[0055] Regular nodes are created using the `CREATE` statement in Cypher. For example, to create a "Match" node with start and end times, the Cypher statement is as follows: `CREATE(n:Match{"Start Time":"2025-01-02","End Time":"2025-01-28"})`. This statement creates a node in the database with the tag "Match" and adds "Start Time" and "End Time" attributes with values ​​of "2025-01-02" and "2025-01-28" respectively.

[0056] For each set of structured data, first create a node labeled "Structured Data" for unified management, and add default attributes to it, including "Containing Fields" and "Field Description". Other attribute keys and values ​​can be added freely as needed. Then, for each structured data record, create a node labeled "data" for storage. The attribute key of each node is the value of the "Containing Fields" attribute in the corresponding "Structured Data" node; the attribute value is the specific value.

[0057] For example, to store a set of trajectory data, first use the `create` statement to create a "structured data" node: `create(n: "structured data" { "contains fields": "time, longitude, latitude, speed", "field description": "time, longitude, latitude, speed"})`. The graph database will create the corresponding node and assign an `id` as its unique identifier. In this example, assume the graph database assigns the node an `id` of 1. Then, for each specific data record, use the `create` statement to create a "data" node: `match(s) where id(s) = 1 create(s)-[r: specific data]->(n: data{"time": "2025-01-08 10:58:56.220", "lon": "121.888111", "lat": "39.111511", "speed": "10.22"})`. In this example, the `match` statement is used to match the node `s` with `id` 1, and the `create` statement is used to create the "data" node `n` and the relationship between nodes `s` and `n`.

[0058] For file data, create a node with the tag "File" and add the default attributes "File Path" and "Modification Time". The tag can also specify a data type. For example, for a video file, use the `create` statement to create "File" and "Video" nodes: `create(n:["File", "Video"]{"File Path": "d:\root\xxxx.mp4", "Modification Time": "2025-01-04 12:11:58"})`.

[0059] Step 2: Store descendant nodes.

[0060] The relationships between nodes are stored using a graph database. As in the example described in step 1, the `create` statement creates a relationship between node `s` and node `n`, making node `n` a descendant node of node `s`.

[0061] Step 3: Design the data storage mode to store the templates of the descendant nodes of the current node.

[0062] For each node, a set of data nodes can be created as templates for its descendant nodes. Then, a relational "template" is created, pointing from the current node to the head node of its descendant nodes. This allows the template to be invoked when storing new data, ensuring a consistent directory structure for descendant nodes and facilitating management.

[0063] Case examples Figure 1 As shown. Suppose we need to store data for a certain match, the designed data storage directory structure is as follows: Figure 1 As shown, default attributes can also be set for nodes in the template, for example... Figure 1 The "Task Sheet" node in the template can be set with default attributes such as "Participating Unit," "Start Time," and "End Time." The subject is the header node of the template. First, the template nodes and the relationships between them are created in the database using Cypher statements. Then, the ID is used to match the current competition data node with the template header node to create the relationship.

[0064] After creating the template, when storing new subject data, you can copy the template and modify the attributes of the corresponding nodes. This ensures that the directory structure of all stored subject data is consistent, which facilitates subsequent management.

[0065] Step 4: Design the data export function.

[0066] To address personalized export needs, a personalized export function was developed, establishing multi-hop relationships between data based on personalized export configurations. For example, in a competition, the organizers requested videos from several teams, all located in the same folder. The data storage directory structure is as follows: Figure 1As shown. If this requirement is met using the traditional method, operators would need to copy the relevant videos one by one from the file manager and then paste them into a designated unified folder. This method is not only mechanical and monotonous, consuming a lot of time and energy, but also extremely inadequate when facing complex and ever-changing demands. For example, if the organizers subsequently raise new requirements, such as reorganizing the videos according to different teams or different stages of the competition, the traditional method would require another tedious copy-and-paste process, which is inefficient and prone to errors.

[0067] For the example above, first, retrieve the user's personalized export configuration, i.e., the export directory structure is "Competition - Various Videos," while the intermediate folders "Subjects," "Task Sheets," and "Team Submission Data" do not need to be exported. At this point in the database, the competition node and video node are not directly related, so it is necessary to create a relationship between them. To avoid affecting the display of the normal tree structure data in non-query mode, set the relationship type to "query." For example... Figure 2 As shown. This allows you to query match data and individual videos, and then export them accordingly.

[0068] The detailed design of the graph database-based data management software of the present invention is described below.

[0069] This system adopts a C / S (client / server) architecture, such as Figure 3 As shown in the diagram, this architecture is designed to achieve efficient interaction with the file management system, fully leveraging the strengths of both the client and server. The client is responsible for direct interaction with the user, providing a user-friendly interface and convenient operation; the server focuses on data processing, storage, and interaction with the database, ensuring data security and consistency.

[0070] The client-side uses a combination of Vue and Electron technologies. The backend uses the Spring Boot framework. The database is Neo4j, a graph database. Neo4j is a high-performance NoSQL database that stores data in the form of a graph, where nodes represent entities and relations represent the relationships between entities. Neo4j features full support for ACID transactions and supports relational representation and querying of massive amounts of data. When adding new entities or relations, Neo4j can be easily extended. Only new nodes or relations need to be created; there's no need to consider modifying the table structure as with relational databases.

[0071] like Figure 4As shown, when a user expands an unloaded tree node, the client first checks the local cache. If the child node data does not exist, it initiates a GET request ` / api / node / children?nodeId=123`. The backend uses Cypher to query `MATCH (n)-[:Own]->(m) WHERE id(n)=$nodeId RETURN m` to obtain the list of direct child nodes. The returned data includes the node ID, name, and key-value pair attributes (formatted as label:“Subject”, name: "Subject 1";). The client uses virtual scrolling technology to dynamically render nodes and caches the results in memory to avoid duplicate requests. For very large subtrees (e.g., >1000 nodes), the backend supports pagination parameters (limit / offset).

[0072] like Figure 5 As shown, the user triggers a file upload operation through the client interface. After selecting a local file in the Electron file dialog box, the client submits the file binary stream and the current node ID to the backend via the POST / api / file / upload interface. The backend first stores the file in the preset root directory, and then updates the file path attribute of the corresponding node in Neo4j using the Cypher statement "MATCH (n) WHERE id(n)=$nodeId SET n.filepath=$path". After storage is complete, the backend returns a success response (HTTP 200), and the client refreshes the information bar, rendering an "Open File" button next to the file path attribute. This button is bound to a local file system call.

[0073] like Figure 6As shown, users can choose between offline and real-time data access. Offline data access requires users to upload data files. The frontend uses Electron's file selection API to retrieve the uploaded files and sends them to the backend via an HTTP request. Upon receiving the file, the backend parses it, using the appropriate parsing library based on the file format (e.g., CSV, JSON), and imports the parsed data into the graph database. Real-time data access requires users to configure the UDP port number and field mapping rules on the frontend (e.g., "bytes 0-4 correspond to time"). The client submits the configuration via POST / api / udp / setup. The backend starts an independent thread listening on the specified port, and upon receiving data packets, parses the binary stream according to the configuration to generate structured data (e.g., {timestamp: 1620000000}). Subsequently, a Neo4j node is dynamically created and associated with the parent node (Cypher example: CREATE (n:SensorData $props) MERGE (parent)-[:HAS_DATA]->(n)). New node events are pushed to the client in real time via WebSocket (ws: / / backend / updates), triggering dynamic updates to the interface.

[0074] like Figure 7 As shown, when a user clicks "Add Child Node," the client first calls GET / api / template?nodeId=123 to check if the current node is associated with a template. If a template exists, the backend uses the Cypher statement "match (n)-[r:template]->(k) where id (n) = $nodeId with k match (k)-[r*0..]->()". This statement first retrieves the header node k of the template node corresponding to the current node, and then uses a multi-hop query to obtain all descendant nodes and relationships of k, which is the descendant node directory structure of the current node. After clicking "Add," the backend copies the template and adds it as a descendant node of the current node. The client renders the template selection interface, where the user can further modify the attribute values. If no template exists, the user can select a custom child node, enter the label and attribute key-value pair, and add it. After the operation is completed, the client only refreshes the affected subtree area through incremental updates.

[0075] like Figure 8As shown, after the user clicks "Export Data," the backend queries the template of the root node, and the frontend renders it. The user can then configure each level of the template, including whether to export the current folder, whether to export subfolders, and setting several conditions to filter the desired data. After the user completes their settings and clicks "Query," the client encapsulates the filter tree into JSON and sends it to the backend. The backend then constructs a Cypher statement based on the settings and executes it.

[0076] Example:

[0077] The system has a default root node "root". When a user clicks the "Set Template" button, the root node will be displayed in the "Current Node Template" area in the lower left corner of the homepage. After clicking the root node, the user can add child nodes to it, thereby defining the template structure of the root node.

[0078] After completing the template settings, the user returns to the tree structure view in the top left corner, clicks the root node, and selects "Add Child Node." At this point, the "Add by Template" option can be selected. The backend system reads the defined template information and creates corresponding descendant nodes for the root node. Each time this operation is performed, a complete node structure is added. For example, after the user performs two add operations, the system will generate two complete "Competition" nodes and their subordinate structures. The user can manage the attributes of any node, including adding, modifying, or deleting nodes.

[0079] In the file management function, users can click the "Video" node under "Team Submission Data" to upload files. After clicking the "Upload File" button, a file upload window will pop up, supporting both drag-and-drop and file selection. After the file upload is complete, the server will first save the file to the preset storage directory, and then update the file path attribute of the corresponding node in the Neo4j database using the Cypher query statement MATCH (n) WHERE id(n)=$nodeId SET n.filepath=$path. Simultaneously, the client interface will display "Open File" and "Open Folder" buttons next to the file path attribute for convenient and quick file access.

[0080] The system provides two data access modes: offline data access and real-time data access.

[0081] In offline data access mode, users need to upload data files. The frontend retrieves the user-selected file using Electron's file selection API and sends it to the backend server via an HTTP request. After receiving the file, the backend calls the appropriate parsing library to process the data based on the file format (such as CSV, JSON, etc.) and finally imports the parsed data into the graph database.

[0082] In real-time data access mode, users need to configure the UDP port number and field mapping rules on the front end (e.g., setting "bytes 0-4 correspond to timestamps"). The configuration information is submitted to the backend via a POST request ( / api / udp / setup). The backend system will start a separate listening thread to receive data packets on the specified port and parse the binary data stream according to the preset mapping rules to generate structured data (e.g., {timestamp: 1620000000}). Subsequently, the system will dynamically create Neo4j nodes and associate them with their parent nodes (using the Cypher statement: CREATE (n:SensorData $props) MERGE(parent)-[:HAS_DATA]->(n)). Through a WebSocket connection, the system will push new node events to the client in real time, triggering dynamic updates to the interface.

[0083] In terms of data display, once data is accessed, the system will clearly display the accessed structured data in a table format in the information bar on the right.

[0084] The data export function allows users to flexibly configure the exported content according to their needs. After the user clicks "Export Data," the system will query the template information of the root node and render the configurable export options on the front-end interface. Users can set detailed filter conditions for each level of nodes to precisely select the data to be exported.

[0085] After completing the settings, click the "Query" button. The client will encapsulate the complete filter criteria into JSON format and send it to the backend. The backend system will then construct and execute the corresponding Cypher query based on these settings. This process may involve multi-hop queries. For example, when exporting the "Competition - Various Videos" directory structure, if it is necessary to skip intermediate folders such as "Subjects," "Task Books," and "Team Submission Data," the user needs to check both "Export Current Folder" and "Export Sub-folders" for the "Competition" node, while only checking "Export Sub-folders" for intermediate level nodes.

[0086] During query execution, the system creates a temporary "query" type relationship in the database, associating the match node with the video node. This ensures query accuracy without affecting the normal tree-structured data display. After the query results are generated, users can click "Export" to save the data to the default export folder.

[0087] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data management method based on graph database, characterized in that, Includes the following steps: Step 1: Use a graph database for data management, storing each record of structured data as a node; file data is stored using a file management system, and its path is stored in the corresponding node of the graph database. Step 2: Use the directed relation of the graph database to store nodes and child nodes; Step 3: Design the data storage pattern to store the templates of the descendant nodes of the current node; Step 4: Design the data export function, and select whether to establish multi-hop connections between nodes and their descendants based on the options.

2. The data management system based on graph database according to claim 1, characterized in that: The nodes and their attributes can be freely defined, and the nodes include various types of business data nodes, unstructured data type nodes, and structured data type nodes. The attributes of the nodes include the attributes that need to be stored for various types of business data.

3. The data management system based on graph database according to claim 1, characterized in that: The client uses Vue + Electron, the backend uses Spring Boot, and the database uses Neo4j graph database.

4. The data management system based on a graph database according to claim 1, characterized in that, In step 2, the create statement is used to create a relationship between node s and node n, making node n a descendant node of node s.

5. The data management system based on a graph database according to claim 1, characterized in that, In step 3, firstly, for each node, a set of data nodes is created as templates for its descendant nodes; then, a relation "template" is created, pointing from the current node to the head node of the descendant node, so that the template can be called when storing new data, in order to ensure the uniformity and convenient management of the descendant node directory structure.

6. The data management system based on a graph database according to claim 1, characterized in that, In step 4, a personalized export function was developed to meet personalized export requirements, and multi-hop connections between data were established based on the personalized export configuration.

7. A data management system based on a graph database, characterized in that, To implement the graph database-based data management method as described in any one of claims 1-6, the system adopts a client / server architecture to facilitate interaction with a file management system, and includes the following modules: The menu bar module includes a query button and a settings button; the query button can jump to the data query page, and the settings button is used to open the settings page, set the login account and password, and the root directory for file storage. The main page module displays data in a tree structure and uses lazy loading to prevent software crashes caused by excessive data volume. The tree nodes display the node name and other attribute key-value pairs, with each pair separated by a semicolon. The information bar module allows users to view all attribute keys of the currently selected node by clicking on the tree node. Users can add, modify, and delete attribute keys. If the current node is a "file" and its attributes include "file path," "open file" and "open folder" buttons are added after the attribute. The information bar also provides operation function buttons for the current node, including "add attribute," "upload file," "add child node," "set template," "delete current node," and "access data." In the file upload module, clicking the upload file button and selecting the corresponding file will send it to the backend. The backend will automatically store the file in the root directory and add the attribute "file path" to the current node, with the attribute value being the corresponding file path. In the template setting module, after clicking the "Set Template" button, you will be redirected to the template settings page. You can add child nodes by clicking with the mouse and set the child node template for the current node. Add a child node function module, which allows users to choose to add child nodes by template or customize them; The data access module is divided into "real-time data access" and "offline data access". The "real-time data access" is configured with UDP protocol, visual configuration of the fields corresponding to the bit length, backend opening of the listening module, listening for UDP data, and parsing the received data according to the configuration and storing it in the graph database. The "offline data access" requires uploading a data file, and the backend parses the data file and imports it into the graph database. In the data query module, the backend obtains the template of the current node and sends it to the frontend to display the data in a tree structure. Clicking on any tree node allows you to set filter conditions and choose whether to export it.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program can be executed by a processor to implement the steps of the data management method based on a graph database as described in any one of claims 1-6.

Images