Dynamic data organization method and device based on event driving and storage medium

Abstract

The invention discloses a dynamic data organization method and device based on event driving and a storage medium, and the method comprises the steps: constructing a historical event processing knowledge base, extracting multi-dimensional features of a current emergency, converting the multi-dimensional features into vector representation, and processing the multi-dimensional features of the current emergency based on the historical event processing knowledge base, using the large model to generate data demand entries required for processing the current emergency; constructing a unified data representation, and constructing a data association relationship network according to the label set of the data; encoding the data demand entry into a data demand vector, calculating the similarity between the data demand vector and a data semantic vector, and screening out original data associated with emergency semantics; converting the data demand items into sub-graph query statements on a data association relationship network through a large model, and executing the query statements to obtain original data topologically associated with the emergency; and combining the obtained original data of the semantic association of the emergencies with the original data of the topological association of the emergencies to form final joint data so as to support the handling of the emergencies.

Description

Technical Field

[0001] This invention relates to a data organization method, and more particularly to an event-driven dynamic data organization method, device, and storage medium. Background Technology

[0002] Traditional data organization methods employ static, predefined architectures, periodically batch-process data using ETL tools, rely on manual configuration of event listening rules and data relationships, use fixed thresholds for event status monitoring, and manually code data service interfaces. These traditional methods face numerous bottlenecks in handling dynamic business events and real-time data organization, severely hindering the real-time release of data value and business agility.

[0003] Traditional data organization architectures rely on predefined data connection rules for business-specific data integration. While this predefined approach may meet basic data needs initially, it becomes extremely passive in the face of sudden business events. When business changes or processes are adjusted, existing data connection and integration strategies may fail to respond quickly, leading to data processing delays and impacting the timeliness and accuracy of business decisions. Secondly, traditional ETL tools cannot understand the implicit semantics of the business. They are primarily used to extract, transform, and load data from source systems into target systems, focusing on static data transformation and migration. These tools lack the ability to understand business requirements and event semantics, and cannot automatically adapt to business changes and user needs. This limitation results in rigid and inflexible data processing flows when facing complex business scenarios, thus affecting the efficiency and value of data utilization. Finally, cross-domain data migration is costly. In complex distributed systems, cross-node data migration is a complex and costly process. Traditional methods rely on manually configured data migration schemes, which are not only time-consuming and labor-intensive but also prone to data inconsistency and loss risks. With frequent data migrations and adjustments, the high cost becomes even more pronounced, becoming a key factor restricting data organization efficiency.

[0004] Therefore, it is of great significance to provide an event-driven dynamic data organization method, device, and storage medium. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide an event-driven dynamic data organization method, device, and storage medium to improve data utilization efficiency and achieve accurate mapping and efficient supply of demand and data.

[0006] Technical solution: The event-driven dynamic data organization method of the present invention includes:

[0007] Collect historical case studies on handling incidents, build a knowledge base for handling historical incidents, extract multi-dimensional features of current emergencies and convert them into vector representations, and use a large model to generate the data requirements for handling current emergencies based on the knowledge base on handling historical incidents.

[0008] Construct a unified data representation that includes data content, a set of data tags, and data semantic vectors; and build a data relationship network based on the set of data tags.

[0009] Data requirement items are encoded into data requirement vectors, the similarity between the data requirement vectors and the data semantic vectors is calculated, and the original data that is semantically related to the emergency is selected based on the similarity.

[0010] The large model transforms data requirement items into subgraph query statements on a data relationship network, and the execution of the query statements yields the raw data that is topologically related to the sudden event.

[0011] By combining the original data of semantic associations and topological associations of emergencies, the final joint data is formed to support emergency response.

[0012] Furthermore, the process involves collecting historical incident handling cases, constructing a historical incident handling knowledge base, extracting multi-dimensional features of the current emergency and converting them into vector representations, and using a large model based on the historical incident handling knowledge base to generate the data requirement items needed for the current emergency handling, specifically including:

[0013] Historical event handling cases were collected and preprocessed. The data was organized using a question-and-answer block method and then stored in a vector database after being vectorized by an embedded model.

[0014] Extract features of the current emergency, including text features, semantic features, and attribute features, and transform them into an event vector representation;

[0015] The similarity between the current emergency vector and the event vectors in the historical event handling case library is calculated using cosine similarity, and the vectors are sorted according to the similarity. Historical event handling cases with high similarity to the current emergency vector representation are recalled from the historical event handling knowledge base.

[0016] Construct prompt words that include the characteristics of the current emergency and recall historical case studies of emergency handling. Input these prompt words into a large model and output them in a structured manner to generate data requirement entries for the current emergency.

[0017] Furthermore, the preprocessing of historical event handling cases includes removing irrelevant information, correcting erroneous information, and standardizing data formats.

[0018] Furthermore, the text features include keywords, trigger words, and topic words; the semantic features include vector representations of event descriptions; and the attribute features include event type, spatiotemporal attributes, and event associations.

[0019] Furthermore, the method for constructing a unified data representation is as follows:

[0020] For structured data, a tag set is constructed based on business rules, and the structured data is converted into text descriptions and then vectorized to obtain semantic vectors;

[0021] For semi-structured or unstructured data, an algorithm corresponding to the data type is used to extract labels to construct a label set. Then, the corresponding embedding model or visual big model is used to convert the data into a text description and then vectorize it to obtain a semantic vector.

[0022] Furthermore, the process of encoding data requirement items into data requirement vectors, calculating the similarity between data requirement vectors and data semantic vectors, and filtering out raw data semantically related to the emergency based on the similarity specifically includes:

[0023] By using a pre-trained bidirectional context word embedding model, data demand items are encoded into demand vectors, and the total data demand vector representation is obtained by average pooling aggregation.

[0024] Semantic matching is performed by calculating the cosine similarity between the data demand vector and the data semantic vector, and the top N data items with the highest similarity are output as the semantic association result.

[0025] Furthermore, the process of converting data requirement items into subgraph query statements on a data relationship network through a large model, and obtaining raw data related to the topology of the sudden event after executing the query statements, specifically includes:

[0026] Derive the topology schema information from the data association network;

[0027] A Text2Cypher prompt word framework is constructed, which takes data requirement items and topology schema information as input to a large model, and generates Cypher query statements for querying the data relationship network through the large model;

[0028] Verify and execute the generated Cypher query statement to retrieve the original data of the topological associations from the data association network.

[0029] Furthermore, the step of deriving the topology schema information from the data association network specifically involves: calling the neo4j graph database.<CALL apoc.meta.data()> The command exports schema information, including node types, relationship types, and attributes.

[0030] The present invention provides a computer device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the event-driven dynamic data organization method described herein.

[0031] The present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the event-driven dynamic data organization method.

[0032] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) The present invention utilizes the deep semantic understanding and retrieval enhancement of the large model to generate features based on historical event handling cases, and performs feature analysis and data requirement decomposition on events. This not only overcomes the problem that traditional ETL tools cannot understand the implicit semantics of business, but also can automatically adapt to business changes and user needs; (2) Utilizing the large model embedding representation and semantic retrieval technology and the graph query generation Text2Cypher technology, the present invention achieves accurate mapping between data requirements and target data from the semantic layer and topology layer; (3) Utilizing the unified data semantic representation method, the present invention achieves logical convergence of multi-source heterogeneous data without the need to migrate the original data, reducing the risk of data inconsistency and loss, and improving data organization efficiency. Attached Figure Description

[0033] Figure 1 This is a flowchart of the event-driven dynamic data organization method of the present invention.

[0034] Figure 2 This is a schematic diagram of the framework of the dynamic data organization method of the present invention.

[0035] Figure 3 This is a flowchart illustrating the requirements for generating event data based on a large model, as described in this invention.

[0036] Figure 4 This is a diagram illustrating the unified semantic representation of data and the construction of a data association network for this invention.

[0037] Figure 5 This is a flowchart of step S3, data semantic similarity analysis, in an embodiment of the present invention.

[0038] Figure 6 This is a flowchart of step S4, data topology correlation analysis, in an embodiment of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, this embodiment provides an event-driven dynamic data organization method, including:

[0041] Step S1: Collect historical event handling cases, build a historical event handling knowledge base, extract multi-dimensional features of the current emergency and convert them into vector representations, and use a large model to generate the data requirement items required for the handling of the current emergency based on the historical event handling knowledge base.

[0042] Step S11: Collect historical event handling cases, build a historical event handling knowledge base, and store it using a vector database.

[0043] Step S111: Confirm the data source based on the specific business scenario, including collecting historical event handling case data from databases, historical records, reports, and other channels. Clean the collected data, remove irrelevant information, correct errors, and unify the data format to ensure the accuracy and consistency of the data and improve the efficiency of subsequent processing.

[0044] Step S112: For historical event handling case data, a question-and-answer (QA) segmentation method is used to parse the data. The data includes two columns: event description and event handling plan. QA data is usually stored in CSV or TXT format, encoded in UTF-8, and delimited by tabs and commas to separate questions and answers. An embedding model is selected to vectorize the parsed data, and a vector database is used for storage to ensure that the data can be efficiently retrieved and matched.

[0045] Step S12: Extract the multi-dimensional features of the current emergency and convert them into vector representations.

[0046] Event features are categorized into text features, semantic features, and attribute features;

[0047] Text features include extracting key event words from event descriptions, such as keywords like "downtime," "network latency," and "abnormality"; identifying key phrases or short sentences that trigger events, such as "occurred" and "caused"; and identifying the core themes of events, such as "hardware failure" and "network problem."

[0048] Semantic features, including vector representations of event description texts transformed using pre-trained word embedding models;

[0049] Attribute characteristics include information such as the specific type of the event, the objects involved, and time elements, such as: "2025-05-26 09:20:17 A service in a certain system reported a database connection error," etc.

[0050] Step S13: Calculate the similarity between the current emergency event vector and the event vectors in the historical event handling knowledge base using the cosine similarity formula. Sort them according to the similarity value, and recall the top N (N is generally 5) historical event handling cases with high similarity to the current emergency event vector from the historical event handling knowledge base. The most relevant events are retrieved and matched first. The specific formula is as follows:

[0051]

[0052] in, Indicates the similarity between historical events and sudden events. This is a vector representation of a sudden event. Vector representation of historical events. Let be the dimension of the vector representation.

[0053] Step S14: Utilize the large-scale model retrieval and generation capabilities, integrate recalled historical event handling cases, and generate current emergency event data requirement entries, specifically including:

[0054] Step S141: Construct a Prompt prompt template, embedding the characteristics of the emergency and the historical case data of the recalled event into the input part of the prompt;

[0055] Step S142: Input the constructed Prompt prompt into the LLM large model, and output the data requirement items of the current emergency through OutputParse in a structured manner.

[0056] Step S2: Construct a unified data representation that includes data content, a set of data tags, and data semantic vectors. Based on the data tag set, build a data relationship network, such as... Figure 3 As shown;

[0057] Step S21: Construct a unified data representation that includes data content, a set of data tags, and data semantic vectors, and perform unified modeling of the data, including unified semantic representation of structured data and unified semantic representation of semi-structured and unstructured data.

[0058] Step S211: Unified semantic representation of structured data;

[0059] Step S2111: Define rules according to specific business scenarios and construct a set of tags for structured data, such as server ID, timestamp, CPU utilization, and anomaly identifier;

[0060] Step S2112: Convert structured data into text description, select a text embedding model to learn the vector representation of the text description of structured data, and construct a set of semantic vectors for structured data.

[0061] Step S2113: Combine the obtained label set, semantic vector set, and acquisition method of the structured data to construct a unified semantic representation of the structured data.

[0062] Step S212: Unified semantic representation of semi-structured or unstructured data;

[0063] Step S2121: Based on different data types, adopt the corresponding tag extraction method to construct a semi-structured or unstructured data tag set, where the identified tags include: timestamp, server identifier, exception identifier, system component, status, log level, IP address, resource path, and operation;

[0064] For text-based data, algorithms such as named entity recognition and relation extraction are used to identify and classify named entities from the text, such as people, organizations, place names, and times. Specific natural language processing algorithm service development can prepare text annotation data, train relevant models, and release inference services according to specific business scenarios.

[0065] For image data, algorithms such as image segmentation and object detection are used to identify category labels from the images, such as aircraft type recognition, ship recognition, and terrain recognition. Specific computer vision algorithm service development can prepare image annotation data, train relevant models, and release inference services according to specific business scenarios.

[0066] For audio data, the speech is converted into text using a speech-to-text service, and then text-based data processing methods are applied.

[0067] For video data, key page images are extracted from the video by extracting key frames, and then image data processing methods are used.

[0068] Step S2122: Construct semantic vectors for semi-structured or unstructured data;

[0069] For text-based data, a text embedding model (such as bge-large-zh-v1.5) is selected to learn the vectorized representation of the text data;

[0070] For image-based data, a large visual model (such as qwen2.5-vl-instruct) is selected to convert the image into a detailed text description, and then a text-based vector representation learning method is used.

[0071] For audio data, speech is converted into text using a speech-to-text service, and then a text-based vector representation learning method is employed.

[0072] For video data, the video is converted into a collection of images by video frame segmentation, and the images are converted into detailed text descriptions by a large visual model (such as qwen2.5-vl-instruct). The text is then summarized and a text class vector representation learning method is used.

[0073] Step S2123: Combining the data tag set obtained in step S2121, the semantic vector obtained in step S2122, and the acquisition method of the semi-structured or unstructured data itself, construct a unified semantic representation of the semi-structured or unstructured data.

[0074] Step S22: Based on the data's label set, construct a data relationship network using a graph database (such as Neo4j).

[0075] Step S3: Encode the data requirement items into a data requirement vector, calculate the similarity between the data requirement vector and the data semantic vector, and filter out the original data that is semantically related to the emergency based on the similarity.

[0076] Step S31: The data requirement items are transformed into vector representations using a bidirectional contextual word embedding representation method. A deep learning model is used to capture semantic and contextual information in the text to generate richer word vector representations.

[0077] Step S311: Obtain the data requirement items from step S1, including alarm type, alarm time, alarm level, key log information, etc.

[0078] Step S312: Using a pre-trained bidirectional context word embedding model, each data requirement item... Convert to a fixed-dimensional vector representation The model, trained on a large-scale text corpus, is able to capture the semantic changes of words in different contexts. Specifically, it is represented as follows:

[0079]

[0080] in, This represents the forward propagation process of a bidirectional context word embedding model.

[0081] Step S313: Aggregate all demand vector representations using average pooling to generate the final data demand vector representation. :

[0082]

[0083] in, Indicates the number of items required for the event.

[0084] Step S32: By calculating the similarity between the data demand vector and the data semantic vector, the original data that is semantically related to the sudden event is selected based on the similarity. The calculation method is as follows:

[0085]

[0086] in, Represents the data demand vector. Represents a data semantic vector.

[0087] Based on the similarity calculation results, the original data is recalled and reordered to ensure that the original data most relevant to the alarm event requirements is obtained, and the top N data with the highest matching degree are output as the final matching result.

[0088] Step S4: Convert the data requirement items into subgraph query statements on the data association network through the large model, and obtain the original data related to the topology of the emergency after executing the query statement.

[0089] Step S41: Obtain the data requirement items from step S1, including alarm type, alarm time, alarm level, key log information, etc.

[0090] Step S42: Obtain the data association network from step S2. Based on the topology of the data association network, export the schema information such as node type, relationship type and its attributes by calling the "CALL apoc.meta.data()" command in the neo4j graph database, and use it as the input context of the large model.

[0091] Step S43: Construct a Text2Cypher prompt word framework, input the topology information of the data requirement items and the data association network, and output a structured Cypher query statement;

[0092] Step S44: In the data relationship network stored in the graph database (neo4j), the generated Cypher query statement is validated through syntax verification. If the validation is successful, the Cypher query result is returned to obtain the original data associated with the topology of the sudden event; if it fails, the process returns to step S43 to output the structured Cypher query statement.

[0093] Example as follows:

[0094] "Please generate a Cypher query statement based on the following alarm requirements and data relationship network schema:"

[0095] ① Alarm requirements

[0096] Alarm type: {Alarm type}

[0097] Alarm time: {Alarm time}

[0098] Key Logs: {Key Logs}

[0099] Other related information: {Other related information}

[0100] ② Data Relationship Network Schema

[0101] Node type:

[0102] Node1: {Node1 type}

[0103] Node2: {Node2 type}

[0104] Relationship type:

[0105] {Relationship Type 1} (From node 1 to node 2)

[0106] {Relationship Type 2} (From node 2 to node 3)

[0107] Please ensure that the generated Cypher query statement meets the following requirements:

[0108] a. Query target: Based on the information in the alarm request, query the data nodes related to it.

[0109] b. Query criteria: Set query criteria based on information such as alarm time, alarm type, and key logs.

[0110] c. Query results: Returns all nodes related to the alarm request and their relationships.

[0111] d. Query statement structure: Ensure that the query statement structure is correct, including the starting node, query conditions, and result display.

[0112] The generated Cypher query statement should meet the following format:

[0113] MATCH (n: node type { attribute: 'value'}) - [r: relation type] -> (m: node type { attribute: 'value'})

[0114] WHERE n.attribute = 'value' AND r.attribute = 'value' AND m.attribute = 'value'

[0115] RETURN n, r, m.

[0116] Step S5: Combine the original data of semantic association of emergencies and the original data of topological association of emergencies to form the final joint data to support emergency response analysis.

[0117] This embodiment also provides a computer device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the event-driven dynamic data organization method described above.

[0118] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the event-driven dynamic data organization method described above.

Claims

1. An event-driven dynamic data organization method, characterized in that, include: Collect historical case studies on handling incidents, build a knowledge base for handling historical incidents, extract multi-dimensional features of current emergencies and convert them into vector representations, and use a large model to generate the data requirements for handling current emergencies based on the knowledge base on handling historical incidents. Construct a unified data representation that includes data content, a set of data tags, and data semantic vectors; and build a data relationship network based on the set of data tags. Data requirement items are encoded into data requirement vectors, the similarity between the data requirement vectors and the data semantic vectors is calculated, and the original data that is semantically related to the emergency is selected based on the similarity. The large model transforms data requirement items into subgraph query statements on a data relationship network, and the execution of the query statements yields the raw data that is topologically related to the sudden event. By combining the original data of semantic associations and topological associations of emergencies, the final joint data is formed to support the analysis of emergency response.

2. The event-driven dynamic data organization method according to claim 1, characterized in that, The process involves collecting historical incident handling cases, constructing a historical incident handling knowledge base, extracting multi-dimensional features of current emergencies and converting them into vector representations, and using a large model to generate the data requirements for handling current emergencies based on the historical incident handling knowledge base. Specifically, this includes: Historical event handling cases were collected and preprocessed. The data was organized using a question-and-answer block method and then stored in a vector database after being vectorized by an embedded model. Extract features of the current emergency, including text features, semantic features, and attribute features, and transform them into an event vector representation; The similarity between the current emergency vector and the event vectors in the historical event handling case library is calculated using cosine similarity, and the vectors are sorted according to the similarity. Historical event handling cases with high similarity to the current emergency vector representation are recalled from the historical event handling knowledge base. Construct prompt words that include the characteristics of the current emergency and recall historical case studies of emergency handling. Input these prompt words into a large model and output them in a structured manner to generate data requirement entries for the current emergency.

3. The event-driven dynamic data organization method according to claim 2, characterized in that, The preprocessing of historical events includes removing irrelevant information, correcting erroneous information, and standardizing data formats.

4. The event-driven dynamic data organization method according to claim 2, characterized in that, The text features include keywords, trigger words, and topic words; the semantic features include vector representations of event descriptions; and the attribute features include event type, spatiotemporal attributes, and event associations.

5. The event-driven dynamic data organization method according to claim 1, characterized in that, The unified data representation is specifically as follows: For structured data, a tag set is constructed based on business rules, and the structured data is converted into text descriptions and then vectorized to obtain semantic vectors; For semi-structured or unstructured data, an algorithm corresponding to the data type is used to extract labels to construct a label set. Then, the corresponding embedding model or visual big model is used to convert the data into a text description and then vectorize it to obtain a semantic vector.

6. The event-driven dynamic data organization method according to claim 1, characterized in that, The process of encoding data demand items into data demand vectors, calculating the similarity between data demand vectors and data semantic vectors, and filtering out raw data semantically related to emergencies based on the similarity specifically includes: By using a pre-trained bidirectional context word embedding model, data demand items are encoded into demand vectors, and the total data demand vector representation is obtained by average pooling aggregation. Semantic matching is performed by calculating the cosine similarity between the data demand vector and the data semantic vector, and the top N data items with the highest similarity are output as the semantic association result.

7. The event-driven dynamic data organization method according to claim 1, characterized in that, The process involves converting data requirement items into subgraph query statements on a data relationship network using a large model. Executing these queries yields raw data related to the topology of the emergency, specifically including: Derive the topology schema information from the data association network; A Text2Cypher prompt word framework is constructed, which takes data requirement items and topology schema information as input to a large model, and generates Cypher query statements for querying the data relationship network through the large model; Verify and execute the generated Cypher query statement to retrieve the original data of the topological associations from the data association network.

8. The event-driven dynamic data organization method according to claim 7, characterized in that, The process of deriving the topology schema information from the data association network specifically involves: calling the neo4j graph database.<CALL apoc.meta.data()> The command exports schema information, including node types, relationship types, and attributes.

9. A computer device, characterized in that, It includes one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of an event-driven dynamic data organization method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of an event-driven dynamic data organization method as described in any one of claims 1-8.

Images