State reconstruction method based on event traceability

Abstract

The invention discloses a state reconstruction method based on event traceability, particularly relates to the technical field of target extraction, can realize accurate reconstruction of a system state by collecting and utilizing system event data, overcomes limitation and defects of a traditional method, and improves efficiency and accuracy of state reconstruction. The event data is converted into the graphic structure, and the visual and efficient representation method is provided and used for displaying the causal relationship and sequence of the events. According to the technology, large-scale event data can be quickly processed, and the integrity and accuracy of the graph are ensured. Event data are collected and updated in real time, and the system state can be monitored and updated in real time. According to the method, the change of the system state can be quickly responded, and real-time state information is provided, so that the maintainability and the reliability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target extraction, and more specifically, to a state reconstruction method based on event tracing. Background Art

[0002] In today's Internet era, with the continuous expansion of business scale and the growth of users, many systems and applications have accumulated a large amount of operating data, which contains various events of system operation, such as operations, exceptions, failures, etc. The state of the system has also become more difficult to understand and track. There are often complex causal relationships between these events, and these relationships are the key to understanding and reconstructing the state of the system. Traditional state reconstruction methods usually rely on a large amount of log data and timed snapshots, but these data may be incomplete or tampered with, resulting in inaccurate state reconstruction.

[0003] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a state reconstruction method based on event tracing to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A state reconstruction method based on event tracing includes the following steps:

[0007] Step S1, collect various event data generated during the operation of the system, and convert these data into a visual event graph; by collecting and using system event data, the efficiency and accuracy of state reconstruction are improved. And converting event data into a graph structure provides an intuitive and efficient way to represent the causal relationship and sequence of events. It can quickly process large-scale event data and ensure the integrity and accuracy of the graph.

[0008] Step S2, normalize the collected event data, establish a state analysis model, and generate the first comprehensive performance index of the system state; automatically identify key events by analyzing information such as event attributes, impact range, and frequency of occurrence. This method can quickly locate events that have a significant impact on the system state and improve the efficiency and accuracy of state reconstruction. When verifying the accuracy and reliability of the state reconstruction method based on event tracing, a combination of simulation testing and actual scenario testing is used to verify the accuracy and reliability of the state reconstruction method based on event tracing. The performance and effect of the method can be evaluated by comparing the reconstructed state with the actual state.

[0009] Step S3, for the acquired event graph data, first analyze it and automatically identify the key events therein; substitute these key events into the previously established state analysis model, calculate the second comprehensive performance index through the model, and finally compare the first comprehensive performance index with the second comprehensive performance index to derive the degree of influence of the key events on the system state; analyze the acquired event graph data and automatically identify the key events therein. This helps to extract events that have a greater impact on the system state from a large amount of data, thereby improving efficiency. Compare the first comprehensive performance index with the second comprehensive performance index to derive the degree of influence of the key events on the system state. By comparing the comprehensive performance indexes at different time points, the impact of key events on the system state can be more intuitively understood.

[0010] Step S4, by analyzing the impact of key events on the system state and analyzing the causal relationship between key events and other events, the system state is reconstructed and processed; analyzing key events helps to identify the most important problems and events in the system, so that the system can focus on solving the most urgent problems and improve the efficiency and priority of problem handling. By analyzing key events and causal relationships, the system state can be reconstructed and processed in a targeted manner, and effective measures can be taken to solve problems and improve system performance. Analyzing key events and causal relationships helps to predict possible problems in the system and take preventive maintenance measures, thereby reducing the occurrence and impact of system failures.

[0011] Step S5, monitor the changes in the system status in real time, and use distributed and parallel processing technology to process large-scale event data; through distributed and parallel processing technology, large-scale event data processing and analysis are realized. This method can improve the efficiency of processing large-scale data. Distributed processing refers to distributing the collection, processing and analysis tasks of event data to multiple nodes for processing. This method uses technical tools such as distributed databases, message queues and parallel computing frameworks to improve the processing power and efficiency of large-scale systems. Specifically, distributed databases are used to store and manage large amounts of event data, message queues are used to realize asynchronous transmission and decoupling of data, and parallel computing frameworks are used to process data in parallel on multiple nodes to accelerate the processing process. This distributed processing method can effectively cope with the growth of system scale and the increase in data volume, and improve the scalability and performance of the system.

[0012] Step S6: After the system is rebuilt, a combination of simulation testing and actual scenario testing is used to verify the accuracy of the state reconstruction method based on event tracing. By analyzing the impact of key events on the system state, the events that have the greatest impact on performance, stability, etc. in the system can be accurately identified. By analyzing key events and causal relationships, system administrators can identify potential problems in a timely manner and take corresponding measures, thereby improving the operating efficiency and reliability of the system and reducing the possibility of system failures.

[0013] Preferably, when collecting system event data, data filling and repair technology is used to predict and supplement missing or damaged data using known events and status information, and the repaired event data, known events, system status information, and original event data are marked as Q1, K1, K2, and K3, respectively;

[0014] The expression for predicting and supplementing the repaired event data is Q1=K3+f(K1+K2), where f() is used to predict and generate missing or damaged event data based on the known event K1 and the system status information K2.

[0015] Preferably, when converting to an event graph, the data is first normalized, and then the normalized data is converted into a time series graph; then, the constructed event graph is analyzed in real time through a state analysis model to identify key events therein, and analyze the causal relationship and impact relationship between events. These event data include but are not limited to error logs, user operation records, system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposure.

[0016] Preferably, key events include system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposures.

[0017] Preferably, a machine learning algorithm is used to train the state analysis model so that the state analysis model automatically learns and identifies the causal relationship between events, and the training logic is:

[0018] Collect various event data generated in the system, including error logs and user operation records. According to the nature of the task and the characteristics of the data, select appropriate machine learning algorithms to build a state analysis model. Use the preprocessed and feature-engineered data to train the selected machine learning model. Use a validation set or cross-validation method to evaluate the performance of the trained model. Based on the evaluation results, tune the model.

[0019] Preferably, the state analysis model can use known event data to identify and predict the causal relationship of events, thereby deriving the impact of key events on the system state and evaluating the change in the system state. Its evaluation logic is:

[0020] The state analysis model uses known event data to identify and predict the causal relationship of events, and then derive the impact of key events on the system state and evaluate the changes in the system state. Its evaluation logic is:

[0021] Use the trained state analysis model to analyze known event data, identify and predict the causal relationship between events, and determine the impact of key events on the system state based on the identified event causal relationship; determine the impact of key events on the system state based on the identified event causal relationship.

[0022] Preferably, a third comprehensive index is generated in the reconstructed state and compared with the first comprehensive index. If the third comprehensive index is lower than the first comprehensive index, it means that the system state has not been fully restored. The first comprehensive index is 90 and the third comprehensive index is not less than 1.

[0023] Preferably, when processing large-scale event data, the data is stored in the distributed storage system Amazon S3, and the parallel computing framework Apache Flink is used to process the data in parallel.

[0024] Technical effects and advantages of the present invention:

[0025] The present invention can realize accurate reconstruction of system status by collecting and utilizing system event data, overcome the limitations and shortcomings of traditional methods, and improve the efficiency and accuracy of status reconstruction. The method of converting event data into a graphical structure provides an intuitive and efficient representation method for displaying the causal relationship and sequence of events. This technology can quickly process large-scale event data and ensure the integrity and accuracy of the graph. Real-time collection and updating of event data can realize real-time monitoring and updating of system status. This method can quickly respond to changes in system status and provide real-time status information, thereby improving the maintainability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0027] Figure 1 This is a schematic diagram of the structure of Example 1 of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Example 1

[0030] Figure 1 The present invention provides a state reconstruction method based on event tracing, comprising the following steps:

[0031] Step S1, collect various event data generated during the operation of the system, and convert these data into a visual event graph; by collecting and using system event data, the efficiency and accuracy of state reconstruction are improved. And converting event data into a graph structure provides an intuitive and efficient way to represent the causal relationship and sequence of events. It can quickly process large-scale event data and ensure the integrity and accuracy of the graph.

[0032] Step S2, normalize the collected event data, establish a state analysis model, and generate the first comprehensive performance index of the system state; automatically identify key events by analyzing information such as event attributes, impact range, and frequency of occurrence. This method can quickly locate events that have a significant impact on the system state and improve the efficiency and accuracy of state reconstruction. When verifying the accuracy and reliability of the state reconstruction method based on event tracing, a combination of simulation testing and actual scenario testing is used to verify the accuracy and reliability of the state reconstruction method based on event tracing. The performance and effect of the method can be evaluated by comparing the reconstructed state with the actual state.

[0033] Step S3, for the acquired event graph data, first analyze it and automatically identify the key events therein; substitute these key events into the previously established state analysis model, calculate the second comprehensive performance index through the model, and finally compare the first comprehensive performance index with the second comprehensive performance index to derive the degree of influence of the key events on the system state; analyze the acquired event graph data and automatically identify the key events therein. This helps to extract events that have a greater impact on the system state from a large amount of data, thereby improving efficiency. Compare the first comprehensive performance index with the second comprehensive performance index to derive the degree of influence of the key events on the system state. By comparing the comprehensive performance indexes at different time points, the impact of key events on the system state can be more intuitively understood.

[0034] Step S4, by analyzing the impact of key events on the system state and analyzing the causal relationship between key events and other events, the system state is reconstructed and processed; analyzing key events helps to identify the most important problems and events in the system, so that the system can focus on solving the most urgent problems and improve the efficiency and priority of problem handling. By analyzing key events and causal relationships, the system state can be reconstructed and processed in a targeted manner, and effective measures can be taken to solve problems and improve system performance. Analyzing key events and causal relationships helps to predict possible problems in the system and take preventive maintenance measures, thereby reducing the occurrence and impact of system failures.

[0035] Step S5, monitor the changes in the system status in real time, and use distributed and parallel processing technology to process large-scale event data; through distributed and parallel processing technology, large-scale event data processing and analysis are realized. This method can improve the efficiency of processing large-scale data. Distributed processing refers to distributing the collection, processing and analysis tasks of event data to multiple nodes for processing. This method uses technical tools such as distributed databases, message queues and parallel computing frameworks to improve the processing power and efficiency of large-scale systems. Specifically, distributed databases are used to store and manage large amounts of event data, message queues are used to realize asynchronous transmission and decoupling of data, and parallel computing frameworks are used to process data in parallel on multiple nodes to accelerate the processing process. This distributed processing method can effectively cope with the growth of system scale and the increase in data volume, and improve the scalability and performance of the system.

[0036] Step S6: After the system is rebuilt, a combination of simulation testing and actual scenario testing is used to verify the accuracy of the state reconstruction method based on event tracing. By analyzing the impact of key events on the system state, the events that have the greatest impact on performance, stability, etc. in the system can be accurately identified. By analyzing key events and causal relationships, system administrators can identify potential problems in a timely manner and take corresponding measures, thereby improving the operating efficiency and reliability of the system and reducing the possibility of system failures.

[0037] In this embodiment, when collecting system event data, data filling and repair technology is used to predict and supplement missing or damaged data using known events and status information. The repaired event data, known events, system status information, and original event data are marked as Q1, K1, K2, and K3, respectively.

[0038] The expression for predicting and supplementing the repaired event data is Q1=K3+f(K1+K2), where f() is used to predict and generate missing or damaged event data based on the known event K1 and the system status information K2. The use of data filling and repair technology can ensure the integrity and accuracy of event data. This means that when processing system event data, the model can identify missing or damaged data and use the available information to predict and fill these data. The benefit of this is that it avoids incomplete or inaccurate information caused by missing or damaged data, ensuring the quality and reliability of the data. By predicting and supplementing missing data, the model can make full use of the existing information in the system and generate more event data through this information, thereby improving the utilization and value of the data. In addition, repairing missing or damaged data also helps reduce the risk of data loss, protects important event data in the system, and prevents the problem of missing or incomplete information due to data loss. These measures jointly promote the stability and reliability of the system and ensure the integrity and availability of system event data.

[0039] In this embodiment, when converted into an event graph, the data is first normalized, and then the normalized data is converted into a time sequence graph; then, the constructed event graph is analyzed in real time through a state analysis model to identify key events therein, and analyze the causal relationship and influence relationship between events. These event data include but are not limited to error logs, user operation records, system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposure. By identifying key events and analyzing the causal relationship and influence relationship between events, the model can be helped to identify problems and potential risks in the system more quickly. This can help the model take action early to avoid further deterioration of the problem. Through real-time analysis of the event graph, the model can be helped to deeply understand the operation of the system. In turn, the correlation and impact degree between various events can be understood, so as to better understand the overall operation of the system. Through real-time analysis of the event graph, the model can be helped to promptly discover and respond to security issues such as malicious attacks and security vulnerabilities, thereby improving the security and stability of the system.

[0040] In this embodiment, key events include system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposure.

[0041] In this embodiment, a machine learning algorithm is used to train the state analysis model so that the state analysis model automatically learns and identifies the causal relationship between events. The training logic is:

[0042] Collect all kinds of event data generated in the system, including error logs and user operation records. According to the nature of the task and the characteristics of the data, select appropriate machine learning algorithms to build a state analysis model. Use the data that has been preprocessed and feature engineered to train the selected machine learning model. Use the validation set or cross-validation method to evaluate the performance of the trained model. According to the evaluation results, tune the model. Train the state analysis model through machine learning algorithms so that the model can automatically learn and identify the causal relationship between events without manually encoding rules or specifying causal relationships. Actively collect all kinds of event data generated in the system, including error logs, user operation records, etc. These data provide a rich source of information for the model and help build a more comprehensive and accurate state analysis model. When selecting a machine learning algorithm to build a state analysis model, the model will be selected according to the nature of the task and the characteristics of the data to ensure that the model can better adapt to different types of data and problems and improve the performance and effect of the model. At the same time, the model will be tuned according to the evaluation results, so that the performance and accuracy of the model can be continuously improved, so that the state analysis model can better adapt to the changes and needs of the system.

[0043] In this embodiment, the state analysis model uses known event data to identify and predict the causal relationship of events, thereby deriving the impact of key events on the system state and evaluating the change in the system state. The evaluation logic is:

[0044] Use the trained state analysis model to analyze known event data, identify and predict the causal relationship between events, and determine the impact of key events on the system state based on the identified event causal relationship; Based on the identified event causal relationship, determine the impact of key events on the system state. Label the state analysis model as A, the identified event causal relationship as R, the impact degree as C, and the known event data set as D;

[0045] Determination of the impact degree C of key events on system status: C = (f ( D ) ), through which the state analysis model can be used to analyze known event data to determine the degree of impact of key events on the system state. By analyzing the impact of key events on the system state, the changes in the system state can be evaluated in real time. This enables system managers to promptly discover abnormal conditions in the system so that they can take timely actions to deal with them, thereby ensuring the stable operation of the system. Based on the trained state analysis model, the system can automatically analyze event data without manual intervention, thereby saving human resource costs and improving the speed and accuracy of decision-making. By determining the degree of impact of key events on the system state, changes in the system state can be quantitatively evaluated.

[0046] In this embodiment, a third comprehensive index is generated in the reconstructed state and compared with the first comprehensive index. If the third comprehensive index is lower than the first comprehensive index, it means that the system state has not been fully restored, wherein the first comprehensive index is 90 and the third comprehensive index is not less than 1. The first comprehensive index and the third comprehensive index provide a quantitative assessment of the system state. By comparing the two, the degree of recovery of the system state can be quantitatively assessed, and setting the thresholds of the first comprehensive index and the third comprehensive index helps to standardize the process of system state assessment. When the third comprehensive index does not meet expectations, the corresponding management process can be triggered, such as notifying relevant personnel to handle or start a fault repair process, thereby improving management efficiency and the standardization of the work process.

[0047] In this embodiment, when processing large-scale event data, the data is stored in the distributed storage system Amazon S3, and the parallel computing framework Apache Flink is used to process the data in parallel. Using a distributed storage system (such as Amazon S3) can store large-scale event data and has high scalability. Using a parallel computing framework (such as Apache Flink) to process data in parallel can improve processing speed and efficiency. Flink has good parallel processing capabilities, can effectively process large-scale data, and can achieve high-performance data processing in a distributed environment. S3 supports the storage of large amounts of data and can be dynamically expanded to meet growing data needs. Distributed storage systems and parallel computing frameworks usually have good fault tolerance and can maintain the stability and reliability of the system in situations such as node failures or network interruptions. This ensures that the system can continue to operate normally in the face of unexpected situations and does not lose data or processing results.

[0048] Explanation of relevant terms

[0049] Event sourcing: Event sourcing is a design pattern for recording all activities or events of a system so that the state and history of the system can be accurately reconstructed.

[0050] State reconstruction: In event tracing, state reconstruction refers to reconstructing the state of the system at a specific point in time by analyzing the stored event data.

[0051] Key events: Key events are events that have a significant impact on the system state in the event graph.

[0052] Event Graph: An event graph is a graphical structure that represents events and their relationships.

[0053] Expected results: During the state reconstruction process, the expected results refer to the state changes that each key event should cause based on the business logic and rules of the event.

[0054] Verification results: Verification results refer to evaluating the accuracy and reliability of the state reconstruction method by comparing the reconstructed system state with the actual state.

[0055] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0056] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0057] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0058] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A state reconstruction method based on event tracing, characterized by: The steps include: Step S1, collecting various event data generated during the operation of the system and converting these data into a visual event graph; Step S2, normalizing the collected event data, establishing a state analysis model, and generating a first comprehensive performance index of the system state; Step S3, firstly analyze the acquired event graph data to automatically identify the key events therein; substitute these key events into the state analysis model established in advance, calculate the second comprehensive performance index through the model, and finally compare the first comprehensive performance index with the second comprehensive performance index to obtain the degree of influence of the key events on the system state; Step S4, reconstructing and processing the system state by analyzing the impact of the key event on the system state and analyzing the causal relationship between the key event and other events; Step S5, monitor the changes of system status in real time, and use distributed and parallel processing technology to process large-scale event data; Step S6: After the system is rebuilt, a combination of simulation testing and actual scenario testing is used to verify the accuracy of the state reconstruction method based on event tracing.

2. According to claim 1, a state reconstruction method based on event tracing is characterized in that: When collecting system event data, data filling and repair technology is used to predict and supplement missing or damaged data using known events and status information. The repaired event data, known events, system status information, and original event data are marked as Q1, K1, K2, and K3 respectively. The expression for predicting and supplementing the repaired event data is Q1=K3+f(K1+K2), where f() is used to predict and generate missing or damaged event data based on the known event K1 and the system status information K2.

3. The state reconstruction method based on event tracing according to claim 1 is characterized in that: When converting to an event graph, the data is first normalized and then converted into a time series graph; then, the constructed event graph is analyzed in real time through a state analysis model to identify key events and analyze the causal relationship and impact relationship between events. These event data include but are not limited to error logs, user operation records, system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposure.

4. The state reconstruction method based on event tracing according to claim 3 is characterized in that: Key events include system anomalies or errors, system crashes, service interruptions, data loss, malicious attacks, and security vulnerability exposure.

5. The state reconstruction method based on event tracing according to claim 4 is characterized in that: The state analysis model is trained using a machine learning algorithm so that it can automatically learn and identify the causal relationship between events. The training logic is as follows: Collect various event data generated in the system, including error logs and user operation records. According to the nature of the task and the characteristics of the data, select appropriate machine learning algorithms to build a state analysis model. Use the preprocessed and feature-engineered data to train the selected machine learning model. Use a validation set or cross-validation method to evaluate the performance of the trained model. Based on the evaluation results, tune the model.

6. The state reconstruction method based on event tracing according to claim 5 is characterized in that: The state analysis model uses known event data to identify and predict the causal relationship of events, and then derive the impact of key events on the system state and evaluate the changes in the system state. Its evaluation logic is: The state analysis model uses known event data to identify and predict the causal relationship of events, and then derive the impact of key events on the system state and evaluate the changes in the system state. Its evaluation logic is: Use the trained state analysis model to analyze known event data, identify and predict the causal relationship between events, and determine the impact of key events on the system state based on the identified event causal relationship; Based on the identified event causal relationships, determine the impact of key events on the system status.

7. The state reconstruction method based on event tracing according to claim 6 is characterized in that: A third comprehensive index is generated in the reconstructed state and compared with the first comprehensive index. If the third comprehensive index is lower than the first comprehensive index, it indicates that the system state has not been fully restored. The first comprehensive index is 90 and the third comprehensive index is not less than 1.

8. The state reconstruction method based on event tracing according to claim 7 is characterized in that: When processing large-scale event data, the data is stored in the distributed storage system Amazon S3, and the parallel computing framework Apache Flink is used to process the data in parallel.

Images