Large model analysis decision method and system based on real-time data stream

Abstract

The invention discloses a large model analysis decision method and system based on real-time data streams, and the method comprises the steps: collecting original real-time log data, and segmenting the log data into micro-batch data in real time according to a fixed time window; constructing a resource pool containing a plurality of large model instances, and dynamically allocating each micro-batch to the large model instance with the lowest current load according to the corresponding priority under a load balancing strategy; and submitting the micro-batch data to the large model for analysis, and generating a corresponding security policy or decision suggestion. According to the method, reasonable scheduling is carried out by utilizing the large model resource pool, quasi-real-time security analysis decision is realized, and the analysis time is greatly shortened. According to the invention, the detection speed and accuracy are improved, the requirement of manual intervention is reduced, and the overall working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data security and analysis technology, and in particular to a method for implementing quasi-real-time security analysis and decision-making by utilizing micro-batch technology and a large model resource pool. Background Art

[0002] When existing real-time data analysis systems use large models for analysis and calculation, they typically process each newly received data record individually and call the large model to perform in-depth analysis. For example, in a network security monitoring system, whenever a new network connection or user login attempt is detected, the system immediately feeds the event into the large model for analysis to determine whether there is a potential security threat. However, this piecemeal approach results in frequent calls to the large model, which rapidly consumes system resources and creates bottlenecks.

[0003] Because existing technical solutions require a large amount of computing resources each time a large model is called, and the data streams in actual applications are usually continuous and massive, frequent calls to large models not only lead to resource constraints but also prevent other tasks from being processed in a timely manner. In addition, a single large model instance is often unable to handle multiple concurrent requests at the same time, causing some tasks to wait in line, further increasing latency. Even if multiple large model instances are available, traditional solutions often lack an effective resource scheduling mechanism, resulting in poor resource utilization.

[0004] Existing real-time data analysis systems face challenges such as low resource utilization, high response latency, inflexible static resource configuration, and low automation when using large models for computation. These issues severely impact the real-time performance and effectiveness of the systems, particularly in security applications that require rapid response times. Summary of the Invention

[0005] The purpose of the present invention is to provide a large-model analysis and decision-making method and system based on real-time data stream, which can achieve quasi-real-time security analysis and decision-making by dividing real-time data into micro-batch data and utilizing a large-model resource pool for reasonable scheduling.

[0006] The technical solution adopted in the present invention is: A large-scale model analysis and decision-making method based on real-time data streams includes the following steps: Step 1: Collect the original real-time log data and split the data into micro-batches according to fixed time windows in real time; Furthermore, step 1 specifically includes the following steps: Step 1-1, data collection: The data collector regularly collects raw real-time log data from devices such as firewalls, intrusion detection systems (IDS), and routers.

[0007] Step 1-2, Data Cleaning: Use regular expressions and other data cleaning techniques to remove irrelevant fields and noise data, retaining key information such as timestamps, operation types, user IDs, etc. Standardization: Standardize data from different sources to ensure data consistency in subsequent steps.

[0008] Steps 1-3, micro-batch splitting: set a time window and dynamically adjust the time window based on the actual load; aggregate all collected data records in each time window and generate a micro-batch file containing all events and related information within the time period.

[0009] Furthermore, steps 1-3 specifically include the following steps: Step 1-3-1, set the time window: select 5 seconds as the time window (the default is 5 seconds, adjust the time window according to the actual situation), that is, all data records within every 5 seconds are processed as a micro-batch.

[0010] Step 1-3-2, Dynamic Adjustment: The system can dynamically adjust the time window size based on actual load conditions. For example, during high-traffic periods, the time window can be shortened to 3 seconds to improve response speed; during low-traffic periods, it can be extended to 10 seconds to reduce resource consumption.

[0011] Step 1-3-3, data aggregation: In each time window, all collected data records are aggregated and a micro-batch file is generated, which contains all events and related information in the time period.

[0012] Step 2: Resource pool management and scheduling: Build a resource pool containing multiple large model instances, and dynamically assign each micro-batch to the large model instance with the lowest current load based on its corresponding priority under the load balancing strategy; Furthermore, a certain number of spare large model instances are reserved in the resource pool in step 2; health checks are performed regularly on each large model instance in the resource pool, and once an instance is found to be faulty, it is immediately marked as unavailable, and a spare instance is started to replace it.

[0013] Furthermore, in step 2, a timeout period (e.g., 60 seconds) is set for each micro-batch; if processing cannot be completed within the specified time, the system will mark the corresponding batch as abnormal and attempt to reschedule or take other remedial measures.

[0014] Step 3: Large model security analysis and decision-making: Submit the micro-batch data to the large model for analysis and generate corresponding security policies or decision recommendations.

[0015] Furthermore, step 3 specifically includes the following steps: Step 3-1: perform feature extraction on the data records in each micro-batch and convert the text information into a numerical vector representation; Step 3-2: Use the large model to conduct in-depth analysis of the extracted feature vectors to identify potential security threat patterns; Step 3-3: Generate corresponding security policies or decision recommendations based on the security threat model.

[0016] The present invention also discloses a large-scale model analysis and decision-making system based on real-time data flow, which mainly involves the following components and communication connection relationships: Data collector: responsible for collecting raw real-time data from various network nodes and performing preliminary cleaning and formatting.

[0017] Micro-batch processor: Splits the cleaned real-time data into micro-batches according to fixed time windows.

[0018] Resource Pool Manager: Maintains a resource pool containing multiple large model instances and dynamically allocates large model resources based on demand.

[0019] Security analysis module: Uses large models to analyze micro-batch data and generates corresponding security policies or decision recommendations.

[0020] Furthermore, the relevant components of the present invention are connected through an internal network or cloud service, and can use the patent of the present invention to efficiently call large models in real time, thereby achieving quasi-real-time data analysis and computing capabilities.

[0021] The present invention adopts the above technical solution, which has the following technical advantages compared with the existing traditional technology: 1. Micro-batch Data Processing Technology: The most important technical point of this invention is to divide real-time data into micro-batches according to fixed time windows, which solves the resource bottleneck problem caused by frequent calls to large models in traditional methods. In this way, the frequency of large model calls is greatly reduced, improving the overall performance of the system. 2. Large Model Resource Pool Scheduling Technology: The key technical point of this invention is to build a resource pool containing multiple large model instances and dynamically allocate large model resources according to demand. This approach ensures that each micro-batch of data can obtain available large model resources for analysis in a timely manner, avoiding the problem of real-time data being unable to be called due to large model analysis delays. 3. Quasi-real-time Security Analysis and Decision-making Mechanism: This invention also implements a quasi-real-time security analysis and decision-making mechanism by submitting micro-batch data to the large model for analysis and generating corresponding security policies or decision recommendations. This method not only improves the system's response speed, but also reduces the need for manual intervention, improving overall work efficiency.

[0022] This invention addresses the resource bottlenecks and high latency issues associated with real-time data analysis caused by frequent calls to large models in traditional approaches. By introducing micro-batch technology and a large model resource pool, it enables near-real-time security analysis and decision-making, improving system response speed and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 This is a schematic diagram of the process architecture of a large-scale model analysis and decision-making method based on real-time data flow in the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0025] In existing security analysis systems, traditional methods often fail to meet the demand for fast analysis within seconds, facing the rapidly changing and massive data volumes of real-time data streams. This is especially true when invoking large models for analytical decision-making. Each invocation consumes significant computing resources, and large models are typically invoked in batches, resulting in lengthy analysis times and the inability to achieve real-time response. Furthermore, if the large model is invoked individually for each event, the resulting high access frequency can overwhelm the model's resources, impacting overall system performance.

[0026] like Figure 1 As shown, the present invention discloses a large model analysis and decision-making method based on real-time data stream, which includes the following steps: Step 1: Collect the original real-time log data and split the data into micro-batches according to fixed time windows in real time; Furthermore, step 1 specifically includes the following steps: Step 1-1, data collection: The data collector regularly collects raw real-time log data from devices such as firewalls, intrusion detection systems (IDS), and routers.

[0027] Step 1-2, Data Cleaning: Use regular expressions and other data cleaning techniques to remove irrelevant fields and noise data, retaining key information such as timestamps, operation types, user IDs, etc. Standardization: Standardize data from different sources to ensure data consistency in subsequent steps.

[0028] Steps 1-3, micro-batch splitting: set a time window and dynamically adjust the time window based on the actual load; aggregate all collected data records in each time window and generate a micro-batch file containing all events and related information within the time period.

[0029] Furthermore, steps 1-3 specifically include the following steps: Step 1-3-1, set the time window: select 5 seconds as the time window (the default is 5 seconds, adjust the time window according to the actual situation), that is, all data records within every 5 seconds are processed as a micro-batch.

[0030] Step 1-3-2, Dynamic Adjustment: The system can dynamically adjust the time window size based on actual load conditions. For example, during high-traffic periods, the time window can be shortened to 3 seconds to improve response speed; during low-traffic periods, it can be extended to 10 seconds to reduce resource consumption.

[0031] Step 1-3-3, data aggregation: In each time window, all collected data records are aggregated and a micro-batch file is generated, which contains all events and related information in the time period.

[0032] Step 2: Resource pool management and scheduling: Build a resource pool containing multiple large model instances, and dynamically assign each micro-batch to the large model instance with the lowest current load based on its corresponding priority under the load balancing strategy; Furthermore, a certain number of spare large model instances are reserved in the resource pool in step 2; health checks are performed regularly on each large model instance in the resource pool, and once an instance is found to be faulty, it is immediately marked as unavailable, and a spare instance is started to replace it.

[0033] Furthermore, in step 2, a timeout period (e.g., 60 seconds) is set for each micro-batch; if processing cannot be completed within the specified time, the system will mark the corresponding batch as abnormal and attempt to reschedule or take other remedial measures.

[0034] Specifically, during resource pool initialization and large model instance configuration, the resource pool manager starts three or more large model instances and registers them with the resource pool. Each instance runs in a separate container or virtual machine environment to ensure isolation and stability.

[0035] Load balancing strategy: Use Round Robin or other advanced load balancing algorithms (such as least connections first) to ensure that each micro-batch is assigned to the large model instance with the lowest current load.

[0036] Scheduling algorithm: Priority scheduling: Set a priority for each micro-batch. High-priority micro-batches (such as batches involving urgent security incidents) will be scheduled to available large model instances for processing.

[0037] Dynamic resource allocation: When a large model instance is busy, the system automatically selects the next available instance. If all instances are busy, newly arriving micro-batches are queued for processing until an instance becomes free.

[0038] Timeout mechanism: Set a timeout for each micro-batch (e.g., 60 seconds). If processing cannot be completed within the specified time, the system will mark the batch as abnormal and attempt to reschedule or take other remedial measures.

[0039] Fault recovery algorithm: Health check: Regular health checks are performed on each large model instance in the resource pool to ensure its normal operation. Once an instance is found to be faulty, the system will immediately mark it as unavailable and start a backup instance to replace it.

[0040] Redundancy: To improve system reliability, a certain number of backup large model instances are usually reserved in the resource pool. These backup instances are usually on standby and are only activated when the primary instance fails.

[0041] Step 3: Large model security analysis and decision-making: Submit the micro-batch data to the large model for analysis and generate corresponding security policies or decision recommendations.

[0042] Furthermore, step 3 specifically includes the following steps: Step 3-1: perform feature extraction on the data records in each micro-batch and convert the text information into a numerical vector representation; Step 3-2: Use the large model to conduct in-depth analysis of the extracted feature vectors to identify potential security threat patterns; Step 3-3: Generate corresponding security policies or decision recommendations based on the security threat model.

[0043] Specifically, data analysis: Feature extraction: For each data record in a micro-batch, feature extraction is first performed to convert the text information into a numerical vector representation. This step is usually implemented using word embedding technology (such as BERT).

[0044] Pattern recognition: Utilizes large models to conduct in-depth analysis of extracted feature vectors to identify potential security threat patterns. For example, by analyzing user behavior sequences, we can determine whether there are abnormal login behaviors or malicious attack attempts.

[0045] Through the above detailed implementation process, the present invention not only solves the performance bottlenecks and delay problems existing in traditional methods, but also realizes quasi-real-time security analysis and decision-making by introducing micro-batch technology and large model resource pool scheduling, significantly improving the system's response speed and processing efficiency.

[0046] The present invention is applicable to security monitoring systems that need to process real-time data streams, and is particularly suitable for network security protection, log auditing and other application scenarios. The technical solution of the present invention mainly involves the following components and communication connection relationships: Data collector: responsible for collecting raw real-time data from various network nodes and performing preliminary cleaning and formatting.

[0047] Micro-batch processor: splits the cleaned real-time data into micro-batches according to fixed time windows.

[0048] Resource Pool Manager: Maintains a resource pool containing multiple large model instances and dynamically allocates large model resources based on demand.

[0049] Security analysis module: Uses large models to analyze micro-batch data and generates corresponding security policies or decision recommendations.

[0050] Furthermore, the relevant components of the present invention are connected through an internal network or cloud service, and can use the patent of the present invention to efficiently call large models in real time, thereby achieving quasi-real-time data analysis and computing capabilities.

[0051] The large-scale model security analysis and decision-making method based on real-time data stream proposed in the present invention achieves significant benefits in many aspects, as follows: (1) Improve response speed: By dividing real-time data into micro-batch data and using the large-scale model resource pool for reasonable scheduling, quasi-real-time security analysis and decision-making are achieved, greatly shortening the analysis time. (2) Reduce resource consumption: Change the previous practice of calling the large model separately for each event, reduce the calling frequency of the large model, reduce resource consumption, and improve the overall performance of the system. (3) Enhance system flexibility: It can not only enhance the detection capability of known threat patterns, but also better cope with various unknown challenges that may arise in the future. This method not only improves the detection speed and accuracy, but also reduces the need for manual intervention and improves overall work efficiency.

[0052] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

Claims

1. A large-scale model analysis and decision-making method based on real-time data streams, characterized by: It includes the following steps: Step 1: Collect the original real-time log data and split the log data into micro-batches according to fixed time windows in real time; Step 2: Build a resource pool containing multiple large model instances and dynamically allocate each micro-batch to the large model instance with the lowest current load based on the corresponding priority under the load balancing strategy; In step 3, the micro-batch data is submitted to the large model for analysis, and corresponding security policies or decision recommendations are generated.

2. The large-scale model analysis and decision-making method based on real-time data stream according to claim 1 is characterized by: Step 1 specifically includes the following steps: Step 1-1, data collection: the data collector regularly collects raw real-time log data; Step 1-2, data cleaning: Use data cleaning technology to remove irrelevant fields and noise data to retain key information; key information includes timestamp, operation type, user ID; standardization processing Steps 1-3, micro-batch splitting: set the time window and aggregate all collected data records in each time window, and generate a micro-batch file containing all events and related information within the time period.

3. The large-scale model analysis and decision-making method based on real-time data stream according to claim 2 is characterized by: Steps 1-3 specifically include the following steps: Step 1-3-1, set the time window: select a specified time length as the time window, that is, all data records within each set time length are processed as a micro-batch; Step 1-3-2, dynamic adjustment: the system dynamically adjusts the time window size according to the actual load situation; Step 1-3-3, data aggregation: Aggregate all collected data records in each time window and generate a micro-batch file containing all events in the time period and their related information.

4. The large-scale model analysis and decision-making method based on real-time data stream according to claim 1 is characterized by: In step 2, a certain number of spare large model instances are reserved in the resource pool; health checks are performed regularly on each large model instance in the resource pool. Once an instance is found to be faulty, it is immediately marked as unavailable and a spare instance is started to replace it.

5. The large-scale model analysis and decision-making method based on real-time data stream according to claim 1 is characterized by: In step 2, a timeout is set for each micro-batch. If processing cannot be completed within the specified time, the system will mark the corresponding batch as abnormal and attempt to reschedule or take other remedial measures.

6. The large-scale model analysis and decision-making method based on real-time data stream according to claim 1 is characterized by: Step 3 specifically includes the following steps: Step 3-1: perform feature extraction on the data records in each micro-batch and convert the text information into a numerical vector representation; Step 3-2: Use the large model to conduct in-depth analysis of the extracted feature vectors to identify potential security threat patterns; Step 3-3: Generate corresponding security policies or decision recommendations based on the security threat model.

7. A large-scale model analysis and decision-making system based on real-time data streams, according to a large-scale model analysis and decision-making method based on real-time data streams according to any one of claims 1 to 6, characterized in that: The system includes the following components: Data collector: responsible for collecting raw real-time data from various network nodes and performing preliminary cleaning and formatting; Micro-batch processor: divides the cleaned real-time data into micro-batches according to fixed time windows; Resource Pool Manager: maintains a resource pool containing multiple large model instances and dynamically allocates large model resources based on demand; Security analysis module: Uses large models to analyze micro-batch data and generates corresponding security policies or decision recommendations.

8. The large-scale model analysis and decision-making system based on real-time data stream according to claim 7 is characterized in that: Components are connected through internal networks or cloud services, and real-time data uses large models to achieve quasi-real-time data analysis and computing capabilities.

Images