Data security analysis method based on high-concurrency scheduling large model

Abstract

The invention discloses a data security analysis method based on a high-concurrency scheduling large model, and the method comprises the steps: obtaining different types of security events for vectorization calculation, carrying out the recognition and classification and batch of vector features, merging the similar types of security events into a data set of one batch, and outputting data sets of N batches; creating a resource pool with a plurality of large model instances, and dynamically distributing the security events to be processed to the large model instance with the lowest current load according to the corresponding security event batch priority under a load balancing strategy; configuring a special memory module for each large model instance, wherein the memory module is used for storing historical event information and corresponding analysis results; the data stored in the memory module is analyzed through a correlation analysis algorithm, and the relation between similar events occurring at different time points is searched; and finally, a subsequent risk assessment strategy is adjusted based on the latest data analysis result. According to the invention, efficient and accurate data security analysis decision is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security analysis, and particularly relates to a data security analysis method based on high-concurrency scheduling of large models. BACKGROUND

[0002] In the current field of data security analysis, with the increasing complexity of network environments and the explosive growth of data volume, real-time monitoring and analysis become crucial. Facing hundreds of simultaneous security events, the traditional processing method mainly relies on calling large models one by one for analysis. Each security event is individually sent to a large model for in-depth analysis to determine whether there is a potential threat. Although this method can provide relatively accurate security analysis results, it highly depends on the capabilities of large models, and each security event needs to independently occupy a large model resource once, which leads to a huge consumption of computing resources.

[0003] The existing technical method meets the needs of security analysis to some extent, but in actual application, it also exposes some significant problems: (1) Huge consumption of computing resources: Each call to a large model for analysis consumes a large amount of computing resources, especially when a large number of security events need to be processed simultaneously, this resource consumption quickly accumulates, causing a heavy burden on the system. (2) Too long response time: Since each security event needs to go through the analysis process of a large model one by one, this greatly prolongs the overall response time. Especially in high-concurrency scenarios, the time waiting for all events to complete analysis one by one may far exceed the acceptable range, failing to meet the real-time requirements. (3) Bottleneck caused by high access frequency: Processing each event individually means frequent calls to large models, which easily overburdens the resources of large models, not only reducing the processing efficiency of the system, but also possibly causing service interruption or delay, affecting the stability and reliability of the system. (4) Lack of effective resource integration and optimization: The existing processing method fails to fully consider how to effectively integrate and utilize limited computing resources. For example, for similar types of security events, if they can be processed together, it can greatly reduce resource consumption and improve processing speed while ensuring analysis accuracy. SUMMARY

[0004] The purpose of the present application is to provide a data security analysis method based on high-concurrency scheduling of large models, which can effectively manage and schedule computing resources while ensuring analysis accuracy, and realize efficient and fast security event analysis and decision-making.

[0005] The technical solution adopted by the present application is: A data security analysis method based on high-concurrency scheduling of large models, comprising the following steps: Step 1, obtain different types of security events for vector calculation, and identify and classify the vector features in batches. Similar types of security events are merged into a batch of data sets, and N batch data sets are output. Step 2, large model instance resource pool and scheduling mechanism: create a resource pool with multiple large model instances, and dynamically allocate the security events to be processed to the large model instance with the lowest current load according to the corresponding security event batch priority under the load balancing strategy. Step 3, build long memory chain: configure a special memory module for each large model instance to store historical event information and corresponding analysis results; analyze the data stored in the memory module through correlation analysis algorithm to find the connection between similar events occurring at different time points; finally, adjust the subsequent risk assessment strategy based on the latest data analysis results.

[0006] Further, the security events in step 1 include account login events, abnormal period login events, and abnormal file download events.

[0007] Further, in step 2, each large model instance maintains a state flag (such as idle, busy) to track the current working state of the instance.

[0008] Further, step 2 specifically includes the following steps: Step 2-1, large model instance resource pool initialization: create and initialize multiple large model instances according to system requirements and hardware resource conditions and store them in the resource pool; at the same time, maintain a state flag for each large model instance to track the current working state of the instance; Step 2-2, queue scheduling mechanism: establish at least one task queue with different priorities to store the security event batches to be processed, so as to allocate the security event batches to idle large model instances for execution according to the priority; Step 2-3, waiting resource management: dynamically increase or decrease the number of large model instances according to the length of the waiting task queue to save resources; Step 2-4, error retry mechanism: when an exception occurs during the processing of a large model instance, the error type is identified and the error information is recorded, and whether the current task needs to be retried and the number of retries is determined according to the error type; Step 2-5, load balancing and optimization: continuously monitor the work load of each large model instance, and allocate tasks to the large model instance with the lowest current load based on the load balancing strategy; Further, in step 2-3, when the length of the waiting task queue exceeds the set upper limit value during the peak period, the number of large model instances is dynamically increased according to the actual situation; when the length of the waiting task queue is less than the set lower limit value during the trough period, the number of instances is reduced to save resources. Further, in step 2-3, when the waiting time of the task in the waiting state exceeds the preset time, the task is re-assigned to other available large model instances.

[0009] Further, in step 3, the historical event information includes the timestamp of the event occurrence, the specific operation involved, and the context environment.

[0010] Further, in step 3, the content in the memory module is constantly updated as new events occur.

[0011] The above technical scheme is adopted in the present application, and compared with the prior art, the present application has the following beneficial effects: (1) The vectorization and classification grading technology for security events solves the problem of low efficiency of traditional methods in handling high-concurrency security events. Through vectorization and classification grading, the processing speed and accuracy are improved. (2) The large model instance resource pool scheduling technology ensures the reasonable scheduling of large model instances in high-concurrency scenarios, improving resource utilization and service quality. (3) The long-chain memory mode for security risk prediction realizes cross-time period security event correlation analysis by establishing a long-chain memory mode, providing more accurate data security analysis capability, which helps to discover potential security threats in advance.

[0012] The present application solves the problems of low efficiency and untimely response of traditional methods in handling high-concurrency security events. By introducing vectorization and classification grading technology, large model instance resource pool scheduling mechanism and long-chain memory mode, efficient and accurate data security analysis and decision-making are realized, and the response speed and processing efficiency of the system are improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described in detail below in combination with the drawings and specific embodiments. Figure 1 The flowchart of the data security analysis method based on high-concurrency scheduling large model of the present application is shown. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application.

[0015] As shown in the drawings, Figure 1 The present application discloses a data security analysis method based on high-concurrency scheduling large model, which includes the following steps: It includes the following steps: Step 1, different types of security events are acquired for vectorization calculation, and the vector features are identified and classified in batches, similar types of security events are merged into a batch of data sets, and N batches of data sets are output; Further, the security events include account login events, abnormal period login events, and abnormal file download events.

[0016] Specifically, before the security event data calls the large model analysis, the data is first vectorized and classified in batches. Different types of security events (such as account login events, abnormal period login events, and abnormal file download events) are vectorized and calculated. The vector analysis model is used to identify and classify batches of these vector features, and similar types of security events are merged into a batch of data sets.

[0017] Through the above steps, a large number of security events are vectorized and classified, and N batches of data sets are output. Then N batches of data can obtain instances from the large model instance resource pool to realize concurrent calling of large model instances for security event analysis and concurrent calling of large models.

[0018] Step 2, large model instance resource pool and scheduling mechanism: create a resource pool with multiple large model instances, and dynamically allocate the security events to be processed according to the corresponding security event batch priority under the load balancing strategy to the large model instance with the lowest current load; Further, each large model instance in step 2 maintains a state flag (such as idle, busy) for tracking the current working state of the instance.

[0019] Specifically, in order to meet the ability of concurrent access to large models for concurrent analysis of multiple security events, a large model instance resource pool is designed, which can realize the ability of parallel computing and analysis of multiple instances. The following designs the mechanism and algorithm of this resource scheduling.

[0020] Further, step 2 specifically includes the following steps: Step 2-1, large model instance resource pool initialization, according to system requirements and hardware resource conditions, pre-create and initialize multiple large model instances and store them in the resource pool; at the same time, maintain a state flag for each large model instance to track the current working state of the instance; Specifically, define N large model instances: first, according to system requirements and hardware resource conditions, pre-create and initialize a certain number (N) of large model instances, and store them in a resource pool. Each large model instance should have the ability to run independently. State flag: maintain a state flag (such as idle, busy) for each large model instance to track the current working state of the instance.

[0021] Step 2-2, queue scheduling mechanism: establish at least one task queue with different priorities for storing security event batches to be processed, so as to allocate security event batches to idle large model instances according to priority for execution; Specifically, task queue: establish one or more task queues for storing batches of security events to be processed. These queues can be prioritized to ensure that high-priority tasks are processed in a timely manner. Allocation strategy: when a new batch of security events arrives, the system will select an idle large model instance based on the current state of each instance in the resource pool to execute the task. If all instances are busy, the task will be placed in the waiting queue until an instance becomes available.

[0022] Step 2-3, waiting resource management: dynamically increase or decrease the number of large model instances based on the length of the waiting task queue to save resources; Further, in step 2-3, when the length of the waiting task queue exceeds the set upper limit value during peak periods, dynamically increase the number of large model instances according to actual conditions; when the length of the waiting task queue is less than the set lower limit value during trough periods, reduce the number of instances to save resources; Specifically, during peak periods, if the waiting queue is too long, the number of large model instances can be dynamically increased according to actual conditions to speed up processing; during trough periods, the number of instances can be appropriately reduced to save resources.

[0023] Further, in step 2-3, when the waiting time of the waiting task exceeds the preset time, the task is re-assigned to other available large model instances.

[0024] Specifically, for tasks that have been in a waiting state for a long time, a timeout mechanism is set. Once the preset time is exceeded and the task is not processed, it is re-assigned to other large model instances that may have completed a previous task and become available.

[0025] Step 2-4, error retry mechanism: when an exception occurs during the processing of a large model instance, causing the task to fail, identify the error type and record the error information, and decide whether to retry the current task and the number of retries based on the error type; Error detection: during the processing of a large model instance, if an exception occurs causing the task to fail, the system should be able to automatically identify and record error information.

[0026] Intelligent retry: decide whether to retry and the number of retries based on the error type. For example, for some temporary failures (such as connection failures caused by network fluctuations), a limited number of retries can be arranged; for permanent errors (such as incorrect data format), it is directly marked as failed and the administrator is notified.

[0027] Step 2-5, load balancing and optimization: continuously monitor the workload of each large model instance and distribute tasks to the large model instance with the lowest current load based on a load balancing strategy; Specifically, the workload of each large model instance is continuously monitored, including CPU utilization, memory occupation and other key indicators. Based on the above monitoring results, intelligent algorithms are used to optimize task allocation strategies to keep the load balanced among instances and avoid the situation where some instances are overburdened while others are idle. Through the above design, not only can the large model instance resources be effectively managed and scheduled, but also stable and efficient analysis services can be provided in the face of high concurrency scenarios, meeting the real-time requirements.

[0028] Step 3, build long memory chain: configure a special memory module for each large model instance to store historical event information and corresponding analysis results; analyze the data stored in the memory module through correlation analysis algorithm to find the connection between similar events occurring at different time points; finally adjust the subsequent risk assessment strategy based on the latest data analysis results Further, the historical event information in step 3 includes the timestamp of the event, the specific operation involved, and the context environment.

[0029] Further, the content in the memory module is updated continuously with the occurrence of new events in step 3.

[0030] Specifically, because the same type of events are scheduled to the same large model instance, a special memory module can be configured for each large model instance to store historical event information and its analysis results. These information includes but is not limited to the timestamp of the event, the specific operation involved, and the context environment. At the same time, the correlation analysis algorithm can be implemented: advanced algorithms are developed or adopted to analyze the data stored in the memory module to find the connection between similar events occurring at different time points. For example, if a user has tried to log in to the system at non-working hours multiple times in the past week, it may indicate a risk of account theft. Finally, the dynamic updating mechanism: with the occurrence of new events, the content in the memory module is updated continuously to ensure that all relevant information can be reflected in time. At the same time, the subsequent risk assessment strategy is adjusted based on the latest data analysis results.

[0031] The long chain memory mode establishes a framework for continuous learning and adaptation, enabling security analysis to go beyond the current state of affairs and span the time dimension to fully grasp the development trend of network security situation, which is of great significance to improve the overall defense level.

[0032] The present application demonstrates the processing mechanism of security events, including vectorization and classification of security events, scheduling of large model instance resource pool, and processing of event data batches by multiple large model instances. Through vectorization and classification, events are assigned to different large model instances for long chain memory mode construction and correlation analysis, ultimately realizing risk prediction.

[0033] Compared with the prior art, the application has the following beneficial effects: (1) The problems of low efficiency in processing high-concurrency security events in the traditional method are solved by the vectorization and classification grading technology of security events, and the processing speed and accuracy are improved through vectorization and classification grading. (2) The large model instance resource pool scheduling technology is used to ensure the reasonable scheduling of large model instances in a high-concurrency scenario, and the resource utilization and service quality are improved. (3) The security risk prediction of the long-chain memory mode, by establishing a long-chain memory mode, realizes the correlation analysis of security events across time periods, provides more accurate data security analysis capability, and helps to discover potential security threats in advance.

[0034] The application solves the problems of low efficiency and untimely response in processing high-concurrency security events in the traditional method, realizes efficient and accurate data security analysis and decision-making by introducing the vectorization and classification grading technology, the large model instance resource pool scheduling mechanism and the long-chain memory mode, and improves the response speed and processing efficiency of the system.

[0035] Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

Claims

1. A data security analysis method based on high concurrency scheduling large models, characterized by: The method comprises the following steps: Step 1: obtaining different types of security events for vector calculation, identifying and classifying the vector features in batches, merging similar types of security events into a batch of data sets, and outputting N batch data sets; Step 2: large model instance resource pool and scheduling mechanism: creating a resource pool with multiple large model instances, and dynamically allocating the security events to be processed to the large model instance with the lowest current load according to the corresponding security event batch priority under the load balancing strategy; Step 3: building a long memory chain: configuring a special memory module for each large model instance to store historical event information and corresponding analysis results; analyzing the data stored in the memory module through correlation analysis algorithm to find the connection between similar events occurring at different time points; finally, adjusting the subsequent risk assessment strategy based on the latest data analysis results.

2. The data security analysis method based on high concurrency scheduling large model according to claim 1, characterized in that: The security events in step 1 include account login events, abnormal period login events, and abnormal file download events.

3. The data security analysis method based on high concurrency scheduling large model according to claim 1, characterized in that: In step 2, each large model instance maintains a state marker to track the current working state of the instance.

4. The data security analysis method based on high concurrency scheduling large model according to claim 1, characterized in that: Step 2 specifically comprises the following steps: Step 2-1: large model instance resource pool initialization, creating and initializing multiple large model instances according to system requirements and hardware resource conditions and storing them in the resource pool; at the same time, maintaining a state marker for each large model instance to track the current working state of the instance; Step 2-2: queue scheduling mechanism: establishing at least one task queue with different priorities for storing security event batches to be processed, so as to allocate security event batches to idle large model instances according to priority; Step 2-3: waiting resource management: dynamically increasing or decreasing the number of large model instances according to the length of the waiting task queue to save resources; Step 2-4: error retry mechanism: when an exception occurs during the processing of a large model instance, the error type is identified and the error information is recorded, and whether the current task needs to be retried and the number of retries are determined according to the error type; Step 2-5: load balancing and optimization: continuously monitoring the work load of each large model instance, and allocating tasks to the large model instance with the lowest current load based on the load balancing strategy.

5. The data security analysis method based on high concurrency scheduling large model according to claim 4, characterized in that: In step 2-3, when the length of the waiting task queue exceeds the set upper limit value during the peak period, the number of large model instances is dynamically increased according to the actual situation; when the length of the waiting task queue is less than the set lower limit value during the trough period, the number of instances is reduced to save resources.

6. The data security analysis method based on high concurrency scheduling large model according to claim 4, characterized in that: In step 2-3, when the waiting time of the waiting task exceeds the preset time, the task is re-assigned to other available large model instances.

7. The data security analysis method based on high concurrency scheduling large model according to claim 1, characterized in that: In step 3, the historical event information includes the timestamp of the event occurrence, the specific operation involved, and the context environment.

8. The data security analysis method based on high concurrency scheduling large model according to claim 1, characterized in that: In step 3, the content in the memory module is updated continuously with the occurrence of new events.

Images