A method, apparatus, equipment, and storage medium for managing multi-source heterogeneous data for secure operation and maintenance.

Abstract

This invention provides a method, apparatus, device, and storage medium for managing multi-source heterogeneous data in security operations and maintenance. It determines the collection priority of target data sources by using the information entropy gain index of semi-structured log data, achieving adaptive data collection with dynamic scheduling based on information value, thus solving the problem of low collection efficiency. By calculating the heterogeneous confusion entropy of the original data and constructing an adaptive temperature adjustment model, it performs variable-temperature semantic mapping based on dynamically calculated mapping temperature parameters according to the confusion entropy. This ensures that high-confusion-entropy fields corresponding to low temperatures undergo rigid feature preservation, while low-confusion-entropy fields corresponding to high temperatures undergo flexible normalization, solving the problem of mismapping of high-confusion fields caused by fixed mapping strategies. By calculating the causal contribution of candidate training samples to the target security model and selecting high-value samples for inclusion in the database, it solves the problem of low-value samples affecting model training performance.

Description

Technical Field

[0001] This invention relates to the field of security operation and maintenance data management, and in particular to a method, apparatus, equipment and storage medium for managing multi-source heterogeneous data for security operation and maintenance. Background Technology

[0002] In enterprise security operations and maintenance scenarios, security operation and maintenance data is typically scattered across multiple heterogeneous data sources such as firewalls, intrusion detection systems, server logs, and application logs. When security operations and maintenance personnel need to analyze specific security events, such as tracing the source of an abnormal IP address, existing technical solutions usually require operations and maintenance personnel to manually determine the data sources to be queried, log into each system separately to export data, manually align fields and convert formats of the data output from different systems, and finally perform analysis and judgment based on the integrated data.

[0003] The above process suffers from the following core problems: Different security devices use different naming conventions for fields with the same semantic meaning. For example, source IP addresses may be named "client_ip", "src_ip", or "source_address" in different systems, requiring maintenance personnel to spend a significant amount of time on field identification and manual mapping. Simultaneously, existing systems employ full-data collection or periodic batch collection modes, failing to dynamically determine the collection scope based on specific query needs, resulting in long data preparation cycles and severe resource waste. Furthermore, high-value information generated during maintenance, such as query questions, analysis context, and expert conclusions, is not systematically preserved and cannot be transformed into structured samples suitable for training large-scale language models in the security field.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention discloses a management method, apparatus, device, and storage medium for multi-source heterogeneous data in security operations and maintenance, aiming to solve the problems in existing security operations and maintenance scenarios, such as the lack of information value priority decision-making in multi-source heterogeneous data collection, the inability of heterogeneous field mapping to adaptively adjust according to the degree of data confusion, and the lack of causal contribution evaluation in training sample selection.

[0006] The first embodiment of the present invention provides a method for managing multi-source heterogeneous data for security operations and maintenance, including:

[0007] Receive security operation and maintenance query requests, parse the security operation and maintenance query requests, and extract the query intent;

[0008] A dynamic data collection strategy is constructed based on the query intent, and the collection priority of the target data source is determined based on the information entropy gain index of semi-structured log data, generating adaptive data collection instructions.

[0009] According to the adaptive data acquisition instruction, raw semi-structured log data is collected from the heterogeneous security operation and maintenance system, and the heterogeneous confusion entropy of the raw semi-structured log data is calculated. The heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of data fields.

[0010] An adaptive temperature regulation model is constructed, and the mapped temperature parameters are dynamically calculated based on the heterogeneous confusion entropy.

[0011] Variable-temperature semantic mapping is performed based on the mapping temperature parameter: For new fields not registered, the similarity distribution between the new field and the standard field is scaled using the mapping temperature parameter, and a normalized mapping confidence score is calculated; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; the format of the mapped data is unified to generate standardized data, and a context data set is constructed;

[0012] The context data set is encapsulated into candidate training samples, the causal contribution of the candidate training samples to the target security model is calculated, and high-value samples are selected and stored in the database based on the causal contribution.

[0013] Preferably, the method of determining the collection priority of the target data source based on the information entropy gain index of semi-structured log data specifically involves:

[0014] Construct a partially observable Markov decision process model, defining the current security threat state as a hidden state variable;

[0015] For each data source in the candidate data source set, calculate the degree to which the uncertainty of the security threat state is expected to decrease after collecting the data source, and obtain the information entropy gain value corresponding to the data source.

[0016] The acquisition queue is generated in descending order of the information entropy gain value;

[0017] The posterior probability distribution of the current security threat status is updated in real time during the data collection process, and the collection order of subsequent data sources is dynamically adjusted.

[0018] Preferably, the expression for calculating the heterogeneous confusion entropy of the original semi-structured log data is:

[0019]

[0020] Where H(D) represents the heterogeneous confusion entropy of the original data D, and M is the total number of heterogeneous fields contained in the data. Let K be the i-th heterogeneous field, and K be a preset set of standard fields. Predict heterogeneous field f for a pre-defined feature recognition model i The probability of belonging to standard field k; wherein, the larger the value of the heterogeneous confusion entropy, the more unknown confusion fields or adversarial features the data contains.

[0021] Preferably, the step of dynamically calculating the mapping temperature parameter based on the heterogeneous confusion entropy, and performing variable-temperature semantic mapping based on the mapping temperature parameter, specifically includes:

[0022] The mapped temperature parameter T is calculated using the following adaptive temperature adjustment function:

[0023]

[0024] in, and These are the preset upper and lower temperature boundaries, respectively; λ is the sensitivity decay coefficient; and H(D) is the heterogeneous confusion entropy.

[0025] The following variable-temperature Softmax function is used to calculate the mapping of the new field f to the standard field. Normalized mapping confidence :

[0026]

[0027] in, For the new field f and the j-th standard field The semantic feature similarity, where N is the total number of standard fields; when If the maximum value is greater than the preset threshold, perform field mapping; otherwise, retain the new field as an extended feature.

[0028] Preferably, calculating the causal contribution of the candidate training samples to the target security model specifically includes:

[0029] Obtain the gradient of the loss function of the target security model on the validation set;

[0030] Calculate the gradient of the loss function for the candidate training samples;

[0031] Based on the Shapley value approximation algorithm, the inner product of the loss function gradient of the candidate training sample and the loss function gradient on the validation set is calculated to obtain the influence function value;

[0032] The influence function value is used as the causal contribution, where a positive causal contribution indicates that the sample can reduce the error of the model on the validation set, and a negative value indicates that the sample is a harmful or redundant sample.

[0033] Preferably, the step of screening high-value samples for storage based on the causal contribution specifically includes:

[0034] Obtain the contribution filtering threshold;

[0035] If the causal contribution of the candidate training sample is greater than the contribution screening threshold, it is determined to be a high-value sample, stored in the training database, and the feature library index is updated.

[0036] If the causal contribution of the candidate training sample is less than or equal to the contribution screening threshold, it is discarded or downgraded and archived to the cold storage area.

[0037] The causal contribution of samples in the cold storage area is recalculated periodically to adapt to version iterations of the target security model.

[0038] The second embodiment of the present invention provides a management device for multi-source heterogeneous data for security operation and maintenance, comprising:

[0039] The intent parsing module is used to receive security operation and maintenance query requests, parse the security operation and maintenance query requests, and extract the query intent;

[0040] The adaptive acquisition module is used to construct a dynamic acquisition strategy based on the query intent, determine the acquisition priority of the target data source based on the information entropy gain index of semi-structured log data, and generate adaptive data acquisition instructions.

[0041] The entropy evaluation module is used to collect raw semi-structured log data from the heterogeneous security operation and maintenance system according to the adaptive data collection instruction, and calculate the heterogeneous confusion entropy of the raw semi-structured log data. The heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of data fields.

[0042] The mapped temperature parameter calculation module is used to construct an adaptive temperature regulation model and dynamically calculate the mapped temperature parameters based on the heterogeneous confusion entropy.

[0043] The variable-temperature mapping module is used to perform variable-temperature semantic mapping based on the mapping temperature parameter: for new fields not registered, the similarity distribution between the new field and the standard field is scaled using the mapping temperature parameter, and a normalized mapping confidence score is calculated; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; the format of the mapped data is unified to generate standardized data, and a context data set is constructed;

[0044] The value assessment module is used to encapsulate the context data set into candidate training samples, calculate the causal contribution of the candidate training samples to the target security model, and select high-value samples for storage based on the causal contribution.

[0045] The third embodiment of the present invention provides a management device for multi-source heterogeneous data for security operation and maintenance, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a management method for multi-source heterogeneous data for security operation and maintenance as described in any of the above embodiments.

[0046] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, so as to implement a method for managing multi-source heterogeneous data for security operation and maintenance as described in any of the above claims.

[0047] This invention provides a method, apparatus, device, and storage medium for managing multi-source heterogeneous data for security operations and maintenance. By determining the collection priority of target data sources based on the information entropy gain index of semi-structured log data, adaptive data collection with dynamic scheduling according to information value is achieved, solving the problem of low collection efficiency. By calculating the heterogeneous confusion entropy of the original data and constructing an adaptive temperature adjustment model, variable-temperature semantic mapping is performed based on dynamically calculated mapping temperature parameters according to the confusion entropy. This ensures that high-confusion-entropy fields corresponding to low temperatures undergo rigid feature preservation, while low-confusion-entropy fields corresponding to high temperatures undergo flexible normalization, solving the problem of mismapping of high-confusion fields caused by fixed mapping strategies. Furthermore, by calculating the causal contribution of candidate training samples to the target security model and selecting high-value samples for inclusion in the database, the problem of low-value samples affecting model training performance is solved. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a method for managing multi-source heterogeneous data for security operations and maintenance, provided in the first embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of a module of a multi-source heterogeneous data management device for security operation and maintenance provided in the second embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] This invention discloses a management method, apparatus, device, and storage medium for multi-source heterogeneous data in security operations and maintenance, aiming to solve the problems in existing security operations and maintenance scenarios, such as the lack of information value priority decision-making in multi-source heterogeneous data collection, the inability of heterogeneous field mapping to adaptively adjust according to the degree of data confusion, and the lack of causal contribution evaluation in training sample selection.

[0053] Please see Figure 1 The first embodiment of the present invention provides a management method for multi-source heterogeneous data for security operation and maintenance, which can be executed by a management device for multi-source heterogeneous data for security operation and maintenance (hereinafter referred to as the management device or system), specifically, by one or more processors within the management device, to at least implement the following steps:

[0054] S101, Receive a security operation and maintenance query request, parse the security operation and maintenance query request, and extract the query intent;

[0055] In this embodiment, the management device can be a desktop computer, laptop computer, server, or other terminal with data processing capabilities. The management device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment can be realized through the combination of the operating system and application software.

[0056] In one embodiment of the present invention, the system receives a security operations and maintenance query request initiated by security operations and maintenance personnel, and parses the request to extract the query intent. The sources of security operations and maintenance query requests include two types. The first type is alarm requests generated by real-time system detection. These requests are automatically generated by the security operations and maintenance system deployed in the customer's environment during continuous monitoring. When the system detects abnormal spikes in host CPU usage, memory usage exceeding a preset threshold, abnormally prolonged response times at business sites, or a large number of access failures, the system automatically triggers the corresponding security operations and maintenance query request. The second type is natural language query requests input by users through the front end. These requests are proactively initiated by security operations and maintenance personnel based on actual operations and maintenance needs. Users input query content in natural language through the interactive interface provided by the system, such as "Analyze the causes of CPU anomalies on host 192.168.1.100 in the past 24 hours" or "Investigate the root causes of timeouts at business sites in the past week," etc. The scope of security operation and maintenance query requests covers monitoring customer host hardware metrics and business site response status. Host hardware metrics monitoring includes operational metrics such as CPU utilization, memory usage, disk read / write speed, and network bandwidth utilization of cloud hosts or physical hosts. Business site response status monitoring includes service quality metrics such as access availability, response time, error rate distribution, and request success rate of customer business sites.

[0057] The system parses and processes received security operations and maintenance (O&M) query requests, extracting the query intent contained within. The system employs a routing model pre-trained with industry data to identify the intent of these queries. This routing model, based on a large language model and pre-trained with industry data in the security O&M field, possesses a deep understanding of input content from the security domain. The routing model performs semantic analysis and feature extraction on the query request content, identifying query intents that reflect the characteristics of attack or abnormal events, including intent types such as alarm assessment, source tracing analysis, and handling strategies. The extracted query intents serve as input for subsequent construction of dynamic data collection strategies, driving the system to determine the appropriate data collection scope and priority based on different O&M analysis needs.

[0058] S102, construct a dynamic collection strategy based on the query intent, determine the collection priority of the target data source based on the information entropy gain index of semi-structured log data, and generate an adaptive data collection instruction;

[0059] In one embodiment of the present invention, after receiving a query intent, the system first determines a set of candidate data sources related to the query intent. The set of candidate data sources covers various heterogeneous data sources in the enterprise security operations and maintenance environment, including firewall logs, intrusion detection system alarms, server access logs, host performance monitoring data, business site response logs, threat intelligence data, etc. Different query intents correspond to different ranges of candidate data sources. For example, for a query intent targeting abnormal IP tracing, the candidate data source set may include firewall logs, intrusion detection system alarms, threat intelligence data, etc.; for a query intent targeting host performance anomaly analysis, the candidate data source set may include host CPU monitoring data, memory usage data, disk read / write logs, network traffic data, etc.

[0060] After determining the candidate data source set, the system prioritizes the collection of each data source based on the information entropy gain index of the semi-structured log data. The system constructs a partially observable Markov decision process model, defining the current security threat state as a latent variable. The security threat state is a latent variable that the system cannot directly observe; its possible values ​​include normal, low-risk, medium-risk, and high-risk states. The system can only indirectly infer the current security threat state by collecting log data from various data sources. In the partially observable Markov decision process model, the system maintains a probability distribution regarding the security threat state, representing the confidence level of being in various threat states.

[0061] For each candidate data source in the set, the system calculates the expected reduction in uncertainty regarding the security threat state after collecting that data source, obtaining the corresponding information entropy gain value. Specifically, the system first calculates the information entropy of the current security threat state probability distribution, which characterizes the degree of uncertainty in the current state judgment. Then, for each candidate data source, the system predicts the possible observation results after collecting that data source based on historical data and prior knowledge, and calculates the conditional information entropy of the posterior probability distribution of the security threat state after obtaining those observation results. Finally, the difference between the current information entropy and the conditional information entropy is used as the information entropy gain value for that data source. A larger information entropy gain value indicates that collecting that data source will yield more information about the security threat state, and thus contributes more to eliminating the uncertainty in the current state judgment.

[0062] The system sorts candidate data sources in descending order of information entropy gain, generating a collection queue. Data sources with the highest information entropy gain are placed at the top of the queue and collected first; data sources with lower information entropy gain are placed at the back of the queue, and their collection is determined based on actual needs after the preceding data sources have been collected. This information value-based prioritization allows the system to prioritize the collection of data sources most valuable to the current security analysis, avoiding the waste of resources caused by blindly collecting large amounts of low-value data.

[0063] During the data collection process, the system does not execute all collection tasks at once. Instead, it updates the posterior probability distribution of the current security threat state in real time and dynamically adjusts the collection order of subsequent data sources. Once the system completes the collection of data from a data source in the queue and obtains actual observation data, it performs a Bayesian update on the posterior probability distribution of the security threat state based on this observation data, resulting in a more accurate state estimate. Based on the updated state probability distribution, the system recalculates the information entropy gain value of the remaining candidate data sources and adjusts the order of subsequent collection queues accordingly. If the information entropy gain value of some data sources decreases significantly after the state update, the system can remove them from the collection queue or lower their priority; conversely, if the information entropy gain value of some data sources increases significantly due to the state update, the system can advance their priority. Through this dynamic adjustment mechanism, the system can adaptively optimize subsequent collection decisions based on the actual content of the collected data, achieving efficient utilization of collection resources.

[0064] The system generates adaptive data acquisition instructions based on the determined acquisition priority and the dynamically adjusted acquisition queue. These instructions include information such as the target data source identifier, acquisition priority order, acquisition time range, and target object identifier, serving as the basis for subsequent data acquisition modules to perform actual acquisition operations.

[0065] S103, according to the adaptive data acquisition instruction, collect raw semi-structured log data from the heterogeneous security operation and maintenance system, and calculate the heterogeneous confusion entropy of the raw semi-structured log data. The heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of data fields.

[0066] The system parses the target data source identifier, collection priority order, collection time range, and target object identifier contained in the adaptive data collection command, and initiates data collection requests to each target data source sequentially according to the priority order specified in the command. The heterogeneous security operation and maintenance systems corresponding to the target data sources cover a variety of devices and platforms in the enterprise security operation and maintenance environment, including firewall systems, intrusion detection systems, intrusion prevention systems, server log systems, host performance monitoring systems, business site monitoring systems, threat intelligence platforms, etc. The data output by these heterogeneous systems are all semi-structured log data, which have certain structural characteristics but significant differences in field naming, data format, encoding methods, etc. The system interfaces with various heterogeneous systems through pre-configured data source adapters, filters log records that meet the conditions according to the time range and target object identifier parameters in the collection command, and completes the collection of raw semi-structured log data.

[0067] After collecting the raw semi-structured log data, the system calculates the heterogeneous confusion entropy of the data. Heterogeneous confusion entropy, proposed in this invention, is a quantitative indicator used to characterize the degree of confusion and distribution uncertainty of data fields. Its core idea is to reflect the overall confusion level of the original data by measuring the uncertainty of the mapping relationship between heterogeneous fields and standard fields. The system pre-builds a feature recognition model that can analyze the input heterogeneous fields and output the probability distribution of each field belonging to various standard fields. The standard field set K contains standardized field definitions commonly used in security operations and maintenance, such as source IP address, destination IP address, event time, attack type, threat level, port number, and protocol type.

[0068] The heterogeneous confusion entropy is calculated using the following formula: Where H(D) represents the heterogeneous confusion entropy of the original data D, and M is the total number of heterogeneous fields contained in the data. Let K be the i-th heterogeneous field, and K be a preset set of standard fields. Predict heterogeneous field f for a pre-defined feature recognition model i The probability of belonging to the standard field k. The calculation logic of this formula is as follows: for each heterogeneous field in the original data... The system uses a feature recognition model to calculate the probability distribution of its belonging to each standard field. Then calculate the information entropy of the probability distribution. Finally, the average information entropy of all heterogeneous fields is taken and negative to obtain the overall heterogeneous confusion entropy H(D).

[0069] The magnitude of heterogeneous confusion entropy reflects the degree of confusion in the original data. When a heterogeneous field f... i When a feature recognition model can explicitly identify a certain standard field, its probability distribution exhibits a highly concentrated characteristic, that is, a certain... When the probability is close to 1 and the rest are close to 0, the information entropy of this field is low; when a heterogeneous field f i When a field cannot be clearly identified and may belong to multiple standard fields with a relatively uniform probability distribution, its information entropy is high. Therefore, the larger the heterogeneous confusion entropy value, the more unknown confused fields or adversarial features the data contains, the higher the overall confusion level of the data, and the greater the difficulty of subsequent field mapping. Conversely, the smaller the heterogeneous confusion entropy value, the clearer the field structure of the data, the clearer the correspondence with standard fields, and the lower the difficulty of subsequent field mapping. The calculation result of the heterogeneous confusion entropy will serve as the core input for subsequently constructing an adaptive temperature regulation model and dynamically determining the mapped temperature parameters.

[0070] S104, Construct an adaptive temperature regulation model and dynamically calculate the mapped temperature parameters based on the heterogeneous confusion entropy;

[0071] S105, Perform variable-temperature semantic mapping based on the mapping temperature parameter: For new fields not registered, scale the similarity distribution between the new field and the standard field using the mapping temperature parameter, and calculate the normalized mapping confidence; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; unify the format of the mapped data, generate standardized data, and construct a context data set;

[0072] In one embodiment of the present invention, the system constructs an adaptive temperature regulation model, dynamically calculates the mapping temperature parameters based on heterogeneous confusion entropy, performs variable temperature semantic mapping based on the mapping temperature parameters, unifies the format of the mapped data to generate standardized data, and constructs a context data set.

[0073] The system constructs an adaptive temperature adjustment model based on the calculation results of heterogeneous confusion entropy. The core idea of ​​this model is to dynamically adjust the temperature parameter in the field mapping process according to the degree of confusion of the original data, so that the mapping strategy can adaptively match data features with different levels of confusion. The temperature parameter is a key variable controlling the smoothness of the mapping confidence distribution. Low temperature makes the confidence distribution sharper, which is beneficial for preserving discriminative features under high confusion levels; high temperature makes the confidence distribution smoother, which is beneficial for achieving flexible normalization matching under low confusion levels.

[0074] The system uses an adaptive temperature adjustment function to calculate the mapped temperature parameter T. The specific formula is as follows: .in, and These are the preset upper and lower temperature bounds, respectively; λ is the sensitivity attenuation coefficient; and H(D) is the heterogeneous confusion entropy calculated in the previous steps. In this embodiment, the upper temperature bound... It can be set to 2.0, with a lower temperature limit. The value can be set to 0.1, and the sensitivity attenuation coefficient λ can be set to 1.5. These values ​​can be adjusted according to the actual application scenario. From this formula, it can be seen that the mapping temperature parameter T is negatively correlated with the heterogeneous confusion entropy H(D): when the heterogeneous confusion entropy H(D) is large, As the value approaches 0, the mapped temperature parameter T approaches the lower limit of the temperature range. The system employs low-temperature execution mapping; when the heterogeneous confusion entropy H(D) is small, As the value approaches 1, the mapped temperature parameter T approaches the upper temperature bound. The system uses high-temperature execution mapping.

[0075] The system performs variable-temperature semantic mapping based on the calculated mapping temperature parameter T. For fields already registered in the field alias set in the original semi-structured log data, the system directly maps them to the corresponding standard fields according to the preset mapping relationship. For new fields not registered, the system uses the mapping temperature parameter to scale the similarity distribution between the new field and the standard fields and calculates the normalized mapping confidence score. Specifically, the system first calculates the similarity between the new field f and each standard field k in the standard field set. j semantic features similar The similarity is obtained by vectorizing the field names using a domain word vector model and then calculating the cosine similarity. Then, the normalized mapping confidence of the new field f to each standard field is calculated using the variable-temperature Softmax function. The specific formula is as follows: .in, For the new field f and the j-th standard field The semantic feature similarity is given by N, where N is the total number of standard fields and T is the mapped temperature parameter calculated in the preceding steps.

[0076] The temperature parameter T in the variable-temperature softmax function has a crucial impact on the shape of the mapping confidence distribution. When the heterogeneous confusion entropy of the data is high, the system calculates a lower mapping temperature parameter T, resulting in a sharper output distribution of the variable-temperature softmax function. The gap between the highest and second-highest confidence scores is amplified, and the system tends to make more explicit mapping judgments or reject mappings. This rigid feature preservation strategy helps avoid erroneous mappings under high confusion conditions. When the heterogeneous confusion entropy of the data is low, the system calculates a higher mapping temperature parameter T, resulting in a smoother output distribution of the variable-temperature softmax function. The mapping confidence gap between various standard fields is reduced, and the system's fault tolerance for mapping results is enhanced. This flexible normalization strategy helps achieve more flexible field matching under low confusion conditions.

[0077] The system determines how to process the new field based on the calculated normalized mapping confidence score. When When the maximum value is greater than a preset threshold, the system determines that the matching relationship between the new field and the corresponding standard field is reliable, performs field mapping, and maps the new field to the standard field with the highest mapping confidence; when When the maximum value is less than or equal to the preset threshold, the system determines that the new field cannot be reliably mapped to any existing standard field, and retains the new field as an extended feature, temporarily storing it according to the preset naming rules for use in subsequent manual review or model iteration.

[0078] The system performs format standardization processing on the data after field mapping, including converting timestamps of different formats to a standard time format, IP addresses of different representations to a standardized format, and strings of different encoding methods to UTF-8 encoding, generating standardized data. The system further integrates and correlates this standardized data according to entity dimensions, aggregating data records from different data sources that belong to the same query object to construct a context data set corresponding to the current query request. This context data set covers multi-dimensional security and operational information related to the query intent, providing a complete data foundation for subsequent intelligent analysis and training sample generation.

[0079] S106, the context data set is encapsulated into candidate training samples, the causal contribution of the candidate training samples to the target security model is calculated, and high-value samples are selected and stored in the database based on the causal contribution.

[0080] The system associates and encapsulates the constructed context data set with the corresponding query requests and analysis results to form candidate training samples. These candidate training samples contain complete input-output correspondences and can be used for training and optimizing the target security model. However, not all automatically generated candidate training samples have equal training value. Some samples may have issues such as information redundancy, feature duplication, or high similarity to existing training data. Including these low-value samples in the training dataset would not only increase storage and computational overhead but may also negatively impact model training performance. Therefore, the system needs to evaluate the value of candidate training samples to select truly high-quality samples with training value.

[0081] The system employs a causal contribution-based approach to evaluate the value of candidate training samples. Causal contribution is a quantitative metric that measures the impact of a single training sample on model performance. Its core idea is to determine the actual contribution of a sample to improving the model's generalization ability by analyzing the direction and strength of its influence on the model's loss function. The system first obtains the loss function gradient of the target safety model on the validation set. The validation set is a pre-constructed standard dataset used to evaluate model performance; the loss function gradient reflects the sensitivity of the model's error on the validation set to the model parameters. The system calculates the loss function value for each sample in the validation set and uses the backpropagation algorithm to calculate the gradient vector of the loss function relative to the model parameters. The gradient vectors of all validation samples are then summed to obtain the loss function gradient on the validation set.

[0082] The system further calculates the gradient of the loss function for the candidate training samples. The system inputs the candidate training samples into the target security model for forward inference, calculates the loss function value between the model output and the sample labels, and then calculates the gradient vector of this loss function relative to the model parameters using the backpropagation algorithm, thus obtaining the gradient of the loss function for the candidate training samples.

[0083] The system uses the Shapley value approximation algorithm to calculate the dot product of the gradient of the loss function on the candidate training sample and the gradient of the loss function on the validation set, thus obtaining the influence function value. The Shapley value is a classic method in cooperative game theory used to measure the marginal contribution of participants. Applying it to training sample value evaluation can effectively quantify the contribution of a single sample to the overall model performance. The influence function value is calculated through the gradient dot product, which involves taking the dot product of the gradient vector of the loss function on the candidate training sample and the gradient vector of the loss function on the validation set. The result of this dot product operation reflects the consistency between the gradient directions of the candidate training sample and the gradient directions of the validation set: when the two gradient vectors are in the same direction, the dot product is positive; when the two gradient vectors are in opposite directions, the dot product is negative; and when the two gradient vectors are orthogonal, the dot product is close to zero.

[0084] The system uses the calculated influence function value as the causal contribution of candidate training samples. A positive causal contribution indicates that the gradient direction of the sample is consistent with the gradient direction of the validation set. Including this sample in training will cause the model parameters to update in the direction that reduces the validation set error, meaning the sample can reduce the model's error on the validation set and has a positive effect on improving the model's generalization ability. A negative causal contribution indicates that the gradient direction of the sample is opposite to the gradient direction of the validation set. Including this sample in training may cause the model parameters to update in the direction that increases the validation set error, meaning the sample is a harmful or redundant sample and may have a negative impact on model training. The larger the absolute value of the causal contribution, the stronger the influence of the sample on model performance.

[0085] The system filters high-value samples for storage based on causal contribution. A pre-defined contribution threshold is set, which can be configured according to the training requirements of the target security model and storage resource limitations. The system compares the causal contribution of each candidate training sample with the contribution threshold: if the causal contribution is greater than the threshold, the system determines the sample as high-value and stores it in the training database as valid data for subsequent model training, while updating the feature library index to facilitate subsequent sample retrieval and deduplication; if the causal contribution is less than or equal to the threshold, the system determines the sample has insufficient training value and implements different processing strategies based on its specific causal contribution value: harmful samples with negative causal contributions are discarded, and redundant samples with near-zero causal contributions are downgraded and archived to a cold storage area.

[0086] The system periodically recalculates the causal contribution of samples in the cold storage area to adapt to version iterations of the target security model. As the target security model is continuously trained and optimized, its parameters are constantly updated, and samples previously deemed low-value may have high training value in newer model versions. The system retrieves samples from the cold storage area according to a preset cycle, recalculates their causal contribution based on the current version of the target security model, and if the recalculated causal contribution exceeds the contribution screening threshold, the sample is transferred from the cold storage area to the training database, achieving dynamic updates and value reassessment of the training data.

[0087] Please see Figure 2 The second embodiment of the present invention provides a management device for multi-source heterogeneous data for secure operation and maintenance, comprising:

[0088] The intent parsing module 201 is used to receive security operation and maintenance query requests, parse the security operation and maintenance query requests, and extract the query intent;

[0089] The adaptive acquisition module 202 is used to construct a dynamic acquisition strategy based on the query intent, determine the acquisition priority of the target data source based on the information entropy gain index of semi-structured log data, and generate adaptive data acquisition instructions.

[0090] The entropy evaluation module 203 is used to collect raw semi-structured log data from the heterogeneous security operation and maintenance system according to the adaptive data collection instruction, and calculate the heterogeneous confusion entropy of the raw semi-structured log data. The heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of the data fields.

[0091] The mapping temperature parameter calculation module 204 is used to construct an adaptive temperature regulation model and dynamically calculate the mapping temperature parameter based on the heterogeneous confusion entropy.

[0092] The variable-temperature mapping module 205 is used to perform variable-temperature semantic mapping based on the mapping temperature parameter: for new fields that are not registered, the similarity distribution between the new field and the standard field is scaled using the mapping temperature parameter, and a normalized mapping confidence score is calculated; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; the format of the mapped data is unified to generate standardized data, and a context data set is constructed;

[0093] The value assessment module 206 is used to encapsulate the context data set into candidate training samples, calculate the causal contribution of the candidate training samples to the target security model, and filter high-value samples for storage based on the causal contribution.

[0094] The third embodiment of the present invention provides a management device for multi-source heterogeneous data for security operation and maintenance, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a management method for multi-source heterogeneous data for security operation and maintenance as described in any of the above embodiments.

[0095] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, so as to implement a method for managing multi-source heterogeneous data for security operation and maintenance as described in any of the above claims.

[0096] This invention provides a method, apparatus, device, and storage medium for managing multi-source heterogeneous data for security operations and maintenance. By determining the collection priority of target data sources based on the information entropy gain index of semi-structured log data, adaptive data collection with dynamic scheduling according to information value is achieved, solving the problem of low collection efficiency. By calculating the heterogeneous confusion entropy of the original data and constructing an adaptive temperature adjustment model, variable-temperature semantic mapping is performed based on dynamically calculated mapping temperature parameters according to the confusion entropy. This ensures that high-confusion-entropy fields corresponding to low temperatures undergo rigid feature preservation, while low-confusion-entropy fields corresponding to high temperatures undergo flexible normalization, solving the problem of mismapping of high-confusion fields caused by fixed mapping strategies. Furthermore, by calculating the causal contribution of candidate training samples to the target security model and selecting high-value samples for inclusion in the database, the problem of low-value samples affecting model training performance is solved.

[0097] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in a management device for multi-source heterogeneous data oriented towards secure operation and maintenance. For example, the apparatus described in the second embodiment of the present invention.

[0098] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the aforementioned method for managing multi-source heterogeneous data for secure operation and maintenance, connecting various parts of the method through various interfaces and lines.

[0099] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a multi-source heterogeneous data management method for security operation and maintenance. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0100] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0101] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0102] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A management method for multi-source heterogeneous data oriented towards security operation and maintenance, characterized in that, include: Receive security operation and maintenance query requests, parse the security operation and maintenance query requests, and extract the query intent; A dynamic data collection strategy is constructed based on the query intent, and the collection priority of the target data source is determined based on the information entropy gain index of semi-structured log data, generating adaptive data collection instructions. According to the adaptive data acquisition instruction, raw semi-structured log data is collected from the heterogeneous security operation and maintenance system, and the heterogeneous confusion entropy of the raw semi-structured log data is calculated, with the expression as follows: ; Where H(D) represents the heterogeneous confusion entropy of the original data D, and M is the total number of heterogeneous fields contained in the data. Let K be the i-th heterogeneous field, and K be a preset set of standard fields. Predict heterogeneous field f for a pre-defined feature recognition model i The probability of belonging to standard field k; wherein, the larger the value of the heterogeneous confusion entropy, the more unknown confused fields or adversarial features the data contains, and the heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of the data fields; An adaptive temperature adjustment function is constructed to dynamically calculate the mapped temperature parameter based on the heterogeneous confusion entropy. The mapped temperature parameter T is calculated using the following adaptive temperature adjustment function: in, and These are the preset upper and lower temperature boundaries, respectively; λ is the sensitivity decay coefficient; and H(D) is the heterogeneous confusion entropy. Variable-temperature semantic mapping is performed based on the mapping temperature parameter: For new fields not registered, the similarity distribution between the new field and the standard field is scaled using the mapping temperature parameter, and a normalized mapping confidence score is calculated; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; the format of the mapped data is unified to generate standardized data, and a context data set is constructed; The context data set is encapsulated into candidate training samples, the causal contribution of the candidate training samples to the target security model is calculated, and high-value samples are selected and stored in the database based on the causal contribution.

2. The method for managing multi-source heterogeneous data for security operation and maintenance according to claim 1, characterized in that, The information entropy gain index based on semi-structured log data is used to determine the collection priority of the target data source, specifically as follows: Construct a partially observable Markov decision process model, defining the current security threat state as a hidden state variable; For each data source in the candidate data source set, calculate the degree to which the uncertainty of the security threat state is expected to decrease after collecting the data source, and obtain the information entropy gain value corresponding to the data source. The acquisition queue is generated in descending order of the information entropy gain value; The posterior probability distribution of the current security threat status is updated in real time during the data collection process, and the collection order of subsequent data sources is dynamically adjusted.

3. The method for managing multi-source heterogeneous data for security operation and maintenance according to claim 1, characterized in that, The step of performing variable-temperature semantic mapping based on the mapped temperature parameters specifically includes: The following variable-temperature Softmax function is used to calculate the mapping of the new field f to the standard field. Normalized mapping confidence : in, For the new field f and the j-th standard field The semantic feature similarity, where N is the total number of standard fields; when If the maximum value is greater than the preset threshold, perform field mapping; otherwise, retain the new field as an extended feature.

4. The method for managing multi-source heterogeneous data for security operation and maintenance according to claim 1, characterized in that, The calculation of the causal contribution of the candidate training samples to the target security model specifically includes: Obtain the gradient of the loss function of the target security model on the validation set; Calculate the gradient of the loss function for the candidate training samples; Based on the Shapley value approximation algorithm, the inner product of the loss function gradient of the candidate training sample and the loss function gradient on the validation set is calculated to obtain the influence function value; The influence function value is used as the causal contribution, where a positive causal contribution indicates that the sample can reduce the error of the model on the validation set, and a negative value indicates that the sample is a harmful or redundant sample.

5. A method for managing multi-source heterogeneous data for security operation and maintenance according to claim 1, characterized in that, The process of selecting high-value samples for storage based on the causal contribution specifically includes: Obtain the contribution threshold for filtering; If the causal contribution of the candidate training sample is greater than the contribution screening threshold, it is determined to be a high-value sample, stored in the training database, and the feature library index is updated. If the causal contribution of the candidate training sample is less than or equal to the contribution screening threshold, it is discarded or downgraded and archived to the cold storage area. The causal contribution of samples in the cold storage area is recalculated periodically to adapt to version iterations of the target security model.

6. A management device for multi-source heterogeneous data for secure operation and maintenance, characterized in that, include: The intent parsing module is used to receive security operation and maintenance query requests, parse the security operation and maintenance query requests, and extract the query intent; The adaptive acquisition module is used to construct a dynamic acquisition strategy based on the query intent, determine the acquisition priority of the target data source based on the information entropy gain index of semi-structured log data, and generate adaptive data acquisition instructions. The entropy evaluation module is used to collect raw semi-structured log data from the heterogeneous security operation and maintenance system according to the adaptive data collection instruction, and calculate the heterogeneous confusion entropy of the raw semi-structured log data, expressed as: ; Where H(D) represents the heterogeneous confusion entropy of the original data D, and M is the total number of heterogeneous fields contained in the data. Let K be the i-th heterogeneous field, and K be a preset set of standard fields. Predict heterogeneous field f for a pre-defined feature recognition model i The probability of belonging to standard field k; wherein, the larger the value of the heterogeneous confusion entropy, the more unknown confused fields or adversarial features the data contains, and the heterogeneous confusion entropy is used to characterize the degree of confusion and distribution uncertainty of the data fields; The mapped temperature parameter calculation module is used to construct an adaptive temperature adjustment function and dynamically calculate the mapped temperature parameter based on the heterogeneous confusion entropy. The mapped temperature parameter T is calculated using the following adaptive temperature adjustment function: in, and These are the preset upper and lower temperature boundaries, respectively; λ is the sensitivity decay coefficient; and H(D) is the heterogeneous confusion entropy. The variable-temperature mapping module is used to perform variable-temperature semantic mapping based on the mapping temperature parameter: for new fields not registered, the similarity distribution between the new field and the standard field is scaled using the mapping temperature parameter, and a normalized mapping confidence score is calculated; wherein, the mapping temperature parameter is negatively correlated with the heterogeneous confusion entropy, so that high confusion entropy fields correspond to low mapping temperatures to perform rigid feature preservation, and low confusion entropy fields correspond to high mapping temperatures to perform flexible normalization; the format of the mapped data is unified to generate standardized data, and a context data set is constructed; The value assessment module is used to encapsulate the context data set into candidate training samples, calculate the causal contribution of the candidate training samples to the target security model, and select high-value samples for storage based on the causal contribution.

7. A management device for multi-source heterogeneous data for secure operation and maintenance, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a method for managing multi-source heterogeneous data for security operation and maintenance as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement a method for managing multi-source heterogeneous data for security operation and maintenance as described in any one of claims 1 to 5.

Images