Data acquisition method and system for digital model

Abstract

The application discloses a kind of data acquisition method and system of digital model, it is related to data acquisition technical field, including based on the identification of redundant field based on field uniqueness, determine system safety alarm data redundancy and threshold value;System safety alarm data is preprocessed, constructs the node variable feature of system safety alarm data after preprocessing;Based on node variable feature and system safety alarm data, construct acquisition model.The method disclosed in the application identifies redundant field based on field uniqueness, uses the field of system safety alarm data for parsing using the first normal form principle, not only improves the accuracy of redundancy identification, but also realizes the accurate quantization of data redundancy, by preprocessing system safety alarm data, the quality and consistency of system safety alarm data are improved, simultaneously by distribution characteristic function extraction and matrix processing, it provides more comprehensive feature description for the analysis and modeling of the relationship between complex data.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, specifically to a data acquisition method and system for digital models. Background Technology

[0002] Multi-source data aggregation technology is an information integration method widely used in fields such as industrial control, intelligent security, and smart cities. It aggregates, analyzes, and processes diverse information from different data sources in a unified manner. However, in practical applications, existing technologies have revealed many shortcomings in data acquisition and processing. The most prominent issue is data redundancy. Due to the lack of effective strategies for distinguishing valuable data from redundant information during data acquisition by various systems or devices, a large amount of redundant data exists, occupying storage space and affecting system performance. Data redundancy usually stems from non-standard field design, duplicate storage, and disordered acquisition strategies. Although existing deduplication algorithms have improved data quality to some extent, they often fail to identify hidden redundant fields because they do not deeply consider the uniqueness and independence of fields, and there is a risk of accidentally deleting valid data.

[0003] Furthermore, the limitations of the data preprocessing stage have greatly restricted the improvement of data quality. Current preprocessing methods are mostly focused on basic cleaning and transformation operations, such as filling missing values ​​or detecting outliers, but they have failed to form a systematic strategy. They do not adequately explore the structural characteristics and distribution patterns of the data, making it difficult to meet the needs of subsequent analysis and modeling. In particular, the neglect of the processing of correlations and distribution characteristics between data has directly led to poor performance of the acquisition model in practical applications, making it unable to adapt to complex and ever-changing data environments.

[0004] Finally, existing technologies lack the ability to deeply extract and dynamically adjust the features of data nodes in the construction of acquisition models. Acquisition models are usually constructed using fixed parameters or static features, ignoring the dynamic changes in data, resulting in poor model adaptability. For example, in the acquisition of system security alarm data, the complexity and feature differences of data source modules make it difficult for traditional static models to accurately reflect data patterns. In addition, existing acquisition models have limited ability to process real-time data and cannot meet the needs of high-frequency data interaction and rapid response. These shortcomings not only increase the consumption of system computing power and operating costs, but also prolong data processing latency, becoming the main bottleneck in the development of multi-source data aggregation technology. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is that existing data acquisition methods suffer from data redundancy that is difficult to identify accurately, insufficient data preprocessing strategies, poor adaptability of acquisition models, and the problem of how to efficiently acquire and integrate data.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a data acquisition method for a digital model, comprising identifying redundant fields based on field uniqueness, determining the redundancy and threshold of system security alarm data; preprocessing the system security alarm data to construct node variable features of the preprocessed system security alarm data; and constructing an acquisition model based on the node variable features and the system security alarm data.

[0008] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the step of identifying redundant fields based on field uniqueness includes parsing system security alarm data, including calling the first normal form and determining whether the field has the uniqueness of a record based on the atomicity of the field; if the field does not have the uniqueness of a record, then the field is included in the redundant field set; if the field has the uniqueness of a record, then it is determined whether the field has the independence of column values; if the field does not have the independence of column values, then the field is included in the redundant field set; if the field has the independence of column values, then it is not included in the redundant field set.

[0009] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the determination of system security alarm data redundancy and threshold includes: counting the number of fields in the system security alarm data and the number of fields in the redundant field set; defining the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the system security alarm data redundancy; calculating the minimum union of the fields in the redundant field set and counting the number of fields in the minimum union; defining the ratio of the number of fields in the minimum union to the number of fields in the system security alarm data as the redundancy threshold.

[0010] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the preprocessing of the system security alarm data includes cleaning redundant data, handling missing values, and data consistency; cleaning redundant data includes deleting the corresponding fields from the minimum union of the fields in the redundant field set to generate first system security alarm data; handling missing values ​​includes performing missing value processing operations on the first system security alarm data based on a statistical value substitution function to generate second system security alarm data; data consistency includes unifying the second system security alarm data to generate preprocessed system security alarm data.

[0011] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the following steps are included: constructing the node variable features of the preprocessed system security alarm data, which involves identifying the node variables of the preprocessed system security alarm data and determining the source module of the preprocessed system security alarm data; determining the distribution and central trend of the system security alarm data based on the ontology model of the source module, extracting the skewed features of the system security alarm data using a distribution feature function; matrixing the skewed features into a three-dimensional matrix to form a distribution feature matrix, performing gradient calculation on the distribution feature matrix, identifying the boundaries and edges of the distribution feature matrix, generating a horizontal line of the distribution feature matrix, and extracting node variable features; verifying the consistency between the node variable features and the horizontal line based on the connected regions along the horizontal line, using the degree of agreement between the connected regions and the horizontal line, correcting the consistency verification results of the node variable features and the horizontal line using a line segment detection algorithm, and generating a feature weight matrix.

[0012] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the step of constructing an acquisition model based on node variable features and system security alarm data includes calling a statistical model. Constructing an acquisition model to be trained based on the features of node variables includes fitting a distribution model of the horizontal line of the distribution feature matrix according to the horizontal line of the distribution feature matrix; linking the linear regression function of the distribution model, defining a redundancy threshold as the expected value of the linear regression function, adjusting the distribution feature function based on the expected value, and generating the acquisition model to be trained.

[0013] As a preferred embodiment of the data acquisition method for the digital model described in this invention, the construction of the acquisition model based on node variable features and system security alarm data further includes training the acquisition model to be trained using the system security alarm data and outputting the trained acquisition model, including: modifying the weights of the node variable features according to the node variable features of the preprocessed system security alarm data; receiving system security alarm data, parsing system security alarm data, and statistically analyzing the distribution and central trend of system security alarm data when the redundancy of system security alarm data is less than the redundancy threshold; incorporating the expected value into the line segment detection algorithm and correcting the distribution feature function using the least squares method; extracting the skewness features of the data and generating a horizontal line of the distribution feature matrix; and using the horizontal line of the distribution feature matrix to output system security alarm data that meets the redundancy requirements.

[0014] Another objective of this invention is to provide a data acquisition system for digital models that can identify redundant fields based on field uniqueness, determine the redundancy and threshold of system security alarm data, and solve the problem that current data acquisition technologies have data redundancy that is difficult to accurately identify.

[0015] As a preferred embodiment of the data acquisition system for the digital model described in this invention, it includes: a redundancy determination module, a preprocessing module, and a model building module; the redundancy determination module is used to identify redundant fields based on field uniqueness and determine the redundancy and threshold of system security alarm data; the preprocessing module is used to preprocess the system security alarm data and construct node variable features of the preprocessed system security alarm data; the model building module is used to construct an acquisition model based on the node variable features and the system security alarm data.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a data acquisition method for a digital model.

[0017] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a data acquisition method for a digital model.

[0018] The beneficial effects of this invention are as follows: The data acquisition method for digital models provided by this invention identifies redundant fields based on field uniqueness, uses the first normal form principle to parse the fields of system security alarm data, judges the uniqueness and independence of fields, filters out redundant fields and includes them in the redundant field set, ensuring the atomicity and independence of fields, avoiding data conflicts or update impacts caused by redundancy between fields, and by statistically analyzing the number of redundant fields and the total number of fields, defining the redundancy degree and calculating the redundancy threshold using the minimum union, not only improving the accuracy of redundancy identification, but also achieving precise quantification of data redundancy. By preprocessing the system security alarm data and constructing the node variable features of the preprocessed system security alarm data, the quality and consistency of the system security alarm data are improved. At the same time, through distribution feature function extraction and matrix processing, it provides a basis for the analysis of complex data relationships and... The modeling provides a more comprehensive feature description. Based on the preprocessed node variable features, a distribution model is fitted and linked to a linear regression function through the horizontal line of the distribution feature matrix. The redundancy threshold is used as the expected value of the linear regression to optimize and adjust the distribution feature function, generating a collection model to be trained. By using system security alarm data to train the collection model, the feature weights and distribution parameters are dynamically adjusted to achieve accurate real-time data collection. By constructing and optimizing the collection model, the accuracy and real-time performance of data collection are greatly improved, reducing the dependence on subsequent data cleaning and processing. The dynamic feature adjustment capability enhances the model's adaptability to diverse data scenarios and reduces the consumption of server computing resources, achieving efficient and low-cost security alarm data collection. This invention achieves better results in terms of accuracy, efficiency, and reliability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The first embodiment of the present invention provides an overall flowchart of a data acquisition method for a digital model.

[0021] Figure 2 This is a schematic diagram of a data acquisition system for a digital model provided in the third embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0023] Example 1, referring to Figure 1 As an embodiment of the present invention, a data acquisition method for a digital model is provided, comprising:

[0024] Among them, the digital model can be a system security alarm data model. Currently, system security alarm data models mostly use multi-source data aggregation technology, which is prone to data redundancy.

[0025] S100: Identify redundant fields based on field uniqueness to determine the redundancy and threshold of system security alarm data.

[0026] Data redundancy can waste server computing power. Therefore, determining the redundancy of system security alarm data can reduce the redundancy of collected data by utilizing the target redundancy during the data collection process.

[0027] Furthermore, identifying redundant fields based on field uniqueness includes parsing system security alarm data, including invoking the first normal form and determining whether a field has unique records based on its atomicity. If a field does not have unique records, it is included in the redundant field set. If a field has unique records, it is determined whether the field has independent column values. If a field does not have independent column values, it is included in the redundant field set. If a field has independent column values, it is not included in the redundant field set.

[0028] It should be noted that, in one embodiment of the present invention, S100 includes:

[0029] S111, Select a system security alarm data.

[0030] S112, invoke the first normal form, based on the atomicity of the fields, to parse all fields of the system's security alarm data.

[0031] Specifically, each field should represent only one data item, and this data item should be indivisible, which means that multiple values ​​should not be stored in a single field.

[0032] Therefore, when multiple identical device addresses appear in security data, the redundancy of this set of data can be determined.

[0033] S113, Select a field.

[0034] S114, determine whether the field has the uniqueness of a record.

[0035] Specifically, each record should be unique and non-repeating. Based on the characteristic of being uniquely identified, multiple identical device addresses can be uniquely locked without representing other ambiguities, thus preventing data redundancy.

[0036] S115, if the field does not have the uniqueness of a record, then the field is included in the redundant field set, and the process returns to step S113 to select a field until all fields have been selected.

[0037] S116. If the field has the uniqueness of a record, then determine whether the field has the independence of column values.

[0038] S117 If the field does not have the uniqueness of a record, then the field is included in the redundant field set, and the process returns to step S113 to select a field until all fields have been selected.

[0039] S118. If the field has the uniqueness of a record, return to step S113, select a field, and continue until all fields have been selected.

[0040] S119, return to step S111, select one system security alarm data, until all system security alarm data have been selected.

[0041] Redundant fields are identified based on field uniqueness, and the redundancy of system security alarm data is determined. By ensuring the independence of each field, data updates in any field will not affect other fields, thus maintaining data consistency. First normal form provides a basic data organization framework, so data redundancy can be efficiently determined by back-engineering data based on first normal form.

[0042] It should also be noted that determining the redundancy and threshold of system security alarm data includes counting the number of fields in the system security alarm data and the number of fields in the redundant field set; defining the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the system security alarm data redundancy; calculating the minimum union of the fields in the redundant field set and counting the number of fields in the minimum union; and defining the ratio of the number of fields in the minimum union to the number of fields in the system security alarm data as the redundancy threshold.

[0043] It should also be noted that, in one embodiment of the present invention, S100 further includes:

[0044] S121, the number of all fields in the statistical system security alarm data.

[0045] S122, count the number of all fields in the redundant field set.

[0046] S123 defines the redundancy of system security alarm data as the ratio of the number of all fields in the redundant field set to the number of all fields in the statistical system security alarm data.

[0047] S124 calculates the minimum union of all fields in the statistical redundancy field set.

[0048] Specifically, the minimum union of all fields includes all fields in the redundant field set, and no fields in the minimum union of all fields are repeated.

[0049] S125, determine the minimum and centralize the number of all fields.

[0050] S126 defines the redundancy threshold as the ratio of the minimum number of all fields in the set to the total number of all fields in the statistical system security alarm data.

[0051] Specifically, the amount of real-time response data from the system security alarm system varies at different times. By minimizing and centralizing the ratio of the total number of all fields to the total number of all fields in the statistical system security alarm data, the redundancy threshold can be determined macroscopically.

[0052] In reality, the redundancy is generally zero. However, since the system security alarm system is an interactive response system, some nodes will have duplicate data. Therefore, the redundancy threshold actually represents the proportion of data interaction response nodes per unit time.

[0053] S200: Preprocess the system security alarm data and construct the node variable features of the preprocessed system security alarm data.

[0054] Preprocessing of system security alarm data includes, but is not limited to, data cleaning, data transformation, data integration, data standardization, and data dimensionality reduction. Data preprocessing is beneficial to the acquisition model. Through data preprocessing, the quality of data and the accuracy of analysis results can be greatly improved, providing strong support for reducing the amount of computation in the later stages of data processing.

[0055] Furthermore, the preprocessing of system security alarm data includes cleaning redundant data, handling missing values, and data consistency. Cleaning redundant data involves finding the minimum union of fields in the redundant field set, deleting the corresponding fields from the redundant field set, and generating the first system security alarm data. Handling missing values ​​involves performing missing value processing operations on the first system security alarm data based on a statistical value substitution function to generate the second system security alarm data. Data consistency involves standardizing the second system security alarm data to generate preprocessed system security alarm data.

[0056] It should be noted that, in one embodiment of the present invention, S200 includes:

[0057] S211, select one field from the minimum union of all fields in the redundant field set.

[0058] S212, Delete the corresponding field in the redundant field set.

[0059] Specifically, data cleaning involves removing duplicate data, handling missing values, and processing outliers. This process removes noise, outliers, duplicate data, missing data, and other factors that negatively impact data quality, thereby improving data quality and reliability.

[0060] S213, return to step S211, select the smallest union of all fields in the redundant field set, until all fields have been selected.

[0061] S214, Generate the first system security alarm data.

[0062] S215, based on the statistical value substitution function, performs missing value processing operation on the first system security alarm data.

[0063] Specifically, statistical substitution functions are used in data processing or statistical analysis to automatically replace specific parts of data, thereby simplifying calculations and improving efficiency, such as the SUBSTITUTE function and the REPLACE function.

[0064] S216, Generate the second system security alarm data.

[0065] S217, standardize the second system security alarm data and generate preprocessed system security alarm data.

[0066] Specifically, standardizing or unifying data can eliminate inconsistencies by converting data into a uniform format and unit, thereby improving the accuracy and reliability of data analysis and modeling.

[0067] It should be noted that constructing the node variable features of the preprocessed system security alarm data includes identifying the node variables of the preprocessed system security alarm data and determining the source module of the preprocessed system security alarm data; based on the ontology model of the source module, determining the distribution and central trend of the system security alarm data, and extracting the skewed features of the system security alarm data using the distribution feature function; matrixing the skewed features into a three-dimensional matrix to form a distribution feature matrix, performing gradient calculation on the distribution feature matrix, identifying the boundaries and edges of the distribution feature matrix, generating a horizontal line of the distribution feature matrix, and extracting node variable features; based on the connected regions along the horizontal line, using the degree of agreement between the connected regions and the horizontal line, verifying the agreement between the node variable features and the horizontal line, and using a line segment detection algorithm to correct the agreement verification results between the node variable features and the horizontal line, generating a feature weight matrix.

[0068] It should also be noted that the identification of node variables in the preprocessed system security alarm data is mainly achieved through univariate analysis. Univariate analysis can determine the independence of the source of system security alarm data. For example, in a security device integrated system, the periodic self-test data generated by its own traversal program can be considered as a node variable. Based on the function of the traversal program, the characteristics of the node variable can be determined, including the amount of data generated and the speed at which the data is generated.

[0069] Bivariate analysis can also be performed to examine the relationship between two continuous variables. This helps identify the strength and direction of the linear relationship between the variables. The chi-square test can be used to analyze whether two categorical variables are independent, or ANOVA (Analysis of Variance) can be used to analyze whether there are significant differences in the means between different groups. For example, the data between the vulnerability warning system and the vulnerability handling program of the emergency response system. Because the automated process orchestration system also performs vulnerability response, it orchestrates system vulnerabilities and feeds them back to the mobile terminal of the network engineer. Therefore, after determining the redundancy of system security alarm data, the emergency response system can serve as the main program carrier of node variables in this process. By obtaining the characteristics of the main program function of the emergency response system, the amount of data redundancy can be greatly reduced, and the amount of computation on the server can also be reduced in the later stage.

[0070] It should also be noted that, in one embodiment of the present invention, S200 further includes:

[0071] S221, Select a pre-processed system security alarm data.

[0072] S222, Determine the source module of the system's security alarm data.

[0073] Specifically, a system security alarm system may include an event system. The event system is used to accurately manage, aggregate, and respond to various events from various devices and products, quickly locate security events, and effectively achieve automated response to security events.

[0074] S223, based on the ontology model of the source module, determines the distribution and central trend of the system's security alarm data.

[0075] Specifically, the platform will automatically retrieve alarm events from connected devices, and then automatically process events that have been associated with scripts, displaying the events generated by each connected device and their processing status in the event list.

[0076] Therefore, based on the ontology model of the event system, it can be determined that the ontology model of the event system can be analyzed bivariately. The ontology model of the event system will have a large amount of redundant data. By statistically analyzing the distribution and central trend of this redundant data, the skewness characteristics of the redundant data can be extracted.

[0077] S224 uses the distribution characteristic function to extract the skewness characteristics of the data.

[0078] Specifically, the distribution characteristic function is expressed as:

[0079]

[0080] in, Let be the distribution characteristic function, t be the skewness characteristic, i be the imaginary unit, E be the expected value, and X be the redundant data.

[0081] The distribution characteristic function can be the moment generating function of iX, expressed as:

[0082]

[0083] Among them, M X( it ) It is the generating function of the moment.

[0084] Using the cumulative distribution function F X The distribution characteristic function can be transformed and expressed as:

[0085]

[0086] Where x represents redundant data values.

[0087] S225, return to step S221, select one preprocessed system security alarm data, until all preprocessed system security alarm data have been selected.

[0088] By constructing the features of node variables, the distribution feature function can realize the feature extraction of the function set of one or more ontology models, and parse the moment generating function of the distribution feature function. This can greatly reduce data redundancy and reduce the amount of computation on the server later.

[0089] It should also be noted that, in one embodiment of the present invention, S200 further includes:

[0090] S231, invoke all skewed features.

[0091] S232, each skewed feature is matrixed into a three-dimensional matrix to form a distribution feature matrix.

[0092] Specifically, if the function of an ontology model follows a normal distribution, then the skewness of this function will be a three-dimensional matrix of points (data volume, function feature parameters, and mean of redundant data). It's worth noting that the function feature parameters are actually the feature parameters of the expected value function.

[0093] S233 performs gradient calculation on the distribution feature matrix.

[0094] S234, identify the boundaries and edges of the distribution feature matrix to obtain the horizontal line of the distribution feature matrix.

[0095] S235, find connected regions along the horizontal line.

[0096] S236 uses the degree of agreement between the connected region and the horizontal line to verify the agreement between the features of the node variables and the horizontal line.

[0097] S237 uses the LSD algorithm to correct the verification results of the match between the features of the node variables and the horizontal line.

[0098] Specifically, the LSD algorithm, or Line Segment Detection Algorithm, offers high accuracy in adjusting image grayscale parameters. Furthermore, it can adjust the three-dimensional matrix, enabling progressive optimization of grayscale features. This characteristic aligns with the technical direction of optimizing function feature parameters, the mean of redundant data, and the three-dimensional matrix of data volume. Therefore, applying the LSD algorithm to the optimization of skewed feature correction of functions is quite appropriate.

[0099] Simply put, the weights of each function in each ontology model need to be reconciled in the data acquisition model to ensure that the overall input data of the data acquisition model meets the expected values, thereby achieving the data output.

[0100] S300: Construct a data acquisition model based on node variable characteristics and system security alarm data.

[0101] Furthermore, based on node variable features and system security alarm data, the acquisition model is constructed by calling a statistical model. The acquisition model to be trained is constructed by constructing the node variable features by fitting a distribution model to the horizontal line of the distribution feature matrix; linking the linear regression function of the distribution model, defining the redundancy threshold as the expected value of the linear regression function, adjusting the distribution feature function based on the expected value, and generating the acquisition model to be trained.

[0102] Statistical models typically utilize linear functions, such as linear models (GLM) or generalized linear models (GLM). Therefore, constructing a data acquisition model multidimensionalizes the feature matrix of node variables. Through matrix-based and multidimensional features, the harmonic curve of the data acquisition model can be analyzed. By changing the fitting parameters of the harmonic curve, the target redundancy can be achieved.

[0103] By inversely deriving the characteristics of each function based on the harmonic curve, we can achieve data acquisition with the minimum amount of data, thereby reducing the amount of computation on the server in the later stages.

[0104] It is worth mentioning that the harmonic curve has a systematic nature, that is, it has a constraint capability, which can also reduce the amount of computation on the server.

[0105] It should be noted that, in one embodiment of the present invention, S300 includes:

[0106] S311, invokes the horizontal line of the distribution feature matrix.

[0107] S312, a distribution model of the horizontal line fitting the distribution feature matrix.

[0108] S313, the linear regression function of the link distribution model.

[0109] S314, call the redundancy threshold as the expected value of the linear regression function.

[0110] S315, with the desired value as the objective, adjust the distribution characteristic function.

[0111] S316, Generate and construct the acquisition model to be trained.

[0112] Specifically, the horizontal line of the distribution feature matrix follows a linear model (GLM). By establishing the relationship between the expected value of the mean of the response redundant data volume and the linear combination of data volume through the connection function, it can avoid forcibly changing the natural measure of the redundant data volume. The data can achieve the function feature parameters of the distribution feature function to fit the expected value.

[0113] Simply put, by fitting the distribution model of the horizontal lines of the distribution feature matrix, the distribution parameters of each horizontal line are adjusted to achieve the adjustment of each horizontal line, that is, by linking the linear regression function of the distribution model.

[0114] The adjusted horizontal lines are used to adjust the distribution characteristic function. Therefore, when the redundancy of the output data does not meet the expected value, the distribution parameters of each horizontal line can be adjusted.

[0115] It should also be noted that, based on node variable features and system security alarm data, the construction of the acquisition model also includes training the acquisition model to be trained using system security alarm data, and outputting the trained acquisition model. This includes: modifying the weights of the node variable features according to the node variable features of the preprocessed system security alarm data; receiving system security alarm data, parsing the system security alarm data, and statistically analyzing the distribution and central trend of the system security alarm data when the redundancy of the system security alarm data is less than the redundancy threshold; incorporating the expected value into the line segment detection algorithm and correcting the distribution feature function using the least squares method; extracting the skewness features of the data and generating a horizontal line of the distribution feature matrix; and using the horizontal line of the distribution feature matrix to output system security alarm data that meets the redundancy requirements.

[0116] It should also be noted that, in one embodiment of the present invention, S300 further includes:

[0117] S321 receives system security alarm data.

[0118] S322, analyzes system security alarm data, and statistically analyzes the distribution and central trend of system security alarm data.

[0119] Specifically, the distribution characteristic function The correlation parameter is actually the expected value, and t is the skewed feature affected by the expected value. By adjusting the range of t, the distribution feature function can be adjusted.

[0120] S323 uses the least squares method to fit the distribution characteristic function of multiple skewed features.

[0121] S324 receives the fitted function and determines the desired solution value.

[0122] S325 incorporates the expected solution into the least squares method to obtain the modified distribution characteristic function.

[0123] S326 incorporates the expected value into the LSD algorithm to correct the distribution characteristic function.

[0124] Specifically, steps S323 to S326 are implemented as in steps S311 to S316.

[0125] S327, extract the skewness features of the data to obtain the horizontal line of the distribution feature matrix.

[0126] S328 uses the horizontal line of the distribution feature matrix to output the cleaned system safety alarm data.

[0127] Example 2 is an embodiment of the present invention, which provides a data acquisition method for a digital model. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiment.

[0128] This experiment selected multiple system security alarm datasets (named datasets A, B, C, D, and E respectively). Each dataset contains data fields from different devices and modules, involving various types of data such as device alarms, network security events, and system operation logs. The purpose of the experiment is to verify the effectiveness of identifying redundant fields based on field uniqueness, determine the data redundancy and threshold, and improve data quality through optimization processing.

[0129] The experiment was conducted according to the following steps:

[0130] 1. Data selection: Five datasets were extracted from the industrial system, each containing between 800 and 2000 fields, and the data was stored in relational database records.

[0131] 2. Field parsing: Based on the first normal form, fields are parsed to determine their atomicity, ensuring that each field represents only one data item.

[0132] 3. Redundant field identification: Determine the uniqueness and column value independence of each field, and include fields that do not meet the conditions into the redundant field set.

[0133] 4. Redundancy Calculation: Calculate the total number of fields in each dataset, the total number of redundant fields, and the minimum union number of redundant fields, and calculate the redundancy and threshold.

[0134] 5. Optimization Processing: For redundant field sets, perform field cleanup, missing value handling, and consistency operations to generate an optimized dataset.

[0135] The above steps were performed on each dataset, and the superiority of the invention was verified by statistically analyzing the changes in the number of fields, the changes in redundancy, and the processing time before and after the redundancy threshold.

[0136] Refer to Table 1 for comparative analysis of the experimental data.

[0137] Table 1 Experimental Data Recording Table

[0138]

[0139] As can be clearly seen from the data in Table 1, this invention has the following advantages in data redundancy identification and optimization:

[0140] 1. Redundancy control

[0141] After processing, the redundancy threshold of the dataset is significantly reduced. For example, the original redundancy of dataset B is 0.33. After processing by the present invention, the proportion of the minimum union field is reduced to 0.17, and the redundancy is reduced by 48.5%. In other datasets, the redundancy is significantly lower than the initial value, indicating that the present invention has extremely high efficiency in field uniqueness judgment and redundant field optimization.

[0142] 2. Field cleanup and optimization

[0143] This invention analyzes the atomicity and uniqueness of fields using the first normal form, effectively identifying redundant fields and avoiding accidental deletion of key fields through the minimum union operation. For example, the original dataset E had 600 redundant fields, and the optimized minimum union had 250 fields, demonstrating that the optimization process maintained field integrity while reducing data redundancy.

[0144] 3. Processing efficiency and real-time performance

[0145] Experimental data show that the processing time of this invention increases linearly with the size of the dataset. The processing time for the largest dataset E is 40 seconds, demonstrating the efficiency of the algorithm. Dataset D, as a sample with fewer fields, has a processing time of only 18 seconds, further verifying the algorithm's rapid response capability in small-scale data scenarios.

[0146] 4. Compare with existing technologies

[0147] Traditional deduplication algorithms struggle to accurately identify redundant fields and risk accidental deletion. This invention avoids these problems by using field uniqueness and column value independence. Furthermore, traditional methods are inefficient when processing large datasets, while the parsing strategy based on the first normal form and the dynamic redundancy threshold optimization model adopted in this invention improves processing speed and optimization effectiveness.

[0148] The above analysis leads to the conclusion that this invention reduces the proportion of redundant fields, improves the quality and consistency of system security alarm data, and reduces server computing power consumption. It is particularly suitable for improving and optimizing data quality in large-scale, multi-source data scenarios. Therefore, this invention is inventive.

[0149] Example 3, referring to Figure 2 As an embodiment of the present invention, a data acquisition system for a digital model is provided, including a redundancy determination module, a preprocessing module, and a model building module.

[0150] The redundancy determination module is used to identify redundant fields based on field uniqueness and determine the redundancy and threshold of system security alarm data; the preprocessing module is used to preprocess the system security alarm data and construct the node variable features of the preprocessed system security alarm data; the model building module is used to construct the acquisition model based on the node variable features and the system security alarm data.

[0151] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0153] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0154] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0155] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method of data acquisition for a digital model, characterized by, The application relates to a system security alarm data redundancy degree and threshold value determination method and system. The method comprises the following steps: identifying redundant fields based on field uniqueness, determining the system security alarm data redundancy degree and threshold value; preprocessing the system security alarm data, and constructing node variable features of the preprocessed system security alarm data; constructing a collection model based on the node variable features and the system security alarm data; the redundant field identification based on field uniqueness comprises field analysis on the system security alarm data, including calling a first normal form and judging whether the field has record uniqueness based on field atomicity; if the field does not have record uniqueness, the field is included in a redundant field set; if the field has record uniqueness, it is judged whether the field has column value independence; if the field does not have column value independence, the field is included in the redundant field set; if the field has column value independence, it is not included in the redundant field set; the determination of the system security alarm data redundancy degree and threshold value comprises the following steps: counting the number of fields in the system security alarm data and the number of fields in the redundant field set; defining the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the system security alarm data redundancy degree; calculating the minimum set of fields in the redundant field set, and counting the number of fields in the minimum set; defining the ratio of the number of fields in the minimum set to the number of fields in the system security alarm data as the redundancy threshold value; the construction of the collection model based on the node variable features and the system security alarm data further comprises the following steps: training the collection model to be trained by using the system security alarm data, and outputting the trained collection model, including modifying the weight of the node variable feature according to the constructed node variable features of the preprocessed system security alarm data; when the system security alarm data redundancy degree is less than the redundancy threshold value, receiving the system security alarm data, analyzing the system security alarm data, and counting the distribution and central tendency of the system security alarm data; 2. The method of data acquisition of a digital model of claim 1, wherein: including an expected value in a line segment detection algorithm, and correcting a distribution feature function by using a least square method; extracting skewness features of the data, and generating a horizontal line of a distribution feature matrix; outputting the system security alarm data meeting the redundancy degree by using the horizontal line of the distribution feature matrix. The preprocessing of the system security alarm data comprises the following steps:

3. The method of data acquisition of a digital model of claim 2, wherein: cleaning redundant data, missing value processing and data unification; the cleaning of the redundant data comprises the following steps: deleting the corresponding field in the minimum set of fields in the redundant field set, and generating first system security alarm data; the missing value processing comprises the following steps: performing a missing value processing operation on the first system security alarm data based on a statistical value substitution function, and generating second system security alarm data; the data unification comprises the following steps: unifying the second system security alarm data, and generating preprocessed system security alarm data. The construction of the node variable features of the preprocessed system security alarm data comprises the following steps: identifying the node variable of the preprocessed system security alarm data, and determining the source module of the preprocessed system security alarm data; based on the ontology model of the source module, determining the distribution and central tendency of the system security alarm data, and extracting skewness features of the system security alarm data by using a distribution feature function. The skewness features are three-dimensionally matrixed to form a distribution feature matrix, gradient calculation is performed on the distribution feature matrix, the distribution feature matrix boundary and edge are identified, the horizontal line of the distribution feature matrix is generated, and the node variable feature is extracted; Based on the connected region in the horizontal line direction, the coincidence verification of the node variable feature and the horizontal line is performed by using the coincidence degree of the connected region and the horizontal line, the coincidence verification result of the node variable feature and the horizontal line is corrected by using a line segment detection algorithm, and a feature weight matrix is generated.

4. The method of data acquisition of a digital model of claim 3, wherein: The acquisition model is constructed based on the node variable feature and the system security alarm data, and the acquisition model includes a statistical model; the feature of the node variable is constructed to generate a trained acquisition model, including fitting a distribution model of the horizontal line of the distribution feature matrix according to the horizontal line of the distribution feature matrix; A linear regression function of the distribution model is linked, the redundancy threshold is defined as an expected value of the linear regression function, the distribution feature function is adjusted based on the expected value, and the trained acquisition model is generated.

5. A system for data acquisition using the method of digital modeling according to any one of claims 1 to 4, characterized in that: The redundancy determination module, the preprocessing module, and the model construction module are included. The redundancy determination module is used to identify redundant fields based on field uniqueness, determine the redundancy and threshold of the system security alarm data. The preprocessing module is used to preprocess the system security alarm data, and construct the node variable feature of the preprocessed system security alarm data. The model construction module is used to construct an acquisition model based on the node variable feature and the system security alarm data. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the data acquisition method of the digital model in any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the data acquisition method of the digital model in any one of claims 1 to 4.

Images