Data acquisition method and system of digital model

Abstract

The invention discloses a data acquisition method and system for a digital model, and relates to the technical field of data acquisition, and the method comprises the steps: recognizing a redundant field based on the uniqueness of the field, and determining the redundancy and threshold of system security alarm data; preprocessing the system security alarm data, and constructing node variable characteristics of the preprocessed system security alarm data; and constructing an acquisition model based on the node variable characteristics and the system security alarm data. According to the method, the redundant field is identified based on the uniqueness of the field, and the field of the system security alarm data is analyzed by using the first normal form principle, so that the accuracy of redundancy identification is improved, the accurate quantification of the data redundancy is realized, and the accuracy of the system security alarm data is improved by preprocessing the system security alarm data. The method improves the quality and consistency of system security alarm data, and provides a more comprehensive feature description for the analysis and modeling of the relationship between complex data through distribution feature function extraction and matrix processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a data acquisition method and system for a digital model. Background Art

[0002] Multi-source data aggregation technology is an information integration method widely used in industrial control, intelligent security, smart cities and other fields. It aggregates, analyzes and uniformly processes diversified information from different data sources. However, in practical applications, existing technologies have exposed many deficiencies in data collection and processing. The first problem is data redundancy. Since each system or device lacks an effective strategy to distinguish valuable data from redundant information during data collection, a large amount of redundant data exists, which occupies storage space and affects system performance. Data redundancy usually comes from irregular field design, storage duplication and disordered collection strategy. Although the existing deduplication algorithms have improved data quality to a certain extent, they often find it difficult to identify hidden redundant fields because they do not deeply consider the uniqueness and independence of the fields, and there is a risk of accidentally deleting valid data.

[0003] In addition, the limitations of the data preprocessing link also greatly restrict the improvement of data quality. The current preprocessing methods are mostly focused on basic cleaning and conversion operations, such as filling missing values ​​or detecting outliers, but have failed to form a systematic strategy. The structural characteristics and distribution patterns of the data are not sufficiently explored, making it difficult to meet the needs of subsequent analysis and modeling. In particular, the processing methods that ignore the correlation and distribution characteristics between data directly lead to the poor effect of the acquisition model in practical applications and the inability to adapt to the complex and changing data environment.

[0004] Finally, the existing technology lacks the ability to deeply extract and dynamically adjust the features of data nodes in the construction of collection models. The collection models are usually constructed with fixed parameters or static features, ignoring the dynamic changes of data, resulting in poor model adaptability. For example, in the collection of system security alarm data, the complexity and feature differences of data source modules make it difficult for traditional static models to accurately reflect data patterns. In addition, the existing collection models have limited capabilities in processing real-time data and cannot meet the needs of high-frequency data interaction and rapid response. The above shortcomings not only increase the system computing power consumption and operating costs, but also extend the data processing delay, becoming the main bottleneck in the development of multi-source data aggregation technology. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problems solved by the present invention are: the existing data collection methods have the problems of data redundancy being difficult to accurately identify, insufficient data preprocessing strategies, poor adaptability of collection models, and how to efficiently collect and integrate data.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a data acquisition method for a digital model, comprising identifying redundant fields based on field uniqueness, determining the redundancy and threshold of system security alarm data; preprocessing the system security alarm data, and constructing node variable features of the preprocessed system security alarm data; and constructing an acquisition model based on the node variable features and the system security alarm data.

[0008] As a preferred solution of the data acquisition method of the digital model described in the present invention, wherein: the identification of redundant fields based on field uniqueness includes field parsing of system security alarm data, including calling the first normal form, and judging whether the field has record uniqueness based on the atomicity of the field; if the field does not have record uniqueness, the field is included in the redundant field set; if the field has record uniqueness, judging whether the field has column value independence; if the field does not have column value independence, the field is included in the redundant field set; if the field has column value independence, it is not included in the redundant field set.

[0009] As a preferred scheme of the data acquisition method of the digital model described in the present invention, wherein: the determination of the redundancy and threshold of the system security alarm data includes counting the number of fields in the system security alarm data and the number of fields in the redundant field set; defining the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the redundancy of the system security alarm data; calculating the minimum union of the fields in the redundant field set, and counting the number of fields in the minimum union; defining the ratio of the number of fields in the minimum union to the number of fields in the system security alarm data as the redundancy threshold.

[0010] As a preferred scheme of the data acquisition method of the digital model described in the present invention, wherein: the preprocessing of the system safety alarm data includes cleaning redundant data, missing value processing and data consistency; cleaning redundant data includes deleting the fields in the minimum union of the fields in the redundant field set, and generating the first system safety alarm data; missing value processing includes performing a missing value processing operation on the first system safety alarm data based on a statistical value replacement function to generate the second system safety alarm data; data consistency includes consistent the second system safety alarm data to generate the preprocessed system safety alarm data.

[0011] As a preferred solution of the data acquisition method of the digital model described in the present invention, the construction of the node variable characteristics of the preprocessed system safety alarm data includes identifying the node variables of the preprocessed system safety alarm data and determining the source module of the preprocessed system safety alarm data; based on the ontology model of the source module, determining the distribution and central trend of the system safety alarm data, and using the distribution characteristic function to extract the skewness characteristics of the system safety alarm data; three-dimensionally matrixing the skewness characteristics to form a distribution characteristic matrix, and performing gradient calculation on the distribution characteristic matrix, identifying the boundaries and edges of the distribution characteristic matrix, generating horizontal lines of the distribution characteristic matrix, and extracting node variable characteristics; based on the connected area in the direction of the horizontal line, using the degree of fit between the connected area and the horizontal line, verifying the fit between the characteristics of the node variables and the horizontal line, and using the line segment detection algorithm to correct the verification results of the fit between the characteristics of the node variables and the horizontal line to generate a feature weight matrix.

[0012] As a preferred solution of the data acquisition method of the digital model described in the present invention, wherein: the acquisition model is constructed based on the node variable characteristics and the system safety alarm data, including calling the statistical model, and the acquisition model to be trained is constructed based on the characteristics of the node variables, including fitting the distribution model of the horizontal lines of the distribution feature matrix according to the horizontal lines of the distribution feature matrix; linking the linear regression function of the distribution model, defining the redundancy threshold as the expected value of the linear regression function, adjusting the distribution feature function based on the expected value, and generating the acquisition model to be trained.

[0013] As a preferred solution of the data acquisition method of the digital model described in the present invention, wherein: the construction of the acquisition model based on the node variable characteristics and the system safety alarm data also includes using the system safety alarm data to train the acquisition model to be trained, and outputting the trained acquisition model includes modifying the weights of the characteristics of the node variables according to the node variable characteristics of the preprocessed system safety alarm data; when the redundancy of the system safety alarm data is less than the redundancy threshold, receiving the system safety alarm data, parsing the system safety alarm data, and statistically calculating the distribution and central trend of the system safety alarm data; incorporating the expected value into the line segment detection algorithm, and correcting the distribution characteristic function by the least squares method; extracting the skewness characteristics of the data, and generating the horizontal lines of the distribution characteristic matrix; using the horizontal lines of the distribution characteristic matrix, outputting the system safety alarm data that meets the redundancy.

[0014] Another object of the present invention is to provide a digital model data acquisition system that can identify redundant fields based on field uniqueness, determine the redundancy and threshold of system security alarm data, and solve the problem that current data acquisition technology contains data redundancy that is difficult to accurately identify.

[0015] As a preferred solution of the data acquisition system of the digital model described in the present invention, it includes: a redundancy determination module, a preprocessing module, and a model construction module; the redundancy determination module is used to identify redundant fields based on field uniqueness, and determine the redundancy and threshold of system safety alarm data; the preprocessing module is used to preprocess the system safety alarm data and construct node variable characteristics of the preprocessed system safety alarm data; the model construction module is used to construct an acquisition model based on the node variable characteristics and the system safety alarm data.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a data acquisition method for a digital model.

[0017] A computer-readable storage medium stores a computer program, which implements the steps of a data acquisition method for a digital model when executed by a processor.

[0018] Beneficial effects of the present invention: The data acquisition method of the digital model provided by the present invention identifies redundant fields based on field uniqueness, uses the first paradigm principle to parse the fields of system security alarm data, determines the uniqueness and independence of the fields, screens out redundant fields and incorporates them into the redundant field set, thereby ensuring the atomicity and independence of the fields, avoiding data conflicts or update impacts caused by redundancy between fields, and defining the redundancy by counting the number of redundant fields and total fields and combining the minimum union to calculate the redundancy threshold, which not only improves the accuracy of redundant identification, but also realizes the accurate quantification of data redundancy. By preprocessing the system security alarm data and constructing the node variable features of the preprocessed system security alarm data, the quality and consistency of the system security alarm data are improved. At the same time, through the distribution feature function extraction and matrix processing, it is convenient for the analysis and The modeling provides a more comprehensive feature description. It is based on the preprocessed node variable features, fits the distribution model and links the linear regression function through the horizontal lines of the distribution feature matrix, uses the redundancy threshold as the expected value of the linear regression, optimizes and adjusts the distribution feature function, and generates a collection model to be trained. The collection model is trained by using system security alarm data, and the feature weights and distribution parameters are dynamically adjusted to achieve accurate collection of real-time data. By constructing and optimizing the collection model, the accuracy and real-time performance of data collection are greatly improved, and the dependence on later data cleaning and processing is reduced. The dynamic feature adjustment capability improves the adaptability of the model to diversified data scenarios, while reducing the consumption of server computing resources, and achieving efficient and low-cost security alarm data collection. The present invention achieves better results in terms of accuracy, efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 An overall flow chart of a digital model data acquisition method provided for the first embodiment of the present invention.

[0021] Figure 2 A module schematic diagram of a digital model data acquisition system provided in a third embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0023] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a data acquisition method for a digital model, comprising:

[0024] Among them, the digital model can be a system security alarm data model. At present, the system security alarm data model mostly uses multi-source data aggregation technology, which is prone to data redundancy.

[0025] S100: Identify redundant fields based on field uniqueness, and determine system security alarm data redundancy and threshold.

[0026] Among them, data redundancy will cause the computing power of the server to be wasted, so the redundancy of the system security alarm data is determined. During the data collection process, the target redundancy can be used to reduce the redundancy of the collected data.

[0027] Furthermore, identifying redundant fields based on field uniqueness includes performing field parsing on system security alarm data, including calling the first normal form to determine whether the field has record uniqueness based on the atomicity of the field; if the field does not have record uniqueness, the field is included in the redundant field set; if the field has record uniqueness, determining whether the field has column value independence; if the field does not have column value independence, the field is included in the redundant field set; if the field has column value independence, the field is not included in the redundant field set.

[0028] It should be noted that, in one embodiment of the present invention, S100 includes:

[0029] S111, select a system security alarm data.

[0030] S112, calling the first normal form, and parsing all fields of the system security alarm data based on the atomicity of the fields.

[0031] Specifically, each field should represent only one data item, and this data item should be indivisible, which means that multiple values ​​should not be stored in one field.

[0032] Therefore, when multiple identical device addresses appear in the security data, the redundancy of this set of data can be determined.

[0033] S113, select a field.

[0034] S114, judging whether the field has the uniqueness of the record.

[0035] Specifically, each record should be uniquely identified and non-duplicate. Based on the unique identification feature, multiple identical device addresses can be uniquely locked when no other ambiguity is represented, and data redundancy occurs.

[0036] S115, if the field does not have the uniqueness of the record, then the field is included in the redundant field set, and the process returns to step S113 to select a field until all fields are selected.

[0037] S116: If the field has record uniqueness, determine whether the field has column value independence.

[0038] S117, if the field does not have the uniqueness of the record, then the field is included in the redundant field set, and the process returns to step S113, where a field is selected until all fields are selected.

[0039] S118, if the field is unique in the record, return to step S113 and select a field until all fields are selected.

[0040] S119, return to step S111, select one system security alarm data, until all system security alarm data are selected.

[0041] Redundant fields are identified based on field uniqueness, and the redundancy of system security alarm data is determined. By ensuring the independence of each field, data updates in any field will not affect other fields, thereby maintaining data consistency. The first paradigm provides a basic data organization framework, so reverse data based on the first paradigm can efficiently determine data redundancy.

[0042] It should also be noted that determining the redundancy and threshold of system security alarm data includes counting the number of fields in the system security alarm data and the number of fields in the redundant field set; defining the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the redundancy of the system security alarm data; calculating the minimum union of the fields in the redundant field set, and counting the number of fields in the minimum union; defining the ratio of the number of fields in the minimum union to the number of fields in the system security alarm data as the redundancy threshold.

[0043] It should also be noted that, in one embodiment of the present invention, S100 further includes:

[0044] S121, counting the number of all fields in the system security alarm data.

[0045] S122, counting the number of all fields in the redundant field set.

[0046] S123, defining the ratio of the number of all fields in the redundant field set to the number of all fields in the statistical system security alarm data as the system security alarm data redundancy.

[0047] S124, calculating the minimum union of all fields in the statistical redundant field set.

[0048] Specifically, the minimum union of all fields includes all fields in the redundant field set, and there is no duplication of fields in the minimum union of all fields.

[0049] S125, determining the number of all fields in the minimum union.

[0050] S126, defining a redundancy threshold as the ratio of the number of all fields in the minimum union to the number of all fields in the statistical system security alarm data.

[0051] Specifically, the amount of real-time response data of the system security alarm system is different in different periods. The redundancy threshold can be determined macroscopically by the ratio of the number of all fields in the minimum union to the number of all fields in the statistical system security alarm data.

[0052] In fact, the general redundancy is zero. However, since the system security alarm system is an interactive response system, data duplication will occur in some nodes. For this reason, the redundancy threshold actually represents the ratio of the number of data interactive response nodes per unit time.

[0053] S200: Preprocessing the system security alarm data, and constructing node variable features of the preprocessed system security alarm data.

[0054] Among them, the preprocessing of system security alarm data includes but is not limited to data cleaning, data conversion, data integration, data normalization, data dimension reduction, etc. Data preprocessing is beneficial to the acquisition model. Through data preprocessing, the quality of data and the accuracy of analysis results can be greatly improved, providing strong support for reducing the amount of data calculation in the later stage.

[0055] Furthermore, preprocessing the system security alarm data includes cleaning redundant data, missing value processing and data consistency; cleaning redundant data includes deleting the corresponding fields in the redundant field set for the fields in the minimum union of the fields in the redundant field set, and generating the first system security alarm data; missing value processing includes performing a missing value processing operation on the first system security alarm data based on a statistical value replacement function to generate the second system security alarm data; data consistency includes consistent the second system security alarm data to generate the preprocessed system security alarm data.

[0056] It should be noted that, in one embodiment of the present invention, S200 includes:

[0057] S211, selecting a field in the minimum union of all fields in the redundant field set.

[0058] S212, deleting the corresponding fields in the redundant field set.

[0059] Specifically, by deleting duplicate data, processing missing values, processing outliers, etc., data cleaning can remove noise, outliers, duplicate data, missing data and other factors that affect data quality, thereby improving data quality and reliability.

[0060] S213, returning to step S211, selecting a field in the minimum union of all fields in the redundant field set until all fields are selected.

[0061] S214: Generate first system security alarm data.

[0062] S215: Perform a missing value processing operation on the first system safety alarm data based on the statistical value replacement function.

[0063] Specifically, the statistical value replacement function is used to automatically replace specific parts of the data using certain functions in data processing or statistical analysis, thereby simplifying calculations and improving efficiency, such as the SUBSTITUTE function and the REPLACE function.

[0064] S216: Generate second system security alarm data.

[0065] S217, unifying the second system security alarm data to generate pre-processed system security alarm data.

[0066] Specifically, normalizing, or unifying, data can eliminate data inconsistencies, convert data into a unified format and unit, and improve the accuracy and reliability of data analysis and modeling.

[0067] It should be noted that constructing the node variable features of the preprocessed system safety alarm data includes identifying the node variables of the preprocessed system safety alarm data, and determining the source module of the preprocessed system safety alarm data; based on the ontology model of the source module, determining the distribution and central trend of the system safety alarm data, and using the distribution characteristic function to extract the skewness characteristics of the system safety alarm data; three-dimensionally matrixing the skewness characteristics to form a distribution characteristic matrix, and performing gradient calculation on the distribution characteristic matrix, identifying the boundaries and edges of the distribution characteristic matrix, generating horizontal lines of the distribution characteristic matrix, and extracting node variable features; based on the connected area in the direction of the horizontal line, using the degree of fit between the connected area and the horizontal line, verifying the fit between the characteristics of the node variables and the horizontal line, and using the line segment detection algorithm to correct the verification results of the fit between the characteristics of the node variables and the horizontal line to generate a feature weight matrix.

[0068] It should also be noted that the node variables of the pre-processed system safety alarm data are mainly identified through univariate analysis. Univariate analysis can determine the independence of the source of the system safety alarm data, such as the periodic self-test data generated by the traversal program of the safety equipment integration system itself. Then, the traversal program can be used as a node variable. Based on the function of the traversal program, the characteristics of the node variable can be determined, including the amount of data generated, the speed of generating data, etc.

[0069] Bivariate analysis can also be performed on the relationship between two continuous variables, which can help identify the strength and direction of the linear relationship between variables. The chi-square test can be used to analyze whether two categorical variables are independent, or ANOVA (Analysis of Variance) can be used to analyze whether there are significant differences in the mean values ​​between different groups, such as the data between the early warning system of a vulnerable device and the vulnerability handling program of the emergency response system. Since the automated process orchestration system will also respond to vulnerabilities, it will schedule tasks for system vulnerabilities and provide feedback to network engineers' mobile terminals. Therefore, after determining the redundancy of the system security alarm data, the emergency response system can be used as the main program carrier of the node variables in this process. By acquiring the main program function features of the emergency response system, the data redundancy can be greatly reduced, and the amount of computing power of the later server can also be reduced.

[0070] It should also be noted that, in one embodiment of the present invention, S200 further includes:

[0071] S221, select a preprocessed system security alarm data.

[0072] S222, determining the source module of the system security alarm data.

[0073] Specifically, if the system security alarm system has an event system, the event system is used to accurately manage, aggregate and respond to various events connected from various devices and products, quickly locate security incidents and effectively realize automated response to security incidents.

[0074] S223, based on the ontology model of the source module, determine the distribution and central trend of the security alarm data of the system.

[0075] Specifically, the platform will automatically pull the alarm events of the connected devices, and then directly start automatic processing of the events that have been associated with the scripts, and display the events generated by each connected device and the processing status of these events in the event list.

[0076] Then, based on the ontology model of the event system, it can be determined that the ontology model of the event system can be analyzed by bivariate analysis. The ontology model of the event system will have a large amount of redundant data. By statistically analyzing the distribution and central trend of the redundant data, the skewness characteristics of the redundant data can be extracted.

[0077] S224, using the distribution characteristic function to extract the skewness characteristics of the data.

[0078] Specifically, the distribution characteristic function is expressed as:

[0079]

[0080] in, is the distribution characteristic function, t is the skewness characteristic, i is the imaginary unit, E is the expected value, and X is the redundant data that appears.

[0081] The distribution characteristic function can be the moment generating function of iX, expressed as:

[0082]

[0083] Among them, M X( it ) is the moment generating function.

[0084] Using the cumulative distribution function F X , the distribution characteristic function can be deformed and expressed as:

[0085]

[0086] Where x is the redundant data value.

[0087] S225, returning to step S221, selecting a preprocessed system security alarm data until all preprocessed system security alarm data are selected.

[0088] Constructing the features of node variables, the distribution feature function can realize the feature extraction of the function set of one or more ontology models, and parse the moment generating function of the distribution feature function, which can greatly reduce the data redundancy and the amount of computation of the later server.

[0089] It should also be noted that, in one embodiment of the present invention, S200 further includes:

[0090] S231, call all skew features.

[0091] S232, performing three-dimensional matrixing on each skew feature to form a distribution feature matrix.

[0092] Specifically, if the function of an ontology model obeys a normal distribution, then the skewness characteristic of this function will be a three-dimensional matrix point of (data volume, function characteristic parameter, and redundant data volume mean). It is worth mentioning that the function characteristic parameter is actually the characteristic parameter of the expected value function.

[0093] S233, performing gradient calculation on the distribution feature matrix.

[0094] S234, identifying the boundaries and edges of the distribution feature matrix to obtain horizontal lines of the distribution feature matrix.

[0095] S235, searching for connected areas in the horizontal direction.

[0096] S236, using the degree of fit between the connected region and the horizontal line, verify the fit between the characteristics of the node variable and the horizontal line.

[0097] S237, using the LSD algorithm, the characteristics of the node variables are corrected with the verification results of the coincidence of the horizontal line.

[0098] Specifically, the LSD algorithm is a line segment detection algorithm (Line Segment Detection Algorithm) that has a high accuracy in adjusting the grayscale parameters of the image. At the same time, the LSD algorithm can adjust the three-dimensional matrix to achieve the progressive optimization of the grayscale features. This feature of the LSD algorithm is in line with the technical direction of optimizing the function feature parameters, the mean of the redundant data volume, and the three-dimensional matrix of the data volume. Therefore, it is more appropriate to use the LSD algorithm for the correction and optimization of the skewness feature of the function.

[0099] Simply put, the weights of each function of each ontology model in the acquisition model need to be coordinated so that the input data of the overall acquisition model meets the expected value, thereby realizing the data output.

[0100] S300: Build a collection model based on node variable characteristics and system security alarm data.

[0101] Furthermore, based on the node variable characteristics and system safety alarm data, constructing the collection model includes calling the statistical model, and constructing the collection model to be trained for the characteristics of the node variables includes fitting the distribution model of the horizontal lines of the distribution feature matrix according to the horizontal lines of the distribution feature matrix; linking the linear regression function of the distribution model, defining the redundancy threshold as the expected value of the linear regression function, adjusting the distribution feature function based on the expected value, and generating the collection model to be trained.

[0102] Among them, statistical models generally use linear functions, which can be linear models (GLM) and generalized linear models (GLM). Therefore, when constructing an acquisition model, the characteristic matrix of the node variables will be multidimensionalized. Through the matrix and multidimensional characteristics, the harmonized curve of the acquisition model can be analyzed, and the target redundancy can be achieved by changing the fitting parameters of the harmonized curve.

[0103] Based on the harmonic curve, the characteristics of each function can be inferred to achieve data acquisition with the minimum amount of data, thereby reducing the amount of computing power of the server in the later stage.

[0104] It is worth mentioning that the harmonized curve is systematic, that is, it has constraint capabilities, which can also reduce the amount of computing power on the server.

[0105] It should be noted that, in one embodiment of the present invention, S300 includes:

[0106] S311, calling the horizontal line of the distribution feature matrix.

[0107] S312, fitting the distribution model of the horizontal lines of the distribution feature matrix.

[0108] S313, linear regression function of the link distribution model.

[0109] S314, calling the redundancy threshold as the expected value of the linear regression function.

[0110] S315, adjusting the distribution characteristic function with the expected value as the target.

[0111] S316, generating and constructing a collection model to be trained.

[0112] Specifically, the horizontal lines of the distribution characteristic matrix obey the linear model (GLM). The relationship between the expected value of the mean of the response redundant data and the data volume of the linear combination is established through the connection function. It can not forcibly change the natural measurement of the redundant data volume, and the data can realize the function characteristic parameters of the distribution characteristic function fitting towards the expected value.

[0113] Simply, the distribution model of the horizontal lines of the distribution feature matrix is ​​fitted and the distribution parameters of each horizontal line are adjusted to achieve the adjustment of each horizontal line, that is, the linear regression function of the link distribution model is linked.

[0114] The adjusted horizontal lines are used to adjust the distribution characteristic function. Therefore, when the output data redundancy does not meet the expected value, the distribution parameters of the horizontal lines can be adjusted.

[0115] It should also be noted that, based on the node variable characteristics and the system safety alarm data, constructing the acquisition model also includes using the system safety alarm data to train the acquisition model to be trained, and outputting the trained acquisition model includes modifying the weights of the characteristics of the node variables according to the node variable characteristics of the preprocessed system safety alarm data; when the redundancy of the system safety alarm data is less than the redundancy threshold, receiving the system safety alarm data, parsing the system safety alarm data, and statistically calculating the distribution and central trend of the system safety alarm data; incorporating the expected value into the line segment detection algorithm, and correcting the distribution characteristic function through the least squares method; extracting the skewness characteristics of the data and generating the horizontal lines of the distribution characteristic matrix; using the horizontal lines of the distribution characteristic matrix, outputting the system safety alarm data that meets the redundancy.

[0116] It should also be noted that, in one embodiment of the present invention, S300 further includes:

[0117] S321, receiving system security alarm data.

[0118] S322, analyzing the system security alarm data, and counting the distribution and central trend of the system security alarm data.

[0119] Specifically, the distribution characteristic function The associated parameter is actually the expected value, and t is the skewness characteristic affected by the expected value. By adjusting the generation range of t, the distribution characteristic function can be adjusted.

[0120] S323, using the least square method, fitting the distribution characteristic function of multiple skewed characteristics.

[0121] S324, receiving the fitting function and determining the expected solution value.

[0122] S325, incorporating the expected solution value into the least square method to obtain a modified distribution characteristic function.

[0123] S326, incorporating the expected value into the LSD algorithm and correcting the distribution characteristic function.

[0124] Specifically, steps S323 to S326 are implemented as steps S311 to S316.

[0125] S327, extract the skewness characteristics of the data and obtain the horizontal lines of the distribution feature matrix.

[0126] S328, using the horizontal lines of the distribution feature matrix, output the cleaned system safety alarm data.

[0127] Embodiment 2 is an embodiment of the present invention, which provides a data acquisition method for a digital model. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0128] This experiment selected multiple system security alarm data sets (named as data sets A, B, C, D, and E respectively). Each data set contains data fields from different devices and modules, involving multiple types of data such as device alarms, network security events, and system operation logs. The purpose of the experiment is to verify the effectiveness of identifying redundant fields based on field uniqueness, determine data redundancy and thresholds, and improve data quality through optimization processing.

[0129] The experiment was carried out according to the following steps:

[0130] 1. Data selection: Five data sets were extracted from the industrial system, each containing 800 to 2000 fields, and the data were stored in the form of relational database records.

[0131] 2. Field parsing: Parse the fields based on the first normal form, determine the atomicity of the fields, and ensure that each field represents only one data item.

[0132] 3. Redundant field identification: Determine the uniqueness of fields and the independence of column values ​​one by one, and include fields that do not meet the conditions into the redundant field set.

[0133] 4. Redundancy calculation: Count the total number of fields, the total number of redundant fields, and the minimum union number of redundant fields in each data set, and calculate the redundancy and threshold.

[0134] 5. Optimization processing: For redundant field sets, perform field cleaning, missing value processing and consistency operations to generate an optimized data set.

[0135] The above steps are performed on each data set, and the superiority of the present invention is verified by statistically analyzing the change in the number of fields, the change in redundancy, and the processing time before and after the redundancy threshold.

[0136] Refer to Table 1 for a comparative analysis of the experimental data.

[0137] Table 1 Experimental data record table

[0138]

[0139] It can be clearly seen from the data in Table 1 that the present invention has the following advantages in data redundancy identification and optimization processing:

[0140] 1. Redundancy control

[0141] After the data set is processed, the redundancy threshold is significantly reduced. For example, the original redundancy of data set B is 0.33. After the processing of the present invention, the proportion of its minimum union field is reduced to 0.17, and the redundancy is reduced by 48.5%. In other data sets, the redundancy is significantly lower than the initial value, indicating that the present invention has extremely high efficiency in field uniqueness judgment and redundant field optimization.

[0142] 2. Field cleaning and optimization

[0143] The present invention analyzes the atomicity and uniqueness of fields through the first normal form, not only effectively identifying redundant fields, but also avoiding accidental deletion of key fields through the minimum union operation. For example, the original number of redundant fields in data set E is 600, and the minimum number of union fields after optimization is 250, which shows that the optimization process not only maintains field integrity but also reduces data redundancy.

[0144] 3. Processing efficiency and real-time performance

[0145] Experimental data show that the processing time of the present invention increases linearly with the size of the data set. The processing time of the largest data set E is 40 seconds, which shows the high efficiency of the algorithm. The processing time of data set D, as a sample with fewer fields, is only 18 seconds, which further verifies the algorithm's ability to respond quickly in small-scale data scenarios.

[0146] 4. Comparison with existing technologies

[0147] Traditional deduplication algorithms have difficulty in accurately identifying redundant fields and there is a risk of accidental deletion. The present invention avoids these problems by judging the uniqueness of fields and the independence of column values. In addition, traditional methods are inefficient when processing large-scale data sets. The present invention adopts a first-paradigm-based parsing strategy and a dynamic redundancy threshold optimization model to improve processing speed and optimization effect.

[0148] Through the above analysis, it can be concluded that the present invention reduces the proportion of redundant fields, improves the quality and consistency of system security alarm data, and reduces the computing power consumption of the server. It is particularly suitable for improving and optimizing data quality in large-scale, multi-source data scenarios. Therefore, the present invention is creative.

[0149] Example 3, reference Figure 2 , as an embodiment of the present invention, provides a data acquisition system for a digital model, including a redundancy determination module, a preprocessing module, and a model building module.

[0150] The redundancy determination module is used to identify redundant fields based on field uniqueness and determine the redundancy and threshold of system security alarm data; the preprocessing module is used to preprocess the system security alarm data and construct the node variable characteristics of the preprocessed system security alarm data; the model construction module is used to construct the acquisition model based on the node variable characteristics and system security alarm data.

[0151] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0153] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0154] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0155] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data acquisition method for a digital model, characterized in that: include: Identify redundant fields based on field uniqueness and determine the redundancy and threshold of system security alarm data; Preprocess the system security alarm data and construct node variable features of the preprocessed system security alarm data; Build a collection model based on node variable characteristics and system security alarm data.

2. The data acquisition method of the digital model according to claim 1, characterized in that: The identifying redundant fields based on field uniqueness includes performing field parsing on system security warning data, including calling the first normal form to determine whether the field has record uniqueness based on the atomicity of the field; If the field is not unique to the record, the field is included in the redundant field set; If the field has record uniqueness, determine whether the field has column value independence; If the field does not have column value independence, the field is included in the redundant field set; If a field is independent of its column values, it is not included in the redundant field set.

3. The data acquisition method of the digital model according to claim 2, characterized in that: Determining the redundancy and threshold of the system security alarm data includes counting the number of fields in the system security alarm data and the number of fields in the redundant field set; Define the ratio of the number of fields in the redundant field set to the number of fields in the system security alarm data as the system security alarm data redundancy; Calculate the minimum union of fields in the redundant field set and count the number of fields in the minimum union; The ratio of the number of fields in the minimum union to the number of fields in the system security alarm data is defined as a redundancy threshold.

4. The data acquisition method of the digital model as claimed in claim 3, characterized in that: The preprocessing of the system security alarm data includes cleaning redundant data, processing missing values ​​and data consistency; Cleaning redundant data includes deleting corresponding fields in the redundant field set for fields in the minimum union of fields in the redundant field set, and generating first system security alarm data; The missing value processing includes performing a missing value processing operation on the first system security alarm data based on a statistical value replacement function to generate second system security alarm data; The data consistency includes consistent second system security alarm data to generate pre-processed system security alarm data.

5. The data acquisition method of the digital model as claimed in claim 4, characterized in that: The constructing of node variable features of the preprocessed system security warning data includes identifying node variables of the preprocessed system security warning data and determining a source module of the preprocessed system security warning data; Based on the ontology model of the source module, the distribution and central trend of the system security alarm data are determined, and the skewness characteristics of the system security alarm data are extracted using the distribution characteristic function; The skewness features are matrixed into three dimensions to form a distribution feature matrix, and gradient calculation is performed on the distribution feature matrix to identify the boundaries and edges of the distribution feature matrix, generate horizontal lines of the distribution feature matrix, and extract node variable features; Based on the connected area in the horizontal line direction, the degree of fit between the connected area and the horizontal line is used to verify the fit between the characteristics of the node variables and the horizontal line. The line segment detection algorithm is used to correct the verification results of the fit between the characteristics of the node variables and the horizontal line to generate a feature weight matrix.

6. The data acquisition method of the digital model according to claim 5, characterized in that: The constructing of the acquisition model based on the node variable characteristics and the system security alarm data includes calling the statistical model, and constructing the acquisition model to be trained based on the characteristics of the node variables includes fitting the distribution model of the horizontal lines of the distribution feature matrix according to the horizontal lines of the distribution feature matrix; The linear regression function of the link distribution model is linked, the redundancy threshold is defined as the expected value of the linear regression function, the distribution feature function is adjusted based on the expected value, and the acquisition model to be trained is generated.

7. The data acquisition method of the digital model according to claim 6, characterized in that: The constructing of the acquisition model based on the node variable features and the system security alarm data further includes using the system security alarm data to train the acquisition model to be trained, and outputting the trained acquisition model includes modifying the weight of the node variable feature according to the node variable feature of the preprocessed system security alarm data; When the redundancy of the system security alarm data is less than the redundancy threshold, the system security alarm data is received, the system security alarm data is analyzed, and the distribution and central trend of the system security alarm data are counted; The expected value is incorporated into the line segment detection algorithm, and the distribution characteristic function is modified by the least square method; Extract the skewness characteristics of the data and generate horizontal lines of the distribution feature matrix; Utilizing the horizontal lines of the distribution characteristic matrix, the system safety warning data that meets the redundancy is output.

8. A system using the data acquisition method of the digital model according to any one of claims 1 to 7, characterized in that: It includes redundancy determination module, preprocessing module and model building module; The redundancy determination module is used to identify redundant fields based on field uniqueness and determine the redundancy and threshold of system security alarm data; The preprocessing module is used to preprocess the system security alarm data and construct node variable features of the preprocessed system security alarm data; The model building module is used to build a collection model based on node variable characteristics and system safety alarm data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the digital model data acquisition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the digital model data acquisition method according to any one of claims 1 to 7 are implemented.

Images