Data acquisition methods, devices, equipment, and computer storage media

Abstract

This application discloses a data acquisition method, apparatus, device, and computer storage medium. The method includes: acquiring target interface data and target data source type from a target data source device, preset feature vectors corresponding to multiple interface types, and the weights of these preset feature vectors; extracting target keywords from the target interface data that match a first preset data format and preset keywords; performing one-hot encoding on the target keywords to obtain target feature vectors; and, when the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than a similarity threshold for the target type interface, acquiring data from the target data source device using the target type interface at a preset acquisition frequency for the target data source type. This eliminates the need for manual operation, improving the efficiency of interface determination, and allows for dynamic adjustment of the acquisition frequency based on different adapted interfaces, avoiding data loss or resource waste.

Description

Technical Field

[0001] This application belongs to the field of information security technology, and in particular relates to a data acquisition method, apparatus, device and computer storage medium. Background Technology

[0002] With the increasing severity of cybersecurity threats, it is essential to build a security operations platform to centralize, standardize, and digitize security data, and to automate various security management tasks by coordinating and integrating all existing security protection capabilities. However, automating these tasks relies not only on the coordinated efforts of individual security capabilities but also on massive amounts of security data as a driving force. Therefore, it is necessary to collect various types of security data.

[0003] In existing technologies, fixed APIs or open middleware libraries are typically used to connect to secure data. This requires manual configuration and adjustment of data interfaces on various platforms to adapt to different data sources. Data collection is carried out using scheduled tasks or event-driven methods. This not only requires a lot of manual operation, but also requires additional interface configuration when facing new data sources, resulting in low efficiency. Furthermore, when facing different data sources, collecting data according to fixed collection strategies may lead to data loss or waste of resources. Summary of the Invention

[0004] This application provides a data acquisition method, apparatus, device, and computer storage medium that can adapt to different types of interfaces without manual operation, improving the efficiency of interface determination. Moreover, data is collected based on a preset acquisition frequency set by the adapted interface, and the acquisition frequency can be dynamically adjusted according to different adapted interfaces, avoiding data loss or resource waste.

[0005] In a first aspect, embodiments of this application provide a data acquisition method, including:

[0006] Obtain the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple interface types, and the weights of the preset feature vectors;

[0007] Extract target keywords from the target interface data that match the first preset data format and preset keywords;

[0008] The target keywords are one-hot encoded to obtain the target feature vector;

[0009] Based on the weights of the preset feature vectors, determine the weighted distance between the preset feature vectors and the target feature vectors corresponding to multiple types of interfaces;

[0010] If the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the target type interface is used to collect data from the target data source device according to the preset collection frequency of the target data source type.

[0011] In one possible implementation, acquiring target interface data from the target data source device includes:

[0012] Obtain the interface documentation for the target data source device;

[0013] Identify the structural information of the data in the interface document, and use the structural information to segment the data in the interface document to obtain the target interface data.

[0014] In one possible implementation, before obtaining the preset feature vectors and weights of the preset feature vectors corresponding to multiple type interfaces, the method further includes:

[0015] Retrieve interface data corresponding to multiple types of interfaces;

[0016] Extract keyword samples that match the first preset data format and preset keywords from the interface data corresponding to multiple types of interfaces;

[0017] Each keyword in the keyword sample is one-hot encoded to obtain preset feature vectors corresponding to multiple types of interfaces;

[0018] Multiple decision trees are constructed using the random forest algorithm; the first keyword in the keyword sample is randomly selected and used as the first training set; the target decision tree is trained using the first training set to obtain the first decision tree; the second keyword in the keyword sample is used as the first validation set, and the first prediction error rate of the first validation set is determined using the first decision tree. The second keyword is any keyword in the keyword sample other than the first keyword, and the target decision tree is any one of the multiple decision trees.

[0019] For each target keyword in the first keyword, replace the target keyword with a random word to obtain the third keyword;

[0020] The third keyword is used as the second training set. The target decision tree is trained using the second training set to obtain the second decision tree. The second prediction error rate of the first validation set is determined using the second decision tree.

[0021] For each decision tree, calculate the difference between the first prediction error rate and the second prediction error rate;

[0022] Calculate the average of the differences to determine the importance of each target keyword in the first keyword;

[0023] Based on the mapping relationship between weight and importance, determine the weight corresponding to the importance of each target keyword in the first keyword, and obtain the weight of each target keyword in the first keyword.

[0024] In one possible implementation embodiment, before collecting data from the target data source device using the target type interface according to a preset collection frequency of the target data source type, when the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the method further includes:

[0025] Retrieve the keywords corresponding to the target type interface;

[0026] If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is greater than the first preset value, the first similarity threshold is determined as the similarity threshold of the target type interface.

[0027] In one possible implementation embodiment, it further includes:

[0028] If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is not greater than the first preset value, the second similarity threshold is determined as the similarity threshold of the target type interface.

[0029] In one possible implementation embodiment, it further includes:

[0030] For the data collected from the target data source device, construct a target sliding window corresponding to the target data source type;

[0031] Determine the amount of change in data within two consecutive target sliding windows;

[0032] If the number of consecutive occurrences of the change being greater than the second preset value is greater than the third preset value, the sampling frequency of the preset step size is increased.

[0033] If the number of consecutive occurrences of a change less than the fourth preset value is greater than the third preset value, reduce the sampling frequency of the preset step size.

[0034] In one possible implementation embodiment, it further includes:

[0035] Get the network load value of the target type interface;

[0036] If the network load value is less than the fifth preset value, increase the sampling frequency by the preset step size;

[0037] If the network load value is greater than the sixth preset value, reduce the sampling frequency of the preset step size.

[0038] In one possible implementation embodiment, when the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, after collecting data from the target data source device using the target type interface according to the preset collection frequency of the target data source type, the method further includes:

[0039] Extract multiple feature information that matches the second preset data format and preset feature information from the data collected from the target data source device;

[0040] Store multiple feature information.

[0041] In one possible implementation embodiment, after extracting multiple feature information matching the second preset data format and preset feature information from the data collected from the target data source device, the method further includes:

[0042] Multiple feature information are segmented to obtain multiple first words;

[0043] Multiple first words are deduplicated to obtain multiple second words;

[0044] For each feature, calculate the first ratio of the number of occurrences of each second word to the total number of occurrences of multiple second words to obtain the word frequency of each second word;

[0045] Based on the number of multiple feature information and the number of target feature information, the inverse document frequency of each second word is calculated. The target feature information is the feature information of the corresponding second word among multiple feature information.

[0046] The weight of each second word is determined based on its term frequency and inverse document frequency.

[0047] The second words are sorted according to their weights to obtain a second word sequence;

[0048] Select the third word with a relatively high weight from the second word sequence;

[0049] Store multiple feature information, including:

[0050] Save the third vocabulary.

[0051] Secondly, embodiments of this application provide a data acquisition device, including:

[0052] The acquisition module is used to acquire the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple types of interfaces, and the weights of the preset feature vectors.

[0053] The extraction module is used to extract target keywords that match the first preset data format and preset keywords from the target interface data;

[0054] The encoding module is used to perform one-hot encoding on the target keywords to obtain the target feature vector;

[0055] The determination module is used to determine the weighted distance between the preset feature vectors and the target feature vectors corresponding to multiple types of interfaces based on the weights of the preset feature vectors.

[0056] The acquisition module is used to acquire data from the target data source device using the target type interface at a preset acquisition frequency for the target data source type, provided that the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface.

[0057] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0058] Processor and memory storing computer program instructions;

[0059] A method for acquiring data that implements any of the above when the processor executes computer program instructions.

[0060] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, a method for acquiring data as described above is provided.

[0061] Fifthly, embodiments of this application provide a computer program product, characterized in that, when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device is able to execute the data acquisition method described above.

[0062] This application discloses a data acquisition method, apparatus, device, and computer storage medium. The method includes: acquiring target interface data and target data source type of a target data source device, preset feature vectors corresponding to multiple types of interfaces, and weights of the preset feature vectors; extracting target keywords that match a first preset data format and preset keywords from the target interface data; performing one-hot encoding on the target keywords to obtain target feature vectors; determining the weight distance between the preset feature vectors corresponding to multiple types of interfaces and the target feature vector based on the weights of the preset feature vectors; and, when the weight distance between the preset feature vectors corresponding to the target type interface and the target feature vector is less than a similarity threshold of the target type interface, acquiring data from the target data source device using the target type interface according to a preset acquisition frequency of the target data source type.

[0063] In this way, by using the target feature vector corresponding to the interface data and the weighted distance between the preset feature vectors corresponding to multiple types of interfaces, the target type interface for collecting data from the target data source device is determined. When the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the target type interface is used to collect data from the target data source device according to the preset collection frequency of the target data source type. This allows for adaptation of different types of interfaces without manual operation, improving the efficiency of interface determination. Moreover, the data is collected based on the preset collection frequency set by the adapted interface, and the collection frequency can be dynamically adjusted according to different adapted interfaces, avoiding data loss or resource waste. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a schematic flowchart of a data acquisition method provided in one embodiment of this application;

[0066] Figure 2 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0067] Figure 3 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0068] Figure 4 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0069] Figure 5 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0070] Figure 6 This is a flowchart illustrating a data acquisition method provided in another embodiment of this application;

[0071] Figure 7 This is a schematic diagram of the structure of a data acquisition device provided in another embodiment of this application;

[0072] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0073] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0074] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0075] With the increasing severity of cybersecurity threats, it is essential to build a security operations platform to centralize, standardize, and digitize security data, and to automate various security management tasks by coordinating and integrating all existing security protection capabilities. However, automating these tasks relies not only on the coordinated efforts of individual security capabilities but also on massive amounts of security data as a driving force. Therefore, it is necessary to collect various types of security data.

[0076] In existing technologies, fixed APIs or open middleware libraries are typically used to connect to secure data. This requires manual configuration and adjustment of data interfaces on various platforms to adapt to different data sources. Data collection is carried out using scheduled tasks or event-driven methods. This not only requires a lot of manual operation, but also requires additional interface configuration when facing new data sources, resulting in low efficiency. Furthermore, when facing different data sources, collecting data according to fixed collection strategies may lead to data loss or waste of resources.

[0077] To address the problems of existing technologies, embodiments of this application provide a data acquisition method, apparatus, device, and computer storage medium. The data acquisition method provided in this application determines the target type interface for collecting data from a target data source device by using the weighted distance between the target feature vector corresponding to the target interface data and the preset feature vectors corresponding to multiple type interfaces. When the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than a similarity threshold for the target type interface, data from the target data source device is collected using the target type interface at a preset collection frequency for the target data source type. This eliminates the need for manual operation to adapt to different type interfaces, improving the efficiency of interface determination. Furthermore, the preset collection frequency set based on the adapted interface allows for dynamic adjustment of the collection frequency according to different adapted interfaces, avoiding data loss or resource waste.

[0078] The data acquisition method provided in the embodiments of this application will be introduced first below. Figure 1 A flowchart illustrating a data acquisition method provided in one embodiment of this application is shown.

[0079] like Figure 1 As shown in the embodiments of this application, the data acquisition method includes the following steps.

[0080] S110. Obtain the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple types of interfaces, and the weights of the preset feature vectors.

[0081] Here, the preset feature vectors corresponding to multiple interface types are pre-defined. The target interface data is in plain text format, which is easy for machines to process.

[0082] In some embodiments, the interface types cover a variety of interfaces, including HTTP, HTTPS, FTP, SFTP, SMTP, POP3, IMAP, Telnet, SSH, Syslog, SNMP, REST API, and SQL. Preset feature vectors corresponding to multiple interface types can be provided by numerous equipment and platform manufacturers involved in projects across the country.

[0083] S120. Extract target keywords from the target interface data that match the first preset data format and preset keywords.

[0084] Here, the first preset data format and preset keywords are set in advance.

[0085] In some embodiments, the first preset data format may be a URL format. Preset keywords may include, but are not limited to, communication URL, communication action, communication data format, communication path, and namespace. Specific preset keywords include the interface name, description, parameter list, return value, and error code.

[0086] Understandably, target keywords include characteristic information. Target keywords may also include: characteristics of the communication URL (such as URL length, special characters contained in the URL, etc.), types of communication actions, communication data formats (such as JSON, XML, etc.), communication path depth, namespace structure, etc.

[0087] In some embodiments, the communication URL can be extracted by matching the URL format string in the target interface data using regular expressions. The communication action, typically an HTTP method (such as GET, POST, etc.), can be extracted by searching keywords. The communication data format, such as JSON, XML, etc., can be extracted by searching keywords. The communication path and namespace, usually found in the URL, can be extracted by parsing the URL.

[0088] As an example, a typical target API query might begin with an HTTP method like "GET" or "POST," followed by a URL path. We can define a regular expression to match these parameters and extract the HTTP method and URL path. Alternatively, we can predefine keywords such as "parameters," "return values," and "error codes." When these keywords appear in the API query data, we extract them and the information following them as target keywords. Here, the regular expression describes the first predefined data format.

[0089] S130. Perform one-hot encoding on the target keywords to obtain the target feature vector.

[0090] In some embodiments, target keywords are feature-encoded using one-hot encoding. A Boolean vector of "0"s and "1"s represents characteristics of the communication URL, types of communication actions, communication data formats, communication path depth, namespace structure, etc., transforming the target keywords into actual feature vectors. For example, the following describes the process of encoding features using one-hot encoding:

[0091] (1) For the characteristics of the communication URL: decompose the URL into a series of path segments. For example, " / user / profile / edit" can be decomposed into "user", "profile" and "edit". Perform one-hot encoding on these path segments. It can be understood that if there are 1000 different path segments, then the one-hot encoding of each path segment is a vector of length 1000, where the position corresponding to the path segment is 1 and the other positions are 0.

[0092] (2) Regarding the types of communication actions: There are four types of communication actions, namely "GET", "POST", "PUT" and "DELETE". We can perform one-hot encoding on these four communication actions to obtain four vectors of length 4. For example, "GET" can be represented as [1,0,0,0], "POST" can be represented as [0,1,0,0], and so on;

[0093] (3) Regarding communication data formats: There are three communication data formats, namely "JSON", "XML" and "HTML". We can perform one-hot encoding on these three formats to obtain three vectors of length 3. For example, "JSON" can be represented as [1,0,0], "XML" can be represented as [0,1,0], and so on.

[0094] (4) For the depth of the communication path: If the depth of the communication path is within a reasonable range, such as [1, 10], then one-hot encoding can be performed for each depth. For example, a depth of 3 can be represented as [0,0,1,0,0,0,0,0,0,0].

[0095] (5) For namespace structure: For namespace, first decompose it into a series of decomposed parts, for example, "com.example.myapp" can be decomposed into "com", "example" and "myapp", and then these decomposed parts can be one-hot encoded.

[0096] S140. Based on the weights of the preset feature vectors, determine the weighted distance between the preset feature vectors and the target feature vectors corresponding to multiple types of interfaces.

[0097] Here, the weights of the preset feature vectors can be pre-defined. They can be determined based on the degree of influence of each preset feature vector on the accuracy of interface type prediction.

[0098] In some embodiments, the Euclidean distance calculation formula is used to compare the target feature vector with the feature vector of each type of interface in the interface type library to calculate the distance between them. The calculation formula (1) is shown below:

[0099]

[0100] Where xi and yi represent the target feature vector and the preset feature vector, respectively, and i represents the dimension of the vector.

[0101] However, while considering distance differences, it is also necessary to fully consider the weight of each preset feature vector. The higher the weight of a feature vector, the greater the importance of the keyword corresponding to that feature vector in the interface type judgment. The weight is mainly achieved by using the weighted distance between the preset feature vectors corresponding to multiple types of interfaces and the target feature vector. Then, the weighted distances are compared. A smaller distance indicates a higher similarity, and a larger distance indicates a lower similarity. The weighted distance calculation formula (2) after incorporating weights is as follows:

[0102]

[0103] Where i = 1, 2, ..., n, x i Let yi and yi represent the values ​​of x and y in the i-th dimension, respectively, and wi represent the weight in the i-th dimension.

[0104] S150. When the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the target type interface is used to collect data from the target data source device according to the preset collection frequency of the target data source type.

[0105] Here, "data" refers to security data from various platforms. The primary data types targeted include server logs, security device alarms, firewall alarms, audit platform operation logs, security situational awareness events, security detection data, and security management data. This can be expanded to include more data sources, such as cloud environment data, IoT device data, and user behavior data. Besides driving security capabilities and providing data support for security analysis, the collected and processed data may have other applications. For example, it can be used for quality assessments in security operations and to support operational decision-making. Adding functionality for secondary data utilization could be considered. The target data source type can be any of the data types mentioned above.

[0106] In some embodiments, the preset collection frequency for the target data source type is pre-set. The preset collection frequency for the target data source type can be user-inputted, and different data source types have different preset collection frequencies. Based on the predefined real-time requirements of various security data types, the most suitable collection method is automatically determined. Collection methods include scheduled fetching and real-time push. The collection method can also be manually customized according to specific needs. For scheduled fetching, the default is once every 300 seconds, but it can also be customized through manual management to ensure the flexibility of data collection strategy configuration. Both scheduled fetching and real-time push have their corresponding collection frequencies.

[0107] In some embodiments, the most suitable data collection method is automatically determined, which is also a pre-defined relationship set by the user. For example, the following illustrates the automatic suggestion of data collection methods for certain types of security data:

[0108] (1) For host-related security data: host logs or server logs are collected using a timed fetching method, as this type of data typically does not require real-time processing. For host or server performance metrics and other data, real-time push is generally used for data collection.

[0109] (2) For security data from security devices: Logs, alarm information, and event data from devices such as firewalls, IDS, IPS, and EDR generally have high real-time requirements, so real-time push is used for data collection. This allows for immediate alerts when a security incident occurs, enabling timely responses.

[0110] (3) For application resource security data: The collection method for logs, performance metrics, and access records of applications such as databases, web servers, and big data components usually needs to be determined based on the characteristics and requirements of the application. Generally speaking, if the application has a real-time monitoring requirement, then real-time push data collection is used; otherwise, timed retrieval can be used for collection.

[0111] (4) For security data of security management platforms: For operation logs, management logs, detection data and alarm information of platforms such as resource management platforms, 4A platforms, data security management platforms, compliance management platforms and log auditing platforms, data is collected in real time so that alarms can be obtained immediately when management operations occur or security events are detected.

[0112] In some embodiments, when the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, a communication connection with the target data source device is established, and data in the target data source device is collected using the target type interface according to the preset collection frequency of the target data source type.

[0113] In some embodiments, if no matching target type interface is found, an error message can be output indicating that the interface type cannot be recognized.

[0114] In this way, by using the target feature vector corresponding to the interface data and the weighted distance between the preset feature vectors corresponding to multiple types of interfaces, the target type interface for collecting data from the target data source device is determined. When the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the target type interface is used to collect data from the target data source device according to the preset collection frequency of the target data source type. This allows for adaptation of different types of interfaces without manual operation, improving the efficiency of interface determination. Moreover, the data is collected based on the preset collection frequency set by the adapted interface, and the collection frequency can be dynamically adjusted according to different adapted interfaces, avoiding data loss or resource waste.

[0115] Based on this, in some embodiments, obtaining the target interface data of the target data source device in S110 above may specifically include:

[0116] Obtain the interface documentation for the target data source device;

[0117] Identify the structural information of the data in the interface document, and use the structural information to segment the data in the interface document to obtain the target interface data.

[0118] In some embodiments, the interface documentation is typically an online API document or an offline file such as a PDF, Word document, or text file. The document is usually obtained through network requests or by directly reading local files. After obtaining the interface documentation, appropriate preprocessing is performed according to the document's format. For example, if the document is in HTML format, it is typically parsed using the BeautifulSoupHTML parsing library; if the document is in PDF or Word format, it is typically parsed using PDFMiner or python-docx. The document is then converted into a plain text format that is easy for machines to process.

[0119] In some embodiments, interface documents typically have a certain structure, such as different parts like titles, paragraphs, lists, and code blocks. Using this structural information, the document is segmented. After identifying the structure of the interface document and segmenting the data within it using the structural information to obtain the target interface data, target keywords matching a first preset data format and preset keywords are extracted from the target interface data. Here, the interface documents can be provided by numerous equipment manufacturers and platform vendors involved in projects across the country.

[0120] In this way, by referring to the interface documentation of the target data source device, the target interface data can be obtained, and the interface used for data collection can be determined, thus improving universal applicability.

[0121] Based on this, in some embodiments, such as Figure 2 As shown, prior to S110 above, the method may further include:

[0122] S101. Obtain interface data corresponding to multiple types of interfaces;

[0123] S102. Extract keyword samples that match the first preset data format and preset keywords from the interface data corresponding to multiple types of interfaces;

[0124] S103. Perform one-hot encoding on each keyword in the keyword sample to obtain preset feature vectors corresponding to multiple types of interfaces;

[0125] S104. Using the random forest algorithm, construct multiple decision trees; randomly select the first keyword from the keyword sample and use the first keyword as the first training set; use the first training set to train the target decision tree and obtain the first decision tree; use the second keyword from the keyword sample as the first validation set, use the first decision tree to determine the first prediction error rate of the first validation set, where the second keyword is a keyword other than the first keyword in the keyword sample, and the target decision tree is any one of the multiple decision trees;

[0126] S105. For each target keyword in the first keyword, replace the target keyword with a random word to obtain the third keyword;

[0127] S106. Use the third keyword as the second training set, train the target decision tree using the second training set, and obtain the second decision tree; use the second decision tree to determine the second prediction error rate of the first validation set.

[0128] S107. For each decision tree, calculate the difference between the first prediction error rate and the second prediction error rate.

[0129] S108. Calculate the average of the differences to obtain the importance of each target keyword in the first keyword;

[0130] S109. Based on the mapping relationship between weight and importance, determine the weight corresponding to the importance of each target keyword in the first keyword, and obtain the weight of each target keyword in the first keyword.

[0131] Here, the mapping relationship between weight and importance is predefined. The keyword sample contains all the key features of multiple types of interfaces, including the characteristics of the communication URL (such as URL format), the type of communication action (such as GET, POST, PUT, etc.), the communication data format (such as JSON, XML, etc.), the depth of the communication path, the namespace structure, etc.

[0132] Furthermore, it is understandable that if the first keyword is randomly selected from the keyword sample for a decision tree in the forest, then the first keyword selected for the entire forest includes all the keywords in the keyword sample, and thus the weight of all the keywords in the keyword sample can be determined.

[0133] In some embodiments, the influence of each keyword on the prediction result is automatically assigned a weight using the random forest algorithm, and finally a feature vector for each interface type is generated based on the weight of each keyword.

[0134] In this way, the importance of each keyword in the keyword sample is determined using the random forest algorithm, thereby determining the keyword weight. Taking into account the weight of each keyword improves the accuracy of interface type determination.

[0135] Based on this, in some embodiments, when the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, before collecting data from the target data source device using the target type interface according to the preset collection frequency of the target data source type, the method further includes:

[0136] Retrieve the keywords corresponding to the target type interface;

[0137] If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is greater than the first preset value, the first similarity threshold is determined as the similarity threshold of the target type interface.

[0138] Here, the first similarity threshold is a relatively low similarity threshold. This first similarity threshold can be user-inputted. Different types of interfaces may have different or the same first similarity threshold.

[0139] In some embodiments, for multiple types of interfaces with set similarity thresholds, if the weighted distance between the target feature vector and the preset feature vector is lower than the similarity threshold, the target feature vector is considered similar to the preset feature vector. If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is greater than a first preset value, a lower similarity threshold is set. For example, the following is an embodiment of setting similarity thresholds for some types of interfaces:

[0140] (1) For Syslog, Syslog typically contains specific keywords and formats, such as date, timestamp, hostname, service name, etc. Therefore, a similarity threshold can be set based on these characteristics. For example, if the target keywords contain most of the keywords of Syslog type interfaces, such as date, timestamp, etc., then a lower similarity threshold can be set, such as 0.5.

[0141] (2) For SNMP, SNMP typically includes specific OIDs (Object Identifiers) and data types. Therefore, a similarity threshold can be set based on these characteristics. For example, if the target keywords contain most OIDs and data types, a lower similarity threshold, such as 0.5, can be set.

[0142] (3) For REST APIs, REST APIs typically contain specific HTTP methods (such as GET, POST, etc.), URL paths, and data formats (such as JSON). Therefore, a similarity threshold can be set based on these characteristics. For example, if the target keywords contain most of the key characteristics of the REST API model, such as HTTP methods and URL paths, then a lower similarity threshold, such as 0.5, can be set.

[0143] (4) For SQL, SQL usually contains specific SQL statements (such as SELECT, INSERT, etc.) and data structures. Therefore, similarity thresholds can be set based on these characteristics. For example, if the target keywords contain most of the key characteristics of the SQL model, such as SQL statements and data structures, then a lower similarity threshold, such as 0.5, can be set.

[0144] In this way, if the keywords corresponding to the target type interface are mostly similar to the target keywords, it means that the target type interface may be matched. In this case, the similarity threshold can be set to a low value, so that the interface can be strictly selected and the accuracy of interface determination can be improved.

[0145] Based on this, in some embodiments, the following is also included:

[0146] If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is not greater than the first preset value, the second similarity threshold is determined as the similarity threshold of the target type interface.

[0147] Here, the second similarity threshold is a relatively high similarity threshold. This second similarity threshold can be user-inputted. Different types of interfaces may use different or the same second similarity threshold.

[0148] If the number of keywords that match the target keyword and the keyword corresponding to the target type interface does not exceed a first preset value, a higher similarity threshold is set. For example, the following is an embodiment of setting the similarity threshold for some type interfaces:

[0149] (1) For Syslog, Syslog typically contains specific keywords and formats, such as date, timestamp, hostname, service name, etc. Therefore, a similarity threshold can be set based on these characteristics. For example, if the target keywords contain a small portion of the keywords of the Syslog type interface, such as date, timestamp, etc., then a higher similarity threshold can be set, such as 0.8.

[0150] (2) For SNMP, SNMP typically includes specific OIDs (Object Identifiers) and data types. Therefore, a similarity threshold can be set based on these characteristics. For example, if the target keyword contains a small portion of OIDs and data types, a higher similarity threshold, such as 0.8, can be set.

[0151] (3) For REST APIs, REST APIs typically contain specific HTTP methods (such as GET, POST, etc.), URL paths, and data formats (such as JSON). Therefore, similarity thresholds can be set based on these characteristics. For example, if the target keywords contain a small portion of the key characteristics of the REST API model, such as HTTP methods and URL paths, then a higher similarity threshold, such as 0.8, can be set.

[0152] (4) For SQL, SQL usually contains specific SQL statements (such as SELECT, INSERT, etc.) and data structures. Therefore, similarity thresholds can be set based on these characteristics. For example, if the target keyword contains a small part of the key characteristics of the SQL model, such as SQL statements and data structures, then a higher similarity threshold can be set, such as 0.8.

[0153] In this way, if the keywords corresponding to the target type interface are only partially similar to the target keywords, it means that the target type interface may not be matched. In this case, the similarity threshold can be set to a higher value. If the higher similarity threshold cannot determine the similarity, then the interface to be determined is definitely not the target type interface, thus improving the accuracy of interface determination.

[0154] Based on this, in some embodiments, such as Figure 3 As shown, the method may further include:

[0155] S210. For the data collected from the target data source device, construct a target sliding window corresponding to the target data source type;

[0156] S220. Determine the change in data within two consecutive target sliding windows respectively;

[0157] S230. If the number of consecutive occurrences of the change being greater than the second preset value is greater than the third preset value, increase the sampling frequency of the preset step size.

[0158] S240. If the number of consecutive occurrences of the change value being less than the fourth preset value is greater than the third preset value, reduce the sampling frequency of the preset step size.

[0159] Here, the sampling frequency is changed in a single step by a preset step size, which is pre-defined. The size and step size of the target sliding window corresponding to the target data source type are also pre-defined. The size and step size of the target sliding window can be user-inputted, and different target data source types can correspond to different sliding windows.

[0160] In some embodiments, the trend of security data collection changes for various types of interfaces is analyzed to avoid situations where data volume surges or decreases drastically due to factors such as threat attacks, management requirements, or shifts in focus, leading to insufficient or wasted performance in data collection. A sliding window is set up during the analysis of data collection trend changes. Corresponding windows are created for different types of security data, and the size and step size of the sliding window are determined based on the characteristics and collection requirements of various types of security data. For example, the detailed settings for the size and step size of the sliding window are as follows.

[0161] (1) For server logs: Server logs are usually generated continuously and the data volume is large. Therefore, a relatively small sliding window size and step size can be set. For example, the sliding window size is 1 hour and the step size is 15 minutes.

[0162] (2) For safety device alarms: Safety device alarms may occur infrequently, but each alarm may be significant. Therefore, a larger sliding window size and a smaller step size can be set. For example, the sliding window size can be 24 hours and the step size can be 1 hour.

[0163] (3) Regarding firewall alerts: Firewall alerts may occur frequently and have a significant impact on firewall performance. Therefore, a smaller sliding window size and a smaller step size can be set. For example, the window size can be 1 hour and the step size can be 15 minutes.

[0164] (4) Regarding auditing platform operation logs: The frequency of auditing platform operation logs may be low, but each operation may affect system security. Therefore, a large sliding window size and a small step size can be set. For example, the sliding window size is 24 hours and the step size is 1 hour.

[0165] (5) For situational security events: Situational security events may occur randomly, so a medium sliding window size and step size can be set. For example, the sliding window size is 12 hours and the step size is 30 minutes.

[0166] (6) For security detection data: Security detection data may be updated infrequently, but each update may contain important information. Therefore, a larger sliding window size and a smaller step size can be set. For example, the sliding window size can be 24 hours and the step size can be 1 hour.

[0167] (7) For security management data: Security management data may be updated infrequently, but its accuracy is crucial for the effectiveness of security management. Therefore, a large sliding window size and a small step size can be set. For example, the window size could be 24 hours and the step size could be 1 hour.

[0168] Secondly, by continuously moving the computer window, the number of features of various security data is observed and counted. For each type of security data, the changes in the number of features are clearly identified. For example, the detailed implementation content is as follows.

[0169] (1) For server logs: Statistical characteristics in server logs may include the number of log events, the number of different types of events in the logs (such as "login", "logout", "modify system configuration", "modify process thread", etc.), and the frequency of specific keywords in the logs (such as "error", "warning", etc.).

[0170] (2) For safety equipment alarms: Statistical characteristics of safety equipment alarms may include the total number of alarms, the number of alarms at different levels (such as emergency, high risk, medium risk, low risk, etc.), and the number of alarms of a specific type.

[0171] (3) For firewall alarms: Statistical characteristics of firewall alarms may include the total number of alarms, the number of alarms from different source IPs, and the number of different target ports involved in the alarms.

[0172] (4) For audit platform operation logs: Statistical features in audit platform operation logs may include the total number of operations, the number of different types of operations (such as query, update, delete), and the number of operations performed by different users.

[0173] (5) For situational security incidents: Statistical characteristics of situational security incidents may include the total number of incidents, the number of incidents at different levels (such as Level I, Level II, Level III, Level IV), and the number of incidents of a specific type.

[0174] (6) For security detection data: Statistical characteristics in security detection data may include the total number of security issues detected, the number of different types of security issues (such as weak password issues, vulnerability issues, non-compliant configuration issues, APP sensitive protection issues, etc.), and the number of specific types of security issues.

[0175] (7) For security management data: Statistical characteristics in security management data may include the total number of management operations, the number of different types of management operations (such as account management, authorization management, resource management, security policy management, threat blocking management, etc.), and the number of management operations performed by different users.

[0176] Finally, by analyzing the number of features in multiple consecutive windows, the trend of data change is identified. When the amount of data in multiple consecutive sliding windows is increasing, it can be predicted that the amount of data may continue to increase in the near future. That is, if the number of consecutive occurrences of the change being greater than the second preset value is greater than the third preset value, the sampling frequency of the preset step size is increased. When the amount of data in several consecutive windows is decreasing, it can be predicted that the amount of data may continue to decrease in the near future. That is, if the number of consecutive occurrences of the change being less than the fourth preset value is greater than the third preset value, the sampling frequency of the preset step size is decreased.

[0177] In this way, the collection frequency is dynamically adjusted based on fluctuations in the amount of data, ensuring the maximum real-time performance and collection efficiency of the data, while also avoiding resource waste or insufficient performance.

[0178] Based on this, in some embodiments, such as Figure 4 As shown, the method may further include:

[0179] S310. Obtain the network load value of the target type interface;

[0180] S320. When the network load value is less than the fifth preset value, increase the sampling frequency by the preset step size;

[0181] S330. If the network load value is greater than the sixth preset value, reduce the sampling frequency of the preset step size.

[0182] In some embodiments, the network quality of the current path is analyzed, integrating DPI deep probing technology with network performance testing technology. First, a certain number of data packets are sent to the network path, and metrics such as transmission time and packet loss rate are measured. This allows for the calculation of the current network bandwidth, latency, and packet loss rate, providing a precise understanding of the overall network path quality at a macroscopic level. Second, a deep analysis is performed on the data packets being transmitted throughout the network. By capturing data packets and decomposing them into three dimensions—network layer, transport layer, and application layer—the application layer content of the data packets is examined to determine their type. This reveals the network usage of various interface types, the amount of bandwidth used, and the peak and average network traffic, providing a microscopic understanding of network occupancy during various data reception processes.

[0183] Finally, regarding network quality analysis, the data collection frequency is adjusted based on network bandwidth usage and changes in network performance. When the network load on the target type interface is high, the frequency of relevant data collection is automatically reduced; when the network load on the target type interface is low, the frequency of relevant data collection is automatically increased.

[0184] In some embodiments, when the overall network performance load is high, the frequency of all data collection is reduced; when the overall network performance load is low, the frequency of all data collection is increased.

[0185] In some embodiments, the sampling frequency is changed in a single step by a preset step size. The preset step size is pre-set.

[0186] In this way, the data collection frequency is dynamically adjusted based on network quality fluctuations, ensuring maximum real-time data transmission and collection efficiency while avoiding resource waste or performance inadequacy.

[0187] Based on this, in some embodiments, such as Figure 5 As shown, after S150 above, the method may further include:

[0188] S160. Extract multiple feature information that matches the second preset data format and preset feature information from the data collected from the target data source device;

[0189] S170, Save multiple feature information.

[0190] In some embodiments, the collected data is first cleaned, including removing invalid, erroneous, duplicate, or irrelevant records. For example, for host logs, it may be necessary to remove irrelevant information generated by the system, retaining only records related to security events. Secondly, the cleaned data is formatted. For example, firewalls, IDS, IPS, and other devices may use different log formats; JSON format is typically used to unify the log format for subsequent processing. Finally, the formatted data undergoes data transformation. Based on a pre-defined standard dictionary, difficult-to-understand encodings are converted into more readable text information. For example, data types 0, 1, and 2 in endpoint virus detection data are transformed into Trojan viruses, worm viruses, and CIH viruses; threat types 0, 1, and 2 in intrusion detection alerts are transformed into webshell script uploads, DoS attacks, and vulnerability exploit attacks. The data preprocessing methods and processes are pre-defined by the user, with different preprocessing methods corresponding to different data types. For example, the preprocessing implementation for a portion of the collected data is as follows:

[0191] (1) For host-related security data (e.g., server logs, performance metrics): it may be necessary to clear irrelevant information generated by the system and retain only records related to security events. At the same time, performance metrics may need to be converted to numerical values.

[0192] (2) For security data of security devices (e.g., device logs, alarm information, and event data of firewalls, IDS, and IPS): it is necessary to unify the log formats of different devices or manufacturers. This may include extracting key information fields, converting a specific format to a general format, and performing numerical-to-text conversion of some content.

[0193] (3) For application resource security data (e.g., application logs, performance metrics, and access records of databases, web servers, and big data components): it may be necessary to extract key application log information and remove invalid or redundant records. Performance metrics and access records may need to be converted to numerical values.

[0194] (4) For security data of security management platforms (e.g., operation logs, management logs, detection data and alarm information of resource management platforms, 4A platforms, data security management platforms, compliance management platforms, log auditing platforms): it is necessary to integrate data from various management platforms, which may involve data formatting, integration and cleaning.

[0195] In some embodiments, multiple feature information matched with a second preset data format and preset feature information from the preprocessed data serves as the data foundation for subsequent security operations. First, the preprocessed data is parsed using regular expressions, JSON parsing, YARA rule matching, etc. Second, feature information of the parsed data content is extracted, typically using a feature selection method. The key features selected generally include: event type, event occurrence time, source address, destination address, operation type, and result status. For example, the implementation of general content feature information extraction for certain types of security data is as follows:

[0196] (1) For host-related security data: For example, for server logs, it may be necessary to parse the log timestamp, log level, log message, etc., and extract key feature information, such as operation type, operation result, resources involved, etc.

[0197] (2) For security data of security devices: For example, for firewall logs, it may be necessary to parse the log timestamp, source address, destination address, protocol type, action, etc., and extract key feature information, such as attack type, attack result, attack target, etc.

[0198] (3) For application resource security data: For example, for database logs, it may be necessary to parse the log timestamp, operation type, operation object, operation result, etc., and extract key feature information, such as database query type, query result, tables or fields involved, etc.

[0199] (4) For security data of security management platform: For example, for audit logs, it may be necessary to parse the log timestamp, operation type, operation object, operation result, etc., and extract key feature information, such as operator identity, operation type, operation result, etc.

[0200] In some embodiments, the extracted feature information is saved.

[0201] This preprocessing, parsing, and standardization of various types of data facilitates data retrieval and analysis in subsequent security operation workflows, enabling more efficient and accurate cross-platform security data collection and processing.

[0202] Based on this, in some embodiments, such as Figure 6 As shown, after S160 above, the method may further include:

[0203] S161. Segment multiple feature information to obtain multiple first words;

[0204] S162. Remove duplicates from multiple first words to obtain multiple second words;

[0205] S163. For each feature information, calculate the first ratio of the number of occurrences of each second word to the total number of occurrences of multiple second words to obtain the word frequency of each second word;

[0206] S164. Based on the number of multiple feature information and the number of target feature information, calculate the inverse document frequency of each second word. The target feature information is the feature information of the corresponding second word among multiple feature information.

[0207] S165. Determine the weight of each second word based on its term frequency and inverse document frequency;

[0208] S166. Sort the multiple second words according to their weights to obtain a second word sequence;

[0209] S167. Select the third word with a relatively large weight from the second word sequence;

[0210] Specifically, S170 mentioned above may include:

[0211] S171, Preserve the third vocabulary.

[0212] In some embodiments, special features are extracted from the parsed data content. When processing security data, some fields contain complex content and a large amount of high-value information. These may not be effectively extracted using conventional regular expression parsing or key feature methods, resulting in low utilization of security data value, inaccurate data parsing, and low data parsing quality. To address the special field content of complex security data, a bag-of-words model-based parsing method is adopted to purify the complex content, thereby accurately extracting high-value data.

[0213] Specifically, the process begins by removing punctuation, numbers, and special characters from the multiple feature information obtained after data parsing. Then, operations such as word segmentation, stemming, and lemma reconstruction are performed to unify words of different forms into a basic structure. This helps eliminate noise and irrelevant information in complex security data, making the data clearer and more standardized. After completing complex text processing, a vocabulary is constructed by extracting unique words from the processed feature information. The size of the vocabulary can be controlled by setting a word frequency threshold, thereby filtering out words with low frequency. This centralizes the display of high-value words in the security data, enabling subsequent data reading devices in the security operation workflow to more effectively access keywords and thematic information from the geographic data.

[0214] Next, the words in the vocabulary are converted into numerical feature vectors. The frequency of each word in the text is used as the feature vector for extraction, called the term frequency threshold. The term frequency is calculated as shown in formula (3):

[0215]

[0216] Where ft,d represents the number of times word t appears in the feature information d, and ∑t′∈dft′,d represents the sum of the number of times all words appear in d.

[0217] Additionally, the proportion of the number of feature information containing word t to the total number of feature information is extracted as a feature vector, called Inverse Document Frequency (IDF), which is used to evaluate the relevance of the word in this type of data. The higher the relevance, the more important the word is. The calculation method of the inverse document frequency of a word is shown in formula (4):

[0218]

[0219] Where |D| represents the total number of data entries of the type to which this feature information belongs, i.e. the number of feature information entries, and |{d∈D:t∈d}| represents the number of feature information entries containing the word t. The denominator is 1+|{d∈D:t∈d}|. Adding 1 is to avoid dividing by zero when the term does not appear in any feature information.

[0220] To consider the impact of word frequency and IDF inverse document frequency on word value, a TF-IDF word weight calculation method is constructed to extract the weight of words in feature information as a feature vector, which is called the word weight value (TF-IDF value). The calculation formula (5) for word weight is as follows:

[0221] TFIDF(t,d,D)=TF(t,d)×IDF(t,D) (5)

[0222] By calculating the word weight value of each word, the importance of each word can be displayed intuitively. These word weights will be placed in the cache and wait to be used as the basis for judging high-value words later.

[0223] In some embodiments, valuable keywords can be extracted from complex content by adding a threshold setting for TF-IDF values. Multiple second-word terms are sorted according to their weights to obtain a second-word sequence. Generally, a percentage-based principle is used, setting a preset proportion (e.g., the top 20%) of all terms with higher TF-IDF weights as the criterion for data extraction. It should be noted that a manual threshold adjustment function is supported later, allowing for flexible manual correction when there is too much or too little content being extracted and parsed.

[0224] In some embodiments, the selected third vocabulary is stored in a Redis in-memory database or a Memcached cache system to facilitate real-time retrieval of security data during subsequent security operations workflows.

[0225] In this way, in addition to satisfying regular expressions, JSON parsing, YARA rule matching, etc., it also supports a data parsing method based on the bag-of-words model to extract keywords from highly complex content, which improves the efficiency and quality of data processing and provides convenience for data access and analysis in subsequent security operation workflows.

[0226] It should be noted that, in addition to data type conversion, data range scaling, and data encoding methods, other data standardization techniques can be considered for data standardization, such as data alignment and data normalization.

[0227] In the embodiments provided in this application, accurate and efficient communication with various data sources can be automatically established, greatly simplifying the data connection process and improving its accuracy and efficiency. It can automatically select the most suitable data collection method and frequency based on real-time network conditions, data types, and other factors, thereby optimizing data collection efficiency and data integrity. It can preprocess, parse, and standardize various types of security data, enabling effective processing even for complex and unstructured security data, while also facilitating data retrieval and analysis in subsequent security operation workflows. This allows for more efficient and accurate cross-platform security data collection and processing.

[0228] The data acquisition method provided in this application can be applied to a data acquisition system. The data acquisition system includes a secure data interface interface module, a secure data collection module, and a secure data content parsing and processing module. Based on these modules, the secure data interface interface module first interfaces with the data transmission interfaces provided by various platforms and data storage devices. During the data interface process, Euclidean distance analysis incorporating feature weight ratios is used to automatically identify the interface type and other relevant information. After successful data transmission interface interface integration, the secure data collection module collects the secure data provided by the interface. During the collection process, data collection trend analysis and network quality fluctuation analysis are used to automatically optimize the data collection method and frequency, avoiding data collection anomalies caused by factors such as wasted computing power and high network load. Finally, the collected security data will undergo data preprocessing through the security data content parsing and processing module. For relatively standardized and simple data content, multiple feature information is extracted based on technologies such as regular expressions, JSON parsing, and YARA rule matching. For unordered and complex data content with multiple feature information, a bag-of-words model is used to extract features of special complexity in security data, generating standardized, easily understandable, and high-quality security data, which facilitates data access and analysis in subsequent security operation workflows.

[0229] Specifically, the secure data interface integration module includes an interface document parsing unit, a feature extraction unit, an interface type library unit, and a feature matching unit. The interface document parsing unit identifies the structural information of the data in the interface document, segments the data using this information, and obtains the target interface data. The feature extraction unit extracts target keywords from the target interface data that match a first preset data format and preset keywords. The interface type library unit provides an interface type library covering various interface types such as HTTP, HTTPS, FTP, SFTP, SMTP, POP3, IMAP, Telnet, and SSH, containing a unique identifier for each interface type and the feature vector of its corresponding keywords. The feature matching unit determines the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector. This secure data interface integration module, in addition to allowing users to manually fill in data interface keywords and select connection drivers and interface types, can automatically identify interface types based on the interface documents provided by the secure data interface provider, select appropriate interface drivers and communication protocols for stable secure data interface communication. This reduces the workload of traditional manual configuration and potential human errors, thereby improving the efficiency and accuracy of data collection. Furthermore, in the event of changes to the secure data interface, only the interface documentation needs to be updated, and all connection configurations will automatically select the optimal connection method, effectively shortening the real-time response time caused by interface changes.

[0230] Understandably, the secure data interface interface docking module, based on the interface document parsing unit, feature extraction unit, interface type library unit, and feature matching unit, has the ability to proactively identify interface types and protocols based on the data source interface document, rather than relying on the interface type information provided by the data source. Furthermore, it actively selects the most appropriate collection method based on feature comparison to ensure the accuracy of interface type identification.

[0231] The security data collection module includes a collection policy management unit, a collection execution unit, and a collection policy adaptation unit. The collection policy management unit receives the collection frequency. The collection execution unit establishes a communication connection with the target data source device to collect data. The collection policy adaptation unit adjusts the collection frequency based on changes in data volume and network load during the security data collection process. In addition to traditional policy configuration methods such as manual configuration of collection methods and frequencies, this security data collection module can automatically suggest collection methods for various types of security data based on the characteristics and real-time requirements of the data source. During the collection process, it automatically adjusts the security data collection frequency based on fluctuations in data volume and network quality. By selecting the most suitable collection method and frequency, it achieves accurate and efficient communication with the data source, improving the efficiency and accuracy of data collection and avoiding resource waste caused by frequent data collection.

[0232] Understandably, the security data collection module, based on the collection strategy management unit, collection execution unit, and collection strategy adaptive unit, has the characteristics and requirements of data source-based proactive suggestion of data collection methods to avoid incorrect configuration due to insufficient human experience, which would affect the efficiency and accuracy of data collection. In addition, during the security data collection process, the collection frequency strategy is dynamically adjusted based on fluctuations in data quantity and network quality to ensure the maximum real-time performance and collection efficiency of the data, and to avoid resource waste or insufficient performance.

[0233] It should be noted that, based on the interface type provided by the security data interface interface module and the data collection method and frequency provided by the data collection strategy management unit, the collection execution unit establishes a communication connection with the target data source. For example, if the data source is a web service and uses the HTTP protocol, the security data collection module will use an HTTP client library (such as Python's requests library) to establish an HTTP connection.

[0234] Furthermore, current security data collection modules primarily rely on pre-defined interface type libraries and predefined parsing rules. Introducing intelligent technologies such as machine learning, enabling these modules to learn and adapt to new interface types and data formats based on the collected data, would significantly improve their adaptability and efficiency.

[0235] The secure data content parsing and processing module comprises a data preprocessing unit, a data parsing unit, and a data storage unit. The data preprocessing unit preprocesses the data to identify multiple feature information. The data parsing unit further parses complex data from these feature information to identify the third keyword. The data storage unit stores the third keyword. In addition to supporting existing technologies such as regular expressions, JSON parsing, and YARA rule matching, this secure data content parsing module also supports a bag-of-words model-based data parsing method for extracting keywords from highly complex content, improving data processing efficiency and quality, and facilitating data access and analysis in subsequent security operations workflows.

[0236] Understandably, the security data content parsing and processing module, based on the data preprocessing unit, data parsing unit, and data storage unit, has the ability to extract features of particularly complex content. Through text processing, vocabulary construction, and text vectorization analysis, it realizes the transformation from raw text to structured data, reduces the complexity dimension of all words in the content, helps to reduce the data analysis complexity and memory consumption of data access devices in subsequent security operation workflows, improves data processing efficiency, and provides a foundation for subsequent data-driven automated orchestration of security operations.

[0237] In the embodiments provided in this application, the secure data interface docking module automatically identifies the data interface based on Euclidean distance analysis incorporating feature weight ratios. This module can proactively identify the interface type and communication protocol type of the data acquisition source, achieving automatic interface docking, significantly improving data docking efficiency and reducing manual configuration workload. The secure data collection module automatically adjusts the collection strategy based on data acquisition trend analysis and network quality fluctuation analysis. This module can automatically adjust the collection strategy according to the real-time requirements of the data source and network quality, achieving dynamic data collection, avoiding both data loss and resource waste. The secure data content parsing and processing module uses a bag-of-words model to implement a data parsing unit design for extracting features of the special complexity of secure data. This allows for preprocessing, parsing, and standardization of the collected data, extracting key information, adapting to complex and variable data content, and improving the accuracy and efficiency of data processing.

[0238] In summary, the data acquisition system provided in this application has significant advantages in the three stages of data docking, data collection, and data processing, and can collect security data scattered across various platforms more accurately and efficiently.

[0239] Based on the data acquisition method provided in the above embodiments, this application also provides specific implementations of the data acquisition device. Please refer to the following embodiments.

[0240] First see Figure 7 The data acquisition device 400 provided in this application embodiment includes:

[0241] The acquisition module 410 is used to acquire the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple types of interfaces, and the weights of the preset feature vectors.

[0242] Extraction module 420 is used to extract target keywords that match the first preset data format and preset keywords from the target interface data;

[0243] Encoding module 430 is used to perform one-hot encoding on the target keywords to obtain the target feature vector;

[0244] The determination module 440 is used to determine the weighted distance between the preset feature vectors and the target feature vectors corresponding to multiple types of interfaces based on the weights of the preset feature vectors.

[0245] The acquisition module 450 is used to acquire data from the target data source device using the target type interface according to the preset acquisition frequency of the target data source type when the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface.

[0246] Based on this, in some embodiments, the acquisition module 410 can be specifically used for:

[0247] Obtain the interface documentation for the target data source device;

[0248] Identify the structural information of the data in the interface document, and use the structural information to segment the data in the interface document to obtain the target interface data.

[0249] Based on this, in some embodiments, the device 400 may further include:

[0250] The acquisition module 410 is also used to acquire interface data corresponding to multiple types of interfaces before acquiring the preset feature vectors and the weights of the preset feature vectors corresponding to multiple types of interfaces;

[0251] The extraction module 420 is also used to extract keyword samples that match the first preset data format and preset keywords from the interface data corresponding to multiple types of interfaces;

[0252] The encoding module 430 is also used to perform one-hot encoding on each keyword in the keyword sample to obtain preset feature vectors corresponding to multiple types of interfaces.

[0253] The determination module 440 is also used to construct multiple decision trees using the random forest algorithm; randomly select the first keyword from the keyword sample and use the first keyword as the first training set; train the target decision tree using the first training set to obtain the first decision tree; use the second keyword from the keyword sample as the first validation set, and use the first decision tree to determine the first prediction error rate of the first validation set. The second keyword is a keyword in the keyword sample other than the first keyword, and the target decision tree is any one of the multiple decision trees.

[0254] The replacement module is used to replace each target keyword in the first keyword with a random word to obtain the third keyword.

[0255] The determination module 440 is also used to use the third keyword as the second training set, train the target decision tree using the second training set to obtain the second decision tree, and use the second decision tree to determine the second prediction error rate of the first validation set.

[0256] The determination module 440 is also used to calculate the difference between the first prediction error rate and the second prediction error rate for each decision tree;

[0257] The determination module 440 is also used to calculate the average of the differences to obtain the importance of each target keyword in the first keyword;

[0258] The determination module 440 is also used to determine the weight corresponding to the importance of each target keyword in the first keyword according to the mapping relationship between weight and importance, so as to obtain the weight of each target keyword in the first keyword.

[0259] Based on this, in some embodiments, the device 400 may further include:

[0260] The acquisition module 410 is also used to acquire the keywords corresponding to the target type interface before collecting data from the target data source device using the target type interface according to the preset collection frequency of the target data source type when the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface;

[0261] The determination module 440 is further configured to determine the first similarity threshold as the similarity threshold of the target type interface when the number of keywords that are consistent with the target keyword and the keyword corresponding to the target type interface is greater than the first preset value.

[0262] Based on this, in some embodiments, the device 400 may further include:

[0263] The determination module 440 is further configured to determine the second similarity threshold as the similarity threshold of the target type interface when the number of keywords that are consistent with the target keyword and the keyword corresponding to the target type interface is not greater than the first preset value.

[0264] Based on this, in some embodiments, the device 400 may further include:

[0265] The building module is used to construct a target sliding window corresponding to the target data source type for the data collected from the target data source device;

[0266] The determination module 440 is also used to determine the amount of change in data within two consecutive target sliding windows;

[0267] An additional module is added to increase the sampling frequency of a preset step size when the number of consecutive occurrences of the change value being greater than the second preset value is greater than the third preset value.

[0268] The reduction module is used to reduce the sampling frequency of the preset step size when the number of consecutive occurrences of the change amount being less than the fourth preset value is greater than the third preset value.

[0269] Based on this, in some embodiments, the device 400 may further include:

[0270] The acquisition module 410 is also used to acquire the network load value of the target type interface;

[0271] The added module is also used to increase the sampling frequency of a preset step size when the network load value is less than the fifth preset value;

[0272] The reduction module is also used to reduce the sampling frequency of the preset step size when the network load value is greater than the sixth preset value.

[0273] Based on this, in some embodiments, the device 400 may further include:

[0274] The extraction module 420 is also used to extract multiple feature information that matches the second preset data format and preset feature information from the collected data of the target data source device after collecting data from the target data source device according to the preset collection frequency of the target data source type when the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface.

[0275] The storage module is used to store multiple feature information.

[0276] Based on this, in some embodiments, the device 400 may further include:

[0277] The word segmentation module is used to extract multiple feature information that matches the second preset data format and preset feature information from the data collected from the target data source device, and then segment the multiple feature information to obtain multiple first words.

[0278] The deduplication module is used to remove duplicates from multiple first words to obtain multiple second words;

[0279] The determination module 440 is also used to calculate, for each feature information, the first ratio of the number of occurrences of each second word to the total number of occurrences of multiple second words, to obtain the word frequency of each second word;

[0280] The determination module 440 is also used to calculate the inverse document frequency of each second word based on the number of multiple feature information and the number of target feature information, wherein the target feature information is the feature information of the corresponding second word among multiple feature information;

[0281] The determination module 440 is also used to determine the weight of each second word based on the word frequency and inverse document frequency of each second word;

[0282] The sorting module is used to sort multiple second words according to their weights to obtain a second word sequence;

[0283] The selection module is used to select a third word with a relatively high weight from the second word sequence;

[0284] The save module can be used specifically for:

[0285] Save the third vocabulary.

[0286] Each module of the data acquisition device provided in this application embodiment can realize the functions of each step of the data acquisition method provided above, and can achieve its corresponding technical effects. For the sake of brevity, it will not be described in detail here.

[0287] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0288] Figure 8 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0289] An electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0290] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0291] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0292] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0293] The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any of the data acquisition methods in the above embodiments.

[0294] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 8 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0295] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0296] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection. The electronic device can perform the data acquisition method described in the embodiments of the present invention, thereby realizing the data acquisition method described above.

[0297] Furthermore, in conjunction with the data acquisition methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data acquisition methods in the above embodiments.

[0298] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described data acquisition method embodiments.

[0299] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0300] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0301] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0302] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0303] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data acquisition method, characterized in that, include: Obtain the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple types of interfaces, and the weights of the preset feature vectors; Extract target keywords that match the first preset data format and preset keywords from the target interface data; The target keywords are one-hot encoded to obtain the target feature vector; Based on the weights of the preset feature vectors, determine the weighted distance between the preset feature vectors corresponding to the multiple types of interfaces and the target feature vector; When the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the target type interface is used to collect data from the target data source device according to the preset collection frequency of the target data source type; For the data collected from the target data source device, a target sliding window corresponding to the target data source type is constructed; Determine the amount of change in data within multiple consecutive sliding windows of two targets; If the number of consecutive occurrences of the change being greater than the second preset value is greater than the third preset value, the sampling frequency of the preset step size is increased. If the number of consecutive occurrences of the change being less than the fourth preset value is greater than the third preset value, the sampling frequency of the preset step size is reduced.

2. The data acquisition method according to claim 1, characterized in that, The acquisition of target interface data from the target data source device includes: Obtain the interface documentation for the target data source device; Identify the structural information of the data in the interface document, and use the structural information to segment the data in the interface document to obtain the target interface data.

3. The data acquisition method according to claim 1, characterized in that, Before obtaining the preset feature vectors corresponding to multiple types of interfaces and the weights of the preset feature vectors, the method further includes: Obtain the interface data corresponding to the multiple types of interfaces; Extract keyword samples that match the first preset data format and preset keywords from the interface data corresponding to the multiple types of interfaces; Each keyword in the keyword sample is one-hot encoded to obtain a preset feature vector corresponding to the multiple types of interfaces; Multiple decision trees are constructed using the random forest algorithm; a first keyword is randomly selected from the keyword sample and used as the first training set; a target decision tree is trained using the first training set to obtain the first decision tree; a second keyword from the keyword sample is used as the first validation set, and the first prediction error rate of the first validation set is determined using the first decision tree, wherein the second keyword is any keyword in the keyword sample other than the first keyword, and the target decision tree is any one of the multiple decision trees; For each target keyword in the first keyword, a third keyword is obtained by replacing the target keyword with a random word; The third keyword is used as the second training set, and the target decision tree is trained using the second training set to obtain the second decision tree; the second prediction error rate of the first validation set is determined using the second decision tree. For each decision tree, calculate the difference between the first prediction error rate and the second prediction error rate; Calculate the average of the differences to obtain the importance of each target keyword in the first keyword; Based on the mapping relationship between weight and importance, the weight corresponding to the importance of each target keyword in the first keyword is determined, and the weight of each target keyword in the first keyword is obtained.

4. The data acquisition method according to claim 1, characterized in that, Before collecting data from the target data source device using the target type interface according to a preset collection frequency of the target data source type, when the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, the method further includes: Retrieve the keywords corresponding to the target type interface; If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is greater than a first preset value, the first similarity threshold is determined as the similarity threshold of the target type interface.

5. The data acquisition method according to claim 4, characterized in that, Also includes: If the number of keywords that match the target keyword and the keyword corresponding to the target type interface is not greater than a first preset value, then the second similarity threshold is determined as the similarity threshold of the target type interface.

6. The data acquisition method according to claim 1, characterized in that, Also includes: Obtain the network load value of the target type interface; If the network load value is less than the fifth preset value, the sampling frequency is increased by a preset step size; If the network load value is greater than the sixth preset value, the sampling frequency of the preset step size is reduced.

7. The data acquisition method according to claim 1, characterized in that, When the weighted distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface, after collecting data from the target data source device using the target type interface according to the preset collection frequency of the target data source type, the method further includes: From the data collected from the target data source device, extract multiple feature information that matches the second preset data format and preset feature information; The aforementioned feature information is stored.

8. The data acquisition method according to claim 7, characterized in that, After extracting multiple feature information that matches the second preset data format and preset feature information from the data collected from the target data source device, the method further includes: The multiple feature information is segmented to obtain multiple first words; The multiple first words are deduplicated to obtain multiple second words; For each of the aforementioned feature information, a first ratio is calculated between the number of occurrences of each second word and the total number of occurrences of the plurality of second words to obtain the word frequency of each second word; Based on the number of the plurality of feature information and the number of target feature information, the inverse document frequency of each second word is calculated, wherein the target feature information is the feature information of the plurality of feature information that includes the corresponding second word; The weight of each second word is determined based on its term frequency and inverse document frequency. The multiple second words are sorted according to their weights to obtain a second word sequence; From the second word sequence, select a third word in a predetermined proportion according to the order of weight from largest to smallest; The storage of the multiple feature information includes: Save the third vocabulary.

9. A data acquisition device, characterized in that, include: The acquisition module is used to acquire the target interface data and target data source type of the target data source device, the preset feature vectors corresponding to multiple types of interfaces, and the weights of the preset feature vectors; The extraction module is used to extract target keywords that match the first preset data format and preset keywords from the target interface data; The encoding module is used to perform one-hot encoding on the target keywords to obtain the target feature vector; The determination module is used to determine the weighted distance between the preset feature vectors corresponding to the multiple types of interfaces and the target feature vector based on the weights of the preset feature vectors; The acquisition module is used to acquire data from the target data source device using the target type interface at a preset acquisition frequency for the target data source type when the weight distance between the preset feature vector corresponding to the target type interface and the target feature vector is less than the similarity threshold of the target type interface. The construction module is used to construct a target sliding window corresponding to the target data source type for the data collected from the target data source device; The determining module is also used to determine the amount of change in data within multiple consecutive two target sliding windows; An additional module is added to increase the sampling frequency by a preset step size when the number of consecutive occurrences of the change being greater than the second preset value is greater than the third preset value. The reduction module is used to reduce the sampling frequency of the preset step size when the number of consecutive occurrences of the change amount being less than the fourth preset value is greater than the third preset value.

10. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data acquisition method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data acquisition method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data acquisition method as described in any one of claims 1-8.

Images