Data denoising method and device based on environmental dimension, equipment and storage medium

Abstract

The application belongs to the technical field of network security, and discloses a data denoising method and device based on environment dimensions, equipment and storage medium, wherein the method comprises the following steps: obtaining host environment dimension samples in a suspected test machine sample set; determining corresponding host environment features and host routing features according to the host environment dimension samples; matching the host environment features and the host routing features with sample organization rules respectively to obtain corresponding sample families; obtaining the association relationship between the sample families and advanced threat actors, and performing denoising processing on the suspected test machine sample set according to the association relationship. In the application, the corresponding features are determined according to the host environment dimension samples, the association relationship between the sample families matched by the corresponding features and the sample organization rules and the advanced threat actors is obtained, the samples with low possibility of being test machine samples in the suspected test machine samples are removed, the loss of computing resources is reduced, the sample analysis speed is improved, and the sample analysis difficulty is reduced.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a data noise reduction method, apparatus, device, and storage medium based on environmental dimensions. Background Technology

[0002] In the process of testing and countering advanced persistent threat (APT) actors against secure endpoint products, APT actors will in most cases install the endpoint product in a virtual environment and test the ability of their malicious tools to counter the secure endpoint product, thereby assessing the feasibility of the next attack deployment.

[0003] During the confrontation process, advanced threat actors will continuously expose their own habitual characteristics. Therefore, by extracting, identifying and organizing such logs to form a knowledge base, and using multi-dimensional methods to quickly associate and confirm past or ongoing test confrontation activities, it is possible to quickly identify advanced threat actors and predict their next attack trends.

[0004] However, in the process of confronting advanced threat actors, the captured suspected test machine sample data usually contains a large number of noisy samples, that is, the suspected test machine samples contain a large number of normal user data samples. If all of them are analyzed, it will waste a lot of computing resources and greatly increase the difficulty of analysis.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a data noise reduction method, apparatus, device, and storage medium based on environmental dimensions, aiming to solve the technical problem that existing test machine sample data contains a large number of noisy samples, which wastes a lot of analysis and computing resources and greatly increases the difficulty of analysis.

[0007] To achieve the above objectives, the present invention provides a data denoising method based on the environmental dimension, the method comprising the following steps: Obtain host environment dimension samples from the suspected test machine sample set; Determine the corresponding host environment features and host routing features based on the host environment dimension samples; The host environment features and the host routing features are matched with the sample organization rules to obtain the corresponding sample families; Obtain the association between the sample family and advanced threat actors, and perform noise reduction processing on the suspected test machine sample set based on the association.

[0008] Optionally, before the step of matching the host environment features and the host routing features with the sample organization rules to obtain the corresponding sample families, the method further includes: Obtain the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set; Sample organization rules are constructed based on the host environment characteristics and routing characteristics of the test machine.

[0009] Optionally, the step of obtaining the test machine host environment characteristics and test machine routing characteristics corresponding to each test machine sample in the test machine sample set includes: Obtain the motherboard characteristics, version characteristics, and maintenance time characteristics of each test machine sample in the test machine sample set; The test machine host environment characteristics are generated based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine maintenance time characteristics. Obtain the test machine routing features corresponding to each test machine sample in the test machine sample set.

[0010] Optionally, the step of obtaining the test machine routing features corresponding to each test machine sample in the test machine sample set includes: Obtain the terminal-associated routing address corresponding to each test machine sample in the test machine sample set; Identify whether the terminal's associated routing address is a local area network address corresponding to a preset routing vendor; When the terminal's associated routing address is the local area network address, the terminal's associated routing address is used as the routing feature of the test machine.

[0011] Optionally, the step of matching the host environment features and the host routing features with sample organization rules to obtain the corresponding sample families includes: The host environment features are matched with the sample organization rules to obtain the environment matching scores between the host environment features and each sample family; The host routing features are matched with the sample organization rules to obtain the routing matching scores between the host routing features and each sample family; The corresponding sample family is determined based on the environment matching score and the route matching score.

[0012] Optionally, the step of matching the host routing features with the sample organization rules to obtain the routing matching scores between the host routing features and each sample family includes: Obtain the host routing address based on the host routing characteristics; The host routing address is traced and associated to obtain the associated user address; The test machine routing characteristics are obtained according to the sample organization rules, and the terminal associated routing address is determined according to the test machine routing characteristics; The associated user address and the associated routing address of the terminal are used to perform profile recognition to obtain profile recognition results; The profile recognition results are used to determine the routing matching scores between the host routing features and each sample family.

[0013] Optionally, the step of tracing and associating the host routing address to obtain the associated user address includes: The user node port identifier is triggered based on the host routing address, and the user's actual address is obtained based on the user node port identifier; Establish the association between the user's actual address and the host's routing address to obtain the address tracing of the user node; The address of the user node is traced back to the associated user address.

[0014] Optionally, the step of determining the corresponding sample family based on the environment matching score and the route matching score includes: Sort the environment matching score and the route matching score in descending order to obtain the sorting result; The sample family corresponding to the first-ranked matching score in the ranking results is taken as the sample family corresponding to the host environment feature and the host routing feature.

[0015] Optionally, the step of performing noise reduction processing on the suspected test machine sample set based on the correlation includes: Each suspected test machine sample in the suspected test machine sample set is marked according to the aforementioned association; Remove the suspected test machine samples marked as noise samples from the suspected test machine sample set.

[0016] Optionally, the step of marking each suspected test machine sample in the suspected test machine sample set according to the association relationship includes: Based on the aforementioned correlation, obtain the correlation score of each suspected test machine sample in the suspected test machine sample set; Suspected test machine samples whose correlation scores are less than or equal to a preset score threshold are marked as noise samples.

[0017] Optionally, before the step of obtaining host environment dimension samples from the suspected test machine sample set, the method further includes: Obtain the sample logs corresponding to each suspected test machine sample in the initial test machine sample set; The initial test machine sample set is denoised based on the sample logs to obtain a suspected test machine sample set.

[0018] Optionally, the step of performing noise reduction processing on the initial test machine sample set based on the sample log includes: Each suspected test machine sample in the initial test machine sample set is marked according to the sample log; The suspected test machine samples marked as noise samples in the initial test machine sample set are removed.

[0019] Optionally, the step of marking each suspected test machine sample in the initial test machine sample set according to the sample log includes: Based on the sample logs, obtain the log point level corresponding to each suspected test machine sample in the initial test machine sample set; Suspected test machine samples whose log point volume exceeds a preset threshold are marked as noise samples.

[0020] Furthermore, to achieve the above objectives, the present invention also proposes a data noise reduction device based on the environmental dimension, the data noise reduction device based on the environmental dimension comprising: The acquisition module is used to acquire host environment dimension samples from the suspected test machine sample set; The determination module is used to determine the corresponding host environment features and host routing features based on the host environment dimension samples. The matching module is used to match the host environment features and the host routing features with the sample organization rules respectively to obtain the corresponding sample families; The processing module is used to obtain the association between the sample family and advanced threat actors, and to perform noise reduction processing on the suspected test machine sample set according to the association.

[0021] Optionally, the environmental-dimensional data noise reduction device further includes: a construction module; The construction module is used to obtain the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set. Sample organization rules are constructed based on the host environment characteristics and routing characteristics of the test machine.

[0022] Optionally, the acquisition module is further configured to acquire the motherboard characteristics, version characteristics, and maintenance time characteristics of each test machine sample in the test machine sample set. The test machine host environment characteristics are generated based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine maintenance time characteristics. Obtain the test machine routing features corresponding to each test machine sample in the test machine sample set.

[0023] Optionally, the acquisition module is further configured to acquire the terminal associated routing address corresponding to each test machine sample in the test machine sample set; Identify whether the terminal's associated routing address is a local area network address corresponding to a preset routing vendor; When the terminal's associated routing address is the local area network address, the terminal's associated routing address is used as the routing feature of the test machine.

[0024] Optionally, the matching module is further configured to match the host environment features with the sample organization rules to obtain the environment matching scores of the host environment features and each sample family; The host routing features are matched with the sample organization rules to obtain the routing matching scores between the host routing features and each sample family; The corresponding sample family is determined based on the environment matching score and the route matching score.

[0025] Furthermore, to achieve the above objectives, the present invention also proposes an environment-based data denoising device, which includes: a memory, a processor, and an environment-based data denoising program stored in the memory and executable on the processor. The environment-based data denoising program is configured with steps for implementing the environment-based data denoising method described above.

[0026] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing an environment-based data denoising program, which, when executed by a processor, implements the steps of the environment-based data denoising method described above. This invention acquires host environment dimension samples from a suspected test machine sample set; determines corresponding host environment features and host routing features based on the host environment dimension samples; matches the host environment features and host routing features with sample organization rules to obtain corresponding sample families; obtains the association relationship between the sample families and advanced threat actors; and performs noise reduction processing on the suspected test machine sample set based on the association relationship. In this invention, by using the association relationship with advanced threat actors, samples with a low probability of being test machine samples are removed from the suspected test machine samples, thereby reducing the consumption of computing resources, improving the sample analysis speed, and reducing the sample analysis difficulty. This solves the technical problem that existing test machine sample data contains a large number of noisy samples, which wastes a lot of analysis computing resources and greatly increases the analysis difficulty. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of a data noise reduction device based on the environment dimension in the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the environmental-based data noise reduction method of the present invention. Figure 3 This is a flowchart illustrating the second embodiment of the environmental dimension-based data noise reduction method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the environmental dimension-based data noise reduction method of the present invention; Figure 5 This is a structural block diagram of the first embodiment of the data noise reduction device based on the environmental dimension of the present invention.

[0028] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a data noise reduction device based on the environment dimension in the hardware operating environment involved in the embodiments of the present invention.

[0031] like Figure 1 As shown, the environmental-based data noise reduction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0032] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on environmental dimension-based data noise reduction devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0033] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data noise reduction program based on the environment dimension.

[0034] exist Figure 1 In the environmental dimension-based data noise reduction device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the environmental dimension-based data noise reduction device of the present invention can be set in the environmental dimension-based data noise reduction device, and the environmental dimension-based data noise reduction device calls the environmental dimension-based data noise reduction program stored in the memory 1005 through the processor 1001 and executes the environmental dimension-based data noise reduction method provided in the embodiment of the present invention.

[0035] This invention provides a data noise reduction method based on the environmental dimension, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a data noise reduction method based on environmental dimensions according to the present invention.

[0036] In this embodiment, the data noise reduction method based on the environmental dimension includes the following steps: Step S10: Obtain host environment dimension samples from the suspected test machine sample set.

[0037] It should be noted that the execution subject of this embodiment is the environmental dimension-based data noise reduction device. The environmental dimension-based data noise reduction device can be a personal computer, server, cloud server, or other electronic device, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the environmental dimension-based data noise reduction device is used as an example to illustrate the environmental dimension-based data noise reduction method of the present invention.

[0038] In simple terms, an end user is someone who installs and uses a secure endpoint product on their terminal, while a user who installs and uses a secure endpoint product for testing purposes is a test machine user, or simply a test machine. A suspected test machine sample is a data sample collected from the terminals of end users suspected of engaging in test-and-defense behavior. A suspected test machine sample set is a collection of samples constructed by combining multiple suspected test machine samples. Host environment dimension samples are obtained from the suspected test machine sample set. These host environment dimension samples contain feature information about the host environment, which may include the terminal host environment (PCINFO) and the terminal host's routing information, etc.

[0039] Step S20: Determine the corresponding host environment features and host routing features based on the host environment dimension samples.

[0040] It should be understood that the corresponding host environment features and host routing features are determined based on the host environment dimension samples. The host environment features may be the corresponding host motherboard features, host version features, and host operation and maintenance time features, etc., and the host routing features may be features based on the routing used by the test machine.

[0041] Specifically, host motherboard characteristics can include CPU-based characteristics, memory-based characteristics, and hard drive-based characteristics. CPU-based characteristics can be limited by the number of CPU cores, memory-based characteristics by memory size, hard drive-based characteristics by hard drive size, and host version characteristics by the list of motherboard manufacturers. Host environment characteristics can also include product type characteristics, which can be limited by product type. For example, due to the target audience of the product, test machines often install personal or international terminal versions, rather than server versions. Host maintenance time characteristics can be limited by installation time, i.e., by the range of usage time. For example, test machines typically have a short installation and usage period; once the testing objective is achieved, the system is restored and traces are cleared.

[0042] Step S30: Match the host environment features and the host routing features with the sample organization rules respectively to obtain the corresponding sample families.

[0043] It should be noted that before matching the host environment features and the host routing features with the sample organization rules, it is necessary to construct the sample organization rules: obtain the host environment features and routing features of each test machine sample in the test machine sample set; construct the sample organization rules based on the host environment features and routing features of the test machine.

[0044] Specifically, the process of obtaining the test machine host environment characteristics and test machine routing characteristics corresponding to each test machine sample in the test machine sample set can be as follows: obtaining the test machine motherboard characteristics, test machine version characteristics, and test machine operation and maintenance time characteristics corresponding to each test machine sample in the test machine sample set; generating test machine host environment characteristics based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine operation and maintenance time characteristics; and obtaining the test machine routing characteristics corresponding to each test machine sample in the test machine sample set.

[0045] The process of obtaining the routing characteristics of each test machine sample in the test machine sample set can be as follows: obtaining the terminal-associated routing address corresponding to each test machine sample in the test machine sample set; identifying whether the terminal-associated routing address is a local area network address corresponding to a preset routing vendor; and when the terminal-associated routing address is the local area network address, using the terminal-associated routing address as the test machine routing characteristic. Specifically, the MAC address associated with the terminal corresponding to each test machine sample is the terminal-associated routing address, and it is identified whether the vendor corresponding to the above-mentioned routing MAC address is the routing manufacturer, rather than a forgery or other vendor. The MAC address corresponding to the device produced by the routing manufacturer is theoretically unique, and due to geographical reasons, associated users generally have a strong correlation. When the terminal-associated routing address is the MAC address corresponding to the device produced by the routing manufacturer, the MAC address corresponding to the device produced by the routing manufacturer is used as the test machine routing characteristic.

[0046] The process of matching host environment features and host routing features with sample organization rules to obtain corresponding sample families can be understood as follows: Match host environment features with sample organization rules to obtain environment matching scores between host environment features and each sample family; match host routing features with sample organization rules to obtain routing matching scores between host routing features and each sample family; determine the corresponding sample family based on the environment matching scores and routing matching scores. Specifically, sort the environment matching scores and routing matching scores in descending order to obtain a ranking result; the sample family corresponding to the highest-ranked matching score in the ranking result is taken as the sample family corresponding to the host environment features and host routing features.

[0047] Specifically, the process of obtaining the route matching score can be as follows: Obtain the host route address based on host route characteristics; trace and associate the host route address to obtain the associated user address; obtain the test machine route characteristics according to sample organization rules, and determine the terminal associated route address based on the test machine route characteristics; perform profile recognition on the associated user address and the terminal associated route address to obtain the profile recognition result; determine the route matching score between the host route characteristics and each sample family based on the profile recognition result. Specifically, it identifies whether the manufacturer corresponding to the above-mentioned route MAC address is the router manufacturer, and not a forgery or other manufacturer. Theoretically, the MAC address corresponding to the device produced by the router manufacturer is unique. Due to geographical reasons, associated users generally have a strong correlation. When the terminal associated route address is the MAC address corresponding to the device produced by the router manufacturer, trace and associate the MAC address corresponding to the router manufacturer, and perform profile recognition on other users traced to the MAC address. If there are users who are normal users and not advanced threat actors, it indicates low correlation and the sample needs to be filtered out; if there are users who are not normal users but advanced threat actors, it indicates high correlation and the sample can be retained.

[0048] Step S40: Obtain the association between the sample family and advanced threat actors, and perform noise reduction processing on the suspected test machine sample set according to the association.

[0049] It's important to clarify that advanced threats, also known as advanced persistent threats or advanced persistent threats, refer to covert and persistent computer intrusion processes, typically meticulously planned by certain individuals targeting specific objectives. They are usually driven by commercial or political motives, targeting specific organizations or countries, and require maintaining a high degree of stealth over a prolonged period. Advanced persistent threats comprise three elements: advanced, persistent, and threat. "Advanced" emphasizes the use of sophisticated malware and techniques to exploit vulnerabilities in systems. "Persistent" implies that an external force continuously monitors a specific target and extracts data from it. "Threat" refers to attacks planned and executed by humans. APT attacks, or advanced persistent threat attacks, also known as targeted threat attacks, refer to sustained and effective attack activities launched by an organization against a specific target. Individuals exhibiting APT attack behavior are called APT actors, or advanced threat actors.

[0050] It is understandable that advanced threat actors continuously expose their own characteristic information. Therefore, sample families can be associated with the characteristic information of advanced threat actors to obtain the corresponding correlation.

[0051] In practical use, the correlation can be determined through multi-dimensional information, such as using vendor-shared advanced threat actor reports, network asset information, website IP resolution results, advanced threat actor analysis reports, and other multi-dimensional information to determine the correlation between sample families and advanced threat actors. The correlation between sample families and advanced threat actors can be analyzed in real time, or the correlation between sample families and advanced threat actors can be analyzed in advance to generate a corresponding correlation table, which can be directly used for table lookup when needed. This embodiment does not impose any restrictions on this.

[0052] It should be noted that the association relationship represents the correlation between the sample family and advanced threat actors. The association relationship indicates the probability that a suspected test machine sample is a test machine. Therefore, the association relationship can be used to denoise the suspected test machine sample set and exclude samples with low correlation, which are suspected test machine samples with low probability of being test machines.

[0053] This embodiment acquires host environment dimension samples from a suspected test machine sample set; determines corresponding host environment features and host routing features based on the host environment dimension samples; matches the host environment features and host routing features with sample organization rules to obtain corresponding sample families; obtains the association relationship between the sample families and advanced threat actors; and performs noise reduction processing on the suspected test machine sample set based on the association relationship. In this embodiment, by using the association relationship with advanced threat actors, samples with a low probability of being test machine samples are removed from the suspected test machine samples, thereby reducing the consumption of computing resources, improving the sample analysis speed, and reducing the sample analysis difficulty. This solves the technical problem that existing test machine sample data contains a large number of noisy samples, which wastes a lot of analysis computing resources and greatly increases the analysis difficulty.

[0054] refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a data denoising method based on environmental dimensions according to the present invention. Based on the first embodiment described above, the data denoising method based on environmental dimensions in this embodiment specifically includes the following in step S30: Step S301: Match the host environment features with the sample organization rules to obtain the environment matching scores of the host environment features and each sample family.

[0055] It should be noted that the sample family is obtained by performing cluster analysis on the host environment characteristics of samples identified as test machines, classifying them into groups with similar, identical, or related host environment characteristics; the sample organization rules are constructed by combining the organization rules of each sample family; by matching the host environment characteristics of suspected test machine samples with the sample organization rules, sample families similar to or close to the suspected test machine samples can be obtained.

[0056] Specifically, in order to more intuitively represent the matching relationship between host environment features and various sample families, and to facilitate the determination of the sample family corresponding to a suspected test machine sample, this embodiment can match host environment features with sample organization rules to obtain the environment matching score between host environment features and various sample families; and determine the sample family corresponding to the host environment features based on the environment matching score.

[0057] It is easy to understand that a suspected test machine sample may have similarities or the same situations with multiple organization rules in the sample organization rules. In order to more intuitively represent the matching degree and facilitate the acquisition of the most similar sample family, a matching score can be calculated to represent the matching relationship. The matching degree is quantified by the matching score. Then, the sample family corresponding to the host environment characteristics of the suspected test machine sample can be quickly and conveniently determined by the matching score. The specific matching score can be set according to the actual situation. This embodiment does not limit this.

[0058] Step S302: Match the host routing features with the sample organization rules to obtain the routing matching scores of the host routing features and each sample family.

[0059] It is easy to understand that this embodiment can match host routing features with sample organization rules to obtain the routing matching score between the host routing features and each sample family; and determine the sample family corresponding to the host routing features based on the routing matching score.

[0060] Specifically, the process of obtaining the route matching score can be as follows: Obtain the host route address based on host route characteristics; trace and associate the host route address to obtain the associated user address; obtain the test machine route characteristics according to sample organization rules, and determine the terminal associated route address based on the test machine route characteristics; perform profile recognition on the associated user address and the terminal associated route address to obtain the profile recognition result; determine the route matching score between the host route characteristics and each sample family based on the profile recognition result. Specifically, it identifies whether the manufacturer corresponding to the above-mentioned route MAC address is the router manufacturer, and not a forgery or other manufacturer. Theoretically, the MAC address corresponding to the device produced by the router manufacturer is unique. Due to geographical reasons, associated users generally have a strong correlation. When the terminal associated route address is the MAC address corresponding to the device produced by the router manufacturer, trace and associate the MAC address corresponding to the router manufacturer, and perform profile recognition on other users traced to the MAC address. If there are users who are normal users and not advanced threat actors, it indicates low correlation and the sample needs to be filtered out; if there are users who are not normal users but advanced threat actors, it indicates high correlation and the sample can be retained.

[0061] Step S303: Determine the corresponding sample family based on the environment matching score and the route matching score.

[0062] It should be noted that, in order to determine the sample family corresponding to the suspected test machine sample, this embodiment can determine the corresponding sample family based on the environment matching score and the route matching score: sort the environment matching score and the route matching score in descending order to obtain the sorting result; and take the sample family corresponding to the first-ranked matching score in the sorting result as the corresponding sample family.

[0063] It should be noted that the matching score is used to represent the degree of matching between the host environment characteristics and host routing characteristics of the suspected test machine sample and the organization rules of each sample family. Therefore, the higher the matching score, the higher the matching degree. Thus, by sorting the matching scores from largest to smallest, the sample family corresponding to the first-ranked matching score can be obtained as the sample family with the highest matching degree. The selected sample family can then be used as the sample family corresponding to the host environment characteristics and host routing characteristics of the suspected test machine sample.

[0064] This embodiment obtains the environment matching score between the host environment features and each sample family by matching the host environment features with the sample organization rules; it also obtains the routing matching score between the host routing features and each sample family by matching the host routing features with the sample organization rules; and finally determines the corresponding sample family based on the environment matching score and the routing matching score. In this embodiment, by establishing associations with advanced threat actors, samples with a low probability of being test machine samples are removed from suspected test machine samples, thereby reducing computational resource consumption, improving sample analysis speed, and reducing sample analysis difficulty. This solves the technical problem that existing test machine sample data contains a large number of noisy samples, which wastes a lot of analytical computing resources and greatly increases the difficulty of analysis.

[0065] refer to Figure 4 , Figure 4 This is a flowchart illustrating a third embodiment of a data denoising method based on environmental dimensions according to the present invention. Based on the first embodiment described above, the data denoising method based on environmental dimensions in this embodiment specifically includes the following in step S40: Step S401: Obtain the association between the sample family and advanced threat actors, and mark each suspected test machine sample in the suspected test machine sample set according to the association.

[0066] It should be noted that the association relationship represents the correlation between the sample family and advanced threat actors. The association relationship indicates the probability that a suspected test machine sample is a test machine. Therefore, the association relationship can be used to denoise the suspected test machine sample set and exclude samples with low correlation, which are suspected test machine samples with low probability of being test machines.

[0067] It's easy to understand that processing sets generally involves traversal. However, directly modifying the set data during traversal can lead to index anomalies, resulting in missed or accidental deletions. Furthermore, excessive operations during traversal significantly reduce processing efficiency. Therefore, we can first mark each suspected test machine sample in the suspected test machine sample set based on association relationships, marking the samples that need to be removed as noise samples. Then, after traversal is complete, all suspected test machine samples marked as noise samples in the suspected test machine sample set can be removed uniformly, which can improve processing speed.

[0068] Furthermore, in order to facilitate the labeling of each suspected test machine sample in the suspected test machine sample set, the process of labeling each suspected test machine sample in the suspected test machine sample set according to the association relationship in this embodiment can be as follows: obtain the association score of each suspected test machine sample in the suspected test machine sample set according to the association relationship; and label the suspected test machine samples with an association score less than or equal to a preset score threshold as noise samples.

[0069] It should be noted that correlation is difficult to quantify, and it is very difficult to label each suspected test machine sample in the suspected test machine sample set based on correlation. Therefore, correlation scores can be used to quantify the correlation, and the correlation score represents the specific correlation. A preset score threshold is set according to the actual situation. If the correlation score is less than or equal to the preset score threshold, it indicates that the correlation is low. Suspected test machine samples with correlation scores less than or equal to the preset score threshold can be marked as noise samples and removed during noise reduction processing.

[0070] Step S402: Remove the suspected test machine samples marked as noise samples from the suspected test machine sample set.

[0071] It should be understood that removing suspected test machine samples marked as noise samples from the suspected test machine sample set achieves noise reduction of the suspected test machine sample set, thereby reducing the consumption of computing resources, improving the sample analysis speed, and reducing the sample analysis difficulty.

[0072] This embodiment obtains the association between the sample family and advanced threat actors, and marks each suspected test machine sample in the suspected test machine sample set according to the association; then, suspected test machine samples marked as noise samples are removed from the suspected test machine sample set. In this embodiment, by using the association with advanced threat actors, samples with a low probability of being test machine samples are removed from the suspected test machine samples, thereby reducing the consumption of computing resources, improving the sample analysis speed, and reducing the sample analysis difficulty. This solves the technical problem that existing test machine sample data contains a large number of noise samples, which wastes a lot of analysis computing resources and greatly increases the analysis difficulty.

[0073] Furthermore, embodiments of the present invention also propose a storage medium storing an environment-based data denoising program, wherein the environment-based data denoising program is executed by a processor using the steps of the environment-based data denoising method described above.

[0074] Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0075] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the data noise reduction device based on the environmental dimension of the present invention.

[0076] like Figure 5 As shown, the environmental-dimensional data noise reduction device described in this embodiment of the invention includes: Module 10 is used to acquire host environment dimension samples from the suspected test machine sample set.

[0077] It should be noted that "end user" refers to a user who installs and uses a secure endpoint product on a terminal, while "test machine user" refers to a user who installs and uses a secure endpoint product on a terminal for the purpose of testing and adversarial activities. Suspected test machine samples are data samples collected from the terminals of end users suspected of engaging in test-and-adversarial behavior. The suspected test machine sample set is a collection of samples constructed by combining multiple suspected test machine samples. Host environment dimension samples are obtained from the suspected test machine sample set. These host environment dimension samples contain feature information about the host environment, which may include the terminal host environment (PCINFO) and the terminal host's routing information, etc.

[0078] The determination module 20 is used to determine the corresponding host environment features and host routing features based on the host environment dimension samples.

[0079] It should be understood that the corresponding host environment features and host routing features are determined based on the host environment dimension samples. The host environment features may be the corresponding host motherboard features, host version features, and host operation and maintenance time features, etc., and the host routing features may be features based on the routing used by the test machine.

[0080] Specifically, host motherboard characteristics can include CPU-based characteristics, memory-based characteristics, and hard drive-based characteristics. CPU-based characteristics can be limited by the number of CPU cores, memory-based characteristics by memory size, hard drive-based characteristics by hard drive size, and host version characteristics by the list of motherboard manufacturers. Host environment characteristics can also include product type characteristics, which can be limited by product type. For example, due to the target audience of the product, test machines often install personal or international terminal versions, rather than server versions. Host maintenance time characteristics can be limited by installation time, i.e., by the range of usage time. For example, test machines typically have a short installation and usage period; once the testing objective is achieved, the system is restored and traces are cleared.

[0081] The matching module 30 is used to match the host environment features and the host routing features with the sample organization rules respectively to obtain the corresponding sample families.

[0082] It should be noted that before matching the host environment features and the host routing features with the sample organization rules, it is necessary to construct the sample organization rules: obtain the host environment features and routing features of each test machine sample in the test machine sample set; construct the sample organization rules based on the host environment features and routing features of the test machine.

[0083] Specifically, the process of obtaining the test machine host environment characteristics and test machine routing characteristics corresponding to each test machine sample in the test machine sample set can be as follows: obtaining the test machine motherboard characteristics, test machine version characteristics, and test machine operation and maintenance time characteristics corresponding to each test machine sample in the test machine sample set; generating test machine host environment characteristics based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine operation and maintenance time characteristics; and obtaining the test machine routing characteristics corresponding to each test machine sample in the test machine sample set.

[0084] The process of obtaining the routing characteristics of each test machine sample in the test machine sample set can be as follows: obtaining the terminal-associated routing address corresponding to each test machine sample in the test machine sample set; identifying whether the terminal-associated routing address is a local area network address corresponding to a preset routing vendor; and when the terminal-associated routing address is the local area network address, using the terminal-associated routing address as the test machine routing characteristic. Specifically, the MAC address associated with the terminal corresponding to each test machine sample is the terminal-associated routing address, and it is identified whether the vendor corresponding to the above-mentioned routing MAC address is the routing manufacturer, rather than a forgery or other vendor. The MAC address corresponding to the device produced by the routing manufacturer is theoretically unique, and due to geographical reasons, associated users generally have a strong correlation. When the terminal-associated routing address is the MAC address corresponding to the device produced by the routing manufacturer, the MAC address corresponding to the device produced by the routing manufacturer is used as the test machine routing characteristic.

[0085] The process of matching host environment features and host routing features with sample organization rules to obtain corresponding sample families can be understood as follows: Match host environment features with sample organization rules to obtain environment matching scores between host environment features and each sample family; match host routing features with sample organization rules to obtain routing matching scores between host routing features and each sample family; determine the corresponding sample family based on the environment matching scores and routing matching scores. Specifically, sort the environment matching scores and routing matching scores in descending order to obtain a ranking result; the sample family corresponding to the highest-ranked matching score in the ranking result is taken as the sample family corresponding to the host environment features and host routing features.

[0086] Specifically, the process of obtaining the route matching score can be as follows: Obtain the host route address based on host route characteristics; trace and associate the host route address to obtain the associated user address; obtain the test machine route characteristics according to sample organization rules, and determine the terminal associated route address based on the test machine route characteristics; perform profile recognition on the associated user address and the terminal associated route address to obtain the profile recognition result; determine the route matching score between the host route characteristics and each sample family based on the profile recognition result. Specifically, it identifies whether the manufacturer corresponding to the above-mentioned route MAC address is the router manufacturer, and not a forgery or other manufacturer. Theoretically, the MAC address corresponding to the device produced by the router manufacturer is unique. Due to geographical reasons, associated users generally have a strong correlation. When the terminal associated route address is the MAC address corresponding to the device produced by the router manufacturer, trace and associate the MAC address corresponding to the router manufacturer, and perform profile recognition on other users traced to the MAC address. If there are users who are normal users and not advanced threat actors, it indicates low correlation and the sample needs to be filtered out; if there are users who are not normal users but advanced threat actors, it indicates high correlation and the sample can be retained.

[0087] The processing module 40 is used to obtain the association between the sample family and advanced threat actors, and to perform noise reduction processing on the suspected test machine sample set according to the association.

[0088] It's important to clarify that advanced threats, also known as advanced persistent threats or advanced persistent threats, refer to covert and persistent computer intrusion processes, typically meticulously planned by certain individuals targeting specific objectives. They are usually driven by commercial or political motives, targeting specific organizations or countries, and require maintaining a high degree of stealth over a prolonged period. Advanced persistent threats comprise three elements: advanced, persistent, and threat. "Advanced" emphasizes the use of sophisticated malware and techniques to exploit vulnerabilities in systems. "Persistent" implies that an external force continuously monitors a specific target and extracts data from it. "Threat" refers to attacks planned and executed by humans. APT attacks, or advanced persistent threat attacks, also known as targeted threat attacks, refer to sustained and effective attack activities launched by an organization against a specific target. Individuals exhibiting APT attack behavior are called APT actors, or advanced threat actors.

[0089] It is understandable that advanced threat actors continuously expose their own characteristic information. Therefore, sample families can be associated with the characteristic information of advanced threat actors to obtain the corresponding correlation.

[0090] In practical use, the correlation can be determined through multi-dimensional information, such as using vendor-shared advanced threat actor reports, network asset information, website IP resolution results, advanced threat actor analysis reports, and other multi-dimensional information to determine the correlation between sample families and advanced threat actors. The correlation between sample families and advanced threat actors can be analyzed in real time, or the correlation between sample families and advanced threat actors can be analyzed in advance to generate a corresponding correlation table, which can be directly used for table lookup when needed. This embodiment does not impose any restrictions on this.

[0091] It should be noted that the association relationship represents the correlation between the sample family and advanced threat actors. The association relationship indicates the probability that a suspected test machine sample is a test machine. Therefore, the association relationship can be used to denoise the suspected test machine sample set and exclude samples with low correlation, which are suspected test machine samples with low probability of being test machines.

[0092] The environmental dimension-based data denoising device described in this embodiment includes: an acquisition module 10, used to acquire host environment dimension samples in a suspected test machine sample set; a determination module 20, used to determine the corresponding host environment features and host routing features based on the host environment dimension samples; a matching module 30, used to match the host environment features and the host routing features with sample organization rules respectively to obtain corresponding sample families; and a processing module 40, used to acquire the association relationship between the sample families and advanced threat actors, and to perform denoising processing on the suspected test machine sample set based on the association relationship. In this embodiment, by using the association relationship with advanced threat actors, samples with a low probability of being test machine samples in the suspected test machine samples are removed, thereby reducing the consumption of computing resources, improving the sample analysis speed, and reducing the sample analysis difficulty. This solves the technical problem that existing test machine sample data contains a large number of noisy samples, which wastes a lot of analysis computing resources and greatly increases the analysis difficulty.

[0093] In one embodiment, the environmental-dimensional data noise reduction device further includes: a construction module; The construction module is used to obtain the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set. Sample organization rules are constructed based on the host environment characteristics and routing characteristics of the test machine.

[0094] In one embodiment, the acquisition module 10 is further configured to acquire the motherboard features, version features, and maintenance time features of each test machine sample in the test machine sample set. The test machine host environment characteristics are generated based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine maintenance time characteristics. Obtain the test machine routing features corresponding to each test machine sample in the test machine sample set.

[0095] In one embodiment, the acquisition module 10 is further configured to acquire the terminal associated routing address corresponding to each test machine sample in the test machine sample set; Identify whether the terminal's associated routing address is a local area network address corresponding to a preset routing vendor; When the terminal's associated routing address is the local area network address, the terminal's associated routing address is used as the routing feature of the test machine.

[0096] In one embodiment, the matching module 30 is further configured to match the host environment features with the sample organization rules to obtain the environment matching scores of the host environment features and each sample family; The host routing features are matched with the sample organization rules to obtain the routing matching scores between the host routing features and each sample family; The corresponding sample family is determined based on the environment matching score and the route matching score.

[0097] In one embodiment, the matching module 30 is further configured to obtain a host routing address based on the host routing characteristics; The host routing address is traced and associated to obtain the associated user address; The test machine routing characteristics are obtained according to the sample organization rules, and the terminal associated routing address is determined according to the test machine routing characteristics; The associated user address and the associated routing address of the terminal are used to perform profile recognition to obtain profile recognition results; The profile recognition results are used to determine the routing matching scores between the host routing features and each sample family.

[0098] In one embodiment, the matching module 30 is further configured to trigger the user node port identifier based on the host routing address, and obtain the user's actual address based on the user node port identifier; Establish the association between the user's actual address and the host's routing address to obtain the address tracing of the user node; The address of the user node is traced back to the associated user address.

[0099] In one embodiment, the matching module 30 is further configured to sort the environment matching score and the route matching score in descending order to obtain a sorting result; The sample family corresponding to the first-ranked matching score in the ranking results is taken as the sample family corresponding to the host environment feature and the host routing feature.

[0100] In one embodiment, the processing module 40 is further configured to mark each suspected test machine sample in the suspected test machine sample set according to the association relationship; Remove the suspected test machine samples marked as noise samples from the suspected test machine sample set.

[0101] In one embodiment, the processing module 40 is further configured to obtain the association score of each suspected test machine sample in the suspected test machine sample set according to the association relationship; Suspected test machine samples whose correlation scores are less than or equal to a preset score threshold are marked as noise samples.

[0102] In one embodiment, the acquisition module 10 is further configured to acquire sample logs corresponding to each suspected test machine sample in the initial test machine sample set; The initial test machine sample set is denoised based on the sample logs to obtain a suspected test machine sample set.

[0103] In one embodiment, the acquisition module 10 is further configured to mark each suspected test machine sample in the initial test machine sample set according to the sample log; The suspected test machine samples marked as noise samples in the initial test machine sample set are removed.

[0104] In one embodiment, the acquisition module 10 is further configured to acquire the log point level corresponding to each suspected test machine sample in the initial test machine sample set based on the sample log; Suspected test machine samples whose log point volume exceeds a preset threshold are marked as noise samples.

[0105] Other embodiments or specific implementations of the environmental dimension-based data noise reduction device described in this invention can be referred to the above-described embodiments of the environmental dimension-based data noise reduction methods, and will not be repeated here.

[0106] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0107] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0108] In addition, for technical details not described in detail in this embodiment, please refer to the data noise reduction method based on the environment dimension provided in any embodiment of the present invention, which will not be repeated here.

[0109] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0110] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data noise reduction method based on environmental dimensions, characterized in that, The environmental dimension-based data noise reduction method includes: Obtain host environment dimension samples from the suspected test machine sample set; Determine the corresponding host environment features and host routing features based on the host environment dimension samples; The host environment features and the host routing features are matched with the sample organization rules to obtain the corresponding sample families; Obtain the association between the sample family and advanced threat actors, and perform noise reduction processing on the suspected test machine sample set based on the association; The step of matching the host environment features and the host routing features with the sample organization rules respectively to obtain the corresponding sample families includes: The host environment features are matched with the sample organization rules to obtain the environment matching scores between the host environment features and each sample family; Obtain the host routing address based on the host routing characteristics; The host routing address is traced and associated to obtain the associated user address; The test machine routing characteristics are obtained according to the sample organization rules, and the terminal associated routing address is determined according to the test machine routing characteristics; The associated user address and the associated routing address of the terminal are used to perform profile recognition to obtain profile recognition results; The profile recognition results are used to determine the routing matching score between the host routing features and each sample family; The corresponding sample family is determined based on the environment matching score and the route matching score.

2. The data noise reduction method based on the environmental dimension as described in claim 1, characterized in that, Before the step of matching the host environment features and the host routing features with the sample organization rules to obtain the corresponding sample families, the method further includes: Obtain the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set; Sample organization rules are constructed based on the host environment characteristics and routing characteristics of the test machine.

3. The data noise reduction method based on the environmental dimension as described in claim 2, characterized in that, The steps of obtaining the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set include: Obtain the motherboard characteristics, version characteristics, and maintenance time characteristics of each test machine sample in the test machine sample set; The test machine host environment characteristics are generated based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine maintenance time characteristics. Obtain the test machine routing features corresponding to each test machine sample in the test machine sample set.

4. The data noise reduction method based on the environmental dimension as described in claim 3, characterized in that, The step of obtaining the test machine routing features corresponding to each test machine sample in the test machine sample set includes: Obtain the terminal-associated routing address corresponding to each test machine sample in the test machine sample set; Identify whether the terminal's associated routing address is a local area network address corresponding to a preset routing vendor; When the terminal's associated routing address is the local area network address, the terminal's associated routing address is used as the routing feature of the test machine.

5. The data noise reduction method based on the environmental dimension as described in claim 1, characterized in that, The step of tracing and associating the host routing address to obtain the associated user address includes: The user node port identifier is triggered based on the host routing address, and the user's actual address is obtained based on the user node port identifier; Establish the association between the user's actual address and the host's routing address to obtain the address tracing of the user node; The address of the user node is traced back to the associated user address.

6. The data noise reduction method based on the environmental dimension as described in claim 1, characterized in that, The step of determining the corresponding sample family based on the environment matching score and the route matching score includes: Sort the environment matching score and the route matching score in descending order to obtain the sorting result; The sample family corresponding to the first-ranked matching score in the ranking results is taken as the sample family corresponding to the host environment feature and the host routing feature.

7. The data denoising method based on the environmental dimension as described in any one of claims 1 to 6, characterized in that, The step of performing noise reduction processing on the suspected test machine sample set based on the correlation includes: Each suspected test machine sample in the suspected test machine sample set is marked according to the aforementioned association; Remove the suspected test machine samples marked as noise samples from the suspected test machine sample set.

8. The data noise reduction method based on the environmental dimension as described in claim 7, characterized in that, The step of marking each suspected test machine sample in the suspected test machine sample set according to the association relationship includes: Based on the aforementioned correlation, obtain the correlation score of each suspected test machine sample in the suspected test machine sample set; Suspected test machine samples whose correlation scores are less than or equal to a preset score threshold are marked as noise samples.

9. The data denoising method based on the environmental dimension as described in any one of claims 1 to 8, characterized in that, Before the step of obtaining host environment dimension samples from the suspected test machine sample set, the method further includes: Obtain the sample logs corresponding to each suspected test machine sample in the initial test machine sample set; The initial test machine sample set is denoised based on the sample logs to obtain a suspected test machine sample set.

10. The data denoising method based on the environmental dimension as described in claim 9, characterized in that, The step of performing noise reduction processing on the initial test machine sample set based on the sample log includes: Each suspected test machine sample in the initial test machine sample set is marked according to the sample log; The suspected test machine samples marked as noise samples in the initial test machine sample set are removed.

11. The data noise reduction method based on the environmental dimension as described in claim 10, characterized in that, The step of marking each suspected test machine sample in the initial test machine sample set according to the sample log includes: Based on the sample logs, obtain the log point level corresponding to each suspected test machine sample in the initial test machine sample set; Suspected test machine samples whose log point volume exceeds a preset threshold are marked as noise samples.

12. A data noise reduction device based on environmental dimensions, characterized in that, The environmental-based data noise reduction device includes: The acquisition module is used to acquire host environment dimension samples from the suspected test machine sample set; The determination module is used to determine the corresponding host environment characteristics and host routing characteristics based on the host environment dimension samples. The matching module is used to match the host environment features and the host routing features with the sample organization rules respectively to obtain the corresponding sample families; The processing module is used to obtain the association between the sample family and advanced threat actors, and to perform noise reduction processing on the suspected test machine sample set according to the association; The matching module is also used to match the host environment features with the sample organization rules to obtain the environment matching scores of the host environment features and each sample family. Obtain the host routing address based on the host routing characteristics; The host routing address is traced and associated to obtain the associated user address; The test machine routing characteristics are obtained according to the sample organization rules, and the terminal associated routing address is determined according to the test machine routing characteristics; The associated user address and the associated routing address of the terminal are used to perform profile recognition to obtain profile recognition results; The profile recognition results are used to determine the routing matching score between the host routing features and each sample family; The corresponding sample family is determined based on the environment matching score and the route matching score.

13. The data noise reduction device based on the environmental dimension as described in claim 12, characterized in that, The environmental-based data noise reduction device further includes: a construction module; The construction module is used to obtain the host environment characteristics and routing characteristics of each test machine sample in the test machine sample set. Sample organization rules are constructed based on the host environment characteristics and routing characteristics of the test machine.

14. The data noise reduction device based on the environmental dimension as described in claim 13, characterized in that, The acquisition module is also used to acquire the motherboard features, version features, and maintenance time features of each test machine sample in the test machine sample set. The test machine host environment characteristics are generated based on the test machine motherboard characteristics, the test machine version characteristics, and the test machine maintenance time characteristics. Obtain the test machine routing features corresponding to each test machine sample in the test machine sample set.

15. The data noise reduction device based on the environmental dimension as described in claim 14, characterized in that, The acquisition module is also used to acquire the terminal associated routing address corresponding to each test machine sample in the test machine sample set; Identify whether the terminal's associated routing address is a local area network address corresponding to a preset routing vendor; When the terminal's associated routing address is the local area network address, the terminal's associated routing address is used as the routing feature of the test machine.

16. A data noise reduction device based on environmental dimensions, characterized in that, The environment-based data denoising device includes: a memory, a processor, and an environment-based data denoising program stored in the memory and executable on the processor, wherein the environment-based data denoising program is configured to implement the environment-based data denoising method as described in any one of claims 1 to 11.

17. A storage medium, characterized in that, The storage medium stores an environment-based data denoising program, which, when executed by a processor, implements the steps of the environment-based data denoising method as described in any one of claims 1 to 11.

Images