A network asset multi-source data fusion processing method, device and medium

Abstract

The present application relates to the field of network assets, and provides a network asset multi-source data fusion processing method, device and medium, the method comprising: collecting multi-source original network asset data and mapping into standardized asset objects; the standardized asset objects use a plurality of association rules to infer the association relationship, and based on the association relationship, the standardized asset objects are subjected to conflict detection and conflict resolution; the standardized asset objects after conflict resolution are used for iterative optimization, and a unified asset knowledge base is generated. The present application can adapt to massive data, scientifically process uncertainty and effectively resolve conflicts.

Description

Technical Field

[0001] This invention relates to the field of network assets, and more specifically, to a method, device, and medium for multi-source data fusion processing of network assets. Background Technology

[0002] With the acceleration of digitalization, network assets are characterized by their massive scale, diverse forms, and dynamic changes. Network assets are no longer limited to traditional IP devices and servers, but also include cloud instances, containers, IoT devices, web applications, API interfaces, domain name certificates, and more. However, the inherent characteristics of these data sources present significant challenges to data integration.

[0003] 1. Multi-source and heterogeneous: The data comes from a wide range of sources, and its data format, structure, collection frequency and protocol are different, ranging from structured database records to unstructured text banners, making it difficult to directly compare and process them; 2. Uncertainty and Conflict: Data from different sources often conflict with each other. The scanner identification service is Nginx, while the CMDB records are from Apache; an IP address resolves to domain name A at one time and to domain name B at another time.

[0004] 3. Convergent Growth and Computing Crisis: As network asset data continues to iterate and merge, the amount of data will inevitably increase. The most direct impact is that a single computation of massive amounts of data requires huge computing resources.

[0005] Most existing data fusion solutions have significant limitations. Common methods include: simple rule matching, such as precise string matching based on IP address or domain name. This method cannot handle complex scenarios such as aliases, load balancing, and CDN, and is powerless against conflicting data; merging based on manual review: inefficient, unable to cope with massive amounts of dynamically changing asset data, and highly dependent on the operator's experience; the above fusion methods do not provide data traceability for the fused data, resulting in poor interpretability of the fusion results. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method, device, and medium for multi-source data fusion processing of network assets, which can adapt to massive amounts of data, scientifically handle uncertainties, and effectively resolve conflicts.

[0007] In a first aspect, the present invention provides a method for multi-source data fusion processing of network assets, comprising: Collect raw network asset data from multiple sources and map it into standardized asset objects; Standardized asset objects use various association rules to infer association relationships. Conflict detection and resolution are performed on standardized asset objects based on relationships; The standardized asset objects after conflict resolution are used for iterative optimization to generate a unified asset knowledge base.

[0008] In a preferred embodiment, the step of collecting raw network asset data from multiple sources and mapping it to standardized asset objects includes: Raw network asset data is collected asynchronously from multiple heterogeneous data sources; The collected raw network asset data is preprocessed to eliminate data noise and standardize the format; The preprocessed raw network asset data is mapped to a standard data model to generate standardized asset objects. The attributes defined in the standard data model include: asset ID, IP address, port number, network protocol, domain name, certificate hash value, service banner information, HTTP response header, HTML body hash value, hardware fingerprint, operating system type, business unit to which it belongs, data source identifier, and update timestamp.

[0009] In a preferred embodiment, the association rules include: Strong association rule: Infer strong associations based on the hash values ​​of attributes in standardized asset objects; Logical association rules: Logical association inference is performed based on the network protocol in the attributes of standardized asset objects; Fuzzy association rules: logical association inferences are made based on the textual and / or semantic similarity of the attributes of standardized asset objects.

[0010] In a preferred embodiment, the conflict detection and resolution of standardized asset objects based on association relationships includes: Conflict detection is performed on the attribute values ​​of standardized asset objects based on relationships; If there are conflicts between the attribute values ​​of standardized asset objects, the conflicts are resolved; the conflict resolution involves assessing the credibility of the conflicting attribute values ​​and selecting the attribute value with higher credibility.

[0011] In a preferred embodiment, assessing the credibility of conflicting attribute values ​​includes: In a preferred embodiment, the conflict detection is further enhanced by considering the inherent credibility of the data source based on the attribute value, the attribute information entropy, and the time decay factor. If the attribute value is duplicated, the duplicate attribute value is deduplicated to resolve the conflict.

[0012] In a preferred embodiment, the step of iteratively optimizing the standardized asset objects after conflict resolution to generate a unified asset knowledge base includes: The standardized asset objects after conflict resolution are merged with the original standardized asset objects. This process involves iterative optimization through conflict detection, conflict resolution, and data fusion to generate the final fused data. A unified asset knowledge base is generated from the final fused data based on different time windows.

[0013] In a preferred embodiment, the unified asset knowledge base uses time windows as identifiers, and templates are used to store the final fused data in the Elasticsearch database. Data for all time windows is retrieved by querying the template name, and finally, the query results are fused in a time series using lazy loading.

[0014] In a second aspect, the present invention provides an electronic device, comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the instructions stored in the memory to perform the method described above.

[0015] Thirdly, the present invention provides a computer-readable storage medium for storing instructions that, when executed, enable the above-described method to be implemented.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: Intelligent integration: This invention adopts dynamic credibility assessment, which elevates the integration process from simple rule execution to evidence reasoning, and can effectively handle uncertainties and conflicts in real-world scenarios.

[0017] Highly explanatory: The final fusion result of this invention is supported by clear credibility, providing a clear understanding of the fusion source of each attribute and facilitating decision-making.

[0018] High computational efficiency and low computational resource consumption: This invention only performs data fusion within a time window. Historical data is stored using template indexes in an Elasticsearch database. When querying data as a unified asset knowledge base, historical data is then lazy-loaded and displayed. This solves the problem of massive data that cannot be computed and merged due to historical data fusion, and also improves data fusion efficiency. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for multi-source data fusion processing of network assets, provided in an embodiment of the present invention.

[0020] Figure 2 This is a flowchart of conflict detection and conflict resolution in an embodiment of the present invention.

[0021] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] In view of the problems existing in the prior art, the present invention mainly solves the following three problems: 1. How to design a standard data model to standardize network asset data from different sources, normalize the various attribute characteristics of assets, and represent the full picture of asset information as much as possible.

[0025] 2. Assets from different sources may have missing attributes. How can we supplement the missing attributes through association? When merging assets through association, there are often conflicts in asset attributes. How can we resolve conflicts based on the attribute values ​​of asset data from each source and select the most reasonable attribute values?

[0026] 3. Faced with massive amounts of data, how can we reduce computing resources when integrating it with historical data, while maintaining traceability of the integrated asset data?

[0027] In view of this, such as Figure 1 As shown, this embodiment of the invention provides a method for multi-source data fusion processing of network assets, including the following steps: S1: Collect raw network asset data from multiple sources and map it into standardized asset objects.

[0028] Raw network asset data is collected asynchronously from multiple heterogeneous data sources, including active scanners, passive traffic analysis systems, public intelligence sources, and internal management systems.

[0029] The collected raw network asset data undergoes preprocessing to eliminate data noise and standardize its format. This preprocessing includes data filtering, data transformation, and data completion. Specifically, data filtering filters out erroneous alarm data to avoid misleading the analysis process. Data transformation standardizes the format of values ​​with the same attribute in network assets, such as changing time types to yyyy-MM-dd HH:mm:ss. Data completion supplements and completes missing data items in the raw network asset data.

[0030] The preprocessed raw network asset data is mapped to a standard data model to generate standardized asset objects. The attributes defined in the standard data model include: asset ID, IP address, port number, network protocol, domain name, certificate hash value, service banner information, HTTP response header, HTML body hash value, hardware fingerprint, operating system type, business unit to which it belongs, data source identifier, update timestamp, etc.

[0031] S2: Use multiple association rules to infer association relationships for standardized asset objects; The association rules include: Strong association rule (high confidence): Strong association inference is made based on the hash value in the attributes of standardized asset objects. If two standardized asset objects have the same certificate hash value or the same HTML body hash value, the association between the two standardized asset objects is inferred to be strong. Strong association has the highest basic confidence, which is usually set to P_init≥0.95.

[0032] Logical Association Rules (Medium Confidence): Logical association inferences are made based on network protocols in the attributes of standardized asset objects. For example: DNS resolution chain association: A domain name (e.g., www.demo.com) points to a specific IP address (e.g., 192.0.2.1) in its authoritative DNS record, which constitutes the most direct and authoritative evidence of an association between the two. Digital certificate association: In a TLS / SSL certificate obtained from a server (e.g., port 443 of IP 192.0.2.1), if its Subject AlternativeName (SAN) field contains a specific domain name (e.g., ... The presence of a domain like .demo.com strongly suggests the server's affiliation with the target domain family. Application-layer response association: In HTTP / HTTPS protocol interactions, if other domains are embedded in the HTTP response headers (such as the Location redirect header) or response body content returned by the server, it reveals dependencies or configuration associations at the application logic level. These associations can be used to infer that the relationship between two standardized asset objects is a logical association. The basic confidence level for logical associations is set based on the specificity of the logical association rules and historical accuracy, typically between 0.7 and 0.9.

[0033] Fuzzy association rules (low confidence, dynamic modulation): Logical association inference is made based on textual and / or semantic similarity in the attributes of standardized asset objects. For example: calculating subdomain relationships using domain name text similarity; calculating semantic similarity of keywords in service banner information. Fuzzy associations have a low base confidence (e.g., 0.5) and require modulation using the calculated similarity score, expressed as: P_init = base confidence. Similarity score.

[0034] S3: Detect and resolve conflicts in standardized asset objects based on relationships.

[0035] Define the standardized asset object set as ,in Indicates the first Standardized asset objects for different data sources (such as Fofa, Quake, Hunter, etc.). Each type of data source's standardized asset object is represented as follows: ,in For standardized asset objects The Each attribute (such as IP address, domain name). The goal of fusion is to standardize asset objects for each type of data source. Generate globally consistent fused data , Data integration Each attribute in the document is a key attribute of a standardized asset object that integrates each type of data source (such as port, service, IP, etc.).

[0036] Core concept definitions: Conflicting data: Attributes of standardized asset objects from different data sources attribute values and Conflicts exist, including logical conflicts based on logical relationships, or textual and / or semantic conflicts based on fuzzy relationships (such as protocol conflicts). For HTTP, (This is for TCP, and there is a semantic conflict between the two).

[0037] Credibility: 1 Standardized asset objects of data source For attributes attribute values The degree of credibility is denoted as It is calculated based on factors such as data source, time decay, and entropy value of attribute values.

[0038] Conflict resolution: Define the conflict resolution strategy for conflicting data as an exclusion strategy, retaining attribute values ​​with high credibility.

[0039] Based on the above definition, step S3 includes the following sub-steps: S31, Detect conflicts in the attributes of standardized asset objects based on relationships: For the same standardized asset object, given a set of attribute values ​​provided by n data sources. The collision detection function is defined as:

[0040] in, Represents attribute value and attribute values They are semantically or logically incompatible.

[0041] Syntax-level conflicts: Simple conflicts based on direct comparison of attribute values:

[0042] Semantic layer conflict: Semantic incompatibility based on domain knowledge:

[0043] Logical layer conflict: Logical contradictions based on business rules:

[0044] in, This is an indicator function used to convert a condition into a Boolean value or output 0 / 1. This is a preset threshold used for semantic similarity calculation.

[0045] S32, if there are conflicts between the attribute values ​​of standardized asset objects, then conflict resolution is performed; the conflict resolution is to assess the credibility of the conflicting attribute values ​​and take the attribute value with higher credibility.

[0046] The assessment of the credibility of conflicting attribute values ​​includes: a weighted summation based on the inherent credibility of the attribute value's data source, attribute information entropy, and time decay factor to obtain the attribute value's credibility. Wherein, the following definitions apply: Inherent credibility of data source ( ): Based on the comparison of real asset data and data from various manufacturers, and considering the quality of the manufacturers' data, the source of the data is given considerable weight.

[0047] Attribute information entropy ( ): Used to describe the uncertainty of various possible events occurring in an information source (e.g., "system is Windows" and "system is Windows 10", the latter carries more information than the former, and the attribute value with more information can be selected by calculating the attribute information entropy).

[0048] Time decay factor ( Asset status data changes over time, and the reliability of older data decreases over time. , The attenuation coefficient is... It is the time difference for asset detection.

[0049] The final credibility calculation is as follows:

[0050] in For weighted parameters, the attribute value with higher credibility is selected based on the exclusion decision result. Specifically, asset data from the same data source inherently possesses the same credibility; therefore, the credibility calculation for asset data from the same data source simplifies to:

[0051] In some alternative solutions, the conflict detection further includes detecting whether attribute values ​​are duplicated. If duplicates are found, the duplicate attribute values ​​are deduplicated to resolve the conflict. In this embodiment of the invention, the deduplication is defined as a compatibility strategy.

[0052] S4: Iteratively optimize the standardized asset objects after conflict resolution to generate a unified asset knowledge base, specifically including: The standardized asset objects after conflict resolution are fused with the original standardized asset objects. Through iterative optimization of conflict detection, conflict resolution and data fusion, the final fused data is generated. Based on the final fused data of different time windows, a unified asset knowledge base that eliminates conflicts and is globally consistent is generated.

[0053] The unified asset knowledge base contains fused data from different time periods. Time windows are used as identifiers and indexes. The final fused data is stored in the Elasticsearch database using templates. Data from all time windows can be retrieved by querying the template name. Finally, the query results are fused in a time series using lazy loading. This design supports efficient real-time querying and data traceability display.

[0054] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the network asset multi-source data fusion processing method provided in the above embodiments of the present invention. In one embodiment, the electronic device may be a server, a terminal device, or other electronic device. Figure 3 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 3 The example used is the connection between the processor and memory via a bus. The bus... Figure 3The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 3 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0055] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can perform a network asset multi-source data fusion processing method as described above.

[0056] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0057] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0058] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the network asset multi-source data fusion processing method disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0059] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0060] By designing and programming the processor, the code corresponding to the network asset multi-source data fusion processing method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0061] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a network asset multi-source data fusion processing method described above.

[0062] In some alternative embodiments, the present invention also provides a method for multi-source data fusion processing of network assets that can also be implemented as a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the method for multi-source data fusion processing of network assets described above according to various exemplary embodiments of the present invention.

[0063] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0067] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-source data fusion processing of network assets, characterized in that, include: Collect raw network asset data from multiple sources and map it into standardized asset objects; Standardized asset objects use various association rules to infer association relationships. Conflict detection and resolution are performed on standardized asset objects based on relationships; The standardized asset objects after conflict resolution are used for iterative optimization to generate a unified asset knowledge base.

2. The method for multi-source data fusion processing of network assets according to claim 1, characterized in that, The process of collecting raw network asset data from multiple sources and mapping it to standardized asset objects includes: Raw network asset data is collected asynchronously from multiple heterogeneous data sources; The collected raw network asset data is preprocessed to eliminate data noise and standardize the format; The preprocessed raw network asset data is mapped to a standard data model to generate standardized asset objects. The attributes defined in the standard data model include: asset ID, IP address, port number, network protocol, domain name, certificate hash value, service banner information, HTTP response header, HTML body hash value, hardware fingerprint, operating system type, business unit to which it belongs, data source identifier, and update timestamp.

3. The method for multi-source data fusion processing of network assets according to claim 2, characterized in that, The association rules include: Strong association rule: Infer strong associations based on the hash values ​​of attributes in standardized asset objects; Logical association rules: Logical association inference is performed based on the network protocol in the attributes of standardized asset objects; Fuzzy association rules: logical association inferences are made based on the textual and / or semantic similarity of the attributes of standardized asset objects.

4. The method for multi-source data fusion processing of network assets according to claim 3, characterized in that, The conflict detection and resolution of standardized asset objects based on relationships includes: Conflict detection is performed on the attribute values ​​of standardized asset objects based on relationships; If there are conflicts between the attribute values ​​of standardized asset objects, the conflicts are resolved; the conflict resolution involves assessing the credibility of the conflicting attribute values ​​and selecting the attribute value with higher credibility.

5. The method for multi-source data fusion processing of network assets according to claim 4, characterized in that, The assessment of the credibility of conflicting attribute values ​​includes: The credibility of an attribute value is obtained by weighted summation based on the inherent credibility of the data source, the attribute information entropy, and the time decay factor.

6. The method for multi-source data fusion processing of network assets according to claim 4, characterized in that, The conflict detection also includes detecting whether attribute values ​​are duplicated. If duplicates are found, the duplicate attribute values ​​are deduplicated to resolve the conflict.

7. The method for multi-source data fusion processing of network assets according to claim 1, characterized in that, The process of iteratively optimizing standardized asset objects after conflict resolution to generate a unified asset knowledge base includes: The standardized asset objects after conflict resolution are merged with the original standardized asset objects. This process involves iterative optimization through conflict detection, conflict resolution, and data fusion to generate the final fused data. A unified asset knowledge base is generated from the final fused data based on different time windows.

8. The method for multi-source data fusion processing of network assets according to claim 7, characterized in that, In the unified asset knowledge base, time windows are used as identifier indexes. Templates are used to store the final fused data in the Elasticsearch database. Data for all time windows can be retrieved by querying the template name. Finally, the query results are fused in a time series using lazy loading.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the instructions stored in the memory to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instructions that, when executed, cause the method as described in any one of claims 1-8 to be implemented.

Images