A data integration method and device for a heterogeneous device and a WEB application system

Abstract

The application discloses a kind of heterogeneous device and WEB application system data integration method and device.Solution is not to network equipment, security equipment, WEB application system equipment, data collection, data acquisition are not adjusted, secondary development.The method is: by unified login system login access equipment and WEB application system;Control access equipment and WEB application system to keep login state;Access equipment and WEB application system data are obtained;The data obtained are stored in database.Affirmative effect: the collection of heterogeneous data is realized centralized and unified collection, and the data can also be obtained in time and efficiently through the control of collection frequency, ensure the timeliness and effectiveness of data, the data integration efficiency is high, and the workload of operation and maintenance personnel is reduced;Through the classification of data, the obtained data are sorted, and subsequent data processing is facilitated;Open the use of integrated data by third-party systems, realize data sharing.

Description

Technical Field

[0001] This invention belongs to the field of information security and relates to a method for acquiring data from accessed network devices, hosts, and application systems, and more particularly to a method and apparatus for data integration of heterogeneous devices and WEB application systems. Background Technology

[0002] With increasing informatization, the number of network devices, network security devices, and web application systems used on the internet has also increased. These network devices and application systems generate a large amount of data. Collecting and acquiring data from various heterogeneous systems to provide the foundational data for analysis and judgment is a necessary means of data mining and utilization. Examples include: obtaining internet access logs from firewalls, intrusion detection logs from IPS, web access violation logs from WAFs, and internet behavior records from internet behavior management systems.

[0003] To collect data from various network devices, network security devices, and web systems, it's typically necessary to log in to each individual device or system before querying and exporting data. For a large number of devices and systems, this requires significant manpower, and related operation and maintenance personnel are repetitive and mechanically performing a large amount of work, resulting in low efficiency throughout the process. Because data generation is real-time, manual processing makes it impossible to obtain timely and effective data, further delaying subsequent data analysis and mining.

[0004] Data integration between heterogeneous devices and systems in a network is typically achieved through the following method: Web API interface authorization integration. For example, if system A needs to integrate data from system B, system B needs to open its API interface to system A and authorize the API interface. This requires development and adaptation adjustments for both the integrating and integrated systems. However, due to technical, time, and capability factors, not all devices and application systems have the conditions for adjustment and secondary development.

[0005] Therefore, the main challenge is how to achieve unified acquisition and collection of data and data integration without adjusting or redeveloping heterogeneous devices or web application systems. Summary of the Invention

[0006] The main objective of this invention is to solve the problem of data collection and acquisition, and data integration, without modification or secondary development, for devices and systems that do not control access via web login, such as network devices, security devices, and web application systems.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A method for data integration between heterogeneous devices and a web application system, comprising the connected devices and the web application system, includes the following steps:

[0009] S1. Log in to the connected devices and web application systems through the unified login system;

[0010] S2. Maintain login status for connected devices and web application systems;

[0011] S3. Obtain data from connected devices and web application systems;

[0012] S4. Store the acquired data into the database.

[0013] Furthermore, it also includes monitoring the acquisition of information by displaying various indicators formed through statistical analysis and visualization via an interactive interface.

[0014] Furthermore, it also includes providing corresponding API interfaces to provide data query services to third parties;

[0015] Audit all connected devices and web application systems, as well as their operations.

[0016] Furthermore, in step S3, the method for obtaining data from the accessed device and the WEB application system is as follows:

[0017] First, simulate the actions of each step when various devices and web application systems perform data acquisition operations;

[0018] Then, for each device and web application system, the simulated actions of each step are arranged into an execution flow according to the execution order and logical relationship, and encapsulated into different data acquisition API interfaces;

[0019] Finally, the data acquisition API interfaces of each device and web application system are called to obtain the data.

[0020] Furthermore, the method for constructing the simulated action is as follows:

[0021] First, data acquisition operations are performed on various devices and web application systems; then, during the interaction, communication data packets are captured and analyzed to obtain the corresponding operation steps and action information of the operation steps; each action information is written into a corresponding execution action script using a scripting language.

[0022] Furthermore, step S3 controls the frequency of data acquisition based on the set data acquisition interval.

[0023] Furthermore, in step S4, data is stored according to different types; and during storage, each type of data is stored in the database one record at a time.

[0024] The present invention also provides a device for data integration of heterogeneous devices and web application systems. The devices and web application systems are connected to the device via a network. The login module is used to log in to the accessed devices and web application systems through a unified login system.

[0025] The timing control module is used to keep the connected devices and web application systems logged in.

[0026] The data acquisition module is used to acquire data from connected devices and web application systems;

[0027] The data storage module is used to store the acquired data into the database;

[0028] The interactive display module is used to display various indicators formed by the statistical analysis of acquired information through a visual interactive interface for monitoring.

[0029] The service provision module is used to provide data query services to third parties by providing corresponding API interfaces;

[0030] The auditing module is used to audit the various connected devices and web application systems, as well as their operations.

[0031] The present invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute a method for data integration of heterogeneous devices and a web application system at runtime.

[0032] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a method for data integration of heterogeneous devices and a web application system.

[0033] Working principle:

[0034] First, the system automatically logs into all connected devices and web application systems, maintaining a connection through repeated logins. Then, simulated actions are orchestrated to simulate data acquisition interactions for each connected device and web application system, mimicking data querying and retrieval operations within the session. This allows for the acquisition of various data from the connected devices and web application systems, achieving data integration with the target system. The acquired data is then categorized and stored in a database. Finally, the data is organized and analyzed, and relevant metrics are displayed through a visual interface and devices.

[0035] Beneficial effects:

[0036] 1. The method in this invention enables centralized and unified collection of data from heterogeneous devices and systems, and by controlling the collection frequency, data can be acquired in a timely and efficient manner, ensuring the timeliness and effectiveness of the data.

[0037] 2. By using the method of the present invention, data integration is achieved without secondary development of the original equipment and system, thus completing the data acquisition and integration.

[0038] 3. The method of this invention achieves high data integration efficiency and reduces the workload of operation and maintenance personnel.

[0039] 4. By classifying and collecting the data, the acquired data was organized to facilitate further processing.

[0040] 5. The method of the present invention can fully realize the automated acquisition of data from connected devices and WEB application systems.

[0041] 6. The method of the present invention enables third-party systems to utilize integrated data by providing API interfaces, thereby realizing data openness. Attached Figure Description

[0042] Figure 1 This is a network application connection diagram illustrating a data integration method for heterogeneous devices and a web application system provided in an embodiment of the present invention.

[0043] Figure 2 A flowchart illustrating data integration between heterogeneous devices and a web application system, provided as an embodiment of the present invention;

[0044] Figure 3 This invention provides an interactive diagram illustrating the use of the Go login engine to control login.

[0045] Figure 4 A flowchart illustrating the process of acquiring data from accessed devices and web application systems, as provided in this embodiment of the invention;

[0046] Figure 5 A flowchart for constructing simulated actions is provided as an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of a data integration device for heterogeneous devices and a web application system provided in an embodiment of the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Example 1

[0051] like Figure 1 The diagram illustrates a network application connection for a data integration method for heterogeneous devices and a web application system provided in an embodiment of the present invention. It includes: an access device and web application system 101, a data integration server 102, a third-party application 103, a display device 104, and a client 105. The access device and web application system 101, data integration server 102, third-party application 103, display device 104, and client 105 are located in the same network (or intranet) and are connected to the data integration server 102 via the network. The data integration server 102 is equipped with a software system executed by a computer that implements a data integration method for heterogeneous devices and a web application system. This system integrates data, analyzes the integrated data, and displays the processed data through the display device. The client 105 can access the data integration server 102 to access and operate the software system for the data integration method. The third-party application 103 obtains data from the data integration server 102 by accessing the corresponding API interfaces opened by the system in the data integration server 102.

[0052] The data integration server 102 can also be a server cluster consisting of several servers, each equipped with a software system that enables data integration between heterogeneous devices and web application systems. Different types of data integration servers 102 can be configured to specifically acquire a particular type of data, depending on the type of data being acquired.

[0053] The accessed devices and WEB application system 101 include: network hosts with WEB applications installed, network devices, security management devices, security systems, user and behavior systems, etc.

[0054] The network devices include: switches, routers, load balancers, etc.

[0055] The security management equipment includes: WAF, NF, ADS, IPS, IDS, etc.

[0056] The security system, installed on a host or server, includes: VPN, behavior auditing, database auditing, baseline, logs, SOC, threat intelligence, etc.

[0057] The user and behavior system includes 4A, access control, PKI / CA, EDR, etc.

[0058] Example 2

[0059] like Figure 2 The diagram illustrates a method for data integration between heterogeneous devices and a web application system, provided by an embodiment of the present invention. This process enables data integration between heterogeneous devices and a web application system within a network application connection, as described in Embodiment 1.

[0060] according to Figure 2 The data integration method described herein includes the following steps:

[0061] S201. Log in to the connected devices and web application systems through the unified login system;

[0062] S220. Control the accessed devices and web application systems to maintain login status;

[0063] S203. Obtain data from connected devices and web application systems;

[0064] S204. Store the acquired data into the database;

[0065] S205. The information obtained is statistically analyzed to form various indicators, which are then displayed through a visual interactive interface for monitoring.

[0066] In step S203, the frequency of data acquisition is controlled according to the set data acquisition interval. In step S204, data is stored according to different data types; and during storage, each type of data is stored in the database one record at a time.

[0067] Furthermore, by providing corresponding API interfaces, data query services are offered to third parties; and audits are conducted on the various connected devices and web application systems and their operations.

[0068] Furthermore, the unified login system employs a login method that simulates each step of the login process and interacts with the accessed devices and WEB application systems using orchestrated simulated action flows to simulate the single sign-on process.

[0069] By using a reverse proxy service, the identification information of the accessing devices and web application systems is obtained to confirm the device or web application system that needs to be logged in. If using an OpenResty reverse proxy service, the response information from each web application system is obtained through LUA scripting language, including the login URL and username. This information is then used to verify whether the web application system corresponds to the password.

[0070] like Figure 3 As shown in the figure, this embodiment of the invention provides an interactive diagram of using the Go login engine to control login.

[0071] The Go login engine controls the overall process, requesting a Python service (initiating the final login request) to obtain authentication information. At the same time, the Go language is used to build the framework of the entire execution process, forming the login engine. Under the control of the login engine, the login service built with Python is used to obtain authentication information, and OpenResty proxy is used to implement reverse proxy while executing Lua scripts to realize login.

[0072] Specifically:

[0073] Log in

[0074] The Go login engine controls the overall process, requests the Python service (initiating the final login request) to obtain identity authentication information, and uses Openresty to implement reverse proxy while executing Lua scripts to achieve login redirection;

[0075] Session persistence

[0076] Controlled by the Go login engine, it continuously accesses the Python service session persistence interface, and then the Python service sends authentication request information to the target client and receives the result, returning it to the Go login engine, thereby achieving session persistence;

[0077] Function assembly execution

[0078] The login operation to be assembled is provided by the Python service. The Go login engine sends the request to the Python service according to the login request method. The Python service executes the request one by one and returns the real-time execution status.

[0079] Dynamic CAPTCHA recognition can automatically recognize website CAPTCHAs using the Go programming language.

[0080] Example 3

[0081] like Figure 4 The image shows a method for acquiring data from accessed devices and web application systems, provided by an embodiment of the present invention. The specific steps are as follows:

[0082] S401. First, establish simulated actions for each step of the data acquisition operation performed by various devices and WEB application systems;

[0083] S402. Then, for each device and WEB application system, the simulated actions of each step are arranged into an execution flow according to the execution order and logical relationship, and encapsulated into different data acquisition API interfaces;

[0084] S403. Finally, call the data acquisition API interfaces of each device and web application system to obtain data.

[0085] The method used in this embodiment can be the same as the login method used in Embodiment 2 to acquire data. The corresponding GO engine construction process corresponds to the data acquisition process, forming a data acquisition engine. The Python service implements each operation step in the data acquisition process.

[0086] Example 4

[0087] like Figure 5 As shown, based on Example 3, this embodiment of the invention provides a method for constructing simulated actions. The specific steps are as follows:

[0088] S501. First, data acquisition operations are performed on various devices and web application systems;

[0089] S502. Then, during the interaction, the communication data packets are captured and analyzed to obtain the corresponding operation steps and the action information of the operation steps;

[0090] S503. Write each type of action information into a corresponding execution action script using a scripting language.

[0091] Example 5

[0092] like Figure 6 The diagram shown is a structural schematic of a data integration device for heterogeneous devices and a web application system provided in an embodiment of the present invention. It includes acquiring data from the accessed devices and the web application system 608, and interacting with them, including:

[0093] Login module 606 is used to log in to access devices and web application systems through a unified login system;

[0094] The timing control module 607 is used to keep the connected devices and web application systems logged in.

[0095] Data acquisition module 601 is used to acquire data from connected devices and WEB application systems;

[0096] The data storage module 602 is used to store the acquired data into the database;

[0097] The interactive display module 605 is used to display various indicators formed by the statistical analysis of the acquired information through a visual interactive interface for monitoring.

[0098] Service provision module 603 is used to provide data query services to third parties by providing corresponding API interfaces;

[0099] Audit module 604 is used to audit the various connected devices and web application systems and their operations;

[0100] Example 6

[0101] like Figure 7 As shown, this embodiment of the invention provides an electronic device. The electronic device includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0102] Multiple components in an electronic device are connected to an I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] The processing unit executes the various methods and processes described above, such as methods S201-S205, S301-S303, and S401-S403. For example, in some embodiments, methods S201-S205, S301-S303, and S401-S403 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S201-S205, S301-S303, and S401-S403 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S201-S205, S301-S303, and S401-S403 by any other suitable means (e.g., by means of firmware).

[0104] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0105] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0108] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data integration method for heterogeneous devices and a web application system, comprising the connected devices and the web application system, characterized in that, Includes the following steps: S1. Log in to the connected devices and web application systems through the unified login system; S2. Maintain login status for connected devices and web application systems; S3. Obtain data from the connected devices and web application systems, specifically including: First, simulate the actions of each step when various devices and web application systems perform data acquisition operations; Then, for each device and web application system, the simulated actions of each step are arranged into an execution flow according to the execution order and logical relationship, and encapsulated into different data acquisition API interfaces; Finally, the data acquisition API interfaces of various devices and web application systems are called to obtain the data; The method for constructing the simulated actions is as follows: First, data acquisition operations are performed on various devices and WEB application systems; then, during the interaction process, communication data packets are captured and analyzed to obtain the corresponding operation steps and action information of the operation steps; each action information is written into a corresponding execution action script using a scripting language. S4. Store the acquired data into the database.

2. The method according to claim 1, characterized in that, It also includes monitoring the acquisition of information by displaying various indicators formed through statistical analysis and visualization via an interactive interface.

3. The method according to claim 1, characterized in that, It also includes providing corresponding API interfaces to external parties to provide data query services for third parties; and auditing the various devices and web application systems and operations that are connected.

4. The method according to claim 1, characterized in that, Step S3 controls the frequency of data acquisition based on the set data acquisition interval.

5. The method according to claim 1, characterized in that, In step S4, data is stored according to different types; and during storage, each type of data is stored in the database one record at a time.

6. A data integration device for heterogeneous devices and a web application system, wherein the devices and the web application system are connected to the device via a network, characterized in that, include: The login module is used to log in to access devices and web application systems through a unified login system. The timing control module is used to keep the connected devices and web application systems logged in. The data acquisition module is used to acquire data from connected devices and web application systems; The data storage module is used to store the acquired data into the database; The interactive display module is used to display various indicators formed by statistical analysis of the acquired information through a visual interactive interface for monitoring. The service provision module is used to provide corresponding API interfaces to external parties and provide data query services to third parties. The auditing module is used to audit the various connected devices and web application systems, as well as their operations.

7. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1-5 when it is run.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method described in any one of claims 1-5.

Images