Data acquisition method, device and equipment and computer readable storage medium

Abstract

The invention relates to a data acquisition method, apparatus and device, and a computer readable storage medium. The method comprises the steps of performing traffic monitoring on a traffic access port of a database; when it is monitored that a traffic transmission event occurs at the traffic access port, copying a traffic packet corresponding to the traffic transmission event to obtain a copied traffic packet; and analyzing the copied traffic packet to obtain target data. According to the method and the device, the data packet passing through the database flow access port is monitored, the data packet passing through the database flow access port is copied, and the copied data packet is further analyzed to obtain the target data so as to realize data acquisition of the database, so that resources of the database do not need to be occupied, and the influence on the performance of the database is greatly reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to a data acquisition method, apparatus, device, and computer-readable storage medium. Background Art

[0002] A database is an organized collection of structured information or data, typically stored electronically in a computer system. Databases play a central role in the services provided by most business processes and applications. Therefore, collecting database data to identify and promptly resolve potential issues is crucial for providing high-quality services.

[0003] In the prior art, data collection from a database is generally performed by monitoring database logs, and obtaining the execution statements of the database when an event occurs within the database.

[0004] However, monitoring database logs will occupy the resources of the database itself, affecting the performance of the database. Summary of the Invention

[0005] In order to solve the above technical problems, the present disclosure provides a data acquisition method, apparatus, device and computer-readable storage medium to reduce the impact of data collection on database performance.

[0006] In a first aspect, an embodiment of the present disclosure provides a data acquisition method, comprising:

[0007] Monitor the traffic in and out of the database;

[0008] When a traffic transmission event occurs at the traffic inlet and outlet ports, copying the traffic packet corresponding to the traffic transmission event to obtain a copied traffic packet;

[0009] The copied traffic packet is parsed to obtain target data.

[0010] In some embodiments, monitoring the traffic in and out ports of the database includes:

[0011] A monitoring agent is installed in the operating system where the database is located, and the monitoring agent is used to monitor the traffic inlet and outlet ports of the database.

[0012] In some embodiments, the method further comprises:

[0013] The target data is sent to a background management server so that the background management server stores and / or analyzes the target data.

[0014] In some embodiments, the method further comprises:

[0015] In response to the management instructions issued by the background management server, the traffic monitoring tasks are managed and controlled.

[0016] In some embodiments, the management instruction includes at least one or more of the following:

[0017] Traffic monitoring start command, traffic monitoring pause command, traffic monitoring modification command, traffic monitoring deletion command.

[0018] In a second aspect, an embodiment of the present disclosure provides a data acquisition device, comprising:

[0019] The monitoring module is used to monitor the traffic in and out of the database;

[0020] A copy module, configured to copy a traffic packet corresponding to the traffic transmission event to obtain a copied traffic packet when a traffic transmission event occurs at the traffic inlet and outlet ports;

[0021] The parsing module is used to parse the copied traffic packet to obtain target data.

[0022] In some embodiments, the apparatus further comprises:

[0023] The sending module is used to send the target data to the background management server, so that the background management server stores and / or analyzes the target data.

[0024] In some embodiments, the apparatus further comprises:

[0025] The control module is used to control the traffic monitoring tasks in response to the management instructions issued by the background management server.

[0026] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0027] Memory;

[0028] processor; and

[0029] computer programs;

[0030] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0031] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0032] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the data acquisition method as described above is implemented.

[0033] The data acquisition method, apparatus, device and computer-readable storage medium provided by the embodiments of the present disclosure monitor the database traffic inlet and outlet ports, copy the data packets passing through the database traffic inlet and outlet ports, and further parse the copied data packets to obtain target data to realize data collection for the database without occupying the resources of the database itself, thereby greatly reducing the impact on database performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 A flow chart of a data acquisition method provided in an embodiment of the present disclosure;

[0037] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0038] Figure 3 A flow chart of a data acquisition method provided in another embodiment of the present disclosure;

[0039] Figure 4 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present disclosure;

[0040] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0043] Existing database data collection technologies typically obtain database execution statements by monitoring database logs. This impacts database performance and prevents seamless integration, leading to performance degradation. Furthermore, database account and password authentication are required, creating security issues. To address this issue, the present disclosure provides a data acquisition method, which is described below in conjunction with specific examples.

[0044] Figure 1 This is a flow chart of the data acquisition method provided by the embodiment of the present disclosure. This method can be applied to Figure 2 The application scenario shown includes an external device 21, a database 22, and a backend management server. The external device 21 can specifically be a server, a smartphone, a PDA, a tablet computer, a wearable device with a display, a desktop computer, a laptop computer, an all-in-one computer, a smart home device, etc. The database 22 is configured with a traffic inlet and outlet port 23. The device where the database is located can be a server, a smartphone, a PDA, a tablet computer, a wearable device with a display, a desktop computer, a laptop computer, an all-in-one computer, a smart home device, etc. It is understandable that the data acquisition method provided in the embodiments of the present disclosure can also be applied in other scenarios.

[0045] Below Figure 1 The data acquisition method shown in FIG. 1 is introduced, and the specific steps of the method are as follows:

[0046] S101. Monitor the traffic in and out of the database.

[0047] Databases have traffic ingress and egress ports for data exchange and access. For example, for MySQL databases, port 3306 is the default port for MySQL databases and serves as the traffic ingress and egress port. This means that if you need to access a MySQL database on a cloud server from an external device, such as a remote client, you must connect through port 3306.

[0048] When an external device exchanges data with a database or an external device needs to access a database, data traffic is transmitted through the traffic inlet and outlet ports. Therefore, by monitoring the traffic inlet and outlet ports of the database, it is possible to identify whether there are changes in the database, such as changes in stored data or changes in external access and query information.

[0049] S102: When a traffic transmission event is detected at the traffic inlet and outlet ports, a traffic packet corresponding to the traffic transmission event is copied to obtain a copied traffic packet.

[0050] When an external device interacts with a database, it needs to access the database through the traffic in / out port by sending a request or other means. At this time, a traffic transmission event will occur at the traffic in / out port, that is, a traffic packet passes through the traffic in / out port.

[0051] The traffic transmission direction in the traffic transmission event may be from an external device to a database, or from a database to an external device.

[0052] At this time, the traffic packet passing through the traffic input and output ports is copied and the copied traffic packet is obtained, while the original traffic packet passes through the traffic input and output ports normally and is not affected.

[0053] S103: Parse the copied traffic packet to obtain target data.

[0054] After parsing the acquired replicated traffic packets, the data contained in the replicated traffic packets is obtained, namely the target data. Based on the target data, the content of the interaction between the external device and the database can be understood, and user behavior can be tracked, thereby realizing data collection and monitoring of the database.

[0055] Specific data is clear, target data,

[0056] Specifically, the data included in the traffic packet or the replicated traffic packet includes, but is not limited to: a request from an external device to access a database, data ultimately read from the database by the external device, and the like.

[0057] That is, this solution performs traffic backtracing through the traffic inlet and outlet ports of the database to achieve real-time network traffic detection, which can deeply analyze traffic packets and track user behavior and the flow of network traffic.

[0058] The embodiment of the present disclosure monitors the traffic inlet and outlet ports of the database; when a traffic transmission event is detected at the traffic inlet and outlet ports, the traffic packet corresponding to the traffic transmission event is copied, the copied traffic packet is obtained, and the copied traffic packet is parsed to obtain target data. By monitoring the traffic inlet and outlet ports of the database, the data packets passing through the traffic inlet and outlet ports of the database are copied, and the copied data packets are further parsed to obtain target data to realize data collection for the database, without occupying the resources of the database itself, thereby greatly reducing the impact on the performance of the database.

[0059] Based on the above embodiment, monitoring the traffic inlet and outlet ports of the database includes: installing a monitoring agent in the operating system where the database is located, and the monitoring agent is used to monitor the traffic inlet and outlet ports of the database.

[0060] Specifically, a monitoring agent is installed in the Windows or Linux operating system where the database is located to monitor the traffic in and out of the database.

[0061] Based on the above embodiment, the method further includes: managing and controlling the traffic monitoring task in response to a management instruction issued by a background management server.

[0062] In some embodiments, the monitoring agent can also filter the traffic packets obtained. Specifically, according to the target data obtained by parsing the traffic packets,

[0063] Optionally, the monitoring agent is managed in response to a management instruction issued by a background management server.

[0064] Specifically, the management instruction includes at least one or more of the following: a flow monitoring start instruction, a flow monitoring pause instruction, a flow monitoring modification instruction, and a flow monitoring deletion instruction.

[0065] When the management instruction is a traffic monitoring start instruction, the monitoring agent is controlled to start the monitoring task for the traffic input and output interface; when the management instruction is a traffic monitoring pause instruction, the monitoring agent is controlled to pause the monitoring task for the traffic input and output interface; when the management instruction is a traffic monitoring modification instruction, the relevant configuration of the monitoring task for the traffic input and output interface is modified, such as the target database that needs to be monitored, etc.; when the management instruction is a traffic monitoring deletion instruction, the traffic monitoring task of the monitoring agent is deleted.

[0066] The disclosed embodiment installs a monitoring agent in the operating system where the database is located, which can not only monitor the traffic inlet and outlet ports of the database, copy the data packets passing through the traffic inlet and outlet ports of the database, but also further parse the copied data packets to obtain target data to realize data collection for the database without occupying the resources of the database itself, greatly reducing the impact on the database performance, and at the same time can also manage the traffic monitoring tasks executed by the monitoring agent according to the management instructions of the background management server, thereby improving the flexibility of data acquisition.

[0067] Furthermore, the data acquisition method provided by the embodiment of the present disclosure supports protocol parsing of multiple databases, has strong consistency, and is not restricted by system versions.

[0068] Figure 3 A flow chart of a data acquisition method provided in another embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the method includes the following steps:

[0069] S301: Monitor the traffic in and out of the database.

[0070] Specifically, a monitoring agent is installed in the operating system where the database is located, and the monitoring agent is used to monitor the traffic inlet and outlet ports of the database.

[0071] Databases have traffic ingress and egress ports for data exchange and access. For example, for MySQL databases, port 3306 is the default port for MySQL databases and serves as the traffic ingress and egress port. This means that if you need to access a MySQL database on a cloud server from an external device, such as a remote client, you must connect through port 3306.

[0072] When an external device exchanges data with a database or an external device needs to access a database, data traffic is transmitted through the traffic inlet and outlet ports. Therefore, by monitoring the traffic inlet and outlet ports of the database, it is possible to identify whether there are changes in the database, such as changes in stored data or changes in external access and query information.

[0073] S302: When a traffic transmission event is detected at the traffic inlet and outlet ports, a traffic packet corresponding to the traffic transmission event is copied to obtain a copied traffic packet.

[0074] When an external device interacts with a database, it needs to access the database through the traffic in / out port by sending a request or other means. At this time, a traffic transmission event will occur at the traffic in / out port, that is, a traffic packet passes through the traffic in / out port.

[0075] The traffic transmission direction in the traffic transmission event may be from an external device to a database, or from a database to an external device.

[0076] At this time, the traffic packet passing through the traffic input and output ports is copied and the copied traffic packet is obtained, while the original traffic packet passes through the traffic input and output ports normally and is not affected.

[0077] S303: parse the copied traffic packet to obtain target data.

[0078] After parsing the acquired replicated traffic packets, the data contained in the replicated traffic packets is obtained, namely the target data. Based on the target data, the content of the interaction between the external device and the database can be understood, and user behavior can be tracked, thereby realizing data collection and monitoring of the database.

[0079] Specifically, the target data is plain text data.

[0080] Specifically, the data included in the traffic packet or the replicated traffic packet includes, but is not limited to: a request from an external device to access a database, data ultimately read from the database by the external device, and the like.

[0081] S304: Send the target data to a background management server, so that the background management server stores and / or analyzes the target data.

[0082] After parsing the replicated traffic packet, the monitoring agent obtains the target data and sends the target data to the background management server. The target data is plain text data.

[0083] After receiving the target data, the backend management server stores the target data; and / or analyzes the target data, and obtains data and event information in the database based on the target data.

[0084] S305: In response to the management instruction issued by the background management server, the traffic monitoring task is managed and controlled.

[0085] The background management server can manage the monitoring agent, thereby controlling the traffic monitoring tasks.

[0086] Specifically, the traffic monitoring start command, the traffic monitoring pause command, the traffic monitoring modification command, and the traffic monitoring deletion command.

[0087] Correspondingly, when the management instruction is a traffic monitoring start instruction, the monitoring agent is controlled to start the monitoring task for the traffic input and output interface; when the management instruction is a traffic monitoring pause instruction, the monitoring agent is controlled to pause the monitoring task for the traffic input and output interface; when the management instruction is a traffic monitoring modification instruction, the relevant configuration of the monitoring task for the traffic input and output interface is modified, such as the target database that needs to be monitored, etc.; when the management instruction is a traffic monitoring deletion instruction, the traffic monitoring task of the monitoring agent is deleted.

[0088] The disclosed embodiment monitors the traffic inlet and outlet ports of the database, copies the data packets passing through the database traffic inlet and outlet ports, and further parses the copied data packets to obtain target data to achieve data collection for the database without occupying the resources of the database itself, thereby greatly reducing the impact on database performance.

[0089] In addition, the embodiment of the present disclosure can also manage the traffic monitoring tasks performed by the monitoring agent according to the management instructions of the background management server, thereby improving the flexibility of data acquisition.

[0090] Figure 4 This is a schematic diagram of the structure of the data acquisition device provided in the embodiment of the present disclosure. The data acquisition device can be the device where the database is located as described in the above embodiment, or the data acquisition device can be a component or assembly in the device where the database is located. The data acquisition device provided in the embodiment of the present disclosure can execute the processing flow provided in the data acquisition method embodiment, such as Figure 4As shown, the data acquisition device 40 includes: a monitoring module 41, a copying module 42, and a parsing module 43; wherein the monitoring module 41 is used to monitor the traffic input and output ports of the database; the copying module 42 is used to copy the traffic packet corresponding to the traffic transmission event when a traffic transmission event is monitored at the traffic input and output ports to obtain a copied traffic packet; the parsing module 43 is used to parse the copied traffic packet to obtain the target data.

[0091] Optionally, the monitoring module 41 is specifically configured to install a monitoring agent in the operating system where the database is located, and the monitoring agent is configured to monitor the traffic inlet and outlet ports of the database.

[0092] Optionally, the data acquisition device 40 further includes a sending module 44, and the sending module 44 is used to send the target data to a background management server, so that the background management server stores and / or analyzes the target data.

[0093] Optionally, the data acquisition device 40 further includes a management and control module 45, which is configured to manage and control the traffic monitoring task in response to a management instruction issued by the background management server.

[0094] Optionally, the management instruction includes at least one or more of the following: a traffic monitoring start instruction, a traffic monitoring pause instruction, a traffic monitoring modification instruction, and a traffic monitoring deletion instruction.

[0095] Figure 4 The data acquisition device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0096] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. The electronic device may be a device where the database described in the above embodiment is located. The electronic device provided in an embodiment of the present disclosure may execute the processing flow provided in the data acquisition method embodiment, such as Figure 5 As shown, the electronic device 50 includes: a memory 51, a processor 52, a computer program and a communication interface 53; wherein the computer program is stored in the memory 51 and is configured so that the processor 52 executes the data acquisition method as described above.

[0097] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the data acquisition method described in the above embodiment.

[0098] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the data acquisition method described above is implemented.

[0099] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0100] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0102] It should be noted that it should be understood that when one or more programs stored in the computer-readable medium are executed by the electronic device, the electronic device may also be enabled to execute other data acquisition methods provided by the examples of the present disclosure.

[0103] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules or units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a module or unit does not, in some cases, limit the module or unit itself.

[0106] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0108] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A data acquisition method, characterized in that: The method comprises: Monitor the traffic in and out of the database; When a traffic transmission event occurs at the traffic inlet and outlet ports, copying the traffic packet corresponding to the traffic transmission event to obtain a copied traffic packet; The copied traffic packet is parsed to obtain target data.

2. The method according to claim 1, characterized in that The monitoring of the traffic inlet and outlet ports of the database includes: A monitoring agent is installed in the operating system where the database is located, and the monitoring agent is used to monitor the traffic inlet and outlet ports of the database.

3. The method according to claim 1, characterized in that The method further comprises: The target data is sent to a background management server so that the background management server stores and / or analyzes the target data.

4. The method according to claim 1, wherein The method further comprises: In response to the management instructions issued by the background management server, the traffic monitoring tasks are managed and controlled.

5. The method according to claim 4, characterized in that The management instructions include at least one or more of the following: Traffic monitoring start command, traffic monitoring pause command, traffic monitoring modification command, traffic monitoring deletion command.

6. A data acquisition device, characterized in that: include: The monitoring module is used to monitor the traffic in and out of the database; A copy module, configured to copy a traffic packet corresponding to the traffic transmission event to obtain a copied traffic packet when a traffic transmission event occurs at the traffic inlet and outlet ports; The parsing module is used to parse the copied traffic packet to obtain target data.

7. The device according to claim 6, characterized in that The device further comprises: The sending module is used to send the target data to the background management server, so that the background management server stores and / or analyzes the target data.

8. The device according to claim 6, characterized in that The device further comprises: The control module is used to control the traffic monitoring tasks in response to the management instructions issued by the background management server.

9. An electronic device, characterized in that: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images