A method and system for real-time data collection and storage based on application layer protocol technology

Abstract

The application discloses a kind of based on application layer protocol technology's data real-time acquisition storage method, system, system includes standard transmission protocol formulation module, transmission tool deployment module, data transmission channel network construction module and data real-time acquisition storage module, method specifically is as follows: client uses standard transmission protocol according to business requirement and sends real-time data to data transmission channel network entrance;Data transmission channel network distributes received data to data forwarding node and carries out data forwarding, wherein data forwarding node forwards data to data cache component according to standard transmission protocol;Data warehousing node obtains data from data cache component according to standard transmission protocol, stores data in warehouse / file system after data format verification is carried out.This application is suitable for multiple scenes, and has strong versatility.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically relating to a method and system for real-time data acquisition and storage based on application layer protocol technology. Background Technology

[0002] With the rapid development of the digital economy, big data applications are penetrating various industries at an accelerated pace. The value mining of data elements has become a key factor in the digital transformation of enterprises in all industries, especially service-oriented enterprises such as finance and the Internet, which all have the need for real-time data collection scenarios, such as obtaining data through Internet web crawlers, collecting enterprise-related data from various government departments, and collecting data from system embedding points, etc.

[0003] The mainstream real-time data acquisition process mainly consists of three parts: the client (A-end) sends the generated real-time data to a custom data receiving interface (B-end); the data receiving interface forwards the received data to a data caching component (such as Kafka, C-end); and the data consumer (such as Flume, D-end) retrieves the data from the data cache and writes it to the database to complete the data acquisition (i.e., A→B→C→D→database). This business process suffers from poor universality and insufficient standardization for different data access scenarios. For example, different companies and teams often use customized data receiving interfaces on the B-end, lacking a unified standard to support receiving data in different formats. This frequently leads to different business teams within the same company developing their own customized B-end acquisition tools. Furthermore, the data consumer (D-end) writes data to a default directory, failing to dynamically allocate data to different directories based on data characteristics (such as acquisition time or business department), resulting in frequent secondary classification and management of the collected data. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for real-time data acquisition and storage based on application layer protocol technology.

[0005] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0006] A method for real-time data acquisition and storage based on application layer protocol technology:

[0007] The client sends real-time data to the data transmission channel network entry point using standard transmission protocols according to business requirements;

[0008] The data transmission channel network distributes the received data to data forwarding nodes for data forwarding, and the data forwarding nodes forward the data to the data buffer component according to the standard transmission protocol;

[0009] The data entry node retrieves data from the data caching component according to the standard transmission protocol, performs data format verification, and then stores the data in the database / file system.

[0010] A further technical solution is that the standard transmission protocol defines the configuration information for real-time data, including a unique data transmission serial number, a unique department identifier, a unique business identifier, a data row separator, a data field separator, a fixed data storage directory, and a dynamic data storage directory.

[0011] In a further technical solution, both the data forwarding node and the data storage node are data transmission tools. The data transmission tool defines functional components including a data transmission channel network management module, a data transmission configuration management module, a data storage configuration management module, a data feature configuration management module, and a data transmission status statistics management module.

[0012] The data transmission channel network management module is specifically a data transmission channel network composed of a large number of transmission tools, used for data transmission;

[0013] The data transmission configuration management module is responsible for the point-to-point transmission configuration information of the transmission tool, including whether the node actively retrieves data from the upstream node or passively receives it, whether the node actively pushes data to the downstream node or passively waits for the downstream to retrieve it, the component type of the upstream node, and the component type of the downstream node.

[0014] The data storage configuration management module is used to configure the storage strategy of the data received by the node, parse the real-time data according to the row separator and field separator, and store it in the data storage fixed directory\dynamic directory.

[0015] The data feature configuration management module is used to configure the data format features forwarded by the node. By comparing the data received by the node with the node's local configuration information, data that meets the format requirements is forwarded, and data that does not meet the format requirements is discarded.

[0016] The data transmission status statistics management module provides statistics on the real-time data transmission status at the current node.

[0017] A further technical solution involves the following process for constructing the data transmission channel network:

[0018] Step (1): Initialize the transmission tool: After the transmission tool starts, set the transmission tool ID and the cluster ID to which the current transmission tool belongs on the server locally; if no other transmission tools have been deployed in the current network environment, set the cluster ID to the cluster ID of the current network; if other transmission tools have been deployed in the current network environment, set the cluster ID to be consistent with the current other transmission tools.

[0019] Configure node transmission settings, including upstream node component type, data acquisition method, downstream node component type, and data sending method;

[0020] Step (2): Network entry of transmission tool: Start the configured transmission tool and broadcast the ID of the current transmission tool and the cluster ID to the network environment; if a transmission cluster already exists in the current network environment, proceed to step 3; if no other transmission tools have been deployed in the current network, proceed to step 4.

[0021] Step (3): Joining the existing data transmission channel network: After receiving the network entry request broadcast, other transmission tools in the network check whether the received cluster ID is consistent with their own cluster ID. If they are consistent, they save the routing information corresponding to the ID of the transmission tool applying for network entry to the local cluster management information list and update the local cluster management information list. The transmission tool applying for network entry sends its configuration information and data transmission status statistics according to the updated local cluster management information list. Each node in the existing data transmission channel network feeds back its own configuration information and data transmission status statistics to the transmission tool applying for network entry, thus completing information synchronization.

[0022] Step (4): Create a new data transmission channel network: If no feedback is received from other nodes, the current data transmission channel network only contains the application network access transmission tool, and this transmission tool serves as the new data transmission channel network.

[0023] A further technical solution involves the data format verification specifically performed by the data feature configuration management module in the data entry node to verify the format and integrity of the fields stored in the standard protocol business data.

[0024] A further technical solution involves verifying the data storage / file system: the data storage node extracts the business data fields from the standard protocol through the data storage configuration management module, parses them according to the row and field separators in the standard transmission protocol, and stores them in the database / file system; during data storage, if the data storage directory for the field in the standard transmission protocol exists, the data is directly written to the corresponding directory; if it does not exist, the data storage configuration management module dynamically creates the directory in the database / file system and stores the data; then, the data transmission status statistics management module uses the data transmission status statistics management module to collect statistics on successful data writing, and the statistical results are synchronized with other transmission tool nodes through the data transmission channel network, completing the transmission process.

[0025] A further technical solution involves discarding or transferring data that fails verification, and then using the data transmission status statistics management module to synchronize the statistical results after discarding or transferring the data with other transmission tool nodes through the data transmission channel network.

[0026] A system for real-time data acquisition and storage based on application layer protocol technology, comprising:

[0027] The standard transmission protocol definition module is used to define the transmission serial number, attribution information, separation information, and storage information of real-time data.

[0028] The transport tool deployment module is used to define functional components;

[0029] The data transmission channel network construction module establishes the data transmission channel network through the initialization of the transmission tool and the network access of the transmission tool.

[0030] The real-time data acquisition and storage module is used to: send real-time data to the data transmission channel network using the standard transmission protocol; forward the data sent by the client to the data caching component according to the standard transmission protocol; and retrieve the data from the data caching component according to the standard transmission protocol, perform data format verification, and then store the data in the database / file system.

[0031] An electronic device, comprising a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program and, while executing the computer program, implement the above-mentioned method for real-time data acquisition and storage.

[0034] A storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the aforementioned method for real-time data acquisition and storage.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) It has multiple applicable scenarios and strong versatility. The structured data real-time acquisition and storage method of the present invention is a general method applicable to various business scenarios. The number and type of fields of the acquired data can be defined by each business team, without the need for separate custom development. The data acquisition process uses a unified transmission channel, which reduces the workload of each team to repeatedly build data acquisition channels.

[0037] (2) Data storage directories can be dynamically created when data is entered into the database. In the current market, most commonly used data acquisition solutions write the collected data into a unified directory, and data partitioning (database sharding) is usually handled separately by other tasks. In the method of this invention, the integrated processing of data partitioning (database sharding) storage is realized through dynamic management of the data storage directory.

[0038] (3) Strong integration of data acquisition. The transmission tool in this invention defines the transmission mode between upstream and downstream points in the data transmission process. By defining the data receiving mode and data transmission mode of upstream and downstream nodes, the data transmission configuration management capability is realized. This capability supports the implementation of the transmission cluster network in this invention. Attached Figure Description

[0039] Figure 1 This is a flowchart of the network access initialization process for the transmission tool described in this invention.

[0040] Figure 2 This is a schematic diagram of the general data acquisition process described in this invention;

[0041] Figure 3 This is a flowchart illustrating the data acquisition and storage process of the transmission tool in the data transmission channel network described in this invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0043] I. Establish a universal real-time data transmission protocol (comm Data Trans Protocol, or standard transmission protocol for short).

[0044] The standard transmission protocol proposed in this invention defines the attribution information (department, business team), separation information, and storage information of real-time data, specifically:

[0045] (1) transaction_id: A unique data transmission serial number that marks the unique number of data transmission;

[0046] (2) dept_id: A unique identifier for the department to which the data belongs;

[0047] (3) busi_id: A unique identifier for the business team to which the data belongs;

[0048] (4) line Separa Char: Data line separator, indicating the data line separation method in data;

[0049] (5) field Separa Char: Data field separator, marking the separation method of each field in each row of data;

[0050] (6) Fixed Path: The data storage is a fixed directory that identifies the storage directory of the real-time data in the database. For example, the data can be automatically stored in subdirectories according to / dept_id / busi_id.

[0051] (7) Active Path: Dynamic directory for data storage. In typical scenarios, when data is partitioned by day and stored in directories with different date identifiers, the data transmission control tool in this invention will automatically create a directory dynamically based on the content of this field (e.g., call the system folder creation command to create a file storage path) for data storage.

[0052] (8) data: business data (i.e. real-time data), refers to the data body carried in the protocol.

[0053] II. Deploying this transmission control tool (commDateTrans, hereinafter referred to as the transmission tool): It defines functional components including a data transmission channel network management module, a data transmission configuration management module, a data storage configuration management module, a data feature configuration management module, and a data transmission status statistics management module. The service capabilities provided by each functional component are as follows:

[0054] (1) Truns Task Status Count Manager (Data Transmission Status Statistics Management Module): Statistics on the real-time data transmission status at the current node, such as the number of data received, the number of data sent, the number of data forwarding failures, and the distribution of forwarding failure reasons. Through the statistics on transmission status, the consistency of data forwarding in the data transmission channel network is ensured, and the reasons for data forwarding failures are easy to analyze.

[0055] (2) Data Chara Conf Manager (Data Feature Configuration Management Module): Used to configure the data format features forwarded by the node, including field separator type and row separator type; by comparing the data received by the node with the node's local configuration information (i.e. (1)-(7) in the transmission protocol), data that meets the format requirements is forwarded, and data that does not meet the format requirements is discarded. The discard statistics are reported to the Truns Task Status Count Manager component.

[0056] (3) Save Work Conf Manager (Data Storage Configuration Management Module): Used to configure the storage strategy (including file storage path, file format, etc.) in the data received by the node, and to parse the business data in the standard transmission protocol according to the line separator and field separator (for example, splitting the whole string into multiple messages according to the line separator, and splitting a message into multiple fields according to the field separator), and store it in the data storage fixed directory\dynamic directory;

[0057] (4) Truns Worker Conf Manager (Data Transmission Configuration Management Module): Responsible for the point-to-point transmission configuration information of the transmission tool, including whether the node actively retrieves or passively receives data from the upstream node, whether the node actively pushes or passively waits for the downstream node to retrieve data, the component type of the upstream node, the component type of the downstream node, etc.

[0058] (5) Truns Net Manager: A data transmission channel network consisting of a large number of transmission tools, used for data transmission.

[0059] The conditions required for the transmission tool to run are: there must be network connectivity between service nodes, and the operating system running on the server must support access to relevant data caching components (such as Kafka, RabbitMQ, etc.).

[0060] III. The process of establishing a data transmission channel network using the aforementioned transmission tools is as follows: Figure 1 The process is as follows:

[0061] Step 1: Transmission Tool Initialization: After the transmission tool starts, set the transmission tool ID and the cluster ID to which the current transmission tool belongs locally on the server. The transmission tool ID is used to identify itself, and the cluster ID is used to identify the management cluster to which the transmission tool currently belongs. If no other transmission tools are deployed in the current network environment, the cluster ID is set to the cluster ID of the current network. If other transmission tools are deployed in the current network environment, the cluster ID in the configuration information is set to be consistent with the current settings of the other transmission tools.

[0062] Configure the node transmission settings, including the upstream node component type (e.g., client, Kafka, etc.) and data acquisition method (e.g., passive reception, active acquisition), and the downstream node component type (e.g., Kafka, file system, database, etc.) and data sending method (e.g., active push, etc.). Once the node configuration is complete, the transmission tool will execute step 2 to perform the network access operation.

[0063] Step 2: Network Access for Transmission Tools: Start the configured transmission tools. The transmission tools will obtain the local network information of the server and broadcast the ID of the current transmission tools and the cluster ID to the network environment. If a transmission cluster already exists in the current network environment, proceed to Step 3. If no other transmission tools have been deployed in the current network, proceed to Step 4.

[0064] Step 3: Joining the Existing Data Transmission Channel Network: After receiving the network entry request broadcast, other transmission tools in the network check whether the received cluster ID matches their own cluster ID. If they match, they save the routing information corresponding to the ID of the requesting transmission tool to the local cluster management information list and update the local cluster management information list. This information is managed by Truns Net Manager. The requesting transmission tool sends its configuration information and data transmission status statistics according to the updated local cluster management information list. Each node in the existing data transmission channel network feeds back its own configuration information and data transmission status statistics to the requesting transmission tool, completing information synchronization. The requesting transmission tool's network entry is then complete.

[0065] Step 4: Establish a new data transmission channel network: If no feedback is received from other nodes, the current data transmission channel network contains only this one transmission tool. Data transmission based on this transmission tool can serve as a new data transmission channel network.

[0066] IV. This invention enables simultaneous data collection and storage from different business teams: The data transmission client is configured with a standard transmission protocol according to business requirements. After the client sends data to the data transmission channel network, it is transmitted point-to-point through various transmission tools within the network and then written to the database. The data collection and storage process using this invention, tailored to the different business needs of different business teams, is as follows: Figure 2 As shown, the specific steps are as follows:

[0067] Step 1: The client sends real-time data to the data transmission channel network entry point using the standard transmission protocol according to business requirements. For example, if there are two business teams in the enterprise, they will fill in the formatted real-time data to be transmitted into the standard transmission protocol after separating it by rows and fields (as shown in Table 1 after completion). The completed transmission protocol is then sent by the client to the data transmission channel network entry point.

[0068]

[0069]

[0070] Step 2: Data forwarding via the data transmission channel network: After receiving data from the client, the data transmission channel network distributes the data to a data transmission tool via a load balancer for forwarding. For example, after receiving the standard transmission protocol, transmission tool 1 (data forwarding node 1) queries the Truns Worker Conf Manager to find that the downstream node component is Kafka and the data sending method is active push. Transmission tool 1 (data forwarding node 1) sends the data to the Kafka message queue according to the standard transmission protocol.

[0071] Step 3: The transmission tool (data entry node) retrieves data from the data caching component Kafka according to the standard transmission protocol, performs data format verification, and then stores the data into the database / file system; see also Figure 3 :

[0072] Step 3-1: Data format verification: The Data Chara ConfManager in the transmission tool (data entry node) verifies the fields stored in the standard protocol business data, such as format verification (line separator, field separator), integrity verification, etc. Data that passes the verification is stored in the database / file system and then proceeds to step 3-2; data that fails the verification proceeds to step 3-3.

[0073] Step 3-2: The transmission tool (data entry node) defines the upstream node component as Kafka and the data acquisition method as active data subscription mode through Truns Worker Conf Manager. After the standard protocol (marking business x) arrives at the data entry node, the data entry node retrieves the business data field data from the standard protocol through Save Worker Conf Manager, parses it according to the line separator and field separator defined in the standard transmission protocol's line Separa Char and field Separa Char fields, and stores it in the database / file system. When storing data, if the data storage directory defined in the fixed Path and active Path fields of the standard transmission protocol exists, the data is directly written to the corresponding directory (file or table). If it does not exist, Save Worker Conf Manager dynamically creates the directory in the database / file system and stores the data. Then, Truns Task Status Count Manager is used to count the successful data writing information. The statistical results are synchronized with other transmission tool nodes (such as data forwarding nodes, data entry nodes, etc.) through the data transmission channel network, and the transmission process is completed.

[0074] Step 3-3: For data that fails validation, the data entry node performs fail processing (such as discarding or transferring). At the same time, the Truns Task Status Count Manager is used to synchronize the statistical results after processing with other transmission tool nodes through the data transmission channel network, and the transmission process is completed.

[0075] A system for real-time data acquisition and storage based on application layer protocol technology includes a standard transmission protocol formulation module, a transmission tool deployment module, a data transmission channel network construction module, and a real-time data acquisition and storage module, wherein:

[0076] The standard transmission protocol definition module is used to define the transmission serial number, attribution information, separation information, and storage information of real-time data;

[0077] The transmission tool deployment module is used to define functional components, including a data transmission channel network management module, a data transmission configuration management module, a data storage configuration management module, a data feature configuration management module, and a data transmission status statistics management module.

[0078] The data transmission channel network construction module establishes the data transmission channel network through the initialization of the transmission tool and the network access of the transmission tool.

[0079] The real-time data acquisition and storage module is used to: send real-time data to the data transmission channel network using the standard transmission protocol; forward the data sent by the client to the data caching component according to the standard transmission protocol; and retrieve the data from the data caching component according to the standard transmission protocol, perform data format verification, and then store the data in the database / file system.

[0080] The aforementioned real-time data acquisition and storage system based on application layer protocol technology can be implemented as a computer program that can run on a computer device, which can be a terminal or a server. The server can be a standalone server or a server cluster; the terminal can be an electronic device such as a tablet computer, desktop computer, laptop computer, mobile phone, wearable device, or personal digital assistant.

[0081] The computer device includes a network interface, memory, and processor connected via a system bus. The memory may include main memory and a non-volatile storage medium. The non-volatile storage medium can store computer programs and an operating system. The computer program includes program instructions that, when executed, enable the processor to perform any real-time data acquisition and storage method based on application layer protocol technology. The processor provides control and computing capabilities to ensure the smooth operation of the entire computer device. The memory provides a runtime environment for the computer program in the non-volatile storage medium. When the computer program is executed by the processor, it enables the processor to perform any real-time data acquisition and storage method based on application layer protocol technology. The network interface is capable of network communication, such as sending assigned tasks.

[0082] It should be understood that the processor can be a digital signal processor (DSP), a central processing unit (CPU), other general-purpose processors, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other programmable logic devices, discrete hardware components, transistor logic devices, or discrete gates, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0083] This application also provides a computer-readable and writable storage medium storing a computer program, which includes program instructions. The processor executes the program instructions to implement the real-time data acquisition and storage method based on application layer protocol technology of this application. The computer-readable and writable storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk of the computer device's memory. Alternatively, the computer-readable and writable storage medium can be an external storage device of the computer device, such as a Smart Media Card (SMC), plug-in hard disk, flash card, or Secure Digital Card (SD) equipped on the computer device.

[0084] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for real-time data collection and storage based on application layer protocol technology, characterized in that: a client sends real-time data to a data transmission channel network portal using a standard transmission protocol according to business needs; the data transmission channel network distributes the received data to a data forwarding node for data forwarding, and the data forwarding node forwards the data to a data cache component according to the standard transmission protocol; a data warehousing node obtains data from the data cache component according to the standard transmission protocol, stores the data in a warehouse / file system after data format verification; the standard transmission protocol defines configuration information of real-time data, including a unique data transmission serial number, a department unique identifier, a business unique identifier, a data row separator, a data field separator, a data storage fixed directory, and a data storage dynamic directory; the data forwarding node and the data warehousing node are both data transmission tools, and the data transmission tool defines functional components including a data transmission channel network management module, a data transmission configuration management module, a data storage configuration management module, a data feature configuration management module, and a data transmission state statistical management module; the data transmission channel network management module is specifically a data transmission channel network composed of a large number of transmission tools, used for transmitting data; the data transmission configuration management module is responsible for point-to-point transmission configuration information of the transmission tool, including whether the node obtains data from the upstream node in an active manner or a passive manner, whether the node transmits data to the downstream node in an active manner or a passive manner, the upstream node component type, and the downstream node component type; the data storage configuration management module is used to configure the storage strategy in the data received by the node, parse the real-time data according to the row separator and the field separator, and store the data in the data storage fixed directory / dynamic directory; the data feature configuration management module is used to configure the data format features of the data forwarded by the node, and the data that meets the format requirements is forwarded and the data that does not meet the format requirements is discarded by comparing the data received by the node with the local configuration information of the node; the data transmission state statistical management module statistically analyzes the transmission of real-time data at the current node. The construction process of the data transmission channel network is as follows: Step (1): transmission tool initialization: after the transmission tool is started, the transmission tool ID and the cluster ID to which the current transmission tool belongs are set locally on the server; if no other transmission tools have been deployed in the current network environment, the cluster ID is set as the cluster ID of the current network; if other transmission tools have been deployed in the current network environment, the cluster ID is set to be consistent with the current other transmission tools; node transmission configuration is set, including the upstream node component type, the data acquisition manner, the downstream node component type, and the data sending manner; Step (2): transmission tool network entry: start the configured transmission tool, and the transmission tool broadcasts the ID and the cluster ID of the current transmission tool to the network environment; if a transmission cluster already exists in the current network environment, step 3 is executed, and if no other transmission tools have been deployed in the current network, step 4 is executed. ​ ​ ​ ​ ​ ​ ​ ​ 2. The method for data real-time collection and storage based on application layer protocol technology according to claim 1, characterized in that, ​ ​ ​ ​ ​ Step (3): Add existing data transmission channel network: after receiving the network request broadcast, the other transmission tools in the network check whether the received cluster ID is consistent with the cluster ID of the self, if consistent, save the route information corresponding to the network access transmission tool ID to the local cluster management information list, update the local cluster management information list; the network access transmission tool sends its configuration information and data transmission state statistical information according to the updated local cluster management information list, and each node in the existing data transmission channel network feeds back the respective configuration information and data transmission state statistical information to the network access transmission tool, and completes information synchronization; Step (4): New data transmission channel network: if no feedback is received, the current data transmission channel network only contains the network access transmission tool, and the transmission tool serves as a new data transmission channel network.

3. The method for data real-time collection and storage based on application layer protocol technology according to claim 2, characterized in that, The data format verification specifically includes: the data feature configuration management module in the data storage node performs format and integrity verification on the fields stored in the standard protocol business data.

4. The method for real-time data acquisition and storage based on application layer protocol technology according to claim 3, characterized in that, The data storage in the database / file system through verification: the data storage configuration management module in the data storage node takes out the field data of the business data in the standard protocol, parses according to the row delimiter and field delimiter in the standard transmission protocol, and stores in the database / file system; When storing data, if the data storage directory of the field in the standard transmission protocol exists, the data is directly written into the corresponding directory, and if it does not exist, the data storage configuration management module dynamically creates the directory in the database / file system and stores the data. The data transmission state statistical management module is used to count the information of successful data writing, and the statistical result is synchronized with other transmission tool nodes through the data transmission channel network, and the transmission process is completed.

5. The method for data real-time collection and storage based on application layer protocol technology according to claim 3, characterized in that, The data that does not pass the verification is discarded or stored, and the data transmission state statistical management module is used to count the statistical result after discarding or storing, and the statistical result is synchronized with other transmission tool nodes through the data transmission channel network.

6. A system for implementing the method of real-time data collection and storage based on application layer protocol technology according to any one of claims 1-5, characterized in that, It includes: A standard transmission protocol development module for defining the transmission flow number, attribution information, separation information and storage information of real-time data; A transmission tool deployment module for defining functional components; A data transmission channel network construction module for initializing transmission tools and establishing a data transmission channel network through network access of the transmission tools; A data real-time acquisition and storage module for realizing that a client sends real-time data to a data transmission channel network using a standard transmission protocol, a data forwarding node forwards the data sent by the client to a data cache component according to the standard transmission protocol, and a data storage node acquires data from the data cache component according to the standard transmission protocol, and stores the data in a database / file system after data format verification.

7. An electronic device, comprising: It includes a memory and a processor; The memory is used to store a computer program; The processor is used to execute the computer program and realize the method of real-time data acquisition and storage based on application layer protocol technology according to any one of claims 1-5 when executing the computer program.

8. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to make the processor execute the method for collecting and storing data in real time based on the application layer protocol technology according to any one of claims 1-5.

Images