Data acquisition method of edge-end automatic protocol matching and key data extraction

Abstract

The embodiment of the application provides a data acquisition method and system for edge automatic protocol matching and key data extraction, a storage medium, a processor and a computer program product. The method comprises the following steps: receiving a data stream of an access device collected by a hardware collector; caching the data stream into a file system; finding key data in the data stream by using a data searching function to determine a position of the key data; determining a communication protocol of the data according to the position; generating a data filtering script by a server according to the communication protocol; and pushing the data filtering script to the hardware collector, so that the hardware collector filters a next data stream according to the data filtering script.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a data acquisition method, system, storage medium, processor, and computer program product for automatic protocol matching and key data extraction at the edge. Background Technology

[0002] Many existing devices using proprietary protocols suffer from network connectivity issues. The variety of protocols necessitates sequential integration with industry data platforms. Analyzing these protocols and acquiring data is challenging, inefficient, time-consuming, and inherently uncertain. Traditionally, different environments and tools must be deployed for analysis of different devices and protocols, resulting in solutions such as:

[0003] Option 1: Manually collect data, analyze protocols, distinguish data domains, and determine data formats.

[0004] Once the protocol is obtained, a data collector that parses the corresponding protocol is developed.

[0005] • Solution 2: For specific protocol types, such as the Modbus protocol, the existing solution is to upload all data to the cloud, analyze the data items in the cloud, set data features and extraction rules, and then push the data to the customer's cloud platform.

[0006] As can be seen, the above solutions are very time-consuming, prone to omissions, and dependent on cloud resolution platform service providers. In addition, the data needs to flow through a third-party database, resulting in low data security. Summary of the Invention

[0007] The purpose of this invention is to provide a data acquisition method, system, storage medium, processor, and computer program product for automatic protocol matching and key data extraction at the edge.

[0008] To achieve the above objectives, the first aspect of the present invention provides a data acquisition method for automatic protocol matching and key data extraction at the edge, comprising:

[0009] Receive data streams from access devices collected by a hardware acquisition unit;

[0010] The data stream is cached in the file system;

[0011] The key data in the data stream is located by using a data lookup function;

[0012] The communication protocol for the data is determined based on the location;

[0013] The server generates a data filtering script according to the communication protocol;

[0014] The data filtering script is pushed to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

[0015] Optionally, the hardware acquisition device is based on the micropython architecture.

[0016] Alternatively, key data can be entered via an input device.

[0017] Optionally, the hardware data collector can be a wireless data collector.

[0018] Optionally, the access device can be a device that cannot connect to the internet.

[0019] A second aspect of the present invention provides a data acquisition system for automatic protocol matching and key data extraction at the edge, comprising:

[0020] Hardware data acquisition unit, used to collect data streams from access devices;

[0021] A data caching unit is used to cache the data stream sent by the hardware collector;

[0022] An input unit is used to input key data via an input device;

[0023] A data lookup unit is used to find the key data in the data stream through a data lookup function in order to determine the location of the key data.

[0024] A filtering unit is used to determine the data communication protocol based on the location. The server generates a data filtering script based on the communication protocol and pushes the data filtering script to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

[0025] A third aspect of the present invention provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned data acquisition method for edge-end automatic protocol matching and key data extraction.

[0026] A fourth aspect of the present invention provides a processor configured to perform the above-described data acquisition method for edge-end automatic protocol matching and key data extraction.

[0027] This solution utilizes a flexible design to create an improved tool with multiple physical interfaces. Through long-term firmware evolution, the algorithm and data filtering are continuously improved. The tool can interface with the platform via wireless communication, enabling remote configuration, upgrades, and analysis. This solution adapts to communication protocols and automatically filters and selects data, improving data transmission efficiency. Furthermore, data transmission in this solution does not require a third-party database, thus enhancing data security.

[0028] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0030] Figure 1 This schematic diagram illustrates the structural block diagram of a data acquisition system for automatic protocol matching and key data extraction at the edge, according to an embodiment of the present invention.

[0031] Figure 2 The schematic diagram illustrates a data acquisition method for automatic protocol matching and key data extraction at the edge according to an embodiment of the present invention;

[0032] Figure 3 This diagram illustrates the data flow of a data acquisition method for automatic protocol matching and key data extraction at the edge, according to an embodiment of the present invention.

[0033] Figure 4 This diagram illustrates the architecture of a data acquisition system for automatic protocol matching and key data extraction at the edge, according to an embodiment of the present invention.

[0034] Figure 5 The diagram illustrates the internal structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0036] like Figure 1 As shown, in this embodiment, a data acquisition system for automatic protocol matching and key data extraction at the edge is provided, including:

[0037] Hardware acquisition unit 101 is used to acquire data streams from access devices;

[0038] Data caching unit 102 is used to cache the data stream sent by the hardware collector;

[0039] Input unit 103 is used to input key data through an input device;

[0040] Data lookup unit 104 is used to find key data in the data stream through a data lookup function in order to determine the location of the key data;

[0041] The filtering unit 105 is used to determine the data communication protocol based on the location. The server generates a data filtering script according to the communication protocol and pushes the data filtering script to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

[0042] Figure 2 This illustration schematically depicts a data acquisition method for automatic protocol matching and key data extraction at the edge, according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, a data acquisition method for automatic protocol matching and key data extraction at the edge is provided, including the following steps:

[0043] Step 201: Receive the data stream from the access device collected by the hardware acquisition device.

[0044] Step 202: Cache the data stream in the file system.

[0045] Step 203: Use a data lookup function to find key data in the data stream to determine the location of the key data.

[0046] Step 204: Determine the data communication protocol based on the location.

[0047] Step 205: The server generates a data filtering script according to the communication protocol.

[0048] Step 206: Push the data filtering script to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

[0049] The access device refers to a device that cannot connect to the internet. First, a hardware collector can collect data streams from these internet-connected devices. This data stream can be data determined by preset requirements and needs to be obtained from the access device, such as its operating parameters, operating status data, or data collected by the access device itself. After collecting the data streams from the access devices, the hardware collector can upload them to a data caching unit, i.e., cache the data streams in a file system. This hardware collector can be a wireless collector.

[0050] Furthermore, key data, i.e., the data to be searched, can be input via an input device. This input device can be cloud software or an app. Based on the key data input, the data to be searched can be determined, and a data search function can be used to find the key data in the data stream, thus locating its position. After locating the key data, the data communication protocol can be determined based on the data format. Then, the server can generate a data filtering script corresponding to this protocol. Furthermore, the data filtering script can be pushed to the hardware collector, allowing it to filter the next data stream based on the script. This allows for the combined acquisition of the protocol and product development, improving the speed of device access.

[0051] Furthermore, with the improvement of embedded hardware performance, hardware data acquisition devices can use advanced architectures and embedded Python languages ​​like MicroPython to enhance the processing power and range of IoT terminals. Moreover, based on the enhanced processing power and high-level language features, data tagging can be implemented on the terminal, achieving self-learning and adaptive functions. Through the hot-update feature of Python virtual machine scripts, data filtering and selection functions can be implemented, shortening the development cycle.

[0052] like Figure 3 As shown, the first step is hardware integration, which involves using a hardware data collector to capture data streams from the access devices and caching the data streams to the file system. After collection and caching are complete, a completion notification can be sent to the server. The server can then establish a local data retrieval mechanism and notify the cloud to wait for the cached data streams. Finally, key data can be input via an input device. Figure 3 The data stream contains key data. Key data can be located within the data stream using data lookup functions, allowing for statistical analysis of the search results. After user confirmation of the retrieved results, a data filtering script is generated according to the communication protocol and pushed to the hardware acquisition unit. This script then filters the next data stream based on the filtering script.

[0053] like Figure 4The architecture diagram shows that existing access devices that cannot connect to the network can communicate with the hardware acquisition unit via Ethernet / RS485 or other methods. After acquiring the data stream from the access device, the hardware acquisition unit can cache the data stream in a data caching unit. After caching is complete, a completion notification is sent to the feature data forwarding unit. Simultaneously, a PC browser or similar device can act as an input device, allowing the user to input the key data to be searched. The feature data forwarding unit uses a data lookup function to find the key data in the data stream. After determining the location of the key data, it determines the communication protocol based on the format of the key data and pushes a data filtering script to the hardware acquisition unit, enabling the hardware acquisition unit to filter the next data stream according to the data filtering script. This may also include a data display unit to display the data stream acquired by the hardware acquisition unit.

[0054] This solution utilizes a flexible design to create an improved tool with multiple physical interfaces. Through long-term firmware evolution, the algorithm and data filtering are continuously improved. The tool can interface with the platform via wireless communication, enabling remote configuration, upgrades, and analysis. This solution adapts to communication protocols and automatically filters and selects data, improving data transmission efficiency. Furthermore, data transmission in this solution does not require a third-party database, thus enhancing data security.

[0055] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and data acquisition, including automatic protocol matching and key data extraction at the edge, can be achieved by adjusting kernel parameters.

[0056] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0057] This invention provides a storage medium storing a program that, when executed by a processor, implements the aforementioned data acquisition method for automatic edge protocol matching and key data extraction.

[0058] This invention provides a processor for running a program, wherein the program executes the aforementioned data acquisition method for automatic edge protocol matching and key data extraction.

[0059] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores data streams from access devices. The network interface A02 communicates with external terminals via a network connection. When the processor A01 executes the computer program B02, it implements a data acquisition method for automatic protocol matching and key data extraction at the edge.

[0060] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0061] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: receiving a data stream from an access device collected by a hardware collector; caching the data stream in a file system; searching for key data in the data stream using a data lookup function to determine the location of the key data; determining the communication protocol of the data based on the location; generating a data filtering script based on the communication protocol; and pushing the data filtering script to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

[0062] In one embodiment, the hardware acquisition unit is based on the micropython architecture.

[0063] In one embodiment, the key data is input via an input device.

[0064] In one embodiment, the hardware collector is a wireless collector.

[0065] In one embodiment, the access device is a device that cannot connect to the internet.

[0066] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: receiving a data stream from an access device collected by a hardware collector; caching the data stream in a file system; searching for key data in the data stream using a data lookup function to determine the location of the key data; determining the data communication protocol based on the location; the server generating a data filtering script according to the communication protocol; and pushing the data filtering script to the hardware collector so that the hardware collector filters the next data stream according to the data filtering script.

[0067] In one embodiment, the hardware acquisition unit is based on the micropython architecture.

[0068] In one embodiment, the key data is input via an input device.

[0069] In one embodiment, the hardware collector is a wireless collector.

[0070] In one embodiment, the access device is a device that cannot connect to the internet.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0076] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0077] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0078] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0079] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data acquisition method for automatic protocol matching and key data extraction at the edge, characterized in that, The method includes: Receive data streams from access devices collected by a hardware acquisition unit; The data stream is cached in the file system; The key data in the data stream is located by using a data lookup function; The communication protocol for the data is determined based on the location; The server generates a data filtering script according to the communication protocol; The data filtering script is pushed to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script; the hardware collector is based on the MicroPython architecture; the key data is input through an input device; the hardware collector is a wireless collector; the access device is a device that cannot connect to the internet.

2. A data acquisition system for automatic protocol matching and key data extraction at the edge, characterized in that, include: Hardware data acquisition unit, used to collect data streams from access devices; A data caching unit is used to cache the data stream sent by the hardware collector; An input unit is used to input key data via an input device; A data lookup unit is used to find the key data in the data stream through a data lookup function in order to determine the location of the key data. A filtering unit is used to determine the data communication protocol based on the location. The server generates a data filtering script based on the communication protocol and pushes the data filtering script to the hardware collector so that the hardware collector can filter the next data stream according to the data filtering script.

3. The system according to claim 2, characterized in that, The hardware acquisition device is based on the MicroPython architecture.

4. A processor, characterized in that, The data acquisition method is configured to perform edge-end automatic protocol matching and key data extraction as described in claim 1.

5. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by the processor, this instruction configures the processor to perform the data acquisition method for automatic protocol matching and key data extraction at the edge as described in claim 1.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data acquisition method for automatic protocol matching and key data extraction at the edge end as described in claim 1.

Images