A traffic recording system and method for Internet of Things testing

Abstract

The invention discloses a flow recording system and method for Internet of Things testing, belongs to the technical field of Internet of Things, solves the technical problem that the existing recording method is not stable enough, and the system comprises an edge gateway, a proprietary device management platform and a flow management platform; one end of the edge gateway is connected to the Internet of Things device, receives various data of the Internet of Things device, and sends messages to the corresponding Internet of Things device for execution; the other end of the edge gateway is respectively connected to the proprietary device management platform and the flow management platform; the edge gateway screens and processes the data of the Internet of Things device and then uploads it to the proprietary device management platform; the edge gateway receives instructions and rules from the flow management platform, records the data to be sent according to the instructions and rules and temporarily stores it locally, and uploads the temporarily stored recording data to the flow management platform without affecting the normal communication with the proprietary device management platform; the flow management platform receives the recording data and stores it in a database.

Description

Technical Field

[0001] The present invention relates to the technical field of the Internet of Things, and more specifically, it relates to a traffic recording system and method for Internet of Things testing. Background Art

[0002] The Internet of Things device management platform can connect and manage the devices and applications accessing the Internet of Things, and manage and interact according to service standards. An Internet of Things device management platform can simultaneously connect and manage hundreds of thousands or even millions of Internet of Things devices, and at the same time support multiple communication protocols such as http, MQTT, CoAP, LwM2M, etc.

[0003] With the continuous development of the Internet industry, more and more application systems are launched. When the application system needs to be modified, a large number of automated regression scripts are often used to modify the application system. However, due to the continuous evolution of the system at all times, it is difficult to guarantee the correctness of the automated regression scripts. To avoid this problem, it is necessary to convert the real data stream on the line into comprehensive regression test cases. Therefore, the traffic recording technology has emerged.

[0004] Traffic recording refers to recording the data packets sent to the application system, saving the traffic in the form of data to a database or a file, and playing it back in the system under test for regression testing or using it as the pressure data source in stress testing. Using real data on the line for regression testing or stress testing can cover real scenarios more comprehensively, thus showing the importance of traffic recording.

[0005] In the existing technology, mainly tools such as Jvm-Sandbox-Repeater and

[0006] ngx_http_mirror_module and GoReplay based on port recording are used. When using these tools, the server side with relatively concentrated traffic is recorded. When the traffic on the server side is large, it will more or less cause certain impacts and risks to the application service. Especially for the Internet of Things device management platform, there are a large number of connected devices, a large amount of data upload traffic, and multiple and complex communication protocols used. The risk of traffic replication on the server side is very high.

[0007] Patent Application 202210193341.X designed a method for traffic recording and playback based on proxy components. However, this method will have a certain impact on the performance of application services when the traffic is large and will consume some server resources, increasing the instability of the platform for the Internet of Things device management platform. Patent Application 202210668819 designed a traffic recording method based on port monitoring, using a recording unit to monitor and record traffic on ports. However, different data processing is required for different communication protocols to effectively record traffic, and it will also pose certain risks to the application service proxy. Especially for the Internet of Things device management platform using multiple communication protocols, the risks and workload are quite large. Summary of the Invention

[0008] The technical problem to be solved by the present invention is in view of the above deficiencies of the prior art. The first object of the present invention is to provide a traffic recording system for Internet of Things testing.

[0009] The second object of the present invention is to provide a traffic recording method for Internet of Things testing.

[0010] To achieve the above first object, the present invention provides a traffic recording system for Internet of Things testing, including an edge gateway, an owned device management platform, and a traffic management platform;

[0011] One end of the edge gateway is connected to Internet of Things devices, directly communicates with Internet of Things devices, receives various data of Internet of Things devices, and sends messages to the corresponding Internet of Things devices for execution;

[0012] The other end of the edge gateway is respectively connected to the owned device management platform and the traffic management platform; after screening and processing the data of Internet of Things devices, the edge gateway uploads the data to the owned device management platform according to the data type;

[0013] The edge gateway receives instructions and rules from the traffic management platform. After receiving the data of Internet of Things devices, before uploading the data to the owned device management platform, it records the data to be sent according to the received instructions and rules and temporarily stores it locally, and uploads the temporarily stored recorded data to the traffic management platform on the premise of not affecting the normal communication with the owned device management platform;

[0014] The traffic management platform receives the recorded data sent by the edge gateway, cleans the recorded data and stores it in the database for regression testing or stress testing of the system under test;

[0015] The owned device management platform receives device instructions from the edge gateway and conducts unified scheduling and management.

[0016] As a further improvement, the Internet of Things devices in the network are divided into device groups according to business attributes, and each group of devices is assigned an edge gateway. The Internet of Things devices in the same device group are directly connected to the edge gateway.

[0017] Furthermore, according to their own operating conditions, each Internet of Things device sends status changes, exception alerts, and heartbeat information to the edge gateway it is connected to in the form of instructions, and can also transmit data to the edge gateway.

[0018] Furthermore, the edge gateway includes a traffic recording module, a data receiving module, a data uploading module, and a storage module. Among them, the traffic recording module includes a traffic recording control unit, a data filtering and storage unit, a resource monitoring unit, and a recorded traffic uploading unit;

[0019] The traffic recording control unit is used to control the start time of traffic recording, receive the recording policy sent from the traffic management platform, and send a recording signal to the data uploading module according to the recording policy, so that the data uploading module copies a portion of the recording data to be sent within a period of time and sends it to the data filtering and storage unit;

[0020] The data filtering and storage unit is used to filter the copied data sent by the data uploading module according to the data filtering rules sent from the traffic management platform, and store the filtered data in the storage module of the edge gateway;

[0021] The resource monitoring unit is used to monitor the usage of the hardware resources of the edge gateway, including CPU usage rate, network traffic, and the remaining storage capacity of the storage module; when there is free space in the storage module, it sends a recordable signal to the traffic recording control unit, otherwise it sends a non-recordable signal; when the CPU usage rate is low and the network traffic is small, it sends an uploadable signal to the recorded traffic uploading unit, enabling the recorded traffic uploading unit to upload the recorded data stored in the storage module to the traffic management platform;

[0022] The recorded traffic uploading unit is used to send the recorded data stored in the storage module. After receiving the uploadable signal sent by the resource monitoring unit, it retrieves the recorded data from the storage module, uploads it in small batches multiple times, and deletes the uploaded recorded data in the storage module after each upload is completed;

[0023] The data receiving module is used to communicate with the Internet of Things devices, receive the data uploaded by the Internet of Things devices, and send the screened data to the data uploading module;

[0024] The data upload module is used to upload the data sent by the data reception module to the self-owned device management platform; and after receiving the recording signal sent by the traffic recording control unit, a new recording thread is started to copy a copy of the data to be uploaded to the self-owned device management platform and send it to the data filtering and storage unit, and the source data continues to be uploaded to the self-owned device management platform without being affected;

[0025] The storage module is used for data storage and can store the copied recording data and the data uploaded by the Internet of Things devices.

[0026] Further, the recording policy includes recording frequency, recording period, and data filtering rules.

[0027] Further, the recording signal carries a recording duration parameter, and the recording thread is automatically closed after exceeding the duration parameter, or the recording thread is closed after receiving the stop recording signal.

[0028] To achieve the above-mentioned second object, the present invention provides a traffic recording method for Internet of Things testing, including the following steps:

[0029] Step 1, the edge gateway normally receives the data uploaded by the device group;

[0030] Step 2, the edge gateway opens the traffic recording switch according to the instruction received from the traffic management platform;

[0031] Step 3, after the edge gateway monitors that the traffic recording switch is opened, it copies a copy of the data of the Internet of Things device and sends it to the traffic recording module, while the source data continues to be uploaded to the self-owned device management platform;

[0032] Step 4, the traffic recording module monitors the usage of the storage resources of the edge gateway. When the storage resources are sufficient, it filters the copied data according to the adoption frequency and data filtering rules in the instruction received from the traffic management platform and temporarily stores it locally, controls the upper limit of the local storage of the traffic recording, and closes the traffic recording switch when the upper limit is exceeded;

[0033] Step 5, the traffic recording module monitors the CPU usage rate and network traffic of the edge gateway. When the CPU usage rate is low and the network traffic is small, the traffic recording module uploads the recording data stored locally to the traffic management platform and deletes it from the local storage after successful upload.

[0034] Beneficial effects

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] The present invention records traffic on the edge gateway side instead of the Internet of Things device management platform side, so it does not need to care about the communication protocol between the edge gateway and the platform, and at the same time does not impose an additional burden on the Internet of Things platform. It can effectively avoid the risk of traffic recording on the self-owned device management platform side and reduce the workload of adapting to various communication protocols that may be required. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is the system network architecture diagram of the present invention;

[0038] Figure 2 is the structural schematic diagram of the edge gateway in the present invention;

[0039] Figure 3 is the schematic diagram of the traffic recording process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The present invention will be further described below with reference to specific embodiments in the drawings.

[0041] Refer to Figures 1 to 3 , a traffic recording system for Internet of Things testing, including an edge gateway, a self-owned device management platform, and a traffic management platform.

[0042] The edge gateway is an embedded device with computing and storage capabilities and can deploy application programs.

[0043] One end of the edge gateway is connected to the Internet of Things device, communicates directly with the Internet of Things device, receives various data of the Internet of Things device, including various instructions, and sends messages to the corresponding Internet of Things device for execution.

[0044] The other end of the edge gateway is respectively connected to the self-owned device management platform and the traffic management platform.

[0045] After screening and processing the data of the Internet of Things device, the edge gateway uploads it to the self-owned device management platform according to the data type.

[0046] The edge gateway receives instructions and rules from the traffic management platform. After receiving the data of the Internet of Things device, before uploading the data to the self-owned device management platform, it records the data to be sent according to the received instructions and rules and temporarily stores it locally, and uploads the temporarily stored recorded data to the traffic management platform on the premise of not affecting the normal communication with the self-owned device management platform.

[0047] The traffic management platform is used to send instructions to the edge gateway to control the switch, frequency of the edge gateway traffic recording, as well as the screening conditions and recording rules for data. The traffic management platform receives the recorded data sent by the edge gateway, and after cleaning the recorded data, stores it in the database for regression testing or stress testing of the system under test.

[0048] The self-owned device management platform receives device instructions from the edge gateway and conducts unified scheduling management.

[0049] Preferably, the IoT devices in the network are divided into various device groups according to service attributes, such as being divided into various device groups according to regions or functions. Each group of devices is assigned an edge gateway, and the IoT devices in the same device group are directly connected to this edge gateway to reduce the number of deployed edge gateways and lower costs.

[0050] Each IoT device sends status changes, exception alarms, and heartbeat information to the edge gateway it is connected to in the form of instructions according to its own operating conditions, and can also transmit data to this edge gateway.

[0051] In this embodiment, the edge gateway includes a traffic recording module, a data receiving module, a data uploading module, and a storage module. Among them, the traffic recording module includes a traffic recording control unit, a data filtering and storage unit, a resource monitoring unit, and a recorded traffic uploading unit.

[0052] The traffic recording control unit is used to control the start time of traffic recording, receive the recording policy sent by the traffic management platform. The recording policy includes recording frequency, recording period, and data filtering rules, and sends a recording signal to the data uploading module according to the recording policy, so that the data uploading module copies a portion of the recording data to be sent within a period of time and sends it to the data filtering and storage unit.

[0053] The data filtering and storage unit is used to filter the copied data sent by the data uploading module according to the data filtering rules sent by the traffic management platform, and store the filtered data in the storage module of the edge gateway.

[0054] The resource monitoring unit is used to monitor the usage of the hardware resources of the edge gateway, including CPU usage rate, network traffic, and the remaining storage capacity of the storage module. When there is free space in the storage module, it sends a recordable signal to the traffic recording control unit, otherwise it sends a non-recordable signal. When the CPU usage rate is low and the network traffic is small, such as when the CPU usage rate is less than 30% and the network traffic is less than 40%, it sends an uploadable signal to the recorded traffic uploading unit, enabling the recorded traffic uploading unit to upload the recording data stored in the storage module to the traffic management platform.

[0055] The recorded traffic uploading unit is used to send the recording data stored in the storage module. After receiving the uploadable signal sent by the resource monitoring unit, it retrieves the recording data from the storage module and uploads it in small batches multiple times to reduce the single upload time, thereby reducing the continuous occupation time of the CPU, and deleting the uploaded recording data in the storage module after each upload is completed to promptly free up the storage space of the storage module.

[0056] The data receiving module is used to communicate with the Internet of Things devices, receive the data uploaded by the Internet of Things devices, and screen the data before sending it to the data uploading module.

[0057] The data uploading module is used to upload the data sent by the data receiving module to the self-owned device management platform; and after receiving the recording signal sent by the traffic recording control unit, a new recording thread is started to copy a copy of the data to be uploaded to the self-owned device management platform and send it to the data filtering and storage unit, while the original data continues to be uploaded to the self-owned device management platform without being affected. Among them, the recording signal carries a recording duration parameter, and the recording thread is automatically closed after exceeding the duration parameter, or the recording thread is closed after receiving the stop recording signal.

[0058] The storage module is used for data storage, and can store the copied recorded data and the data uploaded by the Internet of Things devices.

[0059] A traffic recording method for Internet of Things testing, including the following steps:

[0060] Step 1, the edge gateway normally receives the data uploaded by the device group. Specifically, the data receiving module of the edge gateway normally receives the data uploaded by the device group;

[0061] Step 2, the edge gateway turns on the traffic recording switch according to the instruction received from the traffic management platform;

[0062] Step 3, after the edge gateway detects that the traffic recording switch is turned on, it copies a copy of the data of the Internet of Things device and sends it to the traffic recording module. Specifically, the data uploading module copies the data and sends it to the data filtering and storage unit, while the original data continues to be uploaded to the self-owned device management platform through the data uploading module;

[0063] Step 4, the resource monitoring unit of the traffic recording module monitors the storage resource usage of the edge gateway. When the storage resources are sufficient, the copied data is filtered according to the adoption frequency and data filtering rules in the instruction received from the traffic management platform and temporarily stored locally, controlling the upper limit of the local storage of the traffic recording, that is, controlling the margin of the storage module. When the upper limit is exceeded, the traffic recording switch is turned off;

[0064] Step 5, the resource monitoring unit of the traffic recording module monitors the CPU usage rate and network traffic of the edge gateway. When the CPU usage rate is low and the network traffic is small, the recording traffic uploading unit of the traffic recording module uploads the recorded data stored locally to the traffic management platform, and deletes it from the local storage after the upload is successful.

[0065] The above are only the preferred embodiments of the present invention. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several modifications and improvements can be made, which will not affect the implementation effect of the present invention and the practicability of the patent.

Claims

1. A traffic recording system for Internet of Things testing, characterized in that It includes an edge gateway, an in-house device management platform, and a traffic management platform; One end of the edge gateway is connected to IoT devices, directly communicating with IoT devices, receiving various data of IoT devices, and sending messages to the corresponding IoT devices for execution; The other end of the edge gateway is respectively connected to the in-house device management platform and the traffic management platform; After screening and processing the data of IoT devices, the edge gateway uploads the data to the in-house device management platform according to the data type; The edge gateway receives instructions and rules from the traffic management platform. After receiving the data of IoT devices and before uploading the data to the in-house device management platform, it records the data to be sent according to the received instructions and rules and temporarily stores it locally, and uploads the temporarily stored recorded data to the traffic management platform on the premise of not affecting the normal communication with the in-house device management platform; The traffic management platform receives the recorded data sent by the edge gateway, cleans the recorded data, and stores it in the database for regression testing or stress testing of the system under test; The in-house device management platform receives device instructions from the edge gateway and conducts unified scheduling and management.

2. The traffic recording system for Internet of Things testing according to claim 1, wherein According to the business attributes, the IoT devices in the network are divided into various device groups, and each group of devices is assigned an edge gateway. The IoT devices in the same device group are directly connected to the edge gateway.

3. The traffic recording system for Internet of Things testing according to claim 2, characterized in that According to their own operating conditions, each IoT device sends status changes, abnormal alarms, and heartbeat information to the edge gateway to which it is connected in the form of instructions, and can also transmit data to the edge gateway.

4. A traffic recording system for Internet of Things testing according to claim 1, characterized in that, The edge gateway includes a traffic recording module, a data receiving module, a data uploading module, and a storage module. Among them, the traffic recording module includes a traffic recording control unit, a data filtering and storage unit, a resource monitoring unit, and a recorded traffic uploading unit; The traffic recording control unit is used to control the start time of traffic recording, receive the recording strategy sent by the traffic management platform, and send a recording signal to the data uploading module according to the recording strategy, so that the data uploading module copies a portion of the recorded data to be sent within a period of time and sends it to the data filtering and storage unit; The data filtering and storage unit is used to filter the copied data sent by the data uploading module according to the data filtering rules sent by the traffic management platform, and store the filtered data in the storage module of the edge gateway; The resource monitoring unit is used to monitor the usage of the hardware resources of the edge gateway, including CPU usage rate, network traffic, and the remaining storage capacity of the storage module; when there is free space in the storage module, it sends a recordable signal to the traffic recording control unit, otherwise it sends a non-recordable signal; when the CPU usage rate is low and the network traffic is small, it sends an uploadable signal to the recorded traffic uploading unit, enabling the recorded traffic uploading unit to upload the recorded data stored in the storage module to the traffic management platform; The recorded traffic uploading unit is used to send the recorded data stored in the storage module. After receiving the uploadable signal sent by the resource monitoring unit, it retrieves the recorded data from the storage module, uploads it in small batches multiple times, and deletes the uploaded recorded data in the storage module after each upload is completed; The data receiving module is used to communicate with Internet of Things devices, receive the data uploaded by the Internet of Things devices, and screen the data and then send it to the data uploading module; The data uploading module is used to upload the data sent by the data receiving module to the self-owned device management platform; and after receiving the recording signal sent by the traffic recording control unit, a new recording thread is started to copy a copy of the data to be uploaded to the self-owned device management platform and send it to the data filtering and storage unit, and the source data continues to be uploaded to the self-owned device management platform without being affected; The storage module is used for data storage and can store the copied recording data and the data uploaded by the Internet of Things devices.

5. The traffic recording system for Internet of Things testing according to claim 4, characterized in that, The recording policy includes recording frequency, recording period, and data filtering rules.

6. The traffic recording system for Internet of Things testing according to claim 4, wherein The recording signal carries a recording duration parameter. After exceeding the duration parameter, the recording thread is automatically closed, or the recording thread is closed after receiving the stop recording signal.

7. A traffic recording method for Internet of Things testing, characterized in that, It includes the following steps: Step 1: The edge gateway normally receives the data uploaded by the device group; Step 2: The edge gateway turns on the traffic recording switch according to the instruction received from the traffic management platform; Step 3: After the edge gateway detects that the traffic recording switch is turned on, it copies a copy of the data of the Internet of Things device and sends it to the traffic recording module, while the source data continues to be uploaded to the self-owned device management platform; Step 4: The traffic recording module monitors the usage of the storage resources of the edge gateway. When the storage resources are sufficient, it filters the copied data according to the adopted frequency and data filtering rules in the instruction received from the traffic management platform and temporarily stores it locally, and controls the upper limit of the local storage of the traffic recording. When the upper limit is exceeded, the traffic recording switch is turned off; Step 5: The traffic recording module monitors the CPU usage rate and network traffic of the edge gateway. When the CPU usage rate is low and the network traffic is small, the traffic recording module uploads the recording data stored locally to the traffic management platform and deletes it from the local storage after the upload is successful.

Images