A high-concurrency data acquisition and processing method and device and a storage medium

Abstract

The application discloses a high-concurrency data collection and processing method and device and a storage medium, and the method comprises the following steps: after a client establishes a communication connection with a cloud server, the client sends device system data to the cloud server at a time interval a and saves or updates the device system data in a cache database; according to a time period t, the device system data is read from the cache database, the device state table in a structured database is updated according to a device ID primary key index mode, and the read device system data is inserted into a device increment table according to an incremental serial number index mode; the device system data of the previous p days in the structured database device increment table is inquired at a fixed time, the device system data is backed up to a backup database, and the record of the corresponding device system data in the structured database device increment table is deleted. The cloud server data collection and processing method of the application organically combines an adaptable network, a cache and structured storage, so that the collection cost and related technical risks are reduced to the maximum extent.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition and processing, and in particular to a high-concurrency data acquisition and processing method, device and storage medium. Background Art

[0002] With the rapid development of the internet and cloud computing, more and more users are migrating their business systems to cloud services. Building a bridge for business operations through cloud platforms and systems, collecting and processing relevant data, is becoming increasingly important. Optimizing the resources, parameters, and functional structures of data collection and processing middleware in a timely manner, while navigating massive and highly concurrent business requests, is a crucial technical challenge.

[0003] Currently, when facing high-concurrency scenarios, the general operation method is mainly to increase server resources and deal with it through message queues.

[0004] For example, a Chinese patent document with publication number CN110868349A discloses a distributed high-concurrency data collection and processing method and system in an Internet of Things environment. The method includes: creating a process guide ServerBootstrap, initializing thread groups bossGroup and workerGroup of type NioEventLoopGroup, starting a data forwarder process guide of type Bootstrasp, constructing a processing class inherited from ChannelInboundHandlerAdapter, adding handlers Handler for different protocols, constructing a main interceptor, and using the data receiving guide constructed by the above steps to bind the process port to listen to data sent by the hardware device and parse the incoming message queue unit.

[0005] Chinese patent publication number CN111797150A discloses a method for high-concurrency data docking and forwarding, including: a big data receiving system receives data collected from different front-end devices through an NGINX server; the big data receiving system classifies and cleans the front-end data, placing different types of standard data into different queues in Kafka for subscription by the data forwarding system; when the big data platform receives a subscription request, the data forwarding system retrieves the subscription data from Kafka and sends it to the big data platform. The present invention uniformly processes and receives the massive amounts of data collected by the front-end devices. The system first organizes and categorizes the received data and stores it in the middleware Kafka, then retrieves it from the cache queue and forwards it to the upper-level big data platform.

[0006] In practical applications, increasing server resources presents challenges in estimating the matching of computing hardware resources. This lack of flexibility is common in non-high-concurrency scenarios, leading to significant resource waste. Using message queues has throughput limitations, and the first-in-first-out stack approach is relatively inefficient. Summary of the Invention

[0007] The present invention provides a high-concurrency data collection and processing method, which can minimize the high-concurrency data collection and processing costs and related technical risks.

[0008] The technical solutions of the present invention are as follows:

[0009] A high-concurrency data collection and processing method, comprising:

[0010] (1) After the client establishes a communication connection with the cloud server, it sends device system data to the cloud server at a time interval a. After the cloud server receives the device system data of the current client, if the device system data of the current client does not exist in the cache database, it saves it to the cache database; otherwise, it updates it to the cache database to overwrite the original device system data of the current client; in the cache database, if the device system data of a client has not been updated for more than T time, the device system data of the client is deleted;

[0011] (2) Read the device system data from the cache database according to the time period t, update the device status table in the structured database according to the device ID primary key index method based on the read device system data information, and insert the read device system data into the device increment table according to the ascending sequence number index method;

[0012] (3) Regularly query the device system data before the previous p days in the device incremental table of the structured database, and back up the device system data before the previous p days to the backup database, and delete the records of the corresponding device system data in the device incremental table of the structured database.

[0013] Preferably, the interval a for the client to send device system data to the cloud server, the time period t for reading device system data from the cache database, the maximum retention time T for a client's device system data in the cache database, the device system data packet length, and the public network bandwidth between the client and the cloud server are adjustable. In high-concurrency scenarios, these parameters can be adjusted at any time to minimize data collection and processing costs and related device access technology risks.

[0014] Preferably, t≥a; T≥t.

[0015] The values ​​of a, t, and T are adjusted according to the performance of the cache database and the system data of the networked client device. Preferably, a is 10-60 seconds; t is 10-60 seconds; and T is 5-30 minutes.

[0016] Most preferably, a is 20 seconds; t is 30 seconds; and T is 10 minutes.

[0017] The value of p is adjusted according to the performance of the structured database and the system data of the current networked equipment. Preferably, p is 30 days.

[0018] The present invention uses a cloud server data collection and processing method that organically combines an adaptable network, a high-speed cache, and a structured storage, thereby improving the concurrency of data collection and reducing the pressure on the read and write performance of the structured database, thereby minimizing the collection cost and related technical risks. The method of the present invention can be deployed and run with middleware and can be plug-and-play.

[0019] The client can be industrial control equipment, vehicle-mounted equipment, or media playback equipment; the communication link protocol between the client and the cloud server is WebSocket, MQTT, TCP / IP and other communication protocols.

[0020] Preferably, the device system data sent by the client to the cloud server is a system data packet in JSON format.

[0021] Preferably, after receiving the device system data, the cloud server saves it to the cache database in the form of Key-Value; the Key value is the device ID, and the Value value is the device system data string to save it to the cache database.

[0022] Preferably, step (2) comprises:

[0023] The transfer timer counts according to the time period t:

[0024] At the nt time point, the device system data is read from the cache database using the device ID as the key value, and the device status table in the structured database is updated according to the read device system data and the device ID primary key index method;

[0025] At the mt time point, the device system data is read from the cache database using the device ID as the key value, and the read device system data is inserted into the device increment table according to the ascending sequence number index method;

[0026] m and n are integers greater than or equal to 1, and m>n.

[0027] Most preferably, m=4 and n=2.

[0028] The device status table is used to record the latest device status of the client, and the device increment table is used to store the device system data of the client's MT time period.

[0029] Preferably, step (3) comprises:

[0030] (3-1) The backup timer is used to periodically query the device system data records of the device increment table in the structured database for the past p days or more;

[0031] (3-2) Back up the device system data for the past p days or more to the backup database;

[0032] (3-2) Delete the equipment system data records in the equipment increment table of the structured database that are older than p days ago.

[0033] More preferably, data backup is performed regularly every day, and data backup can be performed at a certain time point between 00:00 and 5:00 every day.

[0034] Preferably, step (3-2) includes: converting the device system data for the previous p days or more into CVS table files and cache list data according to the daily time dimension; backing up the CVS table files to the object storage system, and backing up the cache list data to the backup database. This achieves dual backup to increase backup reliability.

[0035] The present invention also provides a high-concurrency data acquisition and processing device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the device is configured to implement the following steps:

[0036] (1) After the client establishes a communication connection with the cloud server, it sends device system data to the cloud server at a certain time interval a. After the cloud server receives the device system data of the current client, if the device system data of the current client does not exist in the cache database, it saves it to the cache database; otherwise, it updates it to the cache database to overwrite the original device system data of the current client; in the cache database, if the device system data of a client has not been updated for more than T time, the device system data of the client is deleted;

[0037] (2) Read the device system data from the cache database according to the time period t, update the device status table in the structured database according to the device ID primary key index method based on the read device system data information, and insert the read device system data into the device increment table according to the ascending sequence number index method;

[0038] (3) Regularly query the device system data of the structured database device incremental table for the past p days, and back up the device system data of the structured database for the past p days to the backup database, and delete the records of the corresponding device system data in the structured database device incremental table.

[0039] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the following steps:

[0040] (1) After the client establishes a communication connection with the cloud server, it sends device system data to the cloud server at a time interval a. After the cloud server receives the device system data of the current client, if the device system data of the current client does not exist in the cache database, it saves it to the cache database; otherwise, it updates it to the cache database to overwrite the original device system data of the current client; in the cache database, if the device system data of a client has not been updated for more than T time, the device system data of the client is deleted;

[0041] (2) Read the device system data from the cache database at a certain time period t, update the device status table in the structured database according to the read device system data information using the device ID primary key index method, and insert the read device system data into the device increment table using the ascending sequence number index method;

[0042] (3) Regularly query the device system data of the structured database device incremental table for the past p days, and back up the device system data of the structured database for the past p days to the backup database, and delete the records of the corresponding device system data in the structured database device incremental table.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The method of the present invention uploads the client's device system data to the cloud server in real time, automatically processes the data without perception according to the network status, and can adjust relevant parameters in a timely manner under high concurrency conditions, so as to minimize the data collection and processing costs and the risks of related device access technology;

[0045] (2) The method of the present invention can satisfy both the performance expansion of a single server and the elastic scaling of clustered servers during the deployment and operation of the middleware;

[0046] (3) The extended function of the method of the present invention can provide users with multi-channel data output services, that is, the processed data can be provided to storage centers such as distributed file systems and data lakes in an internal channel manner;

[0047] (4) The method of the present invention organically integrates the high-speed cache database with the medium-performance structured database, which not only improves the concurrency capability of data collection, but also reduces the read and write performance pressure of the structured database. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flowchart of a high-concurrency data collection and processing method provided by one embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a flow chart of data transfer processing steps provided in one embodiment of the present invention;

[0051] Figure 3 A flowchart of data backup processing steps provided by one embodiment of the present invention;

[0052] Figure 4 This is a structural block diagram of a high-concurrency data acquisition and processing device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0055] The present invention can reduce the collection cost and related technical risks to the greatest extent by establishing a cloud service data collection and processing model that organically combines adaptive networks, caches and structured storage. The method of the present invention can be deployed and run by middleware and achieve plug-and-play.

[0056] The application scenarios and purposes of the present invention are mainly reflected in the following aspects: (1) The system data of the networked equipment is uploaded to the cloud service end in real time through a unified JSON format. The method can automatically process the data imperceptibly according to the network status and can adjust the relevant parameters in time under high concurrency conditions to minimize the data collection and processing costs and the technical risks of related equipment access; (2) The method can also be deployed and operated in a clustered manner in a middleware manner; (3) The extended function of the method can provide users with multi-channel data output services, that is, the processed data can be provided to storage centers such as distributed file systems and data lakes in an internal channel manner.

[0057] The high-concurrency data collection and processing method of the present invention is that after the network-connected device (such as industrial control equipment, vehicle-mounted equipment, media playback equipment, etc.) establishes a communication link with the cloud server (supporting communication protocols such as WebSocket, MQTT, TCP / IP, etc.), the network-connected device sends the device system data to the cloud server at a certain time period. After receiving the data, the cloud server parses the device system data and stores it in the cache database according to the device ID key value mode. The cache data is parsed using an adaptable timer model and transferred to the device status table and device increment table in the structured database. Then, the device increment table of the structured database is monitored in real time, and the device increment table is backed up to a medium or low-performance structured database system at a certain time period, thereby realizing high-concurrency data collection and processing. Its workflow is as follows Figure 1 shown.

[0058] The high-concurrency data collection and processing method of the present invention comprises the following steps:

[0059] 1. Data collection steps

[0060] The working process of the data collection step includes Figure 1 The specific process of ①②③④ is as follows:

[0061] (1) The networked device (such as industrial control equipment, vehicle-mounted equipment, media playback equipment, etc.) acts as a communication client and actively requests to establish a communication link with the cloud server (supporting communication protocols such as WebSocket, MQTT, TCP / IP, etc.). After the connection is successful, the client will send device system data to the cloud server at a certain interval;

[0062] (2) Each networked device sends a JSON format system data packet with a maximum length of b (unit: bytes) at an interval of a (unit: seconds). The JSON format system data packet contains at least the following fields: device unique identifier (devId), device type (type), longitude (lng), latitude (lat), device system information (devSysInfo), device application information (devAppInfo) and message unique identifier (msgId), as shown in Table 1.

[0063] Table 1 Information contained in the JSON format system data packet

[0064]

[0065] (3) The cloud server saves or updates the received JSON format system data packet in the form of Key-Value (Key is the device ID, Value is the JSON format system data packet string) to a cache database with a read / write performance of q (unit: times / second) and a bandwidth of c (unit: Mbps, i.e., megabits per second). In the present invention, q = 200000 and c = 1536;

[0066] (4) The public network bandwidth from the network device to the cloud server is d (unit: Mbps, i.e., megabits per second), and the overhead of each network device connecting to the cloud server is k (1≤k≤5). The number of network devices that can be connected is:

[0067] Since c>d, then:

[0068] In the present invention, d=50, a=20, b=1024, and k=4, and the number of concurrent network access devices supported is:

[0069]

[0070] (5) The device system data packets in the cache database are transferred to the device status table and the device increment table in the structured database at time periods of 2t and 4t (unit: seconds), respectively. In the present invention, t=30.

[0071] 2. Data transfer processing steps

[0072] The working process of the data transfer processing step includes Figure 1 In ⑤⑥, the data transfer process is as follows Figure 2 As shown, the device system data in the structured database is written and updated mainly according to a certain period. The specific process is as follows:

[0073] (1) The transfer timer reads the device system data from the cache database using the device ID as the key value according to a certain time period t (unit: seconds), where t≥a, that is, the transfer timer time period is not less than the interval length of sending the network device system data, and in the present invention, t=30;

[0074] (2) The read device system data is saved in a structured database, and the device system data is updated into the device status table according to the device ID primary key index method. The device status table information is shown in Table 2. The update interval nt (n≥1) is set. In the present invention, n=2 is taken according to the performance of the structured database.

[0075] Table 2 Equipment status table

[0076]

[0077]

[0078]

[0079] (3) Save the read device system data into the structured database, and save the device system data into the device increment table (as shown in Table 3) according to the ascending sequence index method, and set the saving interval time mt (m>n). In the present invention, m=4 is taken according to the performance of the structured database.

[0080] Table 3 Equipment increment table

[0081]

[0082]

[0083] 3. Data backup processing steps

[0084] The working process of the data backup processing steps includes Figure 1 In ⑦⑧, the data backup process is as follows Figure 3 As shown in the figure, data migration of the device incremental table in the structured database is mainly achieved according to a certain period. The specific process is as follows:

[0085] (1) Start the backup timer according to the daily scheduled cycle (e.g., 02:00:00 every day, or a time point from 0 to 5 o'clock);

[0086] (2) The device system data for the previous p days or more is queried and output according to the daily time dimension, and the record data set is merged and converted into a CVS table file and cache list data;

[0087] (3) Upload the CVS table file of the device system data based on the daily time dimension for more than p days to the object storage system;

[0088] (4) Save the device system cache list data based on the daily time dimension for more than p days in the past to the backup database, thereby achieving double backup to increase backup reliability;

[0089] (5) Delete the equipment system data records of more than p days in the equipment increment table of the structured database according to the day time dimension;

[0090] (6) The backup timer ends this operation, wherein the value of p is adjusted according to the performance of the structured database and the current networked device system data. In the present invention, p=30. The better the performance of the structured database, the larger the value of p, and vice versa.

[0091] 4. Data Acquisition and Processing Process

[0092] (1) The data acquisition module realizes the functions of receiving, parsing, and saving the system data of networked devices into the cache database, corresponding to Figure 1 ①②③④ in

[0093] (2) The data transfer and storage processing module realizes the update of the device status table and the insertion of the device increment table into the structured database for the cache data at a certain period t, corresponding to Figure 1 ⑤⑥ in Figure 2 ;

[0094] (3) The data backup processing module realizes the daily timed processing of the device system data that exceeds a certain threshold p days to relieve the performance pressure of the structured database, corresponding to Figure 1 ⑦⑧ in Figure 3 .

[0095] Embodiment

[0096] The key adjustable parameters involved in the method of the present invention are the public network bandwidth d (unit: Mbps), the transfer storage timer time period t (unit: seconds), the system data sending interval time a (unit: seconds) of networked devices, and the maximum value b (unit: bytes) of the system data packet length of networked devices.

[0097] In one embodiment, the parameter settings for medium-scale (about 30,000 networked devices) data acquisition are d = 50, t = 30, a = 20, and b = 1024.

[0098] According to the above parameter settings, the method of the present invention was used to conduct an access experiment on a server with an 8-core CPU, 32G of memory, 500G of hard disk, and a public network bandwidth of 50 Mbps for 6000 devices (the system data interval of each device is 20 seconds). When 5000 devices were online simultaneously at the peak period, the CPU utilization rate of the server was 14%, and the memory utilization rate was 31%. It is estimated that the server can support 30,000 networked devices when fully loaded (CPU does not exceed 75%, memory does not exceed 70%) (note: the memory does not increase in multiples of a certain device quantity), which is close to the theoretically calculated concurrent device number (32000) in the data acquisition model;

[0099] During the deployment and operation of the present invention method with middleware, it can not only meet the performance expansion of a single server but also meet the elastic scaling of the functions of a clustered server;

[0100] When a small number of networked devices (less than 3,000) are connected, a server with a dual-core CPU, 8GB of memory, a 500GB hard drive, and a public network bandwidth of 5Mbps can also meet the data collection and processing needs of the device system.

[0101] The method of the present invention can adjust relevant parameters at any time under high concurrency conditions to minimize data collection and processing costs and related equipment access technology risks;

[0102] The method of the present invention can be integrated into third-party software to provide customized middleware services for users.

[0103] The method of the present invention organically integrates a high-speed cache database with a medium-performance structured database, thereby improving the concurrent capability of data collection and reducing the pressure on the read and write performance of the structured database.

[0104] The present invention also provides a computer-readable storage medium (including but not limited to a disk storage, CD-ROM, optical storage, etc.), which stores a computer program that can be used to execute the above Figure 1 A high-concurrency data collection and processing method is provided.

[0105] The present invention also provides Figure 4 The one shown corresponds to Figure 1 Schematic diagram of the high-concurrency data acquisition and processing device. Figure 4 As described above, at the hardware level, the high-concurrency data acquisition and processing device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned high-concurrency data acquisition and processing method. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0106] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-concurrency data collection and processing method, characterized in that: include: (1) After the client establishes a communication connection with the cloud server, a Send device system data to the cloud server. After the cloud server receives the device system data of the current client, if the device system data of the current client does not exist in the cache database, it will save it to the cache database. Otherwise, it will update it to the cache database to overwrite the original device system data of the current client. In the cache database, if the device system data of a client exceeds T If the time is not updated, the device system data of the client will be deleted; (2) According to the time period t , read the device system data from the cache database, update the device status table in the structured database according to the device ID primary key index method based on the read device system data information, and insert the read device system data into the device increment table according to the ascending sequence number index method, including: Dump timer according to time period t To time: in nt At the time point, the device system data is read from the cache database using the device ID as the key value, and the device status table in the structured database is updated according to the read device system data and the device ID primary key index method; in mt At a certain time point, the device system data is read from the cache database using the device ID as the key value, and the read device system data is inserted into the device increment table according to the ascending sequence index method; m、n An integer greater than or equal to 1 ，m > n ; (3) Regularly query the previous table of the structured database device increment p Device system data for more than 2 days, and p The device system data older than 1 day is backed up to the backup database, and the records of the corresponding device system data in the device incremental table of the structured database are deleted.

2. The high-concurrency data collection and processing method according to claim 1, characterized in that: The time interval for the client to send device system data to the cloud server a , the time period for reading device system data from the cache database t , The maximum storage time of a client's device system data in the cache database T , the device system data packet length and the public network bandwidth between the client and the cloud server are adjustable.

3. The high-concurrency data collection and processing method according to claim 1, characterized in that: The client is an industrial control device, a vehicle-mounted device or a media playback device; the communication protocol between the client and the cloud server is WebSocket, MQTT or TCP / IP.

4. The high-concurrency data collection and processing method according to claim 1, characterized in that: The device system data sent by the client to the cloud server is a system data packet in JSON format.

5. The high-concurrency data collection and processing method according to claim 1, characterized in that: After receiving the device system data, the cloud server saves it to the cache database in the form of key-value; The Key value is the device ID, and the Value value is the device system data string, which is saved in the cache database.

6. The high-concurrency data collection and processing method according to claim 1, characterized in that: Step (3) includes: (3-1) The backup timer is used to time and query the previous data in the incremental table of the structured database device regularly. p Equipment system data records for more than 10 days; (3-2) Moving forward p Back up the equipment system data of more than 1 day to the backup database; (3-2) Delete the previous table in the structured database device increment table p Equipment system data records for more than 1 day.

7. The high-concurrency data collection and processing method according to claim 6, characterized in that: Step (3-2) includes: p Device system data older than 1 day is converted into CVS table files and cache list data based on the daily time dimension; the CVS table files are backed up to the object storage system, and the cache list data is backed up to the backup database.

8. A high-concurrency data acquisition and processing device, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the steps of the high-concurrency data collection and processing method as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the steps of the high-concurrency data acquisition and processing method as claimed in any one of claims 1 to 7.

Images