Data acquisition method, device, platform, equipment, storage medium and product

Abstract

The invention relates to a data acquisition method and device, a platform, equipment, a storage medium and a product. The method comprises the following steps: receiving a service data packet from a gateway; the service data packet comprises a first-level data identifier, and the first-level data identifier is used for indicating a first-level data type of the service data packet; pushing the service data packet into a cache queue corresponding to the first-level data type of the service data packet; performing data processing on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined through the first-level data type. By the adoption of the method, priority setting can be adopted for multiple types of data, and the real-time performance of key data collection is improved on the premise that computing resources and network resources are not increased.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data collection method, device, platform, equipment, storage medium and product. Background Art

[0002] With the rapid development of digital home, the number of devices connected to home gateway is increasing day by day, and home gateway is gradually becoming the central hub of digital home. A large amount of business data from different sources and types is aggregated through home gateway and uploaded to data collection server.

[0003] In related technologies, data collection servers process all reported data in sequence according to a fixed process. Different types of data share the same computing and network resources, which may cause key data to be delayed due to resource competition. When the amount of data increases suddenly or network congestion occurs, the reception and processing of all data will be affected, resulting in reduced system stability and reliability. Summary of the invention

[0004] Based on this, it is necessary to provide a data collection method, device, platform, equipment, storage medium and product that can improve stability and reliability in response to the above technical problems.

[0005] In a first aspect, the present application provides a data collection method. The method comprises:

[0006] receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet;

[0007] Pushing the service data packet into a cache queue corresponding to the primary data type of the service data packet;

[0008] Data processing is performed on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0009] In one of the embodiments, the processing of the service data packets in the cache queue based on the priority level of the cache queue includes:

[0010] Determining a priority level of the cache queue;

[0011] The queue processing order of the cache queue is determined based on the priority level of the cache queue, and data processing is performed on the service data packets of the cache queue based on the queue processing order; wherein the queue processing order of the cache queue with a high priority level is higher than the queue processing order of the cache queue with a low priority level.

[0012] In one of the embodiments, the processing of the service data packets of the cache queue based on the queue processing order includes:

[0013] determining a target cache queue from a plurality of cache queues based on the queue processing order;

[0014] Acquire the secondary data type of the service data packet in the target cache queue; the secondary data type is determined by the secondary data identifier of the service data packet;

[0015] Based on the secondary data type, a time slice round-robin algorithm is used to push data corresponding to the service data packet of the target cache queue into the database.

[0016] In one embodiment, the method further comprises:

[0017] Get the type number of the first-level data type corresponding to the highest priority level;

[0018] Calling the same number of servers as the type to process the service data packets in the cache queue in parallel;

[0019] Among them, business data packets of the same primary data type are sent to the same server for data processing.

[0020] In one embodiment, before receiving the service data packet from the gateway, the process includes:

[0021] Determine the access addresses corresponding to different geographical areas;

[0022] The mapping relationship between the geographical area and the access address is synchronized to the DNS server, so that when the DNS server receives the access request of the gateway, it sends the access address corresponding to the geographical area where the gateway is located to the gateway.

[0023] In a second aspect, the present application also provides a data collection platform, which includes:

[0024] The data decompression and decryption module is used to: receive a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the service data packet; push the service data packet into a cache queue corresponding to the primary data type of the service data packet;

[0025] A data analysis module is connected to the data decompression and decryption module and is used to process the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0026] In a third aspect, the present application also provides a data acquisition device. The device comprises:

[0027] A receiving module, used for receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used for indicating the primary data type of the service data packet;

[0028] A matching module, used to determine a target priority matching the service data packet based on a target matching rule and the primary data identifier;

[0029] The queue module is used to push the service data packet into a cache queue corresponding to the target priority; wherein the priority level of the service data packet and the processing order of the cache queue corresponding to the priority level are positively correlated.

[0030] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet;

[0032] Pushing the service data packet into a cache queue corresponding to the primary data type of the service data packet;

[0033] Data processing is performed on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0034] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0035] receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet;

[0036] Pushing the service data packet into a cache queue corresponding to the primary data type of the service data packet;

[0037] Data processing is performed on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0038] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet;

[0040] Pushing the service data packet into a cache queue corresponding to the primary data type of the service data packet;

[0041] Data processing is performed on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0042] The above-mentioned data collection method, device, platform, equipment, storage medium and product receive a business data packet from a gateway; the business data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the business data packet; the business data packet is pushed into a cache queue corresponding to the primary data type of the business data packet; data processing is performed on the business data packet in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type. The data collection method based on a home gateway or other Internet of Things terminal provided in the present application divides the business data packet into different primary data types through a primary data identifier, and pushes the business data packets of different primary data types into different cache queues, and marks the priority level according to the primary data type, so that data processing can be performed on the business data packets in different cache queues based on the priority level of the cache queue, thereby improving the performance and scalability of data collection, taking priority settings for multiple types of data, and improving the real-time performance of key data collection without increasing computing resources and network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is an application environment diagram of a data collection method in one embodiment;

[0044] FIG2 is a schematic diagram of a flow chart of a data collection method in one embodiment;

[0045] FIG3 is a schematic diagram of a cache queue in one embodiment;

[0046] FIG4 is a schematic diagram of a flow chart of data processing of service data packets in a cache queue in one embodiment;

[0047] FIG5 is a schematic diagram of a process for processing data based on a queue processing order in one embodiment;

[0048] FIG6 is a schematic flow chart of a data collection method in another embodiment;

[0049] FIG7 is a schematic flow chart of a data collection method in another embodiment;

[0050] FIG8 is a block diagram of a data acquisition platform in one embodiment;

[0051] FIG9 is a schematic diagram of a process of uploading gateway data partitions to different public IP addresses in one embodiment;

[0052] FIG10 is a block diagram of a data acquisition device in one embodiment;

[0053] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] Home gateway: A network access device installed in a user's home, usually a fiber optic modem (optical modem). It is a bridge that enables communication between the user's local network and the Internet. Its main functions include photoelectric conversion, routing, network management, and expansion functions, such as converting fiber optic signals into Ethernet signals, allocating IP addresses required for devices in the user's home to access the Internet, providing user authentication, bandwidth management, and network security functions. Some high-end optical modems also support Wi-Fi, voice calls (VoIP), smart home access, and other functions.

[0056] IP address database: a database that matches IP addresses with geographical information.

[0057] Multi-level feedback queue algorithm: It is a queue-based process scheduling method that uses multiple queues to organize processes and assigns different priorities to each queue. These queues are arranged according to different priorities, usually higher priority queues have shorter time slices, while lower priority queues have longer time slices.

[0058] Since the types of business data reported by home gateways are extremely complex, in related technologies, big data collection platforms usually use a unified method to receive data from all home gateways: first, after the data collection platform collects the data, it decompresses and decrypts the data and stores it in the corresponding business data table according to the business type fields carried in it. In this process, if network congestion or server processing delay occurs, the reception and processing process of all data will be affected, and the real-time and reliability of key data will be difficult to guarantee. This may not only affect the normal operation of the business system, but also cause delays in key tasks, reduce the overall system efficiency and user experience, and reduce the stability and reliability of the system.

[0059] The data collection method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, multiple home gateways 102 communicate with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the home gateway 102 is a network access device installed in the user's home. The home gateway 102 is used to communicate with the terminal device hanging under the gateway. The terminal device can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart car device, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0060] In one embodiment, as shown in FIG. 2 , a data collection method is provided, which is described by taking the method applied to the server 104 in FIG. 1 as an example, and includes the following steps:

[0061] Step 202: Receive a service data packet from a gateway, where the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet.

[0062] The primary data identifier may be a field of a service data packet, usually located in a message header, and is unique or predefined. Specifically, the primary data identifier may be used to characterize a gateway plug-in. The primary data type is used to indicate the data type generated by a plug-in in the gateway corresponding to the primary data identifier, which may represent the business scenario to which the plug-in is applied.

[0063] In the embodiment of the present application, the primary data identifier, the primary data type and the gateway plug-in may have a one-to-one correspondence. There are multiple plug-ins in the gateway, and each plug-in is responsible for different functions, such as device data collection, protocol conversion, security authentication, etc. The role of the primary data identifier is to associate the business data with a specific plug-in, indicating the data type collected by the source plug-in of the business data packet.

[0064] Exemplarily, the first-level data identifier "0xff01" corresponds to plug-in A, the first-level data identifier "0xff02" corresponds to plug-in B, and the first-level data identifier "0xff03" corresponds to plug-in C. Plug-in A can be, for example, a gateway diagnostic plug-in, and its business scenario or the first-level data type A corresponding to the gateway diagnostic plug-in can represent data related to the gateway quality difference label business; plug-in B can be a B01 plug-in, and its business scenario or the first-level data type B corresponding to the B01 plug-in can represent data related to the user subscription business; plug-in C can be a log collection plug-in, and its business scenario or the first-level data type C corresponding to the log collection plug-in can represent log-related data.

[0065] Step 204: Push the service data packet into a cache queue corresponding to the primary data type of the service data packet.

[0066] In the embodiment of the present application, service data packets of different primary data types are pushed into different cache queues according to the primary data type, and service data packets of the same primary data type are pushed into the same cache queue.

[0067] Exemplarily, the server receives various types of data from the gateway, first decompresses and decrypts each business data packet, then determines the primary data type based on the primary data identifier in the message header, and pushes the business data packet into the cache queue corresponding to the primary data type based on the primary data type.

[0068] For example, the service data packets corresponding to the primary data type A are pushed into the first cache queue, the service data packets corresponding to the primary data type B are pushed into the second cache queue, and the service data packets corresponding to the primary data type C are pushed into the third cache queue.

[0069] Step 206: Process the service data packets in the cache queue based on the priority level of the cache queue.

[0070] Among them, the priority level of the cache queue is determined by the primary data type.

[0071] Exemplarily, the server pre-sets priority levels corresponding to different first-level data types. For example, the server can store the mapping relationship between the first-level data identifier, gateway plug-in, first-level data type and priority level as a mapping relationship table in the server. Based on the mapping relationship, the server determines the priority level that matches the first-level data type to which different business data packets belong. The priority level that matches the first-level data type corresponds to the priority level of the cache queue into which it is pushed. For example, the priority level of the cache queue entered by a high-priority first-level data type is also high priority.

[0072] For example, the server classifies the first-level data types (used to distinguish different plug-ins in the gateway) in the message header, for example, they can be classified from level 0 to level 5, and the levels from high to low are level 0, level 1, level 2, level 3, level 4, and level 5. The server presets the priority level of the first-level data type A to level 0, the priority level of the first-level data type B to level 0, the priority level of the first-level data type C to level 5, the priority level of the first-level data type D to level 4, and the priority level of the first-level data type E to level 3.

[0073] Please refer to Figure 3. After receiving the service data packet of the gateway, the server pushes the service data packet of the first-level data type A into the first cache queue, and the priority level of the first cache queue can be level 0; pushes the service data packet of the first-level data type B into the second cache queue, and the priority level of the second cache queue can be level 0; pushes the service data packet of the first-level data type C into the third cache queue, and the priority level of the third cache queue can be level 5; pushes the service data packet of the first-level data type D into the fourth cache queue, and the priority level of the fourth cache queue can be level 4; pushes the service data packet of the first-level data type E into the fifth cache queue, and the priority level of the fifth cache queue can be level 3.

[0074] The server of the embodiment of the present application reads and parses business data from different cache queues in order of priority based on the priority levels of the cache queues, and can give priority to processing high-priority first-level data types, such as giving priority to reading business data in the first cache queue and the second cache queue.

[0075] In the above data collection method, a service data packet is received from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the service data packet; the service data packet is pushed into a cache queue corresponding to the primary data type of the service data packet; data processing is performed on the service data packet in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type. The data collection method based on a home gateway or other IoT terminal provided in the present application divides the service data packet into different primary data types through a primary data identifier, and pushes the service data packets of different primary data types into different cache queues, and marks the priority level according to the primary data type, so that data processing can be performed on the service data packets in different cache queues based on the priority level of the cache queue, thereby improving the performance and scalability of data collection, taking priority settings for multiple types of data, and improving the real-time performance of key data collection without increasing computing resources and network resources.

[0076] In one embodiment, as shown in FIG. 4 , the step of performing data processing on the service data packets in the cache queue based on the priority level of the cache queue in the embodiment shown in FIG. 2 includes:

[0077] Step 402: Determine the priority level of the cache queue.

[0078] Step 404: determine a queue processing order of the cache queue based on the priority level of the cache queue, and perform data processing on the service data packets of the cache queue based on the queue processing order.

[0079] The queue processing order of the cache queue at a high priority level is higher than the queue processing order of the cache queue at a low priority level.

[0080] For example, when the server reads the service data in the cache queue, it first determines the priority level of each cache queue, and then determines the queue processing order corresponding to the different cache queues according to the priority level. The priority level of the cache queue and the queue processing order are positively correlated, that is, the higher the priority level, the higher the queue processing order, and the lower the priority level, the lower the queue processing order. For each cache queue that is not the highest priority, when the cache queues with higher priorities are empty, the data of the cache queue is processed.

[0081] For example, the server will process low-priority data only after processing high-priority data, and can only process lower-priority data when the high-priority queue is empty. Please continue to combine Figure 3, the server can first read the business data of the first-level data type A in the first cache queue and the business data of the first-level data type B in the second cache queue. When the first cache queue and the second cache queue are both empty, the server will read the business data of the first-level data type E in the fifth cache queue and the business data of the first-level data type D in the fourth cache queue in turn. When the first cache queue, the second cache queue, the fifth cache queue, and the fourth cache queue are all empty, the server can finally read the business data of the first-level data type C in the third cache queue.

[0082] In one embodiment, as shown in FIG. 5 , the embodiment shown in FIG. 4 performs data processing on the service data packets in the cache queue based on the queue processing order, including:

[0083] Step 502: determine a target cache queue from a plurality of cache queues based on a queue processing order.

[0084] The target cache queue refers to the cache queue that the server currently needs to read according to the queue processing order.

[0085] It is understandable that the target cache queue can be one or multiple cache queues with the same priority level. When the cache queue with the highest priority level is not empty, the target cache queue refers to the cache queue with the highest priority level, such as the first cache queue and the second cache queue with priority level 0.

[0086] Step 504: Obtain the secondary data type of the service data packet in the target cache queue.

[0087] The secondary data type is determined by the secondary data identifier of the business data packet.

[0088] The message body of the service data packet of the embodiment of the present application is provided with a secondary data identifier, which is used to indicate the secondary data type of the service data packet, and is used to distinguish different secondary data types of the same plug-in. For example, the secondary data type can represent different table file types in the database, which is used to push the data in the message body of the service data packet into the corresponding database table file according to the secondary data identifier.

[0089] Step 506: Based on the secondary data type, a time slice round-robin algorithm is used to push data corresponding to the service data packet in the target cache queue into the database.

[0090] Exemplarily, for each target cache queue, the message body of the same priority level is pushed into the corresponding table file of the data analysis platform according to the secondary data type (used to distinguish different data types of the same plug-in) based on the time slice rotation algorithm. Due to the use of the time slice rotation algorithm, the same thread will only open one cache file in the same time slice, thereby improving the write performance.

[0091] In one embodiment, as shown in FIG6 , the data collection method further includes:

[0092] Step 602, obtaining the type number of the primary data type corresponding to the highest priority level.

[0093] Step 604: Call the same number of servers as the type number to process the service data packets in the cache queue in parallel.

[0094] Among them, business data packets of the same primary data type are sent to the same server for data processing.

[0095] Exemplarily, the server of the embodiment of the present application obtains the number of types of the first-level data type at the highest priority level, and configures the server according to the number of types of the first-level data type at the highest priority level.

[0096] For example, the server may be configured with RPC (Remote Procedure Call). In the embodiment shown in FIG2 , steps 202 and 204 may be implemented using an RPC client, and step 206 may be implemented using an RPC server. If there are n first-level data types with the highest priority level at the same time, the RPC client and the RPC server may be configured as a pair of n. The RPC client sends data to different servers in sequence according to the priority of the first-level data type and ensures that the same first-level data type is sent to the same server. For example, the first-level data type with a priority level of 0 has two types, first-level data type A and B. The server is configured with an RPC client and two RPC servers. After the RPC client pushes the service data packet into the corresponding cache queue according to the corresponding first-level data type, one RPC server first reads the first cache queue with a priority of 0, and the other RPC server first reads the second cache queue with a priority of 0. The two RPC servers process in parallel.

[0097] At the same time, each RPC server is a process. When a server process processes data of the same primary data type, it will create different threads according to the different secondary data types. The data parsing and processing within the same process will adopt a time slice rotation method.

[0098] In one embodiment, as shown in FIG7 , before receiving a service data packet from a gateway, the process includes:

[0099] Step 702: Determine access addresses corresponding to different geographical areas.

[0100] Step 704: Synchronize the mapping relationship between the geographical area and the access address to the DNS server, so that when the DNS server receives the access request from the gateway, it sends the access address corresponding to the geographical area where the gateway is located to the gateway.

[0101] Due to the wide variety of business data, the huge amount of data, and the wide distribution of reported home gateways, the big data collection platform needs to group the data of home gateways according to different regions after obtaining it. The existing data collection method is prone to excessive server processing pressure through a single reporting channel, and the subsequent classification and processing of data requires a lot of computing resources. In particular, there are significant differences in the services in different geographical areas, and the type of data reported by home gateways is also extremely complex. In the related technology, after receiving the data, the server obtains the public IP address of the reporting gateway according to the TCP protocol, and distinguishes the region of the data source by querying the IP address library; then, the data is decompressed and decrypted and stored in the corresponding business data table according to the business type field carried therein. However, the process of IP address library query and business classification requires a large amount of server computing resources, resulting in low efficiency.

[0102] The server of the embodiment of the present application can be provided with a DNS configuration function, which can pre-partition the region to obtain different geographical areas, configure different public IPs for different geographical areas, and then use the DNS cloud resolution service of the cloud platform to synchronize the mapping relationship between the geographical area and the public IP to the DNS server. Specifically, different geographical areas correspond to different DNS servers. The server of the embodiment of the present application can synchronize the public IPs corresponding to the pre-set geographical areas to the DNS servers corresponding to the geographical areas through the DNS cloud resolution service of the cloud platform, so that when the DNS server of a certain geographical area receives the access request for the relevant domain name from the gateway of the geographical area, the public IP corresponding to the geographical area can be sent to the gateway of the geographical area. The server of the embodiment of the present application can quickly determine the region where the service data packets received by different public IPs are located according to the mapping relationship between the geographical area and the public IP.

[0103] In an optional implementation, the server of the embodiment of the present application can also call the traffic monitoring function to monitor the received business data packets, automatically or manually configure the server resources according to the traffic changes, if the public IP of a certain geographical area changes due to the change of server resources, modify the mapping table of the geographical area and the public IP according to the public IP, and then call the DNS cloud resolution service of the external cloud platform to synchronize the domain name resolution configuration (new public IP) of the geographical area to the DNS server corresponding to the geographical area. After the gateway of the geographical area accesses the data collection domain name, it resolves to the new public IP.

[0104] In this embodiment, for home gateways and other IoT devices that report large amounts of data and come from different regions, the regions can be labeled during data collection, thereby avoiding the performance consumption of adding region labels later, improving the speed and reliability of data processing, and meeting diverse business needs.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a data acquisition platform for implementing the data acquisition method involved above. The implementation scheme for solving the problem provided by the platform is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data acquisition platform embodiments provided below can refer to the limitations on the data acquisition method above, and will not be repeated here.

[0107] In one embodiment, as shown in FIG8 , a data collection platform is provided, including: a data decompression and decryption module 802 and a data analysis module 804, wherein the data decompression and decryption module 802 is connected to an external DNS server, and the data analysis module 804 is connected to an external data analysis platform, wherein:

[0108] The data decompression and decryption module 802 is used to receive a business data packet from a gateway; the business data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the business data packet; and the business data packet is pushed into a cache queue corresponding to the primary data type of the business data packet.

[0109] The data analysis module 804 is connected to the data decompression and decryption module 802, and is used to process the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

[0110] In one embodiment, the data collection platform also includes a traffic monitoring module 806, a traffic control module 808 and a DNS automatic configuration module 810. The traffic monitoring module 806, the traffic control module 808 and the DNS automatic configuration module 810 are connected in sequence. The traffic monitoring module 806 is also used to connect to the data decompression and decryption module 802, and the DNS automatic configuration module 810 is also used to connect to an external DNS resolution service.

[0111] The data decompression and decryption module 802 receives various types of data from the gateway, first decompresses and decrypts them, then classifies them according to the primary data type in the message header (used to distinguish different plug-ins in the gateway) (for example, it can be classified into 0-5), and then marks the priority according to the primary data type and pushes it into the corresponding cache queue. Among them, the scheduling algorithm can be implemented using a multi-level feedback queue algorithm.

[0112] The data analysis module 804 reads and analyzes the messages in the cache queue, giving priority to high-priority data types, and pushes the message bodies of the same priority level into the corresponding table files of the data analysis platform according to the secondary data types (used to distinguish different data types of the same plug-in) according to the time slice rotation algorithm. Due to the use of the time slice rotation algorithm, only one cache file will be opened in the same thread in the same time slice, thereby improving the writing performance. Among them, the scheduling algorithm can be implemented using a multi-level feedback queue algorithm.

[0113] The specific implementation of the data decompression and decryption module 802 and the data parsing module 804 can be implemented using RPC (Remote Procedure Call). The data decompression and decryption module 802 uses an RPC client, and the data parsing module 804 uses an RPC server. After the RPC client (i.e., the data decompression and decryption module) decompresses and decrypts the data, it will divide the data into multiple cache queues according to the primary data type. The high-priority primary data type will enter the high-priority queue. The RPC client will process the low-priority data only after processing the high-priority data. The lower-priority data can be processed only when the high-priority queue is empty (i.e., if there are n priority queues, the data of the lowest priority queue will be processed only when the first n-1 queues are empty). The corresponding configuration of the client and the server is reasonably performed according to the number of the highest-priority primary data types. For example, if there are n highest-priority primary data types at the same time, the RPC client and the server can be set as a pair of n configurations, that is, one client corresponds to n servers, and the client sends the data to different servers in sequence according to the priority of the primary data type and ensures that the same primary data type is sent to the same server. Here, each RPC server is a process. When a server process processes data of the same primary data type, it will create different threads according to the different secondary data types. The data parsing and processing within the same process will adopt a time slice rotation method.

[0114] The traffic monitoring module 806 records the traffic at the inlet and outlet of the data decompression and decryption module in real time, and can present the traffic on the monitoring panel through open source components such as Prometheus.

[0115] The traffic control module 808 can call the interface of the traffic monitoring module 806 to automatically or manually configure server resources according to traffic changes. If the public IP changes due to changes in server resources, the public IP can be synchronized to the DNS automatic configuration module 810.

[0116] Please refer to Figure 9, the DNS automatic configuration module 810 learns that the public network IP of the data collection domain name resolution of a province has changed, and synchronizes the new domain name resolution configuration of the province to the DNS server of the province by calling the interface of the DNS cloud resolution service of Alibaba Cloud (or other cloud platforms that can provide DNS cloud resolution services). After the gateway of the province accesses the data collection domain name, it resolves to the new IP.

[0117] Based on the same inventive concept, the embodiment of the present application also provides a data acquisition device for implementing the data acquisition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data acquisition device embodiments provided below can refer to the limitations on the data acquisition method above, and will not be repeated here.

[0118] In one embodiment, as shown in FIG. 10 , a data collection device is provided, including: a receiving module 1002 , a matching module 1004 and a queue module 1006 , wherein:

[0119] The receiving module 1002 is used to receive a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the service data packet.

[0120] The matching module 1004 is used to determine the target priority matching the service data packet based on the target matching rule and the primary data identifier.

[0121] The queue module 1006 is used to push the service data packet into the cache queue corresponding to the target priority; wherein the priority level of the service data packet and the processing order of the cache queue corresponding to the priority level are positively correlated.

[0122] Each module in the above data acquisition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0123] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data acquisition method is implemented.

[0124] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data collection method, characterized in that: The method comprises: receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate a primary data type of the service data packet; Pushing the service data packet into a cache queue corresponding to the primary data type of the service data packet; Data processing is performed on the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

2. The method according to claim 1, characterized in that The processing of the service data packets in the cache queue based on the priority level of the cache queue includes: Determining a priority level of the cache queue; The queue processing order of the cache queue is determined based on the priority level of the cache queue, and data processing is performed on the service data packets of the cache queue based on the queue processing order; wherein the queue processing order of the cache queue with a high priority level is higher than the queue processing order of the cache queue with a low priority level.

3. The method according to claim 2, characterized in that The performing data processing on the service data packets of the cache queue based on the queue processing order includes: determining a target cache queue from a plurality of cache queues based on the queue processing order; Acquire the secondary data type of the service data packet in the target cache queue; the secondary data type is determined by the secondary data identifier of the service data packet; Based on the secondary data type, a time slice round-robin algorithm is used to push data corresponding to the service data packet of the target cache queue into the database.

4. The method according to claim 1, characterized in that The method further comprises: Get the type number of the first-level data type corresponding to the highest priority level; Calling the same number of servers as the type to process the service data packets in the cache queue in parallel; Among them, business data packets of the same primary data type are sent to the same server for data processing.

5. The method according to any one of claims 1 to 4, characterized in that: Before receiving the service data packet from the gateway, the method includes: Determine the access addresses corresponding to different geographical areas; The mapping relationship between the geographical area and the access address is synchronized to the DNS server, so that when the DNS server receives the access request of the gateway, it sends the access address corresponding to the geographical area where the gateway is located to the gateway.

6. A data collection platform, characterized in that: The platform includes: The data decompression and decryption module is used to: receive a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used to indicate the primary data type of the service data packet; push the service data packet into a cache queue corresponding to the primary data type of the service data packet; A data analysis module is connected to the data decompression and decryption module and is used to process the service data packets in the cache queue based on the priority level of the cache queue; the priority level of the cache queue is determined by the primary data type.

7. A data acquisition device, characterized in that: The device comprises: A receiving module, used for receiving a service data packet from a gateway; the service data packet includes a primary data identifier, and the primary data identifier is used for indicating the primary data type of the service data packet; A matching module, used to determine a target priority matching the service data packet based on a target matching rule and the primary data identifier; The queue module is used to push the service data packet into a cache queue corresponding to the target priority; wherein the priority level of the service data packet and the processing order of the cache queue corresponding to the priority level are positively correlated.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images