A data fusing control method and device in a high-concurrency data collection scenario

By building a data processing cache and exception cache structure in high concurrent data acquisition scenarios, monitoring traffic and exception number, and realizing data traffic caching and fuse processing, it solves system crashes caused by over-limited bandwidth of data forwarding modules and storage modules and single-module exceptions. Problem, improving system stability and reliability.

CN118381714BActive Publication Date: 2024-11-08JIANGSU JINHENG INFORMATION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410813438.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-08
Estimated Expiration
2044-06-24

AI Technical Summary

Technical Problem

In high concurrent data acquisition scenarios, the data forwarding module and historical data storage module may exceed the server bandwidth limit, resulting in network congestion, and abnormalities of a single module may cause the entire system to crash.

Method used

Using data processing traffic monitoring and multi-level circuit breaker decision table technology, the data processing cache data structure and abnormal cache data structure are constructed in seconds and minutes, and the data traffic and the number of abnormalities are monitored to realize the cache and circuit breaker processing of data traffic to avoid The system crashes.

Benefits of technology

It effectively reduces the risk of system crash caused by a single system exception, improves the stability and reliability of data processing services, and ensures network stability in high concurrency scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118381714B_ABST
    Figure CN118381714B_ABST
Patent Text Reader

Abstract

The present application discloses a data fusing control method and device in a high-concurrency data collection scenario, belonging to the field of Internet of Things data processing technology. The method comprises: constructing data processing cache data structures with two granularities of second level and minute level respectively, and storing the data flow entering the forwarding or storage module; calculating the upper limit of the data flow per second based on the available bandwidth, and calculating the upper limit of the data processing for different set time periods based on the upper limit of the data flow per second and the control requirements of different scenarios; obtaining the corresponding cache data flow value from the data processing cache data structure based on different flow monitoring timing periods, comparing the obtained cache data flow value with the data processing upper limit, judging whether the data cache queue is empty, and if so, releasing the over-limit state, and processing all the data in real time; if not, storing all the data in the data cache queue, and processing the data regularly according to the data processing upper limit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a data fusing control method and device in a high-concurrency data collection scenario. Background Art

[0002] At present, most IoT platforms are faced with the collection and processing of massive high-frequency data, and use multi-cluster deployment to achieve high availability of overall functions. For the current technical architecture, significant results have been achieved in achieving efficient data processing in high-concurrency scenarios. By centrally deploying data collection, data calculation, historical data storage, real-time cache, and data forwarding modules on the same server within the cluster, network data transmission costs are effectively reduced and server resources are maximized. However, this deployment method also raises the following two core challenges:

[0003] When faced with sudden large-scale data forwarding or storage needs, the data forwarding module and historical data storage module may exceed the bandwidth carrying limit of the server, thereby affecting the data transmission efficiency of other modules with lower network traffic requirements, causing network congestion.

[0004] If a module in the cluster malfunctions or fails, the server's CPU, memory and network resources may be quickly exhausted. In this case, not only will the module's own service be interrupted, but it will also affect the normal operation of all other services in the cluster, and may eventually paralyze the entire system. Summary of the invention

[0005] The purpose of the present invention is to provide a data fusing control method and device in a high-concurrency data acquisition scenario, thereby achieving network stability in the case of excessive burst data traffic in a high-concurrency scenario, realizing exception handling in a multi-module high-concurrency processing process, and reducing the risk of the entire system crashing due to a single system exception.

[0006] To achieve the above object, the present invention provides a data fusing control method in a high-concurrency data collection scenario, comprising:

[0007] Deploy the data acquisition module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster;

[0008] Based on high-concurrency scenarios, data processing cache data structures with two granularities, second level and minute level, are built to store data traffic entering the data forwarding module or historical data storage module.

[0009] The upper limit of data flow per second is calculated based on the available bandwidth, and the upper limit of data processing in different set time periods is calculated based on the upper limit of data flow per second and the control requirements of different scenarios;

[0010] Based on different flow monitoring timing periods, corresponding cache data flow values ​​are obtained from the data processing cache data structure, and the obtained cache data flow values ​​are compared with the data processing upper limit values ​​to determine whether an over-limit condition is triggered;

[0011] If the over-limit condition is triggered, determine whether the data cache queue is empty. If the data cache queue is empty, release the over-limit state and process all data in real time. If the data cache queue is not empty, store all data in the data cache queue and process the data regularly according to the data processing upper limit.

[0012] As a further solution of the present invention, the data processing cache data structures with two granularities of second level and minute level are respectively constructed based on high concurrency scenarios to store the data traffic entering the data forwarding module or the historical data storage module, including:

[0013] Construct thread-safe KV structures with granularity of second and minute respectively as data processing cache data structures, wherein the cache key of the data processing cache structure is a time point with granularity of second or minute, and the cache value is a numerical atomic class;

[0014] The data is stored in the data area corresponding to the data processing cache data structure based on the time point at which the data traffic entering the data forwarding module or the historical data storage module enters.

[0015] As a further solution of the present invention, the calculation of the upper limit of data flow per second based on the available bandwidth includes:

[0016] Obtain the current server network bandwidth N and bandwidth allocation coefficient K, and calculate the available bandwidth M of the data forwarding module or the historical data storage module according to M=KN;

[0017] Based on the average size E of the data packets in the data forwarding module or the historical data storage module, the upper limit value FMAX of the data flow per second is calculated according to FMAX = M / E.

[0018] As a further solution of the present invention, the acquired cache data flow value is compared with the data processing upper limit value to determine whether an over-limit condition is triggered, including:

[0019] If the acquired cache data flow value is greater than the data processing upper limit value, it is determined that the over-limit condition is triggered; otherwise, it is determined that the over-limit condition is not triggered.

[0020] On the other hand, the present invention provides a data fusing control method in a high-concurrency data collection scenario, comprising:

[0021] Deploy the data acquisition module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster;

[0022] Build an exception cache data structure with second-level granularity based on high-concurrency scenarios to store the number of data processing exceptions in different modules;

[0023] The number of data processing exceptions of a single module per unit time is obtained, compared with the preset threshold, and the type of fuse measure is determined based on the comparison result.

[0024] As a further solution of the present invention, the second-level exception cache data structure is constructed based on a high-concurrency scenario to store the number of data processing exceptions of each module, including:

[0025] Construct a KV structure with second-level granularity as a second-level exception cache data structure, wherein the cache key of the second-level exception cache data structure is a second-level granularity time point, and the cache value is a numerical atomic class;

[0026] The number of data processing exceptions that occur during the processing of data entering each module is obtained, and based on the time point when the data enters each module, the number of data processing exceptions is stored in a data area corresponding to the second-level exception cache data structure.

[0027] As a further solution of the present invention, the obtaining of the number of data processing anomalies of a single module per unit time, comparing with a preset threshold, and determining the type of fuse measure according to the comparison result includes:

[0028] Obtain the number of data processing exceptions of a single module within a unit time. If the number of data processing exceptions is greater than a first set threshold, enter the first level fuse, the interrupt processing duration is the first preset time, and issue a system alarm to record the system log;

[0029] If the number of data processing exceptions is greater than the second set threshold or the first processing still fails after the first level fuse is completed, the second level fuse is entered, the processing process is interrupted for a second preset time, and a system alarm is issued to record the system log, wherein the second preset time is an integer multiple of the first preset time;

[0030] If the number of data processing exceptions is greater than the third set threshold or the first processing still fails after the second-level fuse ends, the third-level fuse is entered, the processing process is interrupted for a third preset time, and a system alarm is issued to record the system log, where the third preset time is an integer multiple of the second preset time;

[0031] If the number of data processing exceptions is greater than the fourth set threshold or the first processing still fails after the third-level fuse ends, the fourth-level fuse is entered, the current data processing module service is interrupted, and a system alarm is issued to record the system log.

[0032] In another aspect, the present invention provides a data fusing control device in a high-concurrency data collection scenario, comprising:

[0033] The cluster deployment module deploys the data collection module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster;

[0034] The data processing cache data structure module builds data processing cache data structures with two granularities, namely, second-level and minute-level, based on high-concurrency scenarios to store data traffic entering the data forwarding module or the historical data storage module;

[0035] The data processing upper limit calculation module calculates the upper limit of data flow per second based on the available bandwidth, and calculates the upper limit of data processing for different set time periods based on the upper limit of data flow per second and the control requirements of different scenarios;

[0036] A comparison module, based on different flow monitoring timing periods, obtains corresponding cache data flow values ​​from the data processing cache data structure, compares the obtained cache data flow values ​​with the data processing upper limit value, and determines whether an over-limit condition is triggered;

[0037] The over-limit trigger module, if the over-limit condition is triggered, determines whether the data cache queue is empty. If the data cache queue is empty, the over-limit state is released and all data is processed in real time; if the data cache queue is not empty, all data is stored in the data cache queue and the data is processed regularly according to the data processing upper limit.

[0038] In another aspect, the present invention provides a data fusing control device in a high-concurrency data collection scenario, comprising:

[0039] The cluster deployment module deploys the data collection module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster;

[0040] The exception cache data structure construction module builds an exception cache data structure with second-level granularity based on high-concurrency scenarios, and stores the number of data processing exceptions in different modules;

[0041] The fuse processing module obtains the number of data processing exceptions of a single module per unit time, compares it with a preset threshold, and determines the type of fuse measure based on the comparison result.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The data fusing control method in the high-concurrency data collection scenario of the present invention uses data processing flow monitoring and excessive data caching mechanism to achieve network stability guarantee in the case of excessive burst data flow in high-concurrency scenarios. By monitoring the number of exceptions of various modules in data processing, the multi-level fusing decision table technology realizes the exception handling of multi-module high-concurrency processing processes, reducing the risk of the whole system crashing due to a single system exception. This structure can effectively improve the stability of data processing services for massive IoT collection and computing scenarios, and achieve higher system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the current high-concurrency system architecture diagram of the Internet of Things;

[0045] Figure 2 is a flow chart of a data fusing control method in a high-concurrency data collection scenario in an embodiment;

[0046] Figure 3 is a schematic diagram of a data processing volume cache structure in an embodiment;

[0047] Figure 4 is a flow chart after an over-limit condition is triggered in one embodiment;

[0048] Figure 5 is a flow chart of a data fusing control method in a high-concurrency data collection scenario in another embodiment;

[0049] Figure 6 is a schematic diagram of a data anomaly quantity cache structure in an embodiment;

[0050] Figure 7 is a schematic diagram of the structure of a data fusing control device in a high-concurrency data collection scenario in an embodiment;

[0051] Figure 8 is a structural schematic diagram of a data fusing control device in a high-concurrency data collection scenario in another embodiment;

[0052] Fig. 9 It is a structural schematic diagram of a data fusing control device in a high-concurrency data collection scenario in another embodiment. DETAILED DESCRIPTION

[0053] In order to enable persons skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only one or more partial embodiments of this specification, rather than all embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0054] It should be noted that, in the absence of conflict, one or more embodiments and features in the embodiments in this specification can be combined with each other. One or more embodiments of this specification will be described in detail below with reference to the drawings and in combination with the embodiments.

[0055] like Figure 1 As shown in the figure, most IoT platforms currently face the collection and processing of massive high-frequency data. The functional modules of the data processing process mainly include: 1. Data collection module 2. Data calculation module 3. Historical data storage module 4. Real-time cache module 5. Data forwarding module. And the high availability of the overall function is achieved by multi-cluster deployment. For the current technical architecture, significant results have been achieved in achieving efficient data processing in high-concurrency scenarios. By centrally deploying data collection, data calculation, historical data storage, real-time cache and data forwarding modules on the same server within the cluster, the network data transmission cost is effectively reduced and the server resources are maximized. However, this deployment method also raises the following two core challenges: When facing sudden large-scale data forwarding or storage requirements, the data forwarding module and the historical data storage module may exceed the bandwidth carrying limit of the server, thereby affecting the data transmission efficiency of other modules with lower network traffic requirements, causing network congestion. If a module in the cluster runs abnormally or fails, the server's CPU, memory and network resources may be quickly exhausted. In this case, not only will the module's own service be interrupted, but it will also affect the normal operation of all other services in the cluster, and may eventually lead to the paralysis of the entire system.

[0056] Embodiment 1

[0057] In order to solve the above problems, Figure 2 As shown, an embodiment of the present invention provides a data fusing control method in a high-concurrency data collection scenario, including:

[0058] S201, deploying a data acquisition module, a data calculation module, a historical data storage module, a real-time cache module, and a data forwarding module on the same server within the cluster;

[0059] S202, constructing data processing cache data structures with two granularities of second level and minute level respectively based on high concurrency scenarios, and storing data traffic entering the data forwarding module or the historical data storage module;

[0060] S203, calculating the upper limit of data flow per second based on the available bandwidth, and calculating the upper limits of data processing for different set time periods based on the upper limit of data flow per second and the control requirements of different scenarios;

[0061] S204, obtaining corresponding cache data flow values ​​from the data processing cache data structure based on different flow monitoring timing periods, and comparing the obtained cache data flow values ​​with the data processing upper limit values ​​to determine whether an over-limit condition is triggered;

[0062] S205, if the over-limit condition is triggered, determine whether the data cache queue is empty. If the data cache queue is empty, release the over-limit state and process all data in real time; if the data cache queue is not empty, store all data in the data cache queue and process the data regularly according to the data processing upper limit.

[0063] The data fusing control method in the high-concurrency data collection scenario of the present invention uses data processing flow monitoring and excessive data caching mechanism to achieve network stability guarantee in the case of excessive burst data flow in high-concurrency scenarios. By monitoring the number of exceptions of various modules in data processing, the multi-level fusing decision table technology realizes the exception handling of multi-module high-concurrency processing processes, reducing the risk of the whole system crashing due to a single system exception. This structure can effectively improve the stability of data processing services for massive IoT collection and computing scenarios, and achieve higher system reliability.

[0064] Furthermore, the data processing cache data structures with two granularities of second level and minute level are constructed based on the high concurrency scenario to store the data traffic entering the data forwarding module or the historical data storage module (ie S202), including:

[0065] Construct thread-safe KV structures with granularity of second and minute respectively as data processing cache data structures, wherein the cache key of the data processing cache structure is a time point with granularity of second or minute, and the cache value is a numerical atomic class;

[0066] The data is stored in the data area corresponding to the data processing cache data structure based on the time point at which the data traffic entering the data forwarding module or the historical data storage module enters.

[0067] In high-concurrency scenarios, multiple threads may access and modify the cache at the same time, so thread safety needs to be ensured to avoid data inconsistency or race conditions. In high-concurrency scenarios, the following behaviors can be expected:

[0068] Concurrent reading: Multiple threads can read values ​​from the cache at the same time. The reading operation of ConcurrentHashMap is thread-safe, so data consistency can be guaranteed during concurrent reading.

[0069] Concurrent write: Multiple threads can store key-value pairs in the cache at the same time. Since ConcurrentHashMap is used, it provides thread-safe write operations, thus ensuring data consistency during concurrent writes.

[0070] Atomic increment and decrement operations: In high-concurrency scenarios, multiple threads may perform increment or decrement operations on the same cache value at the same time. By using AtomicInteger to encapsulate the cached value and using atomic operations to perform increment and decrement, the atomicity and thread safety of these operations can be ensured to avoid race conditions.

[0071] Concurrent removal operations: Multiple threads can try to remove key-value pairs from the cache at the same time. The removal operation of ConcurrentHashMap is also thread-safe, so data consistency can be guaranteed during concurrent removal.

[0072] KV structure usually refers to the key-value structure, which is a common data storage method that stores and organizes data in the form of key-value pairs. Each key uniquely identifies a value, similar to the structure of a dictionary or hash table.

[0073] In computer science and software engineering, KV structures are often used to implement various data storage, caching, configuration management and other functions. For example, in database systems, key-value storage engines (such as Redis) use KV structures to store data; in distributed systems, configuration centers (such as ZooKeeper, etcd) also often use KV structures to manage configuration information; in programming languages, hash tables (such as dictionaries in Python) are also a typical KV structure.

[0074] In Java, you can use ConcurrentHashMap as the KV structure, and for the cached values, you can use thread-safe numeric atomic classes such as AtomicInteger. This design ensures that concurrent access to the cache is safe and avoids race conditions and thread safety issues in a multi-threaded environment.

[0075] The key of the cache is the time with granularity of seconds or minutes, and the value of the cache is a numeric atomic class. The internal numeric atomic class uses CAS (Compare and Swap) operations to ensure thread safety. The data of the last minute is stored with a time granularity of 1 second, and the data of the last 10 minutes is stored at the minute level. The current inflow forwarding and storage data traffic will be recorded in the corresponding data area at the corresponding inflow time point for storage. The storage structure diagram is as follows Figure 3 shown.

[0076] Further, the calculation of the upper limit of data flow per second based on the available bandwidth includes:

[0077] Obtain the current server network bandwidth N and bandwidth allocation coefficient K, and calculate the available bandwidth M of the data forwarding module or the historical data storage module according to M=KN;

[0078] Based on the average size E of the data packets in the data forwarding module or the historical data storage module, the upper limit value FMAX of the data flow per second is calculated according to FMAX = M / E.

[0079] According to the current server network bandwidth N, and the bandwidth allocation coefficient K (0.1-0.5), the available bandwidth M of the forwarding or storage module is calculated, M=KN. According to the average size E of the forwarding or storage module data packet, the maximum upper limit of the data flow per second FMAX = M / E can be calculated. According to FMAX, it is calculated according to different control cycles. The control cycle requirements here are set according to the actual needs of different scenarios. In one embodiment, the upper limit of data processing volume can be calculated every 5 seconds, every 10 seconds, every 1 minute, every 3 minutes, and every 5 minutes. The system sets the flow monitoring timed task cycle for different control granularities, such as every 1 second, every 2 seconds, every 12 seconds, every 36 seconds, etc. When the system reaches the set cycle, it obtains the corresponding data volume value from the cache and compares it with the set value. If the cached processing data volume is greater than the set maximum value, it is determined that the current processing volume exceeds the limit and triggers the next step of the overlimit processing process.

[0080] Furthermore, the obtained cache data flow value is compared with the data processing upper limit value to determine whether an over-limit condition is triggered, including:

[0081] If the acquired cache data flow value is greater than the data processing upper limit value, it is determined that the over-limit condition is triggered; otherwise, it is determined that the over-limit condition is not triggered.

[0082] If the over-limit condition is triggered, the current data is written into the first-in-first-out cache queue to be processed. If the data in the data cache queue is not empty, the timing processing process is started to process the data in the cache according to the set upper limit maximum value. When all data flows into the processing module, it is necessary to determine whether the queue is empty. If the queue is empty, the over-limit state is released, and all incoming data enters the real-time processing logic. This process uses the first-in-first-out queue to cache excess data and realizes the timing processing requirements of cached data. The flowchart after the over-limit condition is triggered is as follows: Figure 4 shown.

[0083] Embodiment 2

[0084] like Figure 5 As shown, an embodiment of the present invention provides a data fusing control method in a high-concurrency data collection scenario, including:

[0085] S501, deploying a data acquisition module, a data calculation module, a historical data storage module, a real-time cache module, and a data forwarding module on the same server within the cluster;

[0086] S502, building an exception cache data structure with second-level granularity based on high-concurrency scenarios, and storing the number of data processing exceptions of different modules;

[0087] S503, obtaining the number of data processing exceptions of a single module within a unit time, comparing it with a preset threshold, and determining the type of fuse measure according to the comparison result.

[0088] Furthermore, the second-level exception cache data structure is constructed based on the high-concurrency scenario to store the number of data processing exceptions of each module (ie, S502), including:

[0089] Construct a KV structure with second-level granularity as a second-level exception cache data structure, wherein the cache key of the second-level exception cache data structure is a second-level granularity time point, and the cache value is a numerical atomic class;

[0090] The number of data processing exceptions that occur during the processing of data entering each module is obtained, and based on the time point when the data enters each module, the number of data processing exceptions is stored in a data area corresponding to the second-level exception cache data structure.

[0091] The overall structure of the data cache is a thread-safe KV structure such as ConcurrentHashMap. The cached value is based on a thread-safe numeric atomic class such as AtomicInteger. The cache key is the time with a granularity of seconds, and the cached value is the atomic class of the number of exceptions. The atomic class uses CAS (Compare and Swap) operations to ensure thread safety. The exception data of the last minute is stored with a time granularity of 1 second. The number of exceptions that occur during the processing of the data currently flowing into each module will be recorded in the corresponding data area at the corresponding inflow time point for storage. The storage structure is as follows: Figure 6 shown.

[0092] Furthermore, the obtaining of the number of data processing anomalies of a single module per unit time, comparing it with a preset threshold, and determining the type of fuse measure according to the comparison result (ie, S503) includes:

[0093] Obtain the number of data processing exceptions of a single module within a unit time. If the number of data processing exceptions is greater than a first set threshold, enter the first level fuse, the interrupt processing duration is the first preset time, and issue a system alarm to record the system log;

[0094] If the number of data processing exceptions is greater than the second set threshold or the first processing still fails after the first level fuse is completed, the second level fuse is entered, the processing process is interrupted for a second preset time, and a system alarm is issued to record the system log, wherein the second preset time is an integer multiple of the first preset time;

[0095] If the number of data processing exceptions is greater than the third set threshold or the first processing still fails after the second-level fuse ends, the third-level fuse is entered, the processing process is interrupted for a third preset time, and a system alarm is issued to record the system log, where the third preset time is an integer multiple of the second preset time;

[0096] If the number of data processing exceptions is greater than the fourth set threshold or the first processing still fails after the third-level fuse ends, the fourth-level fuse is entered, the current data processing module service is interrupted, and a system alarm is issued to record the system log.

[0097] The fusing strategy of a single module is divided into fusing trigger conditions, fusing measures, and fusing end conditions, and the fusing logic is organized in the form of a decision table.

[0098] The example table is shown below:

[0099] ;

[0100] During the actual operation of the system, the module abnormal fuse processing logic is carried out according to the above-set strategy table. Logs related to the triggering and termination of the fuse are recorded, and the system's push capability is used to promptly push the system's abnormal information to the system operation and maintenance personnel.

[0101] like Figure 7 As shown, corresponding to the embodiment of the aforementioned data fusing control method, the present application also provides an embodiment of a data fusing control device in a high-concurrency data collection scenario. The device includes:

[0102] The cluster deployment module 701 deploys the data collection module, the data calculation module, the historical data storage module, the real-time cache module and the data forwarding module on the same server within the cluster;

[0103] The data processing cache data structure module 702 constructs data processing cache data structures at two granularities, namely, second level and minute level, based on high concurrency scenarios, and stores data traffic entering the data forwarding module or the historical data storage module;

[0104] The data processing upper limit calculation module 703 calculates the upper limit of data flow per second based on the available bandwidth, and calculates the upper limits of data processing for different set time periods based on the upper limit of data flow per second and the control requirements of different scenarios;

[0105] The comparison module 704 obtains corresponding cache data flow values ​​from the data processing cache data structure based on different flow monitoring timing periods, and compares the obtained cache data flow values ​​with the data processing upper limit value to determine whether an over-limit condition is triggered;

[0106] The over-limit trigger module 705, if the over-limit condition is triggered, determines whether the data cache queue is empty. If the data cache queue is empty, the over-limit state is released and all data is processed in real time; if the data cache queue is not empty, all data is stored in the data cache queue and the data is processed regularly according to the data processing upper limit value.

[0107] It should be noted that the embodiments of the data fuse control device in this specification and the embodiments of the data fuse control method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding data fuse control method mentioned above, and the repeated parts will not be repeated.

[0108] like Figure 8 As shown, corresponding to the embodiment of the aforementioned data fuse control method, the present application also provides an embodiment of a data fuse control device. The device includes:

[0109] The cluster deployment module 801 deploys the data acquisition module, the data calculation module, the historical data storage module, the real-time cache module and the data forwarding module on the same server within the cluster;

[0110] The exception cache data structure construction module 802 constructs an exception cache data structure with a second-level granularity based on a high-concurrency scenario, and stores the number of data processing exceptions of different modules;

[0111] The fuse processing module 803 obtains the number of data processing exceptions of a single module within a unit time, compares it with a preset threshold, and determines the type of fuse measure according to the comparison result.

[0112] It should be noted that the embodiments of the data fuse control device in this specification and the embodiments of the data fuse control method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding data fuse control method mentioned above, and the repeated parts will not be repeated.

[0113] like Fig. 9 As shown, an embodiment of the present application further provides a data fusing control device in a high-concurrency data collection scenario, the device comprising:

[0114] The cluster deployment module 901 deploys the data acquisition module, the data calculation module, the historical data storage module, the real-time cache module and the data forwarding module on the same server within the cluster;

[0115] The data processing cache data structure module 902 constructs data processing cache data structures at two granularities, namely, second level and minute level, based on high concurrency scenarios, and stores data traffic entering the data forwarding module or the historical data storage module;

[0116] The data processing upper limit calculation module 903 calculates the upper limit of data flow per second based on the available bandwidth, and calculates the upper limit of data processing for different set time periods based on the upper limit of data flow per second and the control requirements of different scenarios;

[0117] The comparison module 904 obtains corresponding cache data flow values ​​from the data processing cache data structure based on different flow monitoring timing periods, and compares the obtained cache data flow values ​​with the data processing upper limit value to determine whether an over-limit condition is triggered;

[0118] The over-limit trigger module 905, if the over-limit condition is triggered, determines whether the data cache queue is empty. If the data cache queue is empty, the over-limit state is released and all data is processed in real time; if the data cache queue is not empty, all data is stored in the data cache queue and the data is processed regularly according to the data processing upper limit value;

[0119] The exception cache data structure construction module 906 constructs an exception cache data structure with a second-level granularity based on a high-concurrency scenario, and stores the number of data processing exceptions of different modules;

[0120] The fuse processing module 907 obtains the number of data processing exceptions of a single module within a unit time, compares it with a preset threshold, and determines the type of fuse measures according to the comparison result.

[0121] It should be understood that the device provided in this embodiment can execute the method in the above-mentioned embodiment one and the method in the above-mentioned embodiment two at the same time, and it should be considered that there is no restriction on the order of execution when the device executes the above-mentioned methods, that is, the device can implement the method in embodiment one when limiting the data volume or the overall data flow in the normal running state of the program, and the device can implement the method in embodiment two when monitoring and processing exceptions or errors during the running of the program. If the normal running state and the abnormal state of the program occur at the same time, the above-mentioned two methods are executed separately / in parallel.

[0122] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of one or more claims of this specification.

Claims

1. A method for improving data service reliability in a high-concurrency data collection scenario, characterized in that: include: Deploy the data acquisition module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster; Based on high-concurrency scenarios, data processing cache data structures with two granularities, second level and minute level, are built to store data traffic entering the data forwarding module or historical data storage module. The upper limit of data flow per second is calculated based on the available bandwidth, and the upper limit of data processing in different set time periods is calculated based on the upper limit of data flow per second and the control requirements of different scenarios; Based on different flow monitoring timing periods, corresponding cache data flow values ​​are obtained from the data processing cache data structure, and the obtained cache data flow values ​​are compared with the data processing upper limit values ​​to determine whether an over-limit condition is triggered; If the over-limit condition is triggered, determine whether the data cache queue is empty. If the data cache queue is empty, release the over-limit state and process all data in real time; If the data cache queue is not empty, all data are stored in the data cache queue, and the data is processed regularly according to the data processing upper limit; Build an exception cache data structure with second-level granularity based on high-concurrency scenarios to store the number of data processing exceptions in different modules; The number of data processing exceptions of a single module per unit time is obtained, compared with the preset threshold, and the type of fuse measure is determined based on the comparison result.

2. The method for improving data service reliability in a high-concurrency data collection scenario according to claim 1, characterized in that: The data processing cache data structures with two granularities of second level and minute level are constructed based on high concurrency scenarios to store the data traffic entering the data forwarding module or the historical data storage module, including: Construct thread-safe KV structures with granularity of second and minute respectively as data processing cache data structures, wherein the cache key of the data processing cache data structure is a time point with granularity of second or minute, and the cache value is a numerical atomic class; The data is stored in the data area corresponding to the data processing cache data structure based on the time point at which the data traffic entering the data forwarding module or the historical data storage module enters.

3. The method for improving data service reliability in a high-concurrency data collection scenario according to claim 1, characterized in that: The calculation of the upper limit of data flow per second based on the available bandwidth includes: Obtain the current server network bandwidth N and bandwidth allocation coefficient K, and calculate the available bandwidth M of the data forwarding module or the historical data storage module according to M=KN; Based on the average size E of the data packets in the data forwarding module or the historical data storage module, the upper limit value FMAX of the data flow per second is calculated according to FMAX = M / E.

4. The method for improving data service reliability in a high-concurrency data collection scenario according to claim 1, characterized in that: Compare the obtained cache data flow value with the data processing upper limit value to determine whether the over-limit condition is triggered, including: If the acquired cache data flow value is greater than the data processing upper limit value, it is determined that the over-limit condition is triggered; otherwise, it is determined that the over-limit condition is not triggered.

5. The method for improving data service reliability in a high-concurrency data collection scenario according to claim 1, characterized in that: The second-level exception cache data structure is constructed based on the high-concurrency scenario to store the number of data processing exceptions of each module, including: Construct a KV structure with second-level granularity as a second-level exception cache data structure, wherein the cache key of the second-level exception cache data structure is a second-level granularity time point, and the cache value is a numerical atomic class; The number of data processing exceptions that occur during the processing of data entering each module is obtained, and based on the time point when the data enters each module, the number of data processing exceptions is stored in a data area corresponding to the second-level exception cache data structure.

6. The method for improving data service reliability in a high-concurrency data collection scenario according to claim 1, characterized in that: The obtaining of the number of data processing exceptions of a single module within a unit time, comparing it with a preset threshold, and determining the type of fuse measures according to the comparison result includes: Obtain the number of data processing exceptions of a single module within a unit time. If the number of data processing exceptions is greater than a first set threshold, enter the first level fuse, the interrupt processing duration is the first preset time, and issue a system alarm to record the system log; If the number of data processing exceptions is greater than the second set threshold or the first processing still fails after the first level fuse is completed, the second level fuse is entered, the processing process is interrupted for a second preset time, and a system alarm is issued to record the system log, wherein the second preset time is an integer multiple of the first preset time; If the number of data processing exceptions is greater than the third set threshold or the first processing still fails after the second-level fuse ends, the third-level fuse is entered, the processing process is interrupted for a third preset time, and a system alarm is issued to record the system log, where the third preset time is an integer multiple of the second preset time; If the number of data processing exceptions is greater than the fourth set threshold or the first processing still fails after the third-level fuse ends, the fourth-level fuse is entered, the current data processing module service is interrupted, and a system alarm is issued to record the system log.

7. A device for improving data service reliability in a high-concurrency data collection scenario, characterized in that: include: The cluster deployment module deploys the data collection module, data calculation module, historical data storage module, real-time cache module and data forwarding module on the same server within the cluster; The data processing cache data structure module builds data processing cache data structures with two granularities, namely, second-level and minute-level, based on high-concurrency scenarios to store data traffic entering the data forwarding module or the historical data storage module; The data processing upper limit calculation module calculates the upper limit of data flow per second based on the available bandwidth, and calculates the upper limit of data processing for different set time periods based on the upper limit of data flow per second and the control requirements of different scenarios; A comparison module, based on different flow monitoring timing periods, obtains corresponding cache data flow values ​​from the data processing cache data structure, compares the obtained cache data flow values ​​with the data processing upper limit value, and determines whether an over-limit condition is triggered; The over-limit trigger module, if the over-limit condition is triggered, determines whether the data cache queue is empty. If the data cache queue is empty, the over-limit state is released and all data is processed in real time; if the data cache queue is not empty, all data is stored in the data cache queue and the data is processed regularly according to the data processing upper limit value; The exception cache data structure construction module builds an exception cache data structure with second-level granularity based on high-concurrency scenarios, and stores the number of data processing exceptions in different modules; The fuse processing module obtains the number of data processing exceptions of a single module per unit time, compares it with a preset threshold, and determines the type of fuse measure based on the comparison result.

Citation Information

Patent Citations

  • Whole-process monitoring alarm platform and method thereof

    CN111752795A

  • Unified service traffic-limiting fusing platform and method thereof

    CN111787073A

  • Exception monitoring method based on multi-level cache

    CN114546786A

  • Dynamic resource scheduling method and system under high-concurrency task complex scene

    CN115495231A

  • Data service monitoring system

    CN117540321A