IoT card traffic monitoring method, system, device and medium

Abstract

This application provides a method, system, device, and medium for monitoring the traffic of an IoT SIM card. The method acquires traffic monitoring data from traffic monitoring requests for at least one IoT SIM card, constructs a traffic processing link according to a preset access layer, computing layer, and storage layer, and then dynamically adjusts the processing strategies of the access layer, computing layer, and storage layer based on the current resource load status and IoT SIM card service behavior. This controls computing pressure and / or storage pressure under traffic surge scenarios, obtaining monitoring results for the traffic monitoring requests. This achieves stable operation of IoT SIM card traffic monitoring under sudden traffic surge conditions and ensures the real-time nature and completeness of the monitoring results.

Description

Technical Field

[0001] This application relates to traffic monitoring technology, and more particularly to a method, system, device, and medium for monitoring traffic of an IoT card. Background Technology

[0002] Existing IoT SIM card traffic monitoring systems typically collect traffic monitoring data generated by IoT SIM cards based on operator interfaces, dedicated lines, virtual private networks, or edge gateways. The traffic monitoring data is then parsed, statistically analyzed, and stored on a centralized or distributed server to enable IoT SIM card usage statistics, abnormal traffic alerts, and bill reconciliation.

[0003] However, in the design of existing system architectures, bandwidth, computing resources and storage resources are often planned based on the average daily business load. This leads to queuing and backlog of traffic monitoring data in the collection, computing and storage links during peak traffic periods, significantly increased processing latency and even data loss, which in turn affects the real-time performance and completeness of monitoring results. Summary of the Invention

[0004] This application provides a method, system, device, and medium for monitoring the traffic of an IoT SIM card, in order to solve the technical problem of how to avoid monitoring delays when sudden traffic surges caused by actions such as batch firmware upgrades and periodic centralized reporting occur during the traffic monitoring process of an IoT SIM card.

[0005] Firstly, this application provides a method for monitoring the data traffic of an IoT SIM card, including: Obtain a traffic monitoring request for at least one IoT SIM card, the traffic monitoring request corresponding to traffic monitoring data generated by the IoT SIM card accessing the network side; Based on the traffic monitoring data, a traffic processing link is constructed according to a preset access layer, computing layer, and storage layer; Based on the current resource load status and IoT card service behavior, the processing strategies of the access layer, the computing layer, and the storage layer are dynamically adjusted using the traffic processing link to control computing pressure and / or storage pressure during traffic peak scenarios, and to obtain monitoring results for the traffic monitoring request.

[0006] Secondly, this application provides a data traffic monitoring system for IoT cards, comprising: The monitoring request acquisition module is used to acquire traffic monitoring requests for at least one IoT card, wherein the traffic monitoring request corresponds to traffic monitoring data generated by the IoT card accessing the network side; The traffic processing link construction module is used to construct a traffic processing link based on the traffic monitoring data, according to a preset access layer, computing layer, and storage layer. The peak traffic monitoring module is used to dynamically adjust the processing strategies of the access layer, the computing layer, and the storage layer based on the current resource load status and IoT card service behavior, using the traffic processing link, in order to control computing pressure and / or storage pressure in traffic peak scenarios and obtain monitoring results about the traffic monitoring request.

[0007] Thirdly, this application provides an electronic device, comprising: Processor; and, Memory for storing the executable instructions of the processor; The processor is configured to perform any of the possible methods described in the first aspect by executing the executable instructions.

[0008] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the possible methods described in the first aspect.

[0009] The IoT SIM card traffic monitoring method, system, device, and medium provided in this application acquire traffic monitoring data from traffic monitoring requests for at least one IoT SIM card, construct a traffic processing link according to a preset access layer, computing layer, and storage layer, and then dynamically adjust the processing strategies of the access layer, computing layer, and storage layer based on the current resource load status and IoT SIM card service behavior. This controls computing pressure and / or storage pressure under traffic surge scenarios, obtains monitoring results for traffic monitoring requests, thereby achieving stable operation of IoT SIM card traffic monitoring under sudden traffic surge conditions and ensuring the real-time and completeness of monitoring results. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 This is a flowchart illustrating a traffic monitoring method for an IoT SIM card according to an example embodiment of this application; Figure 2 This is a flowchart illustrating a specific implementation of S120 according to an example embodiment of this application; Figure 3 This is a flowchart illustrating a specific implementation of S130 according to an example embodiment of this application; Figure 4 This is a flowchart illustrating a traffic monitoring method for an IoT card according to another example embodiment of this application; Figure 5This is a schematic diagram of the structure of an IoT card traffic monitoring system according to an example embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of this application.

[0012] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0013] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0014] In IoT SIM card traffic monitoring systems, the network bandwidth, number of connections, and buffer queue length at the access layer are all limited. When the rate of traffic monitoring data arriving per unit time is consistently higher than the system's stable throughput rate, it will inevitably lead to continuous accumulation or even overflow of the access queue.

[0015] The computing layer needs to perform protocol parsing, aggregation statistics, and anomaly rule matching for each traffic monitoring record, which requires a lot of CPU and memory resources. In traffic peak scenarios, the parsing of high-concurrency records and the updating of sliding windows significantly increase processing latency, which in turn puts further back pressure on the upstream access queue.

[0016] Meanwhile, the storage layer database or time-series database has a physical limit on write IOPS and index update capabilities, which can easily lead to write queuing and lock contention in short-term high-concurrency write scenarios.

[0017] However, due to the centralized timed reporting (e.g., unified firmware upgrade, batch meter reading, centralized center jump reporting) and unified operation window configuration unique to IoT card services, business requests that could have been distributed over time are highly synchronized, amplifying instantaneous peak traffic. This makes systems designed based on daily average load more prone to link cascading bottlenecks during sudden floods.

[0018] Therefore, it is urgent to address how to effectively suppress the disorderly growth of the collection queue, computing queue, and storage queue under the constraints of limited access bandwidth, computing power, and storage write capacity when sudden traffic surges occur during IoT SIM card traffic monitoring caused by actions such as batch firmware upgrades and periodic centralized reporting. This would prevent the monitoring system from becoming overloaded, experiencing service crashes, or suffering severe delays in anomaly detection. At the same time, it is crucial to ensure, even in resource-constrained scenarios, the real-time detection of traffic anomalies in critical IoT SIM cards and the accurate understanding of overall traffic trends.

[0019] To solve the above problems, Figure 1 This is a flowchart illustrating a traffic monitoring method for an IoT SIM card according to an example embodiment of this application. Figure 1 As shown, the IoT SIM card traffic monitoring method provided in this embodiment includes: S110, Obtain a traffic monitoring request for at least one IoT SIM card.

[0020] In this step, a traffic monitoring request may be obtained for at least one IoT SIM card, wherein the traffic monitoring request corresponds to traffic monitoring data generated by the IoT SIM card accessing the network side.

[0021] Specifically, the system can receive raw traffic monitoring data generated by the IoT SIM card accessing the network side from the network side through at least one of the following access methods: operator interface, leased line, virtual private network, or edge gateway. The raw traffic monitoring data includes at least one or more fields such as IoT SIM card identifier, integrated circuit, International Mobile Subscriber Identity, access timestamp, uplink traffic value and / or downlink traffic value, access cell information, and protocol type.

[0022] Then, the original traffic monitoring data is parsed and the fields are standardized to map the heterogeneous data generated by different operators and different access methods into traffic monitoring data records under the preset monitoring data model.

[0023] Next, based on the IoT card identifier carried in the traffic monitoring data record, the traffic monitoring data belonging to the same IoT card are classified to form a card-level traffic monitoring request for at least one IoT card.

[0024] Furthermore, traffic monitoring data records can be grouped based on IoT SIM card identifiers, tenant identifiers, and / or service type information. Multiple traffic monitoring data entries from the same IoT SIM card within a preset time window can be aggregated into a single card-level traffic monitoring request, and / or traffic monitoring data from multiple IoT SIM cards can be aggregated into a project-level or tenant-level aggregated traffic monitoring request. Then, monitoring dimension and priority tags are attached to the card-level and / or aggregated traffic monitoring requests. The monitoring dimension tag indicates whether it is card-level, project-level, or tenant-level monitoring, while the monitoring priority tag is used to perform differentiated rate limiting and degradation processing on different requests during traffic surges. Finally, the generated card-level and / or aggregated traffic monitoring requests are input to a preset access layer as the starting data unit for subsequently building traffic processing links and executing monitoring result output.

[0025] S120. Based on traffic monitoring data, construct a traffic processing link according to the preset access layer, computing layer and storage layer.

[0026] Figure 2 This is a flowchart illustrating a specific implementation of S120 according to an example embodiment of this application. For example... Figure 2 As shown, the above-mentioned S120 includes: S121. At the access layer, traffic monitoring data is subjected to hierarchical buffering and rate limiting control to obtain the first traffic data after processing by the access layer.

[0027] In this step, a first buffer queue can be configured on the network access side to buffer the raw monitoring records accessed through carrier interfaces, leased lines, virtual private networks, or edge gateways. Furthermore, a second buffer queue can be configured on the service access process side to perform memory-level buffering of the traffic monitoring data output from the first buffer queue. Finally, a third buffer queue can be configured on the persistence side to persistently buffer the traffic monitoring data output from the second buffer queue that has not yet entered the computing layer for processing.

[0028] Then, bounded capacity thresholds are set for the first buffer queue, the second buffer queue, and the third buffer queue, and rate limiting and data dropping policies are implemented based on the capacity thresholds to control the scale of burst traffic accumulation on the access side.

[0029] Furthermore, the above-mentioned rate limiting and data discarding strategies based on capacity thresholds can specifically include: Configure the maximum queue length and / or maximum input bandwidth for the corresponding data source based on at least one dimension, including IoT SIM card identifier, tenant identifier, or access channel identifier. Specifically, during system initialization or configuration changes, based on historical traffic monitoring data, perform statistical analysis on the average traffic, peak traffic, and service importance of data sources corresponding to different IoT SIM card identifiers, tenant identifiers, and / or access channel identifiers to obtain the target maximum queue length and target maximum input bandwidth for each data source. Write the target maximum queue length and target maximum input bandwidth into the queue management configuration of the access layer. This allows the first, second, and third buffer queues to perform parameter matching based on the IoT SIM card identifier, tenant identifier, or access channel identifier corresponding to the current data source when performing enqueueing, and to determine whether to allow enqueueing and / or limit the amount of traffic monitoring data that the data source can receive per unit time.

[0030] When the actual input rate of any data source is detected to exceed a preset processing rate threshold and the duration exceeds a preset duration, rate limiting control is triggered for the data source to reduce the access rate of the traffic monitoring data corresponding to the data source. Specifically, an input rate statistics unit is configured for each data source on the access layer side. The input rate statistics unit performs real-time statistics on the number of traffic monitoring data entries and / or bytes arriving per unit time based on a sliding time window to obtain the actual input rate. Then, the actual input rate is compared with the preset processing rate threshold for that data source. When the actual input rate is detected to exceed the processing rate threshold for multiple consecutive sliding time windows and the cumulative duration exceeds the preset duration, a rate limiting control indication is generated. Based on the rate limiting control indication, the pull frequency of the data source is reduced, the single pull batch is reduced, and / or a proportional rejection process is performed on newly arriving traffic monitoring data, thereby reducing the actual input rate of the data source to within the allowable range.

[0031] When the queue length of any buffer queue reaches the corresponding capacity threshold, low-priority traffic monitoring data is discarded, retaining only the monitoring data necessary for aggregation statistics and anomaly detection. Specifically, before traffic monitoring data enters the first, second, and / or third buffer queues, each piece of traffic monitoring data is tagged with a priority label and a necessity label for aggregation statistics and anomaly detection based on the IoT SIM card's service importance, tenant level, monitoring scenario, and indicator type. Furthermore, the current queue length of each buffer queue is monitored in real time. When the queue length of any buffer queue is greater than or equal to the preset capacity threshold for that buffer queue, a queue compression operation is triggered. In the queue compression operation, data items with low-priority labels and non-necessary labels are prioritized for discarding. Time-slice aggregation can be performed on multiple consecutive detailed monitoring data from the same IoT SIM card, retaining only the monitoring data that reflects the total traffic and key anomaly characteristics of that time slice, thus reducing queue occupancy while ensuring the integrity of the data necessary for subsequent aggregation statistics and anomaly detection.

[0032] However, given the implementation of access-layer hierarchical buffering and rate limiting control for IoT SIM card monitoring data, and the need to discard some low-priority traffic monitoring data when the buffer queue reaches its capacity threshold, a further solution is needed: how to ensure that the same IoT SIM card retains at least a preset lower limit of monitoring data samples with a relatively even temporal distribution within any critical monitoring time window, while preventing queue overflow. This maintains the basic continuity of the time series used for anomaly detection, avoiding missed anomalies, false alarms, and a decline in subsequent behavioral analysis and audit traceability capabilities due to severe time series perforation.

[0033] To address this, the timeline can be divided into consecutive time windows of a preset length, so that each traffic monitoring data point corresponds to one of these time windows. Then, for each IoT SIM card and each time window, a count of the currently retained traffic monitoring data is maintained, and a minimum number of retained samples and / or a maximum allowable discard ratio are configured for each IoT SIM card in each time window. Next, when traffic monitoring data needs to be discarded, it is only discarded if the count of the corresponding IoT SIM card in the current time window is greater than the minimum number of retained samples and / or the corresponding discard ratio does not exceed the maximum allowable discard ratio. This ensures that the number of monitoring data samples retained by the same IoT SIM card in any time window is not less than a preset lower limit, thereby maintaining the basic continuity of the time series used for anomaly detection.

[0034] Specifically, IoT cards can be categorized into at least two types based on their business importance: critical IoT cards, high-priority IoT cards, ordinary IoT cards, and low-priority IoT cards. Then, a first time window length and a first minimum retained sample count are configured for critical IoT cards; a second time window length and a second minimum retained sample count are configured for ordinary IoT cards; and a third time window length and a third minimum retained sample count are configured for low-priority IoT cards. The first time window length is less than or equal to the second time window length, the second time window length is less than or equal to the third time window length, and the first minimum retained sample count is greater than the second minimum retained sample count, and the second minimum retained sample count is greater than or equal to the third minimum retained sample count.

[0035] When performing the discarding process, priority is given to discarding monitoring data of low-priority IoT cards that exceed the third minimum retention sample number within their corresponding time window. Next, ordinary IoT cards are discarded. When the buffer queue capacity constraint cannot be met, restricted discarding control is then performed on critical IoT cards to ensure the time-series monitoring integrity of critical IoT cards in rate-limited scenarios.

[0036] Furthermore, without exceeding the buffer queue capacity limit, it effectively ensures that the number of monitoring data samples retained by the same IoT card within any time window is not less than the preset lower limit, and the coverage of samples on the time axis is more uniform. This reduces the statistical distortion caused by time series punching, maintains the reliability of features such as total amount, mean, variance, and rate of change within the sliding window, reduces the false alarm rate and false positive rate of anomaly detection in traffic peak scenarios, and provides sufficient time series information support for subsequent behavior analysis, fault tracing, and auditing at the IoT card level.

[0037] Furthermore, it's worth noting that in IoT SIM card traffic monitoring systems, the network access side, service access process side, and persistence side typically operate as relatively independent modules. Their upstream data retrieval rates—such as the rate at which call detail records, traffic details, or real-time statistical records are retrieved in batches from operator interfaces—are often controlled by fixed configurations or simple time scheduling. Meanwhile, the processing capabilities of the downstream computing and storage layers are limited by the available CPU, memory, I / O bandwidth, and other resources. If the computing or storage layers experience a decline in processing capacity during peak traffic scenarios such as firmware upgrades or batch meter readings, and the upstream access layer continues to retrieve and push large amounts of traffic monitoring data at the predetermined rate, it will cause the multi-level buffer queues at the access layer to continuously expand. This will lead to queue overflow, passive data loss, and a sharp increase in monitoring latency, ultimately resulting in untimely monitoring of IoT SIM card traffic anomalies, incomplete monitoring results, or even monitoring failure.

[0038] To address this, for S121, backpressure information can be generated based on the operational status parameters of the computing layer and / or storage layer. This backpressure information is then fed back to the access layer, where it indicates the downstream processing capacity status. Finally, based on the backpressure information, the rate at which traffic monitoring data is retrieved from the operator interface and / or the collection frequency of certain monitoring indicators are dynamically adjusted to achieve backpressure control of the traffic monitoring data.

[0039] Specifically, lightweight monitoring probes can be deployed within the computing and storage layers to periodically sample operational status parameters such as processing queue length, CPU utilization, memory usage, database / storage system write latency, and I / O queue length. These continuous indicators can then be normalized into discrete backpressure levels or specific sets of backpressure parameters using threshold mapping functions or multi-level load grading algorithms.

[0040] Then, the generated backpressure information is transmitted back to the access layer in real time through the internal message channel or control bus. The access layer's pull scheduling module dynamically adjusts the pull strategy for the operator interface according to the backpressure level, including reducing the single batch, lengthening the pull cycle, and applying stricter frequency control parameters to low-priority data sources. Simultaneously, it drives the monitoring indicator acquisition module to reduce the sampling frequency of some non-critical indicators or to aggregate and collect them at a coarser time granularity.

[0041] After the back pressure is relieved, the access layer smoothly increases the pull rate and acquisition frequency in reverse according to the recovery of the back pressure level, so as to avoid jitter and oscillation behavior from full load to pause and then back to full load.

[0042] This enables the data access rate and monitoring index collection frequency of the access layer to be adaptively adjusted according to the real-time processing capabilities of the computing layer and / or storage layer. In scenarios with traffic surges or resource constraints, this avoids extreme queue accumulation, indiscriminate data loss, and excessive monitoring latency caused by the upstream continuously injecting data under high pressure when the downstream processing capacity is insufficient. It ensures that monitoring of key traffic behaviors and abnormal events can still be completed with a controllable queue size and latency even under limited resource conditions.

[0043] S122. In the computing layer, hierarchical feature calculation and anomaly detection degradation control are performed on the first traffic data to obtain the second traffic data after processing by the computing layer.

[0044] In this step, the traffic monitoring characteristics for IoT cards can be divided into at least primary characteristics and secondary characteristics. The primary characteristics include at least one or more of the following: total traffic per unit time, number of packets, zero traffic indication, and traffic doubling indication.

[0045] Optionally, features with computational complexity higher than that of a first-level feature can be defined as second-level and / or third-level features. Second-level features include at least one of the sliding window mean and sliding window variance, while third-level features include composite statistical features across multiple dimensions. Specifically, for each candidate feature, its single-record processing time, required memory overhead, and window state storage space can be pre-evaluated to obtain a complexity score. When the complexity score is higher than a first complexity threshold set for a first-level feature but lower than a second complexity threshold, the candidate feature is defined as a second-level feature; when the complexity score is higher than the second complexity threshold, the candidate feature is defined as a third-level feature; and different computation thread pools and state maintenance strategies are assigned to first-level, second-level, and third-level features respectively in the feature calculation module.

[0046] Then, when the resource load is within the normal threshold, the first-level, second-level, and third-level features are calculated in parallel on the first traffic data to generate a complete feature set. Specifically, the system state can be marked as normal based on the current CPU utilization, memory usage, and processing latency of the computing layer. Then, in the normal state, a multi-level feature calculation pipeline is started, dividing the first traffic data into multiple data shards according to the IoT card identifier, and submitting each data shard to the first-level feature calculation submodule, the second-level feature calculation submodule, and the third-level feature calculation submodule for parallel processing. After completing the calculation of each level of features, the corresponding feature results are aligned and aggregated according to the timestamp and IoT card identifier to form a complete feature set for subsequent anomaly detection.

[0047] When the computing resource load is detected to exceed a preset computing pressure threshold, degradation control is initiated. This maintains continuous computation of Level 1 features, performs sampling computation and / or delayed batch computation of Level 2 features, and suspends computation of Level 3 features and / or performs selective computation for high-risk IoT cards. Specifically, the system status is marked as degraded after CPU utilization and / or processing queue length are detected to exceed the computing pressure threshold. In degraded state, Level 1 feature computation (total traffic per unit time, number of packets, zero traffic indication, and traffic doubling indication) is continuously performed only on all IoT cards. Level 2 feature computation is performed only on certain time slices or on certain IoT cards according to a preset sampling ratio, and / or Level 2 feature computation is aggregated into batch computations with longer time windows. Simultaneously, Level 3 feature computation is suspended for ordinary IoT cards, and selective computation of Level 3 features is retained only for IoT cards pre-marked as high-risk, thus reducing overall computing pressure while ensuring basic anomaly detection capabilities.

[0048] Through the above steps, when the resource load is within the normal threshold, complete parallel computation of first-level features, second-level features, and third-level composite statistical features can be performed simultaneously on the first traffic data to form a complete feature set, ensuring high-precision and fine-grained anomaly detection capabilities during off-peak periods. When any of the following indicators is detected to exceed the preset computational pressure threshold: processing queue length, processing latency, CPU utilization, and / or memory occupancy, automatic degradation control is initiated: low-complexity first-level features are continuously computed for all IoT cards to ensure that basic anomaly types such as sudden flow interruptions and traffic surges are not missed; sampling computation and / or delayed batch computation are used for second-level features to reduce real-time computation while retaining the ability to identify mid-term trend anomalies; real-time computation of third-level features is suspended for ordinary IoT cards, and selective computation is performed only for IoT cards pre-marked as high-risk. This significantly reduces instantaneous computational pressure and queue backlog during traffic peaks, ensuring the continuity and real-time response of key monitoring capabilities, and effectively preventing the failure of anomaly detection due to the computational layer being overwhelmed during traffic peaks.

[0049] Furthermore, the aforementioned degrade control can be implemented by acquiring at least one or more of the following metrics: current processing queue length, processing latency, CPU utilization, and / or memory usage. When any metric exceeds its corresponding computational pressure threshold, the current system state is marked as a peak state. During the peak state, a set of features is selected for computation based on a preset feature priority table, ranked from highest to lowest priority, to ensure that basic anomaly types are not missed. Once the peak state is resolved and the metrics return to below their corresponding thresholds, full computation for all features resumes.

[0050] By comprehensively collecting and processing indicators such as queue length, processing latency, CPU utilization, and memory usage, the system accurately identifies peak conditions in the computing layer. During peak conditions, it selectively performs calculations on feature sets based on a preset feature priority table, reducing overall computational pressure while ensuring that basic anomaly types are not missed. Furthermore, it automatically resumes full calculations for all features after the peak conditions subside, thereby achieving continuous assurance of adaptive scheduling and monitoring capabilities for computing resources. This enhances the stability and effectiveness of IoT SIM card traffic monitoring in complex business scenarios.

[0051] S123. In the storage layer, perform hierarchical storage and batch write processing on the second traffic data to obtain the third traffic data that has been successfully stored.

[0052] It is worth noting that monitoring data usually needs to support near real-time queries and alarms, as well as meet the requirements for long-term retention. Therefore, a multi-storage system architecture with online and historical databases is commonly used. For example, aggregated results and alarm records for a recent period are stored in an in-memory key-value database or time-series database, while full detailed records are stored in a disk-based database, distributed file system, or object storage.

[0053] However, in practice, various monitoring data are often written directly to the corresponding storage system as single or fine-grained logs. That is, when data is written to disk, all types of data are often simply regarded as the same type of log event, and the same storage medium and writing strategy are used uniformly. As the scale of IoT cards continues to expand, monitoring indicators continue to increase, and traffic peaks occur frequently, both online and offline storage are prone to problems such as severe write amplification, disk I / O jitter, and sharp increase in write latency, which in turn affect the timeliness and reliability of monitoring results entering the database.

[0054] Therefore, in this step, the second traffic data can first be divided into hot data and cold data. Hot data includes at least the traffic aggregation results and abnormal alarm events generated based on the second traffic data within a preset time window, while cold data includes at least the original monitoring records corresponding to the second traffic data. The hot data is then written to a first storage system, which is either an in-memory key-value database or a time-series database. The cold data is written to a second storage system, which can be a disk-based database, a distributed file system, or object storage. Before writing the hot data and / or cold data to their respective storage systems, the data to be written is batch-aggregated based on preset time and / or quantity thresholds to form batch write units, which are then written to their corresponding storage systems.

[0055] Furthermore, by defining near-real-time frequently accessed aggregation results and abnormal alarm events as hot data, and defining the original monitoring records as cold data, these data are written to the first and second storage systems with different performance characteristics. Simultaneously, before writing, the data to be written is batch aggregated based on preset time thresholds and / or quantity thresholds. This addresses the technical problems of excessive write pressure on the storage system, unstable monitoring data entry, and large fluctuations in query latency during traffic peaks. It also achieves the technical effect of reducing write amplification and write jitter during traffic peaks, and improving the success rate and stability of monitoring data entry.

[0056] Furthermore, if all second-level traffic data, including aggregated statistics, anomaly markers, and raw details, is synchronously written to the existing first and / or second storage systems during peak traffic periods, without reserving dedicated write bandwidth for critical small-volume data (such as alarm markers and window aggregate statistics) when storage pressure increases, and without designing switchable bypass channels for massive amounts of raw detail data, then when write pressure surges, the continuous influx of raw details will rapidly increase the main storage write queue and disk I / O usage, directly squeezing the write resources for aggregated results and anomaly markers. Consequently, in peak traffic scenarios, there are technical problems such as storage write resources being heavily occupied by non-critical detail data, difficulty in timely and reliable storage of critical monitoring results, and insufficient overall stability of the storage system.

[0057] Therefore, one possible implementation of tiered storage and batch write processing for the second traffic data at the storage layer could be to first detect the write queue length, number of input / output operations, write latency, and / or CPU utilization of the first and / or second storage systems. When any storage metric exceeds the corresponding storage pressure threshold, peak bypass processing is triggered. Furthermore, during peak bypass processing, only aggregated statistical results and anomaly markers are written to the first storage system, the corresponding raw detailed data is compressed and written to the bypass storage medium, and the data in the bypass storage medium is offline processed and backfilled during low-load periods.

[0058] Specifically, a storage pressure detection module, a peak bypass decision module, and a bypass data management module can be introduced inside the storage layer. The storage pressure detection module continuously acquires key operating indicators of the first and second storage systems, including but not limited to write queue length, write IOPS, average write latency, and CPU utilization, and compares them with preset storage pressure thresholds. When any indicator exceeds the threshold, a peak trigger signal is output to the peak bypass decision module.

[0059] After receiving the trigger signal, the peak bypass decision module marks the system status as peak bypass state and issues a write path adjustment command, causing the write pipeline to perform differentiated processing on the newly generated second traffic data: smaller aggregated statistical results and anomaly markers continue to be written to the first storage system through the original write channel, while the corresponding original detailed data is routed to the bypass data management module, which then buckets, compresses, and writes the detailed data to the bypass storage medium (such as an independent object storage space or a low-priority file system path) according to the IoT card identifier, time interval, and other dimensions.

[0060] Then, after the system load recovers and various storage indicators fall below their corresponding thresholds, the peak bypass decision module switches back to normal mode. The bypass data management module then reads the compressed original detailed data in batches from the bypass storage medium according to a preset backfill strategy and rate control. After decompression, reordering, and deduplication verification, the data is written back to the target secondary storage system or historical database at a controlled rate. This gradually restores the integrity of the monitoring data in the main storage without significantly increasing the instantaneous write pressure on the main storage. This ensures that when facing peak storage pressure, critical monitoring results are prioritized for storage, while large volumes of detailed data are temporarily stored through the bypass channel and supplemented in subsequent periods. This achieves comprehensive optimization of real-time performance, stability, and data integrity under limited physical storage resources.

[0061] S130. Based on the current resource load status and IoT card service behavior, dynamically adjust the processing strategies of the access layer, computing layer and storage layer using the traffic processing link.

[0062] In this step, the processing strategies of the access layer, computing layer, and storage layer can be dynamically adjusted based on the current resource load status and IoT card service behavior, in order to control computing pressure and / or storage pressure in traffic peak scenarios and obtain monitoring results on traffic monitoring requests.

[0063] Figure 3 This is a flowchart illustrating a specific implementation of S130 according to an example embodiment of this application. For example... Figure 3 As shown, the above-mentioned S130 includes: S131. Obtain the resource load status information at the current moment.

[0064] Specifically, it can be to obtain the resource load status information at the current moment, wherein the resource load status information includes at least one or more of the following: access layer queue length, computing layer CPU utilization, and / or storage layer write queue length.

[0065] S132. Obtain IoT card service behavior plan information.

[0066] Specifically, it could involve obtaining IoT SIM card service behavior plan information, which includes firmware upgrade plans, parameter distribution plans, and / or batch meter reading cycles.

[0067] S133. Based on the IoT card business behavior plan information, predict the timing and intensity of traffic peaks within a future preset time window.

[0068] One possible implementation involves statistically analyzing the actual traffic curves corresponding to each type of business activity plan based on historical traffic monitoring data to obtain typical peak characteristic parameters for each type of business activity. Then, based on the current business activity plan information and the typical peak characteristic parameters, the expected peak traffic, peak duration, and affected IoT SIM card set within a preset time window are determined. The expected peak traffic is compared with the current system's carrying capacity to obtain expansion demand information and drive resource pre-expansion.

[0069] In another possible implementation, the prediction of the timing and intensity of traffic peaks can be based on firmware upgrade plans, parameter distribution plans, and / or batch meter reading cycles. For each type of IoT SIM card service behavior plan, the corresponding plan effective time, duration, number of IoT SIM cards involved, and / or IoT SIM card distribution area can be obtained by parsing the plan. Then, historical samples that match the current IoT SIM card service behavior plan in terms of service type, plan effective time period, and / or number of IoT SIM cards involved can be selected from historical traffic monitoring data. Based on the historical samples, at least one or more of the following parameters can be statistically analyzed for each IoT SIM card under the corresponding service behavior: average traffic increment, peak amplification, peak arrival time offset, and peak duration, in order to generate typical peak characteristic parameters corresponding to the IoT SIM card service behavior plan.

[0070] Then, based on the number of IoT cards and / or the distribution area of ​​IoT cards involved in the IoT card business behavior plan information, the typical flood peak characteristic parameters are linearly amplified, time-division and zone-based superimposed and / or corrected according to the preset safety margin, so as to obtain the target peak flow, the target peak duration and the estimated flood peak start time and / or flood peak end time within the preset time window.

[0071] Finally, the target peak flow, target peak duration, and estimated peak start and / or peak end times are used as prediction results of the peak flow period and peak intensity, and provided to the access layer, computing layer, and storage layer to drive resource pre-expansion and / or threshold pre-adjustment.

[0072] S134. Perform resource pre-expansion and / or threshold pre-adjustment on the access layer, computing layer and storage layer.

[0073] Then, within a preset preparation time before the flood peak occurs, resources are pre-expanded and / or thresholds are pre-adjusted for the access layer, computing layer, and storage layer to improve the capacity of the traffic processing link during the predicted flood peak period.

[0074] Furthermore, on the access layer side, the number of access processes can be increased and / or the buffer queue capacity threshold can be raised, and rate limiting configuration can be enabled in advance. On the compute layer side, the number of compute instances used to perform feature calculations and anomaly detection can be increased in advance, and a lower degradation trigger threshold can be configured to make degradation control take effect earlier. On the storage layer side, the number of nodes and / or write concurrency of the first storage system and / or the second storage system can be increased, and peak bypass strategy can be enabled in advance.

[0075] In this embodiment, by acquiring traffic monitoring data from traffic monitoring requests for at least one IoT SIM card, a traffic processing link is constructed according to a preset access layer, computing layer, and storage layer. Then, based on the current resource load status and IoT SIM card service behavior, the processing strategies of the access layer, computing layer, and storage layer are dynamically adjusted using the traffic processing link to control computing pressure and / or storage pressure under traffic peak scenarios, and to obtain monitoring results for traffic monitoring requests. This enables stable operation of IoT SIM card traffic monitoring under sudden traffic peak conditions and ensures the real-time and completeness of monitoring results.

[0076] exist Figure 1 In the embodiment shown, the monitoring data received by the access layer is sent to the computing layer without discrimination. The computing layer performs card-level statistics and anomaly detection on all IoT cards according to a unified monitoring cycle. As a result, even if some IoT cards only carry low-value or non-real-time sensitive services during peak periods, the system still allocates computing and storage resources to them as equivalent to those of critical IoT cards.

[0077] In response, Figure 1 Based on the illustrated embodiment, Figure 4 This is a flowchart illustrating a traffic monitoring method for an IoT SIM card according to another exemplary embodiment of this application, as shown below. Figure 4 As shown, in Figure 1 In S130 of the illustrated embodiment, it further includes: S210. When any resource load indicator is detected to exceed the corresponding peak threshold, the IoT card is divided into critical IoT cards and non-critical IoT cards.

[0078] Specifically, when any resource load indicator is detected to exceed the corresponding peak threshold, the IoT cards are divided into critical IoT cards and non-critical IoT cards, including: obtaining at least one of the following as resource load indicators: access layer queue length, computing layer CPU utilization, storage layer write queue length and / or write latency.

[0079] When any resource load indicator exceeds the pre-configured peak threshold and the duration exceeds the preset duration, the system status will be marked as peak state.

[0080] During peak flood conditions, based on at least one of the following: the industry, business level, business scenario, alarm history, and / or manually configured priority information of the IoT card, all IoT cards in the network are assessed for importance. IoT cards that meet the preset importance conditions are marked as critical IoT cards, and the remaining IoT cards are marked as non-critical IoT cards, so as to form a set of IoT cards with critical / non-critical attribute tags.

[0081] S220: Maintain granular monitoring of key IoT cards.

[0082] In this step, the key IoT cards are monitored at the original granularity, including card-level traffic statistics and anomaly detection within the original monitoring period.

[0083] Specifically, after the system status is marked as a peak state, for IoT cards marked with key IoT cards, their original first monitoring cycle remains unchanged, and the traffic monitoring data of each key IoT card continues to be collected, statistically analyzed, and detected as anomalies at the card level within the first monitoring cycle.

[0084] Among them, card-level statistics include at least one or more of the following: total traffic per unit time, number of messages, zero traffic indication and / or traffic doubling indication. Card-level anomaly detection includes at least anomaly rule detection based on primary features and / or secondary features, so as to ensure that key IoT cards maintain the same monitoring frequency and monitoring granularity as those in off-peak conditions during peak periods.

[0085] S230, Adjust the monitoring granularity for non-critical IoT cards.

[0086] In this step, the monitoring granularity is adjusted for non-critical IoT cards, including adjusting the monitoring period from the first monitoring period to a second monitoring period that is longer than the first monitoring period, and / or performing only project-level or tenant-level aggregated traffic monitoring on non-critical IoT cards.

[0087] Specifically, after the system status is marked as a flood peak, the monitoring cycle of IoT cards marked with non-critical IoT cards will be adjusted from the first monitoring cycle to the second monitoring cycle, and the second monitoring cycle will be longer than the first monitoring cycle.

[0088] During the second monitoring cycle, card-level statistics and anomaly detection for individual non-critical IoT cards are abandoned. Instead, traffic monitoring data for multiple non-critical IoT cards are aggregated only by project identifier and / or tenant identifier to generate aggregated traffic statistics at the project or tenant level. Anomaly detection is then performed at the project or tenant level to reduce the monitoring overhead of non-critical IoT cards while retaining their traffic monitoring and anomaly detection capabilities at the aggregation level.

[0089] Optionally, during off-peak hours, the first monitoring cycle can be set to a duration of 5 to 10 seconds, and traffic data for each IoT SIM card can be collected, statistically analyzed, and anomaly detected according to this first monitoring cycle. During peak hours, the second monitoring cycle can be set to a duration of 30 seconds or more, and traffic data for non-critical IoT SIM cards can be collected, statistically analyzed, and anomaly detected according to this second monitoring cycle. After the peak hours subside, the monitoring cycle for non-critical IoT SIM cards can be restored to the first monitoring cycle.

[0090] Figure 5 This is a schematic diagram illustrating the structure of an IoT card traffic monitoring system according to an example embodiment of this application. Figure 5 As shown, the IoT SIM card traffic monitoring system 300 provided in this embodiment includes: The monitoring request acquisition module 310 is used to acquire traffic monitoring requests for at least one IoT card, wherein the traffic monitoring requests correspond to traffic monitoring data generated by the IoT card accessing the network side. The traffic processing link construction module 320 is used to construct a traffic processing link based on the traffic monitoring data, according to a preset access layer, computing layer and storage layer. The peak traffic monitoring module 330 is used to dynamically adjust the processing strategies of the access layer, the computing layer, and the storage layer based on the current resource load status and IoT card service behavior, using the traffic processing link, so as to control the computing pressure and / or storage pressure in the traffic peak scenario and obtain the monitoring results of the traffic monitoring request.

[0091] Figure 6 This is a schematic diagram of the structure of an electronic device according to an example embodiment of this application. For example... Figure 6 As shown, the electronic device 400 provided in this embodiment includes: a processor 401 and a memory 402; wherein: Memory 402 is used to store computer programs, and the memory may also be flash memory.

[0092] Processor 401 is used to execute the execution instructions stored in the memory to implement the various steps in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0093] Alternatively, the memory 402 can be either standalone or integrated with the processor 401.

[0094] When the memory 402 is a device independent of the processor 401, the electronic device 400 may further include: Bus 403 is used to connect the memory 402 and the processor 401.

[0095] This embodiment also provides a readable storage medium storing a computer program, which, when executed by at least one processor of an electronic device, enables the electronic device to perform the methods provided in the various embodiments described above.

[0096] This embodiment also provides a program product including a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the methods provided in the various embodiments described above.

[0097] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0098] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for monitoring the traffic of an IoT SIM card, characterized in that, include: Obtain a traffic monitoring request for at least one IoT SIM card, the traffic monitoring request corresponding to traffic monitoring data generated by the IoT SIM card accessing the network side; Based on the traffic monitoring data, a traffic processing link is constructed according to a preset access layer, computing layer, and storage layer; Based on the current resource load status and IoT card service behavior, the processing strategies of the access layer, the computing layer, and the storage layer are dynamically adjusted using the traffic processing link to control computing pressure and / or storage pressure during traffic peak scenarios, and to obtain monitoring results for the traffic monitoring request.

2. The method for monitoring the traffic of an IoT SIM card according to claim 1, characterized in that, The step of constructing a traffic processing link based on the traffic monitoring data, according to a preset access layer, computing layer, and storage layer, includes: At the access layer, the traffic monitoring data is subjected to hierarchical buffering and rate limiting control to obtain the first traffic data after processing by the access layer. In the computing layer, hierarchical feature calculation and anomaly detection and degradation control are performed on the first traffic data to obtain the second traffic data after processing by the computing layer; In the storage layer, the second traffic data is subjected to hierarchical storage and batch write processing to obtain the third traffic data that has been successfully stored.

3. The method for monitoring the traffic of an IoT card according to claim 2, characterized in that, The step of performing tiered buffering and rate limiting control on the traffic monitoring data at the access layer includes: Configure a first buffer queue for the traffic monitoring data on the network access side; Configure a second buffer queue for the traffic monitoring data on the service access process side; Configure a third buffer queue for the traffic monitoring data on the persistent side; Bounded capacity thresholds are set for the first buffer queue, the second buffer queue, and the third buffer queue, respectively, and rate limiting and data dropping strategies are executed based on the capacity thresholds to control the scale of burst traffic accumulation on the access side.

4. The method for monitoring the traffic of an IoT card according to claim 3, characterized in that, The step of setting bounded capacity thresholds for the first buffer queue, the second buffer queue, and the third buffer queue, and executing rate limiting and data discarding policies based on the capacity thresholds, includes: Configure the maximum queue length and / or maximum input bandwidth for the corresponding data source based on at least one of the following dimensions: IoT SIM card identifier, tenant identifier, or access channel identifier; When the actual input rate of any data source is detected to be greater than the preset processing rate threshold and the duration exceeds the preset duration, rate limiting control is triggered on the data source to reduce the access rate of the traffic monitoring data corresponding to the data source. When the queue length of any buffer queue is detected to have reached the corresponding capacity threshold, low-priority traffic monitoring data is discarded, and only the monitoring data necessary for aggregation statistics and anomaly detection is retained.

5. The method for monitoring the traffic of an IoT card according to claim 3, characterized in that, The step of performing hierarchical buffering and rate limiting control on the traffic monitoring data at the access layer further includes: Back pressure information is generated based on the operating status parameters of the computing layer and / or the storage layer; The back pressure information is fed back to the access layer, wherein the back pressure information is used to indicate the downstream processing capability status; Based on the back pressure information, adjust the rate at which the traffic monitoring data is retrieved from the operator interface and / or adjust the collection frequency of some monitoring indicators.

6. The method for monitoring the traffic of an IoT card according to claim 2, characterized in that, The computational layer performs hierarchical feature calculation and anomaly detection and degradation control on the first traffic data, including: The traffic monitoring characteristics of the IoT card are divided into multiple levels of features; And when the resource load is within the normal threshold, perform a full calculation of multi-level features in parallel on the first traffic data to generate a complete feature set; When the computing resource load is detected to exceed the preset computing pressure threshold, degradation control is initiated.

7. The method for monitoring the traffic of an IoT card according to claim 6, characterized in that, The step of initiating degradation control when the computing resource load is detected to exceed a preset computing pressure threshold includes: Obtain at least one or more of the following metrics: current processing queue length, processing latency, CPU utilization, and / or memory usage. When any indicator exceeds the corresponding calculated pressure threshold, the current system state is marked as a flood peak state; During the flood peak, based on a preset feature priority table, feature sets are selected for calculation according to priority from high to low to ensure that basic anomaly types are not missed. When the flood peak state is lifted and the index returns to below the corresponding threshold, the full calculation for all features is resumed.

8. The method for monitoring the traffic of an IoT card according to claim 2, characterized in that, The process of performing tiered storage and batch write processing on the second traffic data in the storage layer includes: The second flow data is divided into hot data and cold data; The hot data is written into a first storage system, which is a memory-based key-value database or a time-series database. The cold data is written to a second storage system, which is a disk-based database, a distributed file system, or an object storage system. Before writing the hot data and / or the cold data into the corresponding storage system, the data to be written is batch aggregated to form batch writing units based on a preset time threshold and / or quantity threshold, and then the batch writing units are written into the corresponding storage system.

9. The method for monitoring the traffic of an IoT card according to claim 1, characterized in that, The method of dynamically adjusting the processing strategies of the access layer, the computing layer, and the storage layer based on the current resource load status and IoT card service behavior includes: Obtain the resource load status information at the current moment, which includes at least one or more of the following: access layer queue length, computing layer CPU utilization, and / or storage layer write queue length. Obtain IoT card service activity plan information; Based on the IoT card service behavior plan information, predict the timing and intensity of traffic peaks within a future preset time window; Within a preset preparation time before the flood peak occurs, the access layer, the computing layer, and the storage layer are pre-expanded in terms of resources and / or pre-adjusted in terms of thresholds.

10. The method for monitoring the traffic of an IoT card according to claim 9, characterized in that, The pre-emptive period before the flood peak occurs, during which resources are pre-expanded and / or thresholds are pre-adjusted for the access layer, the computing layer, and the storage layer, includes: On the access layer side, increase the number of access processes and / or raise the buffer queue capacity threshold; On the computation layer side, the number of computation instances used to perform feature computation and anomaly detection is pre-expanded, and a lower degradation trigger threshold is configured. On the storage layer side, increase the number of nodes and / or write concurrency of the first and / or second storage systems.

11. The method for monitoring the traffic of an IoT card according to claim 9, characterized in that, The step of predicting the peak traffic period and intensity within a future preset time window based on the IoT card service behavior plan information includes: Based on historical traffic monitoring data, statistical analysis is performed on the actual traffic curves corresponding to each type of business behavior plan to obtain typical peak characteristic parameters corresponding to each type of business behavior. Based on the current business activity plan information and the typical flood peak characteristic parameters, determine the expected peak traffic, peak duration, and affected IoT SIM card set within the preset time window; The expected peak traffic is compared with the current system's carrying capacity to obtain expansion demand information and drive resource pre-expansion.

12. The method for monitoring the traffic of an IoT card according to claim 1, characterized in that, The method of dynamically adjusting the processing strategies of the access layer, the computing layer, and the storage layer based on the current resource load status and IoT card service behavior also includes: When any resource load indicator is detected to exceed the corresponding peak threshold, the IoT card will be classified into critical IoT cards and non-critical IoT cards. Maintain the original granularity of monitoring for the key IoT cards, including performing card-level traffic statistics and anomaly detection within the original monitoring period; Adjusting the monitoring granularity of the non-critical IoT card includes changing the monitoring period from a first monitoring period to a second monitoring period that is longer than the first monitoring period.

13. The method for monitoring the traffic of an IoT card according to claim 12, characterized in that, The step of adjusting the monitoring period from a first monitoring period to a second monitoring period that is longer than the first monitoring period includes: During off-peak hours, the first monitoring period is set to a duration between 5 and 10 seconds, and traffic data for each IoT card is collected, statistically analyzed, and anomaly detected according to the first monitoring period. During peak flood conditions, the second monitoring period is set to a duration of 30 seconds or more, and the traffic data of the non-critical IoT cards is collected, statistically analyzed, and anomaly detected according to the second monitoring period. After the flood peak is lifted, the monitoring cycle of the non-critical IoT card will be restored to the first monitoring cycle.

14. A traffic monitoring system for an IoT SIM card, characterized in that, include: The monitoring request acquisition module is used to acquire traffic monitoring requests for at least one IoT card, wherein the traffic monitoring request corresponds to traffic monitoring data generated by the IoT card accessing the network side; The traffic processing link construction module is used to construct a traffic processing link based on the traffic monitoring data, according to a preset access layer, computing layer, and storage layer. The peak traffic monitoring module is used to dynamically adjust the processing strategies of the access layer, the computing layer, and the storage layer based on the current resource load status and IoT card service behavior, using the traffic processing link, in order to control computing pressure and / or storage pressure in traffic peak scenarios and obtain monitoring results about the traffic monitoring request.

15. An electronic device, characterized in that, include: processor; as well as, Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 13 by executing the executable instructions.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 13.

Images