Data processing method and device, communication module and storage medium

Abstract

The embodiment of the invention relates to a data processing method and device, a communication module and a storage medium, when it is detected that a network of the communication module is disconnected, data to be uploaded to a server side is cached to cache regions according to priorities, the communication module comprises a plurality of cache regions, and each cache region is used for storing data of different priorities; when detecting that the network of the communication module is recovered, determining a data uploading strategy of the communication module according to the bandwidth and the cache load of the communication module; and uploading the data in the cache region to a server according to the data uploading strategy. Therefore, the data of the communication module can be cached differentially according to the priority when the network is disconnected, key data is prevented from being lost or covered due to competition of cache resources, and the utilization efficiency of the cache resources is improved. After the network is recovered, the uploading strategy is dynamically adjusted based on the bandwidth and the cache load, network congestion is reduced, key data are guaranteed to be preferentially and timely uploaded, and the real-time performance and safety of data transmission are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication module technology, and in particular to a data processing method, apparatus, communication module and storage medium. Background Technology

[0002] In the field of IoT edge computing and embedded communication technology, existing IoT devices typically address weak or outage conditions by setting up local caches on the terminal side to temporarily store data generated during network outages in Flash or SD cards, and then uploading it all at once after the network is restored, in order to ensure that data is not lost during network outages.

[0003] However, existing solutions generally employ a uniform caching structure, storing data of varying importance equally without distinguishing between security data and ordinary logs. This can lead to critical data being overwritten by low-value data when cache space is limited. Furthermore, the lack of a scheduling mechanism that combines bandwidth status and data timeliness after network recovery can easily cause upload congestion or delayed recovery of critical data. Therefore, ensuring that data from the terminal can be securely and efficiently uploaded to the server has become a pressing issue that needs to be addressed. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the embodiments of the present invention provide a data processing method, apparatus, communication module and storage medium.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, including: When the network disconnection of the communication module is detected, the data to be uploaded to the server is cached in the cache area according to the priority. The communication module contains multiple cache areas, and each cache area is used to store data with different priorities. When network recovery of the communication module is detected, the data upload strategy of the communication module is determined based on the bandwidth and cache load of the communication module. The data in the cache is uploaded to the server according to the data upload strategy.

[0006] In one possible implementation, caching the data to be uploaded to the server in a cache according to priority includes: Obtain the cache utilization rate for each of the aforementioned cache areas; The priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined. When the cache utilization rate of the first cache area is less than the first threshold, the data is cached in the first cache area; When the cache utilization rate of the first cache area is greater than or equal to the first threshold, the data is cached in the second cache area. The priority of the second cache area is lower than that of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

[0007] In one possible implementation, determining the priority of the data based on its data type to be uploaded to the server, and determining the first cache corresponding to the priority, includes: When the data type is a first type, the data is determined to be of the first priority, and the cache corresponding to the first priority is determined. The data of the first type represents the data processed in real time. When the data type is the second type, the data is determined to be of the second priority, and the regular cache corresponding to the second priority is determined. The data of the second type represents the data processed at each first time interval. The second priority is lower than the first priority, and the capacity of the cache is smaller than the capacity of the regular cache. When the data type is the third type, the data is determined to be of the third priority, and the low-speed cache corresponding to the third priority is determined. The third type of data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache is less than the capacity of the low-speed cache.

[0008] In one possible implementation, after caching the data to be uploaded to the server into a cache according to priority, the method further includes: The cache area is cleaned up using a first-in-first-out strategy and a timestamp eviction mechanism. For the regular cache area, the cached data is cleaned up in a rolling manner according to time windows; For the low-speed cache area, compression and background batch uploading are used to transfer the cached data.

[0009] In one possible implementation, determining the data upload strategy of the communication module based on the bandwidth and cache load of the communication module includes: When the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is uploaded to the server in descending order of priority of the cache area. When the bandwidth is less than the fifth threshold, or the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups according to the priority of the cache area from high to low, and one data group is uploaded to the server every third time interval. The fifth threshold is less than the third threshold, and the sixth threshold is greater than the fourth threshold.

[0010] In one possible implementation, after the network recovery of the communication module is detected, the method further includes: Obtain the network disconnection duration of the communication module; When the network disconnection duration exceeds the seventh threshold, a second upload operation is performed on the data cached in the cache area to the server.

[0011] In one possible implementation, after caching the data to be uploaded to the server into a cache according to priority, the method further includes: Determine the retention period for data of each priority level; When the data's retention period expires, the data will be deleted.

[0012] In a second aspect, embodiments of the present invention provide a data processing apparatus, comprising: The caching module is used to cache the data to be uploaded to the server in a cache area according to priority when the network disconnection of the communication module is detected. The communication module includes multiple cache areas, and each cache area is used to store data with different priorities. The determination module is used to determine the data upload strategy of the communication module based on the bandwidth and cache load of the communication module when the network recovery of the communication module is detected. The upload module is used to upload data in the cache to the server according to the data upload strategy.

[0013] Thirdly, embodiments of the present invention provide a communication module, including: a processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method described in any one of the first aspects above.

[0014] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the data processing method described in any one of the first aspects.

[0015] The data processing scheme provided in this invention caches data to be uploaded to the server according to priority when a network disconnection is detected in the communication module. The communication module includes multiple caches, each storing data of different priorities. When the network of the communication module is restored, a data upload strategy is determined based on the bandwidth and cache load of the communication module. Data in the caches is then uploaded to the server according to the data upload strategy. This allows for differentiated caching of data by priority during network outages, preventing critical data from being lost or overwritten due to cache resource contention and improving cache resource utilization efficiency. After network restoration, the upload strategy is dynamically adjusted based on bandwidth and cache load to reduce network congestion, ensuring priority and timely upload of critical data and improving the real-time performance and security of data transmission. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another data processing method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a communication module provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0019] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method specifically includes: S11. When the network disconnection of the communication module is detected, the data to be uploaded to the server is cached in the cache area according to the priority. The communication module contains multiple cache areas, and each cache area is used to store data with different priorities.

[0020] The data processing method provided in this invention is applied to a communication module, which can be a WiFi module, a 4G / 5G module, etc., and can be applied to at least the following scenarios: Industrial Internet of Things (e.g., production line equipment status monitoring (temperature, voltage data), workshop security alarm signal transmission), smart security (e.g., camera abnormal behavior capture, real-time transmission of access control alarm information), and home scenarios (e.g., key status and alarm information transmission of smart home devices (smart door locks, gas alarms)). The communication module can have a built-in data classification module, multiple buffers, a network status detection module, and a dynamic upload control module, etc., to achieve independent data caching and upload scheduling without relying on an external main control chip. Specifically, when the network is down, the data of the communication module can be cached differently according to priority, and after the network is restored, the upload strategy can be dynamically adjusted based on bandwidth and cache load for data upload.

[0021] In this embodiment, when the communication module determines that the network is disconnected through its built-in network monitoring module (for example, the network status monitoring module detects the connection status by periodically sending heartbeat packets (Ping). Once no response is received for three consecutive times, it is determined that the network is disconnected), the local caching mechanism is immediately triggered. The data to be uploaded to the server in real time by the communication module (e.g., security alarms, operating parameters, logs, etc.) is first divided into high, medium, and low priority levels according to preset rules or a lightweight model by the built-in classification module, representing the priority of the data to be uploaded to the server.

[0022] Multiple buffers are pre-configured within the communication module. Each buffer stores data corresponding to a specific priority level; that is, each priority level corresponds to a separate buffer. Data is then written to the corresponding buffer within the communication module according to its priority.

[0023] In one possible implementation, the data to be uploaded to the server is cached in a cache area according to priority, including: Get the cache utilization rate of each cache area; determine the priority of the data based on the data type to be uploaded to the server, and determine the first cache area corresponding to the priority; when the cache utilization rate of the first cache area is less than the first threshold, cache the data in the first cache area; when the cache utilization rate of the first cache area is greater than or equal to the first threshold, cache the data in the second cache area, the priority of the second cache area is lower than the priority of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

[0024] In this embodiment, the usage of each buffer is periodically monitored during operation, and the current occupancy rate of each buffer is used as the buffer utilization rate, which is then used as the basis for determining data write scheduling. When new data is generated, the communication module first matches the corresponding priority tag according to the data type. This priority tag is used to determine the first buffer where the data should be written.

[0025] After determining the first cache area, the system reads the current cache utilization rate of the first cache area. If the cache utilization rate is less than the preset first threshold, it means that the first cache area can still accept new data. The data is then directly stored in the first cache area, thereby ensuring the fast disk write-to-disk write-to-disk and priority protection of high-priority data.

[0026] When the cache utilization of the first cache area is greater than or equal to the first threshold, indicating that the first cache area is nearing full or has reached full capacity, the communication module triggers a cache degradation write strategy to prevent critical caches from being further squeezed out. This strategy involves finding the second cache area corresponding to the next lower priority level and transferring the data to the second cache area. The second cache area is typically used to store data types with lower priority than the first cache area, and its design purpose is to bear some of the data write pressure when high-priority areas are insufficient. Before writing, the system also checks the current cache utilization of the second cache area. If its cache utilization is lower than the corresponding second threshold, data writing is allowed to avoid cache overflow or cascading blocking. This implements a cache strategy that prioritizes writing to high-priority caches and smoothly degrades them after the high-priority areas are full. This ensures that critical data is written to disk first while also improving the overall utilization efficiency of cache resources and the system's sustainable operation capability during network outages.

[0027] In one possible implementation, the priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined, including: When the data type is type 1, the data is determined to have the first priority, and a corresponding high-speed cache area is determined. Type 1 data represents data processed in real time. When the data type is type 2, the data is determined to have the second priority, and a corresponding regular cache area is determined. Type 2 data represents data processed at a first interval. The second priority is lower than the first priority, and the capacity of the high-speed cache area is smaller than the capacity of the regular cache area. When the data type is type 3, the data is determined to have the third priority, and a corresponding low-speed cache area is determined. Type 3 data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache area is smaller than the capacity of the low-speed cache area.

[0028] In this embodiment, when the communication module receives data output from a sensor, device status monitoring module, or log module, it identifies the data type. Based on pre-defined data classification rules, the system can determine whether the data belongs to a first type, a second type, or a third type.

[0029] When data is identified as Type 1, it is categorized as data requiring real-time processing, such as security alarms, fault trigger signals, or real-time control feedback information. This type of data has high timeliness and is therefore given first priority, being written to a cache for priority storage. The cache has fast access speed and a small capacity (e.g., 16KB) to ensure that this type of critical data can be written to disk immediately and prevented from being overwritten by other data.

[0030] When data is identified as type II, it is treated as periodically updated data, i.e., regular operating parameters, such as device operating status parameters, temperature, current, voltage, and speed, which are typically refreshed at short, fixed time intervals. This type of data is classified as second priority and is stored in the regular cache. The capacity of the regular cache is larger than that of the high-speed cache (e.g., 64KB), enabling it to handle higher volumes of periodic data while maintaining a certain write speed.

[0031] When data is identified as type three, the system treats it as low-frequency or low-latency-sensitive background data, such as non-critical logs, debugging information, periodic status snapshots, statistical records, or device-retained information. This type of data is generally processed at longer time intervals. The system classifies this type of data as the third priority and writes it to a low-speed cache for storage. The low-speed cache has a maximum capacity (e.g., 128KB), allowing the system to store more non-real-time data during long network outages without consuming high-priority cache resources. Priority refers to the importance of the data, i.e., the order in which it is uploaded to the server, with first priority > second priority > third priority.

[0032] In one possible implementation, after caching the data to be uploaded to the server in a cache according to priority, the method further includes: For the high-speed cache, a first-in-first-out strategy and a timestamp eviction mechanism are used to clean up the cached data; for the regular cache, the cached data is cleaned up in a rolling manner according to a time window; for the low-speed cache, the cached data is transferred using compression and background batch upload.

[0033] In this embodiment, during the continuous operation of the communication module, each cache area will adopt different cleanup and transfer strategies according to the data type and priority it carries, so as to ensure that the cache resources are available for a long time and avoid overflow.

[0034] For the high-priority cache storing the most critical data, which has the highest real-time requirements and the most stringent time-sensitive protection requirements, as the cache gradually approaches full, the system will prioritize clearing the earliest written data in a first-in-first-out (FIFO) manner. Simultaneously, it will combine timestamp records to determine if the data has exceeded its effective protection period, and only perform the eviction operation after the retention period has expired. This method ensures that the high-priority cache continuously has space to accommodate new critical data, thereby maintaining the ability to prioritize real-time and urgent information for disk storage.

[0035] For the regular cache storing second-priority data, the system performs rolling cleanup of the stored data according to time windows. As the time window moves forward, older data that is earlier than the current time window is automatically identified as releasable and cleared, thus achieving periodic data reclamation management. This approach is suitable for the tiered retention and updating of runtime parameter data, avoiding disordered growth while ensuring that data within a certain time span is accessible.

[0036] For the low-speed cache area storing third-priority data, since the data stored in this area is mostly logs or long-cycle information, the system compresses the data before transferring it during cleanup to reduce space usage. After the network recovers or the system is idle, the data in the low-speed cache area will be synchronized to the server in batches in the background, thereby avoiding bandwidth interference to the uploading process of high-priority data.

[0037] By employing the aforementioned differentiated cleanup strategy, different cache areas are managed in a targeted manner based on the importance and frequency of data usage, thereby achieving high-priority protection, periodic management, and efficient transfer of delayed data under limited storage conditions.

[0038] S12. When the network recovery of the communication module is detected, the data upload strategy of the communication module is determined based on the bandwidth and cache load of the communication module.

[0039] In this embodiment, when the network monitoring module detects network reconnection, the communication module performs a real-time assessment of the restored bandwidth quality and reads the current load status of each cache, including information such as storage occupancy and data wait time, to obtain bandwidth and cache load. Thresholds corresponding to bandwidth and cache load can be set to determine bandwidth status and cache load status, respectively. Different data upload strategies are applied when bandwidth and cache load are in different states.

[0040] Specifically, the communication module automatically generates an upload strategy based on preset rules: when bandwidth is sufficient and the overall load of the cache is low, data is uploaded to the server in descending order of priority, that is, high-priority data is uploaded first, and medium and low-priority data are uploaded gradually after the high-priority data is uploaded; when bandwidth is poor or the cache load is high, a batch upload method is adopted, and the upload ratio of low-priority data is limited to ensure that critical data can be restored first.

[0041] In one possible implementation, when the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is uploaded to the server in descending order of priority corresponding to the cache area; when the bandwidth is less than the fifth threshold, or the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups in descending order of priority corresponding to the cache area, and one data group is uploaded to the server every third time interval, where the fifth threshold is less than the third threshold and the sixth threshold is greater than the fourth threshold.

[0042] In this embodiment, once the communication module detects that the network has been restored, the system first assesses the current available bandwidth and the overall usage of the local cache. Based on the assessment results, the system selects different data upload strategies to balance real-time performance and network stability.

[0043] When the current bandwidth is detected to be greater than the third threshold and the cache load is less than the fourth threshold (e.g., bandwidth > 10 Mbps and cache load < 70%), it indicates sufficient upload capacity. At this time, the data upload strategy of the communication module is set to sequential upload mode. In this mode, data to be uploaded is selected in descending order of priority according to the corresponding cache area. Critical data in the high-speed cache area is transmitted to the server first, and then the remaining data in the regular cache area and the low-speed cache area are processed in turn, ensuring that critical data can be synchronized as quickly as possible.

[0044] When the system detects that bandwidth is below the fifth threshold or cache load is above the sixth threshold (bandwidth < 5Mbps or cache load > 80%), it indicates that continuing to upload in large batches may cause network congestion or further increase cache pressure. Therefore, it switches to batch upload mode. In this mode, the system still extracts data from the cache according to priority from high to low, but does not upload all the data at once. Instead, it divides it into multiple smaller batches. A certain time interval is maintained between each batch, allowing for rhythmic upload control by uploading a data group and then waiting for a period of time. This ensures that critical data is transmitted first under bandwidth-constrained conditions, avoiding network overload.

[0045] Through this dynamic scheduling strategy, the communication module can intelligently adjust the upload process based on real-time network conditions and cache occupancy status, which not only improves the upload efficiency of critical data, but also enhances the stability and adaptability of network resource scheduling.

[0046] In one possible implementation, after network recovery of the communication module is detected, the following is also included: Obtain the network disconnection duration of the communication module; when the network disconnection duration exceeds the seventh threshold, perform a second upload operation to the server for the data cached in the cache area.

[0047] In this embodiment, when the communication module network is detected to be disconnected, the current timestamp Tstart is recorded; during the period when the communication module is in a network disconnected state, the disconnection time is continuously accumulated using a local runtime counter; when the communication module network is detected to be restored, the current timestamp Tend is recorded; the network disconnection duration is calculated based on the two timestamps.

[0048] A seventh threshold is preset to distinguish between momentary network outages and prolonged network outages. When the network outage duration exceeds the seventh threshold, a second upload logic is triggered; when the network outage duration is less than or equal to the seventh threshold, the outage is considered a short-term anomaly, and the normal cache upload process is followed.

[0049] During the second upload, the system iterates through the data in the cache that has not yet been uploaded. For each piece of data to be uploaded, it marks it with a "second upload" flag (or increases its upload priority). During the upload scheduling strategy execution phase (i.e., the upload phase after communication recovery), data packets with the "second upload flag" are prioritized and transmitted before the normal upload task. A second transmission is performed after the first transmission to ensure that the data is transmitted completely to the server. If there is data in the cache that has not been uploaded within a set time window, each data packet can be acknowledged (ACK) during the compensation upload phase. Data that fails to receive an ACK is retransmitted. This ensures that high-timeliness and high-reliability data in the cache has the capability for compensation upload after recovery. During upload, priority queues can be scheduled in advance to ensure business continuity.

[0050] S13. Upload the data in the cache to the server according to the data upload policy.

[0051] In this embodiment, after determining the data upload strategy, the data is uploaded to the server according to the upload strategy.

[0052] Specifically, the communication module prioritizes and uploads data from the cache in stages: high-priority data is retrieved from the high-speed cache and sent to the server first to ensure that alarm information or critical operational data can be quickly recovered; after high-priority data transmission is completed, medium-priority data is uploaded as needed based on bandwidth availability and cache release progress; when the system detects that the upload pressure is gradually easing, low-priority data is then uploaded in batches in the background. The entire upload process is managed by the communication module's internal state machine, which can automatically pause or resume when network conditions fluctuate, ensuring that the overall upload process is stable and controllable.

[0053] In one possible implementation, after caching the data to be uploaded to the server in a cache according to priority, the method further includes: Determine the retention period for data of each priority level; when the retention period for data is reached, delete the data.

[0054] In this embodiment, a preset retention period is set for each priority level. For example: high-priority data has a short retention period (e.g., 30s to 120s) because it is usually real-time data; medium-priority data has a moderate retention period (e.g., 5min to 30min); and low-priority data can have a longer retention period (e.g., more than 1 hour), corresponding to non-real-time logs, analysis data, etc. When data is written to the cache, the control program simultaneously records a cache timestamp Tcache for that data and associates it with its corresponding priority.

[0055] When the retention period for data is reached, the data is marked as ready for deletion. Deletion can be performed immediately or in batch processing within the cache management thread; during deletion, the cache space occupied by the data is released, and the corresponding pointer / handle is removed from the cache index structure. Data in the upload queue but not yet sent will also be removed if its retention period reaches its limit to prevent expired data from continuing to consume upload bandwidth.

[0056] The data processing method provided in this invention caches data to be uploaded to the server according to priority in a cache area when a network disconnection of the communication module is detected. The communication module includes multiple cache areas, each storing data of different priorities. When the network of the communication module is detected to be restored, a data upload strategy for the communication module is determined based on the bandwidth and cache load of the communication module. The data in the cache area is then uploaded to the server according to the data upload strategy. This allows for differentiated caching of data by priority in the communication module during network outages, preventing critical data from being lost or overwritten due to cache resource contention, thus improving cache resource utilization efficiency. After network restoration, the upload strategy is dynamically adjusted based on bandwidth and cache load to reduce network congestion, ensuring priority and timely upload of critical data, and improving the real-time performance and security of data transmission.

[0057] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method specifically includes: Step 1: Data Classification and Prioritization The equipment continuously collects various types of data during operation, including security alarm signals (such as fire and intrusion), equipment operating status (such as temperature and voltage), environmental sensor data (such as humidity and light intensity), and system logs. The system prioritizes each piece of data using preset rules or a lightweight classification model. High priority: Involving safety, emergency events, or real-time control commands, such as fire alarms or motor overloads; Medium priority: Common operating parameters, such as temperature, speed, and current; Low priority: Non-critical logs, debug information, periodic status snapshots.

[0058] Taking a WiFi module as an example, the classification process is completed inside the WiFi module, without relying on an external processor, ensuring low-latency response.

[0059] Step 2: Building a Tiered Caching Mechanism The WiFi module is internally divided into three independent buffer areas: Cache-H: 16KB in size, used to store high-priority data, using a first-in-first-out (FIFO) + timestamp-based eviction mechanism; Regular cache (Cache-M): 64KB in size, used for medium-priority data, and supports rolling cleanup by time window; Low-speed cache (Cache-L): 128KB in size, used for low-priority data, supports compressed storage and background batch upload.

[0060] Each cache is managed through a memory-mapped unit (MMU) and supports dynamic space adjustment. When a cache level is close to full, the system automatically triggers a cache degradation mechanism, migrating some low-priority data to a low-speed area to free up space in the high-speed area.

[0061] Step 3: Network Disconnection Detection and Cache Switching The WiFi module has a built-in network status monitoring module that periodically sends heartbeat packets (Ping) to check the connection status. If no response is received after three consecutive attempts, it is considered a network outage, and the system immediately: Stop data upload, start the local cache write process, record the network disconnection start timestamp (T_start), enable the cache monitoring thread, and perform real-time statistics on the cache usage rate of each level.

[0062] Step 4: Dynamic compensation mechanism triggered Once the network is restored, the system first checks the current bandwidth status (estimated by RSSI value and throughput) and cache load. It then determines the compensation strategy based on the following logic: If bandwidth > 10Mbps and cache load < 70%: prioritize uploading high-priority data, followed by medium-priority data, and finally low-priority data; If bandwidth < 5Mbps or cache load > 80%: adopt the "batch upload" strategy, upload only 10 high-priority data in each round, and continue after a 1-second interval; If the network outage lasts for more than 2 hours: the system will automatically trigger the "critical data retransmission" mechanism to re-confirm and upload high-priority data.

[0063] The compensation process is controlled by a state machine to ensure that the upload process can be interrupted and resumed.

[0064] Step 5: Cache cleanup and resource reclamation The system periodically performs cache cleanup tasks: High-priority data: Retention time ≤ 1 hour, automatically deleted after timeout; Medium priority data: retention time ≤ 24 hours; Low-priority data: retention time ≤ 7 days, supports storage after compression.

[0065] The cleanup strategy is based on a two-dimensional approach of "priority + timeliness" to avoid accidentally deleting critical data.

[0066] Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device specifically includes: Cache module 31 is used to cache the data to be uploaded to the server into a cache area according to priority when the network disconnection of the communication module is detected. The communication module includes multiple cache areas, and each cache area is used to store data with different priorities. The determination module 32 is used to determine the data upload strategy of the communication module based on the bandwidth and cache load of the communication module when the network recovery of the communication module is detected. Upload module 33 is used to upload data in the cache to the server according to the data upload strategy.

[0067] In one possible implementation, the caching module is specifically used to obtain the cache utilization rate of each of the cache areas; The priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined. When the cache utilization rate of the first cache area is less than the first threshold, the data is cached in the first cache area; When the cache utilization rate of the first cache area is greater than or equal to the first threshold, the data is cached in the second cache area. The priority of the second cache area is lower than that of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

[0068] In one possible implementation, the determining module is further configured to determine that the data is of a first priority when the data type is a first type, and to determine the cache corresponding to the first priority, wherein the data of the first type represents data processed in real time; When the data type is the second type, the data is determined to be of the second priority, and the regular cache corresponding to the second priority is determined. The data of the second type represents the data processed at each first time interval. The second priority is lower than the first priority, and the capacity of the cache is smaller than the capacity of the regular cache. When the data type is the third type, the data is determined to be of the third priority, and the low-speed cache corresponding to the third priority is determined. The third type of data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache is less than the capacity of the low-speed cache.

[0069] In one possible implementation, the processing module 34 is used to clean up the cached data using a first-in-first-out strategy and a timestamp eviction mechanism. For the regular cache area, the cached data is cleaned up in a rolling manner according to time windows; For the low-speed cache area, compression and background batch uploading are used to transfer the cached data.

[0070] In one possible implementation, the determining module is specifically used to determine the data upload strategy of the communication module as follows when the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold: upload the current cached data of each cache area to the server in descending order of priority of the cache area; When the bandwidth is less than the fifth threshold and the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups according to the priority of the cache area from high to low, and one data group is uploaded to the server every third time interval. The fifth threshold is less than the third threshold and the sixth threshold is greater than the fourth threshold.

[0071] In one possible implementation, the upload module is specifically used to obtain the network disconnection duration of the communication module; When the network disconnection duration exceeds the seventh threshold, a second upload operation is performed on the data cached in the cache area to the server.

[0072] In one possible implementation, the determining module is specifically used to determine the retention time corresponding to data of each priority. When the data's retention period expires, the data will be deleted.

[0073] The data processing device provided in this embodiment may be as follows: Figure 3 The apparatus shown can perform, for example Figure 1-2 All steps of the data processing method in the middle, thereby achieving Figure 1-2 For details on the technical effects of the data processing method shown, please refer to [link / reference]. Figure 1-2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0074] Figure 4 This is a schematic diagram of the structure of a communication module provided in an embodiment of the present invention. Figure 4 The communication module 400 shown includes at least one processor 401, a memory 402, at least one network interface 404, and other user interfaces 403. The various components in the communication module 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to implement communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general designated all buses as Bus System 405.

[0075] The user interface 403 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0076] It is understood that the memory 402 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0077] In some implementations, memory 402 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 4021 and application program 4022.

[0078] The operating system 4021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 4022.

[0079] In this embodiment of the invention, by calling the program or instructions stored in the memory 402, specifically the program or instructions stored in the application program 4022, the processor 401 executes the method steps provided in each method embodiment, including, for example: When the network disconnection of the communication module is detected, the data to be uploaded to the server is cached in the cache area according to the priority. The communication module contains multiple cache areas, and each cache area is used to store data with different priorities. When network recovery of the communication module is detected, the data upload strategy of the communication module is determined based on the bandwidth and cache load of the communication module. The data in the cache is uploaded to the server according to the data upload strategy.

[0080] In one possible implementation, the cache utilization rate of each of the cache areas is obtained; The priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined. When the cache utilization rate of the first cache area is less than the first threshold, the data is cached in the first cache area; When the cache utilization rate of the first cache area is greater than or equal to the first threshold, the data is cached in the second cache area. The priority of the second cache area is lower than that of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

[0081] In one possible implementation, when the data type is a first type, the data is determined to have a first priority, and a cache corresponding to the first priority is determined, wherein the data of the first type represents data processed in real time; When the data type is the second type, the data is determined to be of the second priority, and the regular cache corresponding to the second priority is determined. The data of the second type represents the data processed at each first time interval. The second priority is lower than the first priority, and the capacity of the cache is smaller than the capacity of the regular cache. When the data type is the third type, the data is determined to be of the third priority, and the low-speed cache corresponding to the third priority is determined. The third type of data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache is less than the capacity of the low-speed cache.

[0082] In one possible implementation, a first-in-first-out (FIFO) strategy and a timestamp eviction mechanism are used to clean up the cached data in the cache area; For the regular cache area, the cached data is cleaned up in a rolling manner according to time windows; For the low-speed cache area, compression and background batch uploading are used to transfer the cached data.

[0083] In one possible implementation, when the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold, the data upload strategy of the communication module is determined as follows: the data currently cached in each cache area is uploaded to the server in descending order of priority corresponding to the cache area. When the bandwidth is less than the fifth threshold and the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups according to the priority of the cache area from high to low, and one data group is uploaded to the server every third time interval. The fifth threshold is less than the third threshold and the sixth threshold is greater than the fourth threshold.

[0084] In one possible implementation, the network disconnection duration of the communication module is obtained; When the network disconnection duration exceeds the seventh threshold, a second upload operation is performed on the data cached in the cache area to the server.

[0085] In one possible implementation, the retention period for data of each priority level is determined; When the data's retention period expires, the data will be deleted.

[0086] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 402. Processor 401 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0087] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0088] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0089] The communication module provided in this embodiment can be as follows: Figure 4 The communication module shown can perform the following: Figure 1-2 All steps of the data processing method in the middle, thereby achieving Figure 1-2 For details on the technical effects of the data processing method shown, please refer to [link / reference]. Figure 1-2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0090] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0091] When one or more programs in the storage medium can be executed by one or more processors to implement the data processing method described above that is executed on the device side.

[0092] The processor is used to execute a data processing program stored in the memory to implement the following steps of a data processing method executed on the device side: When the network disconnection of the communication module is detected, the data to be uploaded to the server is cached in the cache area according to the priority. The communication module contains multiple cache areas, and each cache area is used to store data with different priorities. When network recovery of the communication module is detected, the data upload strategy of the communication module is determined based on the bandwidth and cache load of the communication module. The data in the cache is uploaded to the server according to the data upload strategy.

[0093] In one possible implementation, the cache utilization rate of each of the cache areas is obtained; The priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined. When the cache utilization rate of the first cache area is less than the first threshold, the data is cached in the first cache area; When the cache utilization rate of the first cache area is greater than or equal to the first threshold, the data is cached in the second cache area. The priority of the second cache area is lower than that of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

[0094] In one possible implementation, when the data type is a first type, the data is determined to have a first priority, and a cache corresponding to the first priority is determined, wherein the data of the first type represents data processed in real time; When the data type is the second type, the data is determined to be of the second priority, and the regular cache corresponding to the second priority is determined. The data of the second type represents the data processed at each first time interval. The second priority is lower than the first priority, and the capacity of the cache is smaller than the capacity of the regular cache. When the data type is the third type, the data is determined to be of the third priority, and the low-speed cache corresponding to the third priority is determined. The third type of data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache is less than the capacity of the low-speed cache.

[0095] In one possible implementation, a first-in-first-out (FIFO) strategy and a timestamp eviction mechanism are used to clean up the cached data in the cache area; For the regular cache area, the cached data is cleaned up in a rolling manner according to time windows; For the low-speed cache area, compression and background batch uploading are used to transfer the cached data.

[0096] In one possible implementation, when the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold, the data upload strategy of the communication module is determined as follows: the data currently cached in each cache area is uploaded to the server in descending order of priority corresponding to the cache area. When the bandwidth is less than the fifth threshold and the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups according to the priority of the cache area from high to low, and one data group is uploaded to the server every third time interval. The fifth threshold is less than the third threshold and the sixth threshold is greater than the fourth threshold.

[0097] In one possible implementation, the network disconnection duration of the communication module is obtained; When the network disconnection duration exceeds the seventh threshold, a second upload operation is performed on the data cached in the cache area to the server.

[0098] In one possible implementation, the retention period for data of each priority level is determined; When the data's retention period expires, the data will be deleted.

[0099] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0101] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: When the network disconnection of the communication module is detected, the data to be uploaded to the server is cached in the cache area according to the priority. The communication module contains multiple cache areas, and each cache area is used to store data with different priorities. When network recovery of the communication module is detected, the data upload strategy of the communication module is determined based on the bandwidth and cache load of the communication module. The data in the cache is uploaded to the server according to the data upload strategy.

2. The method according to claim 1, characterized in that, The step of caching the data to be uploaded to the server in the cache according to priority includes: Obtain the cache utilization rate for each of the aforementioned cache areas; The priority of the data to be uploaded to the server is determined based on the data type, and the first cache corresponding to the priority is determined. When the cache utilization rate of the first cache area is less than the first threshold, the data is cached in the first cache area; When the cache utilization rate of the first cache area is greater than or equal to the first threshold, the data is cached in the second cache area. The priority of the second cache area is lower than that of the first cache area, and the cache utilization rate of the second cache area is less than the second threshold.

3. The method according to claim 2, characterized in that, The step of determining the priority of the data based on the data type to be uploaded to the server, and determining the first cache corresponding to the priority, includes: When the data type is a first type, the data is determined to be of the first priority, and the cache corresponding to the first priority is determined. The data of the first type represents the data processed in real time. When the data type is the second type, the data is determined to be of the second priority, and the regular cache corresponding to the second priority is determined. The data of the second type represents the data processed at each first time interval. The second priority is lower than the first priority, and the capacity of the cache is smaller than the capacity of the regular cache. When the data type is the third type, the data is determined to be of the third priority, and the low-speed cache corresponding to the third priority is determined. The third type of data represents data processed at a second interval. The third priority is lower than the second priority, the second interval is longer than the first interval, and the capacity of the regular cache is less than the capacity of the low-speed cache.

4. The method according to claim 3, characterized in that, After caching the data to be uploaded to the server into a cache according to priority, the method further includes: The cache area is cleaned up using a first-in-first-out strategy and a timestamp eviction mechanism. For the regular cache area, the cached data is cleaned up in a rolling manner according to time windows; For the low-speed cache area, compression and background batch uploading are used to transfer the cached data.

5. The method according to claim 1, characterized in that, The step of determining the data upload strategy of the communication module based on the bandwidth and cache load of the communication module includes: When the bandwidth is greater than the third threshold and the cache load is less than the fourth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is uploaded to the server in descending order of priority of the cache area. When the bandwidth is less than the fifth threshold and the cache load is greater than the sixth threshold, the data upload strategy of the communication module is determined as follows: the current cached data of each cache area is divided into multiple data groups according to the priority of the cache area from high to low, and one data group is uploaded to the server every third time interval. The fifth threshold is less than the third threshold and the sixth threshold is greater than the fourth threshold.

6. The method according to claim 1, characterized in that, After the network of the communication module is detected to have recovered, the method further includes: Obtain the network disconnection duration of the communication module; When the network disconnection duration exceeds the seventh threshold, a second upload operation is performed on the data cached in the cache area to the server.

7. The method according to claim 1, characterized in that, After caching the data to be uploaded to the server into a cache according to priority, the method further includes: Determine the retention period for data of each priority level; When the data's retention period expires, the data will be deleted.

8. A data processing apparatus, characterized in that, include: The caching module is used to cache the data to be uploaded to the server in a cache area according to priority when the network disconnection of the communication module is detected. The communication module includes multiple cache areas, and each cache area is used to store data with different priorities. The determination module is used to determine the data upload strategy of the communication module based on the bandwidth and cache load of the communication module when the network recovery of the communication module is detected. The upload module is used to upload data in the cache to the server according to the data upload strategy.

9. A communication module, characterized in that, include: A processor and a memory, the processor being configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 7.

Images