Buried point data acquisition and uploading control method and system based on memory optimization
By dividing the memory cache into multiple memory units and dynamically converting them into local cache, combined with the timestamp index mechanism, the problem of excessive memory usage in point-of-sale data collection is solved, and the continuity of data upload and system stability are achieved.
Patent Information
- Application Number
- CN202510761689.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-09
- Publication Date
- 2025-10-10
AI Technical Summary
Existing point-of-sale data collection solutions are prone to excessive memory usage, system performance degradation, high risk of data loss, and low upload efficiency in high-concurrency scenarios.
By dividing the memory cache into multiple memory units of the same size and dynamically converting them into local caches based on memory usage, the timestamp index mechanism is used for data switching and uploading to optimize memory usage.
It achieves smooth system memory usage, avoids system freezes, ensures the continuity and integrity of data upload, and improves system stability and efficiency.
Smart Images
Figure CN120763012A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of data collection and memory optimization control, in particular to a memory-optimized data collection and uploading control method and system. BACKGROUND
[0002] In modern Internet applications, data collection is an important means of analyzing user behavior and optimizing product functions. However, as the amount of data increases, the system often faces the problem of high memory usage or high memory fluctuation when collecting and uploading data. Especially in high concurrency scenarios, a large amount of data is written into the memory cache at the same time, which may cause the system to run out of memory resources, and further cause the risk of system freezing, crashing or even crashing.
[0003] The existing data collection scheme usually uses memory cache as a data staging area, and then uploads in batches when the data accumulates to a certain amount. However, this scheme has the following problems:
[0004] 1. Uncontrollable memory usage: when the amount of data increases rapidly, the memory cache may quickly fill up, causing system performance to decline.
[0005] 2. Data loss risk: if the system crashes due to insufficient memory, the unuploaded data may be lost.
[0006] 3. Low uploading efficiency: the data in the memory cache needs to wait for uploading to release the memory, and the uploading process may be delayed due to network fluctuations and other reasons, further exacerbating memory pressure.
[0007] Therefore, there is an urgent need for a technical solution that can ensure data uploading efficiency while dynamically adjusting memory usage SUMMARY
[0008] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a memory-optimized data collection and uploading control method and system, which optimizes system memory usage through a dynamic conversion strategy between memory cache and local cache. This method can ensure normal data uploading while smoothing system memory usage, avoiding problems such as system freezing, crashing and the like caused by high memory usage or high memory fluctuation.
[0009] In order to achieve the above purpose, the technical solution adopted by the present application is: a memory-optimized data collection and uploading control method, comprising,
[0010] Step 1: data collection and memory cache initialization: store the collected data in the memory cache;
[0011] Step 2: Dynamic conversion of memory cache and local cache: switching control of memory cache and local cache according to monitoring of memory usage;
[0012] Step 3, data loading and uploading: when the set conditions are met, load the local cache into the memory cache and upload it to the server.
[0013] Step 1 also includes: dividing the memory cache for caching the trace data into multiple memory units of the same size, and caching the trace data in memory units as the basic unit.
[0014] Each memory unit contains a data header, which records the timestamp, data length and data check code of the trace data.
[0015] Step 2 includes real-time monitoring of memory usage, and when the memory usage reaches a set threshold, the memory cache and local cache conversion strategy is started, and the trace data in the memory cache is converted to local cache for file storage.
[0016] The memory cache and local cache conversion strategy includes: according to the timestamp index of the memory unit data header, the system converts the earliest memory unit in time to a local cache file, and releases the corresponding memory space, until the memory usage is lower than the set conversion threshold.
[0017] Taking a memory unit as the basic unit, the trace data in a memory unit is stored in a local cache file, and the local cache file is named by timestamp.
[0018] Step 3 includes real-time monitoring of memory usage, and when the memory resource usage is detected to be less than the set loading threshold, the local cache files are loaded back into the memory cache in order according to the timestamp order of the local cache file name, and then the data in the memory cache is uploaded to the server according to the timestamp order of the data.
[0019] During the uploading of data to the server, the current uploading progress is recorded in real time, and when an exception occurs, the unuploaded data is retained in the local cache in the form of a local cache file until the exception is recovered again. The relationship between memory resource usage and loading threshold is used to determine whether to load and upload data.
[0020] Data upload exception includes network exception, and after the exception occurs, the unuploaded data in the memory cache is converted to local cache for file storage.
[0021] The trace data collection and uploading control system based on memory optimization includes
[0022] The data collection and memory cache initialization module is used to temporarily store the collected trace data in the memory cache;
[0023] The dynamic conversion module of the memory cache and the local cache is used to control the switching of the memory cache and the local cache of the buried point data according to the monitoring of the memory usage;
[0024] The data loading and uploading module loads the local cache into the memory cache and uploads it to the server when the set condition is met.
[0025] The innovation and advantages of the present application are as follows:
[0026] 1. Dynamic conversion of memory cache and local cache: by cutting the data in the memory cache into multiple memory units of the same size, and dynamically converting part of the memory units into local cache according to the memory usage, the memory occupation is reduced.
[0027] 2. Timestamp-based indexing mechanism: using the timestamp of the data header as an index to ensure the orderliness and integrity of the data.
[0028] 3. Memory occupation smoothing: by dynamically adjusting the proportion of memory cache and local cache, the high memory occupation is avoided, and the system stability is improved.
[0029] 4. Data upload efficiency optimization: when the memory is sufficient, load the data in the local cache into the memory in the order of the timestamp to wait for uploading, to ensure the continuity and efficiency of data uploading.
[0030] The content expressed by each figure in the specification of the present application and the marks in the figure are briefly described as follows:
[0031] Figure 1 The figure is the principle architecture diagram of the collection and uploading control of the present application.
[0032] Figure 2 The figure is the flowchart of the conversion of the memory cache and the local cache to realize data collection and uploading of the present application. DETAILED DESCRIPTION
[0033] The specific embodiments of the present application are further described in detail by comparing the figures and describing the optimal embodiments.
[0034] The present invention provides a method for optimizing system memory usage through a dynamic conversion strategy between memory cache and local cache. This method can smooth the system memory usage while ensuring normal data upload, and avoid problems such as system freezes and freezes caused by excessive memory usage or excessive memory fluctuations. By converting between memory cache and local cache, it is possible to smooth the system memory usage while ensuring normal data upload, and avoid problems and risks such as excessive memory usage or excessive memory fluctuations. This solution reduces memory usage by cutting the memory cache data into multiple memory units of the same size, indexing them according to the timestamp of the data header, and converting these memory units into local caches in sequence according to the current memory situation. When the memory is sufficient, the uploaded data is sorted according to the timestamp represented by the file name, and then loaded into the memory in sequence for upload.
[0035] like Figure 1 、 2 As shown, the method for collecting and uploading data based on memory optimization includes:
[0036] Step 1: Data collection and memory cache initialization: temporarily store the collected tracking data in the memory cache;
[0037] Step 2: Dynamically switch between memory cache and local cache: Based on the monitoring of memory usage, the tracking data is switched between memory cache and local cache.
[0038] Step 3: Data loading and uploading: When the set conditions are met, the local cache is loaded into the memory cache and uploaded to the server.
[0039] like Figure 1 The following is a schematic diagram of the system architecture. The memory-optimized tracking data collection and upload control system includes architecture modules such as data collection, memory cache, memory monitoring, file cache, and data upload. Specifically, it includes:
[0040] The data collection and memory cache initialization module is used to implement step 1, and is used to temporarily store the collected tracking data in the memory cache;
[0041] The dynamic conversion module between memory cache and local cache is used to implement step 2, and is used to control the switching between memory cache and local cache for buried data based on the monitoring of memory usage;
[0042] The data loading and uploading module is used to implement step 3. When the set conditions are met, the local cache is loaded into the memory cache and uploaded to the server.
[0043] In step 1, the memory cache used to cache tracking data is divided into multiple memory units of equal size, and the tracking data is cached in memory units. This allows the overall data to be split into parts, and each small memory unit is cached. It also provides a foundation for subsequent conversion of the memory cache to local storage.
[0044] Each memory unit contains a data header, which records the timestamp, data length and data check code of the embedded data.
[0045] Step 2 includes real-time monitoring of memory usage. When the memory usage reaches the set threshold Y1, the memory cache and local cache conversion strategy is started to convert the memory cache data into the local cache for file storage.
[0046] The in-memory cache and local cache conversion strategy includes the following: Based on the timestamp index of the memory unit data header, the system converts the oldest memory units into local cache files in sequence and releases the corresponding memory space until the memory usage falls below the set conversion threshold Y2. After each memory unit data is converted into a local storage file, the storage space of the memory unit is released, thereby freeing up memory and reducing usage.
[0047] In this embodiment, the memory unit is used as the basic unit, and the buried point data in a memory unit is stored in a local cache file, wherein the local cache file is named with a timestamp to form a local cache file.
[0048] Step 3 includes real-time monitoring of memory usage. When it is detected that the memory resource usage is less than the set loading threshold Y3, the local cache files are loaded back into the memory cache in sequence according to the timestamp order of the local cache file names, and then the data in the memory cache is uploaded to the server in the order of the data's timestamp order.
[0049] During the process of uploading data to the server, the current upload progress is recorded in real time. When an exception occurs, the unuploaded data will be retained in the local cache as a local cache file until the exception is recovered. Then, it will be determined whether to load and upload the data based on the relationship between memory resource usage and the loading threshold.
[0050] When data upload is abnormal, including network abnormality, after the abnormality occurs, the unuploaded data in the memory cache is converted to the local cache for file storage, and the unuploaded data in the memory cache is converted and controlled between the memory cache and the local cache in the same manner as in step 2.
[0051] In this embodiment, the thresholds Y1, Y2, and Y3 can be adjusted and calibrated according to system configuration data.
[0052] The embodiment provides a kind of based on memory cache and local cache conversion strategy's point-in-time data acquisition and upload optimization method, comprising the following steps:
[0053] (1) the memory cache is divided into multiple same size memory units, each memory unit contains a data header, and the time stamp information is recorded in the data header;
[0054] (2) real-time monitoring of memory usage, when memory occupancy reaches the preset threshold, the earliest memory unit is converted into a local cache file according to the time stamp order;
[0055] (3) when memory resources are sufficient, the local cache file is loaded into the memory cache in turn according to the time stamp order of the local cache file name;
[0056] (4) the data in the memory cache is uploaded to the server according to the time stamp order.
[0057] In the embodiment, the size of the memory unit can be dynamically adjusted according to the system configuration, and the size of the memory unit can be dynamically adjusted according to the actual demand, or the size can be set manually or remotely.
[0058] In the scheme, the local cache file is named by time stamp, which ensures the order of the file, and provides a basis for converting the local file into memory cache.
[0059] The method in the scheme also includes an exception handling mechanism for retaining unuploaded data when an exception occurs during uploading, and continuing uploading after system recovery.
[0060] As shown in Figure 2 The technical scheme of the application comprises the following steps:
[0061] Step 1: data acquisition and memory cache initialization
[0062] The system temporarily stores the data in the memory cache when collecting the point-in-time data.
[0063] The memory cache is divided into multiple same size memory units, each memory unit contains a data header, and the time stamp information of the unit is recorded in the data header.
[0064] Step 2: dynamic conversion of memory cache and local cache
[0065] The system monitors the memory usage in real time. When the memory occupancy reaches the preset threshold, the conversion strategy of memory cache and local cache is started.
[0066] According to the time stamp index of the data header, the system converts the earliest memory unit into a local cache file in turn, and releases the corresponding memory space.
[0067] • Local cache files are named with timestamps to ensure the order of files.
[0068] Step 3: Data loading and uploading when memory is sufficient
[0069] • When the system detects that memory resources are sufficient, it loads local cache files into the memory cache in the order of their timestamps.
[0070] • The system uploads data from the memory cache to the server in the order of their timestamps.
[0071] Step 4: Exception handling and data recovery
[0072] • If the system encounters an exception during uploading (such as network interruption), the unuploaded data will be retained in the local cache and will be reloaded and uploaded after the system recovers.
[0073] • The timestamp indexing mechanism ensures the completeness and continuity of data uploading.
[0074] The specific implementation steps for each step in this solution are as follows:
[0075] Implementation 1: Division of memory cache and local cache
[0076] • Assuming the total size of the system memory cache is 100MB and each memory unit is 1MB, the memory cache can be divided into 100 memory units.
[0077] • The data header of each memory unit contains the following information:
[0078] o Timestamp: Records the time of data generation.
[0079] o Data Length: Records the length of data in the current memory unit.
[0080] o Checksum: Used to verify the integrity of the data.
[0081] Implementation 2: Memory occupation monitoring and conversion strategy
[0082] • The system sets the memory occupation threshold to 80%. When the memory occupation exceeds 80%, the conversion strategy is started.
[0083] • The system converts the earliest memory unit to a local cache file according to the timestamp order. For example, the memory unit with timestamp T1 is written to the local file "T1.dat" and 1MB of memory space is released.
[0084] • Repeat the above process until the memory occupation is reduced to a safe range.
[0085] Implementation 3: Data upload and local cache loading
[0086] When the memory usage is lower than 50%, the system starts loading local cache files into the memory.
[0087] The system loads the files back into the memory cache in order of the timestamps of the file names (e.g., "T1.dat," "T2.dat").
[0088] After loading is complete, the system uploads the data in the memory cache to the server.
[0089] Implementation 4: Exception Handling Mechanism
[0090] If an exception occurs during the upload process, the system records the current upload progress and retains the unuploaded data in the local cache.
[0091] After the system is restored, continue loading and uploading data from the recorded upload progress.
[0092] This invention effectively solves the problem of excessive memory usage during the acquisition and upload of embedded data by dynamically switching between memory caching and local caching. This method not only smooths system memory usage but also ensures the continuity and integrity of data uploads, thus possessing high practical value and promotional significance.
[0093] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the scope of protection of the present invention.
Claims
1. A method for collecting and uploading tracking data based on memory optimization, characterized by: include, Step 1: Data collection and memory cache initialization: temporarily store the collected tracking data in the memory cache; Step 2: Dynamically switch between memory cache and local cache: Based on the monitoring of memory usage, the tracking data is switched between memory cache and local cache. Step 3: Data loading and uploading: When the set conditions are met, the local cache is loaded into the memory cache and uploaded to the server.
2. The memory-optimized method for collecting and uploading data according to claim 1, wherein: Step 1 also includes: dividing the memory cache used to cache the buried point data into multiple memory units of the same size, and caching the buried point data with the memory unit as the basic unit.
3. The memory-optimized method for collecting and uploading data according to claim 2, wherein: Each memory unit contains a data header, which records the timestamp, data length and data check code of the embedded data.
4. The memory-optimized embedded point data collection and upload control method according to any one of claims 1 to 3, characterized in that: Step 2 includes real-time monitoring of memory usage. When the memory usage reaches the set threshold, the memory cache and local cache conversion strategy is started to convert the buried data in the memory cache to the local cache for file storage.
5. The memory-optimized method for collecting and uploading data according to claim 4, wherein: The memory cache and local cache conversion strategy includes: according to the timestamp index of the memory unit data header, the system converts the oldest memory units into local cache files in sequence and releases the corresponding memory space until the memory occupancy is lower than the set conversion threshold.
6. The method for collecting and uploading data based on memory optimization according to claim 5, characterized in that: Taking the memory unit as the basic unit, the tracking data in a memory unit is stored in a local cache file, where the local cache file is named with a timestamp.
7. The memory-optimized method for collecting and uploading embedded data according to any one of claims 1 to 6, characterized in that: Step 3 includes real-time monitoring of memory usage. When it is detected that the memory resource usage is less than the set loading threshold, the local cache files are loaded back into the memory cache in sequence according to the timestamp order of the local cache file names, and then the data in the memory cache is uploaded to the server in the order of the data's timestamp order.
8. The method for collecting and uploading data based on memory optimization according to claim 7, characterized in that: During the process of uploading data to the server, the current upload progress is recorded in real time. When an exception occurs, the unuploaded data will be retained in the local cache as a local cache file until the exception is recovered. Then, it will be determined whether to load and upload the data based on the relationship between memory resource usage and the loading threshold.
9. The method for collecting and uploading data based on memory optimization according to claim 8, characterized in that: Exceptions to data upload include network exceptions. After an exception occurs, the unuploaded data in the memory cache is converted to the local cache for file storage.
10. The memory-optimized data collection and upload control system is characterized by: include Data collection and memory cache initialization module, used to temporarily store the collected tracking data in the memory cache; A dynamic conversion module between memory cache and local cache is used to control the switching between memory cache and local cache for tracking data based on the monitoring of memory usage; The data loading and uploading module loads the local cache into the memory cache and uploads it to the server when the set conditions are met.