Data statistical caching method of mobile terminal and related device
Patent Information
- Application Number
- CN202510622354.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-08-15
Smart Images
Figure CN120492500A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data statistics caching method and related devices for a mobile terminal. Background Art
[0002] With the rapid development of the mobile internet, the demand for data statistics in mobile apps is experiencing a dual trend of surging data volumes and increasingly refined collection dimensions. To meet real-time business requirements for scenarios like user behavior analysis and performance monitoring, developers typically use local databases to temporarily store statistical data and then asynchronously upload it to the server. However, existing technologies present numerous significant challenges in practical applications: complex data contexts and redundant processing logic; high-frequency writes leading to inefficient storage; and excessive memory usage, which poses the risk of OOM (Out of Memory) crashes.
[0003] In summary, the existence of these problems will lead to poor performance and low efficiency of the data statistics caching solution of the mobile terminal in the existing technology. Summary of the Invention
[0004] In view of the above problems, this application provides a data statistics caching method and related devices for a mobile terminal, which can achieve the purpose of data statistics caching for the mobile terminal with high efficiency. The specific solution is as follows:
[0005] A first aspect of the present application provides a data statistics caching method for a mobile terminal, comprising:
[0006] The data stream component is used to collect the initial data set; each initial data collected at the same data collection point in the initial data set is a data stream group; each data stream in the data stream group corresponds to each initial data in the initial data set one by one;
[0007] Determine the preset declarative function computing logic of each data stream group from the data stream assembly storage component, assemble each data stream into a new data stream group according to the preset declarative function computing logic, and store the new data stream group in memory; the declarative function computing logic is a combination of batch output logic and preset logic, and the preset logic is at least one of optional logic, waiting logic, combination logic, and object pool logic;
[0008] When the number of generated new data streams reaches the preset output data volume in the batch output logic, the generated new data streams are written in batches to the local database of the mobile terminal.
[0009] In one possible implementation, the data stream assembly storage component includes a data stream assembly function component, which assembles individual data streams into a new data stream according to a preset declarative function calculation logic, including:
[0010] When the preset declarative function computing logic includes combination logic, the data flow assembly function component assembles the data flows with the same keywords in each data flow group specified in the combination logic into a new data flow;
[0011] When the preset declarative function computing logic includes optional logic, the data flow assembly function component will treat the data flow with the optional mark as an optional data flow when assembling a new data flow;
[0012] When the preset declarative function calculation logic includes waiting logic, the data stream assembly function component will wait for the data stream with the waiting mark according to the preset time in the waiting logic.
[0013] In one possible implementation, the data stream assembly storage component includes a data stream storage function component, which assembles individual data streams into a new data stream group according to a preset declarative function calculation logic and stores the new data stream group in memory, including:
[0014] When the preset declarative function computing logic includes object pool logic, the data stream storage function component stores the new data stream in the preset unified data object format in the target object pool according to the object pool logic; the target object pool is the object pool preset in the object pool logic that matches the data size of the new data stream.
[0015] In one possible implementation, the data stream storage function component includes a queue group function component. When the number of generated new data streams reaches a preset output data volume in the batch output logic, each generated new data stream is batch-written to a local database on the mobile terminal, including:
[0016] When the data volume of each new data stream reaches the preset output data volume in the batch output logic, each new data stream stored in the queue group function component is output and written to the local database of the mobile terminal.
[0017] In one possible implementation, the data stream storage function component includes a grouping function component that, when the number of generated new data streams reaches a preset output data volume in the batch output logic, batch writes each generated new data stream to a local database on the mobile terminal, including:
[0018] When the data volume of each new data stream is lower than the preset minimum data volume threshold in the batch output logic, each new data stream stored in the hash table of the grouping function component is output and written to the local database of the mobile terminal.
[0019] In one possible implementation, a data stream component is used to collect an initial data set, including:
[0020] The data stream component is used to collect different data stream groups from different data collection points upstream, and each data stream is sent to the downstream data stream assembly storage component according to the collection order of each data stream in the data stream group.
[0021] A third aspect of the present application provides a data statistics caching device for a mobile terminal, comprising:
[0022] The acquisition unit is used to acquire the initial data set using a data stream component; each initial data collected at the same data acquisition point in the initial data set is a data stream group; each data stream in the data stream group corresponds to each initial data in the initial data set;
[0023] An assembly unit, configured to determine a preset declarative function computing logic for each data stream group from a data stream assembly storage component, assemble each data stream into a new data stream group according to the preset declarative function computing logic, and store the new data stream group in memory; the declarative function computing logic is a combination of batch output logic and preset logic, wherein the preset logic is at least one of optional logic, waiting logic, combination logic, and object pool logic;
[0024] The output unit is used to write each generated new data stream into the local database of the mobile terminal in batches when the number of generated new data streams reaches the preset output data volume in the batch output logic.
[0025] A third aspect of the present application provides a data statistics cache device for a mobile terminal, comprising at least one processor and a memory connected to the processor, wherein:
[0026] Memory is used to store computer programs;
[0027] The processor is used to execute the computer program so that the data statistics caching device of the mobile terminal can implement any of the data statistics caching methods of the mobile terminal as described above.
[0028] In a fourth aspect, the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the mobile terminal data statistics caching methods described above.
[0029] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the mobile terminal data statistics caching methods described above.
[0030] By means of the above technical solution, the data statistics caching method for the mobile terminal provided by the present application regards the collected initial data as individual data streams, and uses the data stream component to collect them into various data stream groups. Then, the data assembly storage component is used to determine the preset declarative function calculation logic for each data stream group. Based on any two function calculation logics including the batch output logic in the declarative function calculation logic, each data stream group is assembled into a new data stream and stored in the memory. When the number of new data streams generated reaches the preset output data volume in the batch output logic, the generated new data streams are batch-written into the local database of the mobile terminal. In the present application, declarative function calculation logic is used to realize the assembly between complex data streams, which improves the performance of data caching. In addition, an object pool can be used to reduce memory overhead, and batch output can be used to improve the performance of writing data to the database. It can be seen that the data statistics caching method provided by the present application has high performance and high efficiency in all aspects. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.
[0032] Figure 1 This is a flow chart showing an example of a method for caching data statistics on a mobile terminal provided by this application;
[0033] Figure 2 An example diagram of the workflow of the data flow component provided for this application;
[0034] Figure 3 An example diagram of the workflow of the map function component provided for this application;
[0035] Figure 4 An example diagram of the workflow of the filter function component provided for this application;
[0036] Figure 5 An example diagram of the workflow of the dequeue function component provided for this application;
[0037] Figure 6 An example diagram of the workflow of the optional function component provided for this application;
[0038] Figure 7 An example diagram of the workflow of the wait function component provided for this application;
[0039] Figure 8 Provide an example diagram of the function calculation process for iOS network requests for this application;
[0040] Figure 9 An example diagram of the workflow of the QueueGroup function component provided for this application;
[0041] Figure 10 An example diagram of the workflow of the TimeSerialQueue component provided for this application;
[0042] Figure 11 An example diagram of the workflow of the MapGroup function component provided for this application;
[0043] Figure 12 A schematic diagram of the structure of a data statistics cache device for a mobile terminal provided by this application;
[0044] Figure 13 This is a schematic diagram of the structure of the data statistics cache device on the mobile terminal provided in this application. DETAILED DESCRIPTION
[0045] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.
[0046] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0048] In existing technologies, developers usually use a local database to temporarily store statistical data and then asynchronously upload it to the server to meet the real-time needs of mobile app services for scenarios such as user behavior analysis and performance monitoring. However, this also creates many technical problems:
[0049] Problem 1: The data context is complex and the processing logic is redundant.
[0050] Problem 2: High-frequency writing leads to low storage efficiency.
[0051] Question 3: Excessive memory usage poses the risk of causing OOM crashes.
[0052] First, the above three problems in the prior art are introduced:
[0053] The specific manifestation of Problem 1 is as follows: Mobile terminal statistical events, such as user clickstreams and page exposure, often rely on contextual data, such as user IDs, operation timestamps, and previous and subsequent time series. The contextual association of data mainly refers to the need for data A and data B to be combined and logically calculated according to needs in order to obtain a complete piece of data. For example, operations such as merging, updating, and replacing based on the ID are required to obtain complete data. Existing technologies achieve data integration through multi-table associations or nested queries, which leads to increased competition for local transaction locks, and complex queries are prone to causing performance bottlenecks. For example, when the cache of embedded data is mixed with real-time requests, duplicate statistics or data loss may occur due to logical conflicts.
[0054] Furthermore, multiple threads may simultaneously access the same data, potentially causing contention, leading to data corruption or program exit errors. Existing techniques typically employ a mutex lock to block other threads, ensuring that only one thread can access the data at a time. However, when a large amount of data enters the memory cache, the entire cache must be locked, reducing multi-threaded concurrency performance.
[0055] The second problem manifests itself as follows: the frequency of writing mobile statistics is positively correlated with the frequency of user operations. In scenarios with active users, such as e-commerce flash sales and social media updates, local databases must handle hundreds of write requests per second. Traditional databases like SQLite have limited disk I / O performance, which can easily lead to thread blocking. Furthermore, frequent data overwrites, such as cache updates, further exacerbate storage fragmentation and reduce read and write efficiency.
[0056] Specifically, because the writing performance of a single piece of data in a local database is not high, it is necessary to utilize the database's batch writing capabilities. The usual practice is to add a memory cache. When collecting data, it is first written to the memory cache, and then combined with logic processing. Then, the complete data is batched and written from the memory cache to the local database. However, this existing method of writing data to a local database based on a memory cache can bring some performance issues. For example, due to the small memory size of mobile devices, excessive memory usage can cause the app to be killed by the system. As the amount of collected data increases and the data becomes more and more interconnected, the performance of the memory cache will be severely reduced, causing app blockage and lag.
[0057] Problem three manifests itself specifically in the limited memory resources of mobile devices. Existing caching strategies, such as full memory caching, can lead to sharp memory spikes when data volumes surge. For example, unsharded preloaded data or inefficient cache eviction algorithms can occupy heap memory for extended periods, triggering system forced recycling mechanisms and ultimately causing app crashes. This problem is particularly acute on low-end devices or in background multitasking scenarios.
[0058] In addition, the process of writing the collected data into the memory cache requires the creation and release of a large number of data objects, which will cause problems such as CPU overhead, memory jitter, and memory fragmentation.
[0059] Taking iOS network request performance data statistics as an example, according to the life cycle of a network request, data can be divided into three points: data collected at the start of the request (data A), the end of the request (data B), and indicator collection (data C). If these three data points are written directly to the database, the performance will be low, especially when there are a large number of network requests in a short period of time.
[0060] If you use memory cache to complete assembly before writing, you usually need to maintain a HashMap in memory (the key is the taskId of the network request). When data A arrives, a new Key-Value record is added. When data B arrives, it is merged into the Key-Value data record of the HashMap based on the requested taskId. When data C arrives, it is merged into the Key-Value storage based on the taskId to complete the data assembly.
[0061] However, in fact, indicator collection (data C) is optional (that is, the system sometimes does not return indicator collection data C). If this situation is not handled, the data will not be assembled and will eventually be accumulated in the memory HashMap, causing memory leaks. In this case, the merged data of data C output (data A + data B) can be ignored.
[0062] Then you may need to open a new thread to periodically traverse the entire HashMap to check whether the data has been assembled, and then retrieve the data and store it in the database; or add a delayed asynchronous task for each record to monitor whether it is completed. When a large amount of data comes in, it is easy to cause a large number of tasks to be created in a short period of time, leading to memory overflow;
[0063] These two methods are difficult to support scenarios where large amounts of data are collected, and they increase system complexity, affecting stability and maintainability. In addition, for data security, the HashMap usually needs to be locked during merging, further affecting performance.
[0064] Network request performance is only one type of data in the performance field. Performance data statistics also include data from multiple dimensions such as CPU, memory usage, thread call stack, battery power, frame rate, and lag. In addition, to locate problems, in addition to performance data, it is also necessary to combine specific upper-level business data statistics to assemble and merge multiple data. This will result in an extremely complex lifecycle state, with various optional data and data filtering and assembly logic.
[0065] It can be seen from this that the method in the prior art of temporarily storing collected data in a memory cache and then writing it into a local database has low performance and low efficiency.
[0066] In order to solve the above problems, the present application provides a data statistics caching method and related devices for a mobile terminal, which realizes data statistics caching for the mobile terminal with high performance and high efficiency.
[0067] Optional, see Figure 1 , an example flow chart of a data statistics caching method for a mobile terminal provided in this application.
[0068] Step 101: Use the data flow component to collect the initial data set.
[0069] It should be noted that the initial data set may include data in multiple dimensions such as network request performance data, CPU data, memory usage data, thread call stack data, battery power data, frame rate data, and lag data.
[0070] Specifically, the initial data set generally includes basic user behavior data, device and environment information data, application performance data, user attribute and status data, and other supplementary data.
[0071] Basic user behavior data includes page access and operation record data, session information data, and conversion event data. Page access and operation record data includes basic interactive behavior data such as user click data, sliding data, page jump path data, and dwell time data. For example, button click events, page browsing order, and the time users spend on specific pages can be collected through tracking technology to help analyze user usage habits and interests. Session information data includes user startup and exit times and page browsing records. Startup and exit times mainly record the app startup and exit times, as well as the duration of a single-word session. Page browsing records mainly record the sequence of pages users visit within the app, and are used to map user behavior paths. Conversion event data can be key operations such as registration, login, purchase, and form submission. These events are usually directly related to business goals and are used to calculate conversion rates.
[0072] Device and environment information data primarily includes device attribute data and application version information data. Device attribute data includes device model, operating system version, screen resolution, network type, and other data used to adapt to different devices and optimize performance. Application version information data is the current app version number, which facilitates tracking the functional performance and problem data of different versions.
[0073] App performance data includes performance metrics and resource consumption. Performance metrics include startup time, page load time, and crash logs. Startup time is the time it takes for a user to click an icon until the homepage loads. Page load time is the response speed of each functional module or page. Crash logs record the stack trace and the scene when the app crashed. Resource consumption includes memory usage, CPU utilization, battery consumption, and other data used to optimize app fluency.
[0074] User attribute and status data includes user identification information and basic attribute data. User identification includes anonymous user ID or registered user ID, which is used to distinguish users and analyze individual behavior; basic attribute data includes gender, age group, geographic location, etc. obtained through registration or third-party login.
[0075] Other supplementary data includes installation and uninstallation records and test data. Installation and uninstallation records include the time of first installation, uninstallation events, and analysis of user acquisition efficiency based on channel sources. Test data is data from functional or interface optimization testing, recording the differences in user responses to different versions.
[0076] In this application, each piece of initial data is collected and output. Therefore, each piece of initial data collected using the data stream component (Stream component) can be considered a data stream, and each data stream collected from the same data collection point (also known as the same data collection source) can be considered a data stream group. It is easy to understand that each piece of initial data in the initial data set corresponds to each data stream in the data stream group.
[0077] Optionally, a data stream component is used to collect different data stream groups from different upstream data collection points, and then each data stream is sent to a downstream data stream assembly storage component according to the collection order of each data stream.
[0078] A data stream can also be understood as a lifecycle point. Taking iOS network requests as an example, the network request can be divided into three data streams: request start, request end, and indicator collection. The data stream collection component does not need to care about the downstream assembly logic. It is only responsible for collecting each data stream and transmitting the data stream to the downstream data stream assembly and storage component.
[0079] For example, see Figure 2, an example diagram of the workflow of the data flow component provided in this application.
[0080] like Figure 2 As shown, after starting, it first determines whether all data collection has been completed. When the judgment is yes, that is, all data collection has been completed, the sendEnd method is called to send the end status to the downstream DownStream and then the process ends; when the judgment is no, that is, all data collection has not been completed, after waiting for the data to be collected, the sendData method is called to send the data to the downstream DownStream, and then the process returns to continue to determine whether all data collection has been completed, until the judgment is yes, the sendEnd method is called to send the end status to the downstream DownStream and then the process ends.
[0081] Step 102: Determine the preset declarative function computing logic of each data stream group from the data stream assembly storage component, assemble each data stream into a new data stream group according to the preset declarative function computing logic, and store it in the memory.
[0082] It should be noted that the pre-set declarative function computing logic is primarily used to set the processing logic between the various data stream groups collected by the upstream data stream component, specifically the assembly logic between complex data stream groups. This application improves data caching performance by pre-setting the declarative function computing logic for the data stream groups. Specifically, a data stream assembly storage component is used to complete the pre-set declarative function computing logic for each data stream group.
[0083] First, a data stream assembly storage component is used to determine a preset declarative function computing logic for each data stream group. The declarative function computing logic is a combination of batch output logic and preset logic. The preset logic is specifically at least one of optional logic, waiting logic, combination logic and object pool logic.
[0084] Specifically, declarative function computation logic is presented as a functional declaration, based on the functional responsive programming (FRP) approach within responsive programming. Responsive programming is a declarative programming paradigm for propagating changes in data streams. It allows for convenient expression of static or dynamic data streams within a programming language, with the associated computational model automatically propagating changing values through the data stream. Functional responsiveness is a programming paradigm based on the fundamental components of functional programming, such as map, filter, and batch functions, for responsive programming.
[0085] Specifically, the map function component is mainly used to convert data formats and output them to the downstream after conversion.
[0086] For example, see Figure 3 , an example diagram of the workflow of the map function component provided in this application.
[0087] like Figure 3 As shown, after the start, the bind method is called to bind the upstream Upstream and wait for data to arrive. Then, it is determined whether the ReceiveEnd state has not ended. When it is determined to be yes, that is, the ReceiveEnd state has not ended, Map is used to convert the data and output the converted data to the downstream DownStream. Then, the bind method is returned to bind the upstream Upstream until the ReceiveEnd state is determined to end. The ReceiveEnd method of the downstream DownStream is called to send the end state to the downstream and end.
[0088] The Filter function component is mainly used to return whether the data is valid. If it is valid, it will be output to the downstream; if it is invalid, it will be discarded.
[0089] For example, see Figure 4 , an example diagram of the workflow of the filter function component provided in this application.
[0090] like Figure 4 As shown, after the start, the bind method is called to bind the upstream Upstream and wait for data to arrive, and then it is determined whether the ReceiveEnd state has not ended. When the judgment is yes, that is, the ReceiveEnd state has not ended, the Filter is used to determine whether the data is valid. When the data is determined to be valid, the receiveData method of the downstream DownStream is called to send the data to the downstream, and then the bind method is continued to be called to bind the upstream Upstream and wait for data to arrive. When the data is determined to be invalid, the bind method is continued to be called to bind the upstream Upstream and wait for data to arrive. Until it is determined that the ReceiveEnd state has ended, the ReceiveEnd method of the downstream DownStream is called to send the end state to the downstream and end.
[0091] The Batch function component is mainly used for batch output, caching batch data, and outputting it to the downstream at one time.
[0092] In this application, newly designed or modified function components are added on the basis of the basic components of functional programming, so as to complete the data assembly logic between complex data flow groups.
[0093] The newly designed or modified function components mainly include the dequeue function component, group function component, optional function component and wait function component.
[0094] The Dequeue function component is mainly used to reuse data objects from the object pool.
[0095] For example, see Figure 5 , an example diagram of the workflow of the dequeue function component provided in this application.
[0096] After the start, call the bind method to bind the upstream Upstream and wait for data to arrive, then determine whether to end the ReceiveEnd state. If it is, call the ReceiveEnd method of the downstream DownStream to send the end state to the downstream; if it is not, use Dequeue to take out an available object StreamData from the object pool, call the Serialize method to obtain the serialized HashMap to fill StreamData.content, then call the ReceiveData method of the downstream DownStream to send data to the downstream, and then continue to call the bind method to bind the upstream Upstream and wait for data to arrive. When it is determined that the ReceiveEnd state is over, call the ReceiveEnd method of the downstream DownStream to send the end state to the downstream and end.
[0097] The Group function component is mainly used to assemble upstream data into one data according to the specified key and output it to the downstream.
[0098] The Optional function component is mainly used to mark upstream data with an optional identifier. When merging downstream group function components, the upstream data with the optional identifier can be ignored and other upstream data can be directly merged and output to the downstream.
[0099] Exemplary, participating Figure 6 , an example diagram of the workflow of the optional function component provided in this application.
[0100] like Figure 6 As shown, after the start, the bind method is called to bind the upstream Upstream and wait for data to arrive. Then, it is determined whether the ReceiveEnd state has not ended. If it is, that is, the ReceiveEnd state has not ended, the data is marked as optional, and then the data with the optional mark is sent to the downstream DownStream. The execution continues to return to call the bind method to bind the upstream Upstream and wait for data to arrive. Until the ReceiveEnd state is determined to end, the ReceiveEnd method of the downstream DownStream is called to send the end state to the downstream and end.
[0101] The Wait function component is mainly used to indicate the waiting time of data and mark the upstream data with a wait mark. When the downstream group function component merges upstream data, if the upstream data does not arrive within the waiting time, it will be ignored.
[0102] Exemplary, participating Figure 7 , an example diagram of the workflow of the wait function component provided in this application.
[0103] like Figure 7 As shown, after the start, the bind method is called to bind the upstream Upstream and wait for data to arrive. It is judged whether the ReceiveEnd state has not ended. If it is judged to be, that is, the ReceiveEnd state has not ended, the data is marked with a wait mark, and then the data with the wait mark is sent to the downstream DownStream. The execution continues to return to call the bind method to bind the upstream Upstream and wait for data to arrive until it is judged that the ReceiveEnd state has ended. The ReceiveEnd method of the downstream DownStream is called to send the end state to the downstream and end.
[0104] Next, using iOS network requests as an example, we will introduce each logic in the aforementioned declarative function calculation logic.
[0105] iOS network requests are divided into three data streams: start request - data stream A, end request - data stream B, indicator collection - data stream C, and new data stream output after assembly - data stream E.
[0106] A=Stream().dequeue(P1).filer().map();
[0107] B=Stream().dequeue(P1).filer().map();
[0108] C=Stream().dequeue(P1).filer().map().optional();
[0109] D=Stream().dequeue(P2).group(A,B,requestId).wait(5);
[0110] E=Stream().dequeue(P2).queueGroup(D,C,requestId).batch(20).
[0111] For optional logic, the data flow assembly function component will treat the data flow with the optional identifier as an optional data flow when assembling a new data flow.
[0112] In this example, the optional logic is implemented based on the optional function component. Data stream C has the optional flag, indicating that data stream C is optional. Therefore, when downstream data stream E is assembled using the group function component, if data stream C does not reach or is not collected, data stream C can be directly ignored and only data stream D can be output to the downstream.
[0113] For the waiting logic, the data stream assembly storage component will wait for the data stream with the waiting identifier according to the time preset in the waiting logic.
[0114] In this example, the waiting logic is implemented based on the wait function component. Data stream D has a wait mark of 5 seconds. Data stream D will wait for 5 seconds. After 5 seconds, regardless of whether the optional data stream C arrives, the merged data will be directly output to the downstream. However, if data stream C is not an optional data stream, the merged data will be considered invalid and discarded.
[0115] For combinational logic, the data flow assembly storage component assembles a new data flow based on the data flows with the same keywords in each data flow group specified in the combinational logic.
[0116] In this example, the combination logic is implemented based on the group function component. Specifically, data is merged based on the data fields. When merging, it is necessary to determine whether the merged data stream has the optional tag and the wait tag. For example, in group(A,B), both A and B data must exist to output the merged data; if any data is missing, the merged data cannot be output. For group(C,D), data C has the optional tag, so only data D needs to exist to be output. However, data D has the wait tag, so it needs to wait for 5 seconds before being output.
[0117] The object pool logic is related to the storage of the new data stream after it is assembled into a new data stream.
[0118] In this example, the object pool logic is implemented based on the dequeue function component. It mainly obtains data objects through the object pool and then performs subsequent data processing. Among them, different object pools P1 or P2 can be selected according to the data size. The existence of the object pool can reduce the overhead of object creation and control the size of memory usage.
[0119] For example, see Figure 8 ,This application provides an example diagram of the function calculation process for iOS network requests.
[0120] StreamA and StreamB use group transformations to combine data A and data B into data D, which is then transformed into data stream D. StreamD and StreamC are then grouped together to output StreamE. StreamC is a data stream group with an optional flag. Its C3 data is not output, and the data D3 in the combined StreamE does not contain C3. However, this is still considered a valid output, implementing optional logic. Furthermore, fine-grained function combination effectively hierarchizes the entire assembly logic, reduces the scope of locking, and improves the performance of data statistics caching.
[0121] In summary, each piece of collected data is treated as a data stream, and each stream outputs individual pieces of data. Functional computation is performed on each stream, that is, on each data stream in the data stream group. During the functional computation process, there's no need to worry about the specific data; only the functional transformation logic of the data stream is considered. By combining different functional variations, data assembly operations are simplified. Furthermore, the output of functional computation depends only on the input data, with no other dependencies. This allows for parallel computation, improving the efficiency and performance of data stream assembly. Declarative functional computation formulas are also easy to maintain.
[0122] From the above example, we can see that each data flow group is assembled in a declarative functional manner according to the pre-set batch output logic and preset logic (at least one of the optional logic, waiting logic, combination logic and object pool logic) in the pre-set declarative function computing logic, and each data flow group is assembled in a declarative functional manner, which can complete the assembly between various complex data flow groups.
[0123] It should also be noted that each data stream collection component is only responsible for collecting data and does not need to worry about downstream data assembly, which reduces the coupling between data, achieves separation of concerns, makes it easier to write unit tests, and improves the maintainability of the code.
[0124] The previous section mentioned the components that implement the declarative function computing logic. Next, we will explain the specific implementation of the declarative function computing logic from two aspects: data stream assembly processing and new data stream storage.
[0125] The assembly processing of data streams and the storage of new data streams are implemented based on the data stream assembly storage component. The data stream assembly storage component mainly includes the data stream assembly function component and the data stream storage function component.
[0126] The data flow assembly function components primarily include the basic function components mentioned above and the newly designed or modified function components of this application: map function component, filter function component, batch function component, dequeue function component, group function component, optional function component, and wait function component, etc. Each function component has been described in detail above and will not be repeated here.
[0127] Next, we will introduce the data stream storage function component. Similarly, the dequeue function component, group function component, optional function component, and wait function component can be used as data stream storage function components to assemble and generate storage for new data streams.
[0128] Generally, when the preset declarative function computing logic includes object pool logic, the dequeue function component stores the newly generated new data stream group into the target object pool according to the preset unified data object format. The target object pool is an object pool that matches the data size of the new data stream group. It should be noted that in the object pool of the dequeue function component, the application and release of data objects are based on queue operations, taking data from the head of the queue and putting it back to the end of the queue when releasing it.
[0129] Optionally, a data stream assembly storage component is used to assemble the individual data stream groups into a new data stream group, and the new data stream group is stored in the memory.
[0130] Step 103: When the number of generated new data streams reaches the preset output data volume in the batch output logic, each generated new data stream is written in batches into the local database of the mobile terminal.
[0131] Next, the new data stream in the memory cache is written to the local database. In this application, each new data stream is output according to the batch output logic.
[0132] The data stream storage function component is used to complete the output of each new data stream.
[0133] Optionally, when the data volume of each new data stream reaches the preset output data volume in the batch output logic, each new data stream stored in the queue group function component of the data stream storage function component is output and written to the local database of the mobile terminal.
[0134] The queue component function may specifically be a QueueGroup function component, which uses Queue storage internally and outputs data in batches according to time series, which is suitable for output scenarios.
[0135] For example, see Figure 9 , an example diagram of the workflow of the QueueGroup function component provided in this application.
[0136] After starting, the bind method is called to bind the upstream Upstream and wait for data to arrive. Then, it is determined whether the ReceiveEnd state has ended. If it is not, that is, the ReceiveEnd state has not ended, the QueueGroup is used to store the data stream group in the internal TimeSerialQueue. Then, it is determined whether the data meets the output conditions, and the data waiting for the timeout in the queue is searched. When it is determined that the output conditions are met, the ReceiveData method of the downstream DownStream is called to send the data to the downstream. Then, the bind method is returned to bind the upstream Upstream and wait for data to arrive. When it is determined that the ReceiveEnd state has ended, the ReceiveEnd method of the downstream DownStream is called to send the end status to the downstream and end. Moreover, when the data does not meet the output conditions, the bind method is returned to bind the upstream Upstream and wait for data to arrive. When it is determined that the ReceiveEnd state has ended, the ReceiveEnd method of the downstream DownStream is called to send the end status to the downstream and end.
[0137] The TimeSerialQueue component involved in the aforementioned QueueGroup function component primarily supports wait flags and batch output capabilities. It performs a binary search based on the timestamp to locate the last timed-out data item. It then identifies the data immediately preceding it as timed-out and pops all data items before it out of the queue in batches, improving performance. Furthermore, there's no need to create asynchronous or scheduled tasks for each data item. Automatically triggering the wait logic of the TimeSerialQueue component upon data update is more efficient.
[0138] For example, see Figure 10 , this application provides an example diagram of the workflow of the TimeSerialQueue component.
[0139] After the process starts, data is stored and each piece of input data is added to the end of the queue. The queue's last update time is then updated. The last update time is the value of the timeStamp field in the data object, which can be the timestamp of the metadata in the stream. A binary search is then performed based on the timestamp value: the last timeout data item, timeoutData. The comparison condition is data.timestamp ≤ lastUpdate.Time - waitTimeout. All data items before timeoutData are popped from the queue and batch-emitted to the downstream.
[0140] In addition, the data stream storage function component also includes a grouping function component, which can be specifically called a MapGroup function component. It is stored internally in a hash table format, specifically in a key-value format.
[0141] For example, see Figure 11 , an example diagram of the workflow of the MapGroup function component provided in this application.
[0142] After the start, the bind method is called to bind the upstream Upstream and wait for data to arrive, and then determine whether the ReceiveEnd state has ended. When the judgment is no, that is, the ReceiveEnd state has not ended, MapGroup is used to store the data in the internal HashMap, and then determine whether the upstream Upstream data has a wait mark. When it is determined that a wait mark exists, the timeout judgment asynchronous task is started, and the receiveDate method of the downstream DownStream is called to send data to the downstream. Then, the call to the bind method is returned to bind the upstream Upstream and wait for data to arrive. When it is determined that the ReceiveEnd state has ended, the receiveEnd method of the downstream DownStream is called to send the end status to the downstream and end. When the upstream Upstream data is judged to have no wait mark, the receiveDate method of the downstream DownStream is directly called to send data to the downstream, and then the call to the bind method is returned to bind the upstream Upstream and wait for data to arrive. When it is determined that the ReceiveEnd state has ended, the receiveEnd method of the downstream DownStream is called to send the end status to the downstream and end.
[0143] It should be noted that the grouping function component can also delay the output according to the timeout of the wait tag, which is suitable for output scenarios with small data volumes.
[0144] In summary, the data statistics caching method for the mobile terminal provided by the present application regards the collected initial data as individual data streams, and uses the data stream component to collect them into various data stream groups. Then, the data assembly storage component is used to determine the preset declarative function calculation logic for each data stream group. Based on any two function calculation logics including the batch output logic in the declarative function calculation logic, each data stream group is assembled into a new data stream and stored in the memory. When the number of new data streams generated reaches the preset output data volume in the batch output logic, the generated new data streams are batch-written into the local database of the mobile terminal. The declarative function calculation logic is used in the present application to realize the assembly between complex data streams, which improves the performance of data caching. In addition, the object pool can be used to reduce memory overhead, and the batch output method can be used to improve the performance of writing data to the database. It can be seen that the data statistics caching method provided by the present application has high performance and high efficiency in all aspects.
[0145] The above describes a data statistics caching method for a mobile terminal provided by an embodiment of the present application. The following describes an apparatus for executing the above-mentioned data statistics caching method for a mobile terminal.
[0146] See also Figure 12 , Figure 12 This is a schematic diagram of the structure of a data statistics cache device for a mobile terminal provided by this application. Figure 12 As shown, the data statistics cache device of the mobile terminal includes:
[0147] Collection unit 10, assembly unit 20 and output unit 30; wherein:
[0148] The acquisition unit 10 is configured to acquire an initial data set using a data stream component; each initial data collected at the same data acquisition point in the initial data set constitutes a data stream group; each data stream in the data stream group corresponds one-to-one to each initial data in the initial data set;
[0149] An assembly unit 20 is configured to determine a preset declarative function computing logic for each data stream group from the data stream assembly storage component, assemble each data stream into a new data stream group according to the preset declarative function computing logic, and store the new data stream group in memory; the declarative function computing logic is a combination of batch output logic and preset logic, and the preset logic is at least one of optional logic, waiting logic, combination logic, and object pool logic;
[0150] The output unit 30 is configured to write each generated new data stream into a local database of the mobile terminal in batches when the number of generated new data streams reaches a preset output data volume in the batch output logic.
[0151] In one embodiment, the data stream assembly storage component in the data statistics cache device of the mobile terminal includes a data stream assembly function component, wherein the assembly unit 20 is specifically configured to:
[0152] When the preset declarative function computing logic includes combination logic, the data flow assembly function component assembles the data flows with the same keywords in each data flow group specified in the combination logic into a new data flow;
[0153] When the preset declarative function computing logic includes optional logic, the data flow assembly function component will treat the data flow with the optional mark as an optional data flow when assembling a new data flow;
[0154] When the preset declarative function calculation logic includes waiting logic, the data stream assembly function component will wait for the data stream with the waiting mark according to the preset time in the waiting logic.
[0155] In one embodiment, the data stream assembly storage component in the data statistics cache device of the mobile terminal includes a data stream storage function component, and its assembly unit 20 is specifically used to:
[0156] When the preset declarative function computing logic includes object pool logic, the data stream storage function component stores the new data stream in the preset unified data object format in the target object pool according to the object pool logic; the target object pool is the object pool preset in the object pool logic that matches the data size of the new data stream.
[0157] In one embodiment, the data flow storage function component in the data statistics cache device of the mobile terminal includes a queue group function component and an output unit 30, which is specifically configured to:
[0158] When the data volume of each new data stream reaches the preset output data volume in the batch output logic, each new data stream stored in the queue group function component is output and written to the local database of the mobile terminal.
[0159] In one embodiment, the data stream storage function component in the data statistics cache device of the mobile terminal includes a grouping function component and an output unit 30, which is specifically configured to:
[0160] When the data volume of each new data stream is lower than the preset minimum data volume threshold in the batch output logic, each new data stream stored in the hash table of the grouping function component is output and written to the local database of the mobile terminal.
[0161] In one embodiment, the acquisition unit 10 is specifically configured to:
[0162] The data stream component is used to collect different data stream groups from different data collection points upstream, and each data stream is sent to the downstream data stream assembly storage component according to the collection order of each data stream in the data stream group.
[0163] The present application also provides a data statistics cache device for a mobile terminal. Figure 13 As shown, it shows a schematic diagram of the structure of a data statistics cache device for a mobile terminal suitable for implementing the data statistics cache device provided by the present application. The data statistics cache device for the mobile terminal in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 13 The data statistics cache device on the mobile terminal shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.
[0164] like Figure 13 As shown, the data statistics cache device on the mobile terminal may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. When the data statistics cache device on the mobile terminal is powered on, the RAM 603 also stores various programs and data required for the operation of the data statistics cache device on the mobile terminal. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0165] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the mobile terminal's data statistics cache device to communicate with other devices wirelessly or by wire to exchange data. Figure 13 The data statistics cache device of the mobile terminal is shown to have various devices, but it should be understood that it is not required to implement or have all the devices shown, and more or fewer devices may be implemented or have instead.
[0166] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any mobile terminal data statistics caching method provided in the embodiment of the present application.
[0167] A computer storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any mobile terminal data statistics caching method provided in the embodiment of the present application.
[0168] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0169] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0170] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0171] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training equipment or data center to another website, computer, training equipment or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training equipment, data center, etc. that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)), etc.
Claims
1. A data statistics caching method for a mobile terminal, characterized in that: include: Use data flow components to collect the initial data set; Each initial data collected at the same data collection point in the initial data set is a data stream group; Each data stream in the data stream group corresponds to each initial data in the initial data set in a one-to-one manner; Determining a preset declarative function computing logic for each data stream group from a data stream assembly storage component, assembling each data stream into a new data stream group according to the preset declarative function computing logic, and storing the new data stream group in a memory; the declarative function computing logic is a combination of batch output logic and preset logic, and the preset logic is at least one of optional logic, waiting logic, combination logic, and object pool logic; When the number of the generated new data streams reaches the preset output data amount in the batch output logic, the generated new data streams are written in batches into the local database of the mobile terminal.
2. The data statistics caching method for a mobile terminal according to claim 1, characterized in that: The data stream assembly storage component includes a data stream assembly function component, and the assembly of the data streams into a new data stream according to the preset declarative function calculation logic includes: When the preset declarative function computing logic includes combination logic, the data stream assembly function component assembles the data streams with the same keywords in the respective data stream groups specified in the combination logic into the new data stream; When the preset declarative function computing logic includes optional logic, the data stream assembly function component will regard the data stream with the optional identifier as an optional data stream when assembling the new data stream; When the preset declarative function computing logic includes a waiting logic, the data stream assembly function component will wait for the data stream with the waiting identifier according to the time preset in the waiting logic.
3. The data statistics caching method for a mobile terminal according to claim 1, characterized in that: The data stream assembly storage component includes a data stream storage function component, which assembles the data streams into a new data stream group according to the preset declarative function calculation logic and stores it in the memory, including: When the preset declarative function computing logic includes object pool logic, the data stream storage function component stores the new data stream in the target object pool in a preset unified data object format according to the object pool logic; the target object pool is an object pool preset in the object pool logic that matches the data size of the new data stream.
4. The data statistics caching method for a mobile terminal according to claim 3, characterized in that: The data stream storage function component includes a queue group function component, and when the number of the generated new data streams reaches the preset output data amount in the batch output logic, the generated new data streams are batch written into the local database of the mobile terminal, including: When the data volume of each of the new data streams reaches the preset output data volume in the batch output logic, each of the new data streams stored in the queue group function component is output and written into the local database of the mobile terminal.
5. The data statistics caching method for a mobile terminal according to claim 4, characterized in that: The data stream storage function component includes a grouping function component, and when the number of the generated new data streams reaches the preset output data amount in the batch output logic, the generated new data streams are batch written into the local database of the mobile terminal, including: When the data volume of each of the new data streams is lower than the preset minimum data volume threshold in the batch output logic, each of the new data streams stored in the hash table of the grouping function component is output and written into the local database of the mobile terminal.
6. The data statistics caching method for a mobile terminal according to claim 1, characterized in that: The data stream component is used to collect the initial data set, including: The data stream component is used to collect different data stream groups from different data collection points upstream, and the data streams in the data stream group are sent to the data stream assembly storage component downstream according to the collection order of the data streams.
7. A data statistics cache device for a mobile terminal, characterized in that: include: A collection unit, configured to collect an initial data set using a data flow component; Each initial data collected at the same data collection point in the initial data set is a data stream group; Each data stream in the data stream group corresponds to each initial data in the initial data set in a one-to-one manner; An assembly unit, configured to determine a preset declarative function computing logic for each data stream group from a data stream assembly storage component, assemble the data streams into a new data stream group according to the preset declarative function computing logic, and store the new data stream group in a memory; the declarative function computing logic is a combination of batch output logic and preset logic, and the preset logic is at least one of optional logic, waiting logic, combination logic, and object pool logic; The output unit is configured to write each of the generated new data streams into a local database of the mobile terminal in batches when the number of the generated new data streams reaches a preset output data volume in the batch output logic.
8. A data statistics cache device for a mobile terminal, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so that the data statistics caching device on the mobile terminal can implement the data statistics caching method on the mobile terminal as described in any one of claims 1 to 6.
9. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the data statistics caching method for a mobile terminal as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the mobile terminal data statistics caching method as described in any one of claims 1 to 6.