A data acquisition method and device, electronic equipment and storage medium

Abstract

The application provides a data collection method and device, electronic equipment and storage medium. The method is applied to a mobile terminal device and includes the following steps: obtaining task information of a target task from a cloud device, wherein the task information includes the target task's point-in-time information, data filtering conditions and a time window, and the cloud device is used for storing the task information; determining the target task's data point-in-time according to the point-in-time information, and obtaining point-in-time data meeting the data filtering conditions from the data point-in-time as to-be-processed data; and aggregating the to-be-processed data according to the time window to obtain an aggregation result, and storing the aggregation result as a data file of the target task. In this way, the data point-in-time is determined according to the task information, the data is aggregated, and the data file corresponding to the target task can be obtained. If the data required by the target task needs to be modified, the task information only needs to be modified in the cloud device, the data acquisition mode is relatively flexible, and the limitation is relatively small.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a data acquisition method, apparatus, electronic device, and storage medium. Background Technology

[0002] In scenarios involving the collection of user behavior data, mobile devices have a natural advantage in this area, as they can directly interact with users and obtain their behavioral data.

[0003] Currently, user behavior data is typically acquired through hard-coding on mobile devices. This means that the user behavior data to be acquired is directly embedded into the source code of the program or other executable object, and the user behavior data is acquired during the execution of the program or code.

[0004] However, since hard-coded data can usually only be modified by editing the source code and recompiling the executable file, the current method of obtaining user behavior data is inflexible. It requires development for specific needs and then iterative updates with each version release. The data is difficult to use effectively, and it has great limitations and very poor flexibility. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a data acquisition method, apparatus, electronic device, and storage medium.

[0006] This invention provides a data acquisition method applied to mobile devices, the method comprising:

[0007] The task information of the target task is obtained from the cloud device. The task information includes the tracking information, data filtering conditions and time window of the target task. The cloud device is used to store the task information.

[0008] Based on the data tracking information, determine the data tracking points for the target task, and obtain the data tracking point data that meets the data filtering conditions from the data tracking points as data to be processed;

[0009] The data to be processed is aggregated according to the time window to obtain the aggregation result, and the aggregation result is stored as the data file of the target task.

[0010] Optionally, the target task includes multiple tasks. The step of determining the data tracking points for the target task based on the tracking point information, and obtaining tracking point data that meets the data filtering conditions from the data tracking points as data to be processed, includes:

[0011] Based on the data point information of the target task, a distribution index is generated for each data point, and the distribution index is used to indicate the correspondence between each data point and at least one target task.

[0012] The data tracking points are monitored, and the target task corresponding to the monitored tracking point data is determined according to the distribution index.

[0013] Determine whether the monitored data meets the data filtering conditions of any target task corresponding to the data monitoring point. If it does, then the monitored data is used as the data to be processed for any target task.

[0014] Optionally, the step of aggregating the data to be processed according to the time window to obtain the aggregation result includes:

[0015] The time when the task information is acquired is used as the start time to determine the end time of multiple time windows for the target task.

[0016] When the deadline of any time window is reached, the data to be processed obtained within that time window is aggregated to obtain the aggregated result.

[0017] Optionally, the task information further includes a first aggregation rule, wherein the aggregation of the data to be processed according to the time window to obtain the aggregation result includes:

[0018] According to the time window, the data to be processed is aggregated according to the first aggregation rule to obtain the aggregation result;

[0019] The first aggregation rule includes at least one of the following: data count, maximum value calculation, and minimum value calculation.

[0020] Optionally, the task information further includes a retention period, and storing the aggregation result as a data file for the target task includes:

[0021] If the time difference between the current time and the time when the aggregation result was last stored in the target task's data file exceeds the storage period, the unstored aggregation result will be stored as the target task's data file.

[0022] Optionally, the task information further includes a preset operation, which is used to trigger the storage of the aggregation result. Storing the aggregation result as a data file for the target task includes:

[0023] In response to the preset operation, detect whether the aggregation result of the target task has been updated;

[0024] If an update occurs, the updated aggregation result is stored as the data file for the target task.

[0025] Optionally, the task information further includes query time and a second aggregation rule. After aggregating the data to be processed according to the time window to obtain the aggregation result, the method further includes:

[0026] Obtain the data to be queried within the specified query time from the aggregation results;

[0027] According to the second aggregation rule, the data to be queried is aggregated to obtain the query result;

[0028] The step of storing the aggregation result as a data file for the target task includes:

[0029] The aggregation results and the query results are stored as a data file for the target task.

[0030] Optionally, each data in the aggregation result has a corresponding key-value pair, and the aggregation of the data to be queried according to the second aggregation rule to obtain the query result includes:

[0031] According to the second aggregation rule, the key-value pairs of the data to be queried are aggregated to obtain the query result.

[0032] Optionally, each piece of data to be processed includes multiple data items, and the task information also includes grouped data items and aggregated data items. The grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed.

[0033] The process of aggregating the data to be processed according to the time window to obtain the aggregation result includes:

[0034] Based on the grouped data items, the data to be processed within each time window is grouped;

[0035] Aggregate the aggregated data items of the data to be processed within each group to obtain the aggregation result corresponding to that group.

[0036] The present invention also provides a data acquisition device for use in mobile devices, the device comprising:

[0037] The information acquisition module is used to acquire task information of the target task from the cloud device. The task information includes the tracking information, data filtering conditions and time window of the target task. The cloud device is used to store the task information.

[0038] The data acquisition module is used to determine the data tracking points of the target task based on the tracking point information, and to acquire tracking point data that meets the data filtering conditions from the data tracking points as data to be processed.

[0039] The aggregation module is used to aggregate the data to be processed according to the time window, obtain the aggregation result, and store the aggregation result as the data file of the target task.

[0040] Optionally, the target task includes multiple tasks, and the data acquisition module is used for:

[0041] Based on the data point information of the target task, a distribution index is generated for each data point, and the distribution index is used to indicate the correspondence between each data point and at least one target task.

[0042] The data tracking points are monitored, and the target task corresponding to the monitored tracking point data is determined according to the distribution index.

[0043] Determine whether the monitored data meets the data filtering conditions of any target task corresponding to the data monitoring point. If it does, then the monitored data is used as the data to be processed for any target task.

[0044] Optionally, the aggregation module is used for:

[0045] The time when the task information is acquired is used as the start time to determine the end time of multiple time windows for the target task.

[0046] When the deadline of any time window is reached, the data to be processed obtained within that time window is aggregated to obtain the aggregated result.

[0047] Optionally, the task information further includes a first aggregation rule, wherein the aggregation module is used for:

[0048] According to the time window, the data to be processed is aggregated according to the first aggregation rule to obtain the aggregation result;

[0049] The first aggregation rule includes at least one of the following: data count, maximum value calculation, and minimum value calculation.

[0050] Optionally, the task information also includes a retention period, and the aggregation module is used to:

[0051] If the time difference between the current time and the time when the aggregation result was last stored in the target task's data file exceeds the storage period, the unstored aggregation result will be stored as the target task's data file.

[0052] Optionally, the task information further includes a preset operation, which is used to trigger the storage of the aggregation result. The aggregation module is used to:

[0053] In response to the preset operation, detect whether the aggregation result of the target task has been updated;

[0054] If an update occurs, the updated aggregation result is stored as the data file for the target task.

[0055] Optionally, the task information also includes query time and a second aggregation rule. The aggregation module is further used for:

[0056] Obtain the data to be queried within the specified query time from the aggregation results;

[0057] According to the second aggregation rule, the data to be queried is aggregated to obtain the query result;

[0058] The aggregation results and the query results are stored as a data file for the target task.

[0059] Optionally, each data point in the aggregation result has a corresponding key-value pair, and the aggregation module is used to:

[0060] According to the query rules, the key-value pairs of the data to be queried are aggregated to obtain the query results.

[0061] Optionally, each piece of data to be processed includes multiple data items, and the task information also includes grouped data items and aggregated data items. The grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed.

[0062] The aggregation module is used for:

[0063] Based on the grouped data items, the data to be processed within each time window is grouped;

[0064] Aggregate the aggregated data items of the data to be processed within each group to obtain the aggregation result corresponding to that group.

[0065] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the data acquisition methods described above.

[0066] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the data acquisition methods described above.

[0067] Compared with the prior art, the present invention has the following advantages:

[0068] In this invention, the mobile device obtains task information of the target task from the cloud device. The task information includes the target task's tracking information, data filtering conditions, and time window. The cloud device stores the task information. Then, based on the tracking information, the data tracking points of the target task are determined, and tracking data that meets the data filtering conditions is obtained from the data tracking points as data to be processed. Subsequently, the data to be processed is aggregated according to the time window to obtain the aggregation result, and the aggregation result is stored as the target task's data file.

[0069] In this way, by embedding data points based on task information and aggregating the data, the data file corresponding to the target task can be obtained. If it is necessary to modify the data required by the target task, the task information only needs to be modified on the cloud device. There is no need to re-edit the source code and recompile the executable file. Therefore, the data acquisition method is more flexible and has fewer limitations. Attached Figure Description

[0070] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0071] Figure 1 This is a flowchart of the steps of a data acquisition method according to the present invention;

[0072] Figure 2 This is a schematic diagram of a data acquisition method according to the present invention;

[0073] Figure 3 This is a structural block diagram of a data acquisition device according to the present invention;

[0074] Figure 4 This is a structural block diagram of an electronic device according to the present invention. Detailed Implementation

[0075] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0076] The data acquisition method provided by the embodiments of the present invention will be described in detail below through specific examples.

[0077] Reference Figure 1 The diagram illustrates a flowchart of a data acquisition method according to the present invention, applied to a mobile device. The method may specifically include the following steps:

[0078] S101: Obtain the task information of the target task from the cloud device. The task information includes the target task's tracking information, data filtering conditions, and time window. The cloud device is used to store the task information.

[0079] In this invention, a mobile device can send a task retrieval request to a cloud device, carrying the device information of the mobile device in the request. Then, the cloud device can respond to the received task retrieval request by sending the task information of the target task to the mobile device. Alternatively, the cloud device can also generate the task information of the target task and, based on pre-acquired device information, push the generated task information to the mobile device.

[0080] Task information can include the tracking information, data filtering conditions, and time windows for the target task. This information is stored in a cloud device, and users can configure it in the target task's configuration file through interaction with the cloud device. Furthermore, multiple target tasks can be stored simultaneously in the cloud device, and correspondingly, multiple target tasks can be executed on mobile devices. This allows for the collection of richer and more targeted data for different business needs, which is beneficial for subsequent data analysis based on the data to be processed. For example, if the data to be processed is used to train a new model, the flexible configuration of target tasks makes feature warm-up before the new model goes live much easier.

[0081] Among them, the tracking information of the target task is used to determine the data tracking points of the target task, that is, the source of the data content required by the target task. Data tracking is an important way of data collection. Its basic principle is that when the user's behavior on the mobile device meets certain conditions, such as entering a page or clicking a button, it will automatically trigger recording and storage. Then, this data will be collected as tracking data.

[0082] Data filtering conditions are used to filter the event tracking data. As you can understand, the event tracking data collected through event tracking may be diverse. By using data filtering conditions, the event tracking data required for the target task can be filtered out from the event tracking data and used as the data to be processed in subsequent processing.

[0083] The time window is the time window used for subsequent aggregation calculations. In other words, aggregation processing is performed on the data within each time window, which can save storage resources and computational load.

[0084] In addition, the task information may also include information such as the first aggregation rule, grouped data items, aggregated data items, query time, and the second aggregation rule. It may also include information such as the target task's version, name, ID (Identity document), and data source name. The data source name can uniquely identify the source of the event tracking data; for example, the source of the event tracking data could be user behavior logs or QoS (Quality of Service) logs, etc. Specific settings can be configured according to user needs.

[0085] S102: Based on the data tracking information, determine the data tracking points for the target task, and obtain the data tracking point data that meets the data filtering conditions from the data tracking points as the data to be processed.

[0086] Each data point can include multiple data items. For example, if the data source is user behavior logs, the data items could include page information, behavior information, and time information. If the data source is QoS logs, the data items could include port number, bandwidth information, and protocol information. These data items can be recorded in the data dictionary of the data point. For instance, if the data point represents a user click action, it could be represented as {"rpage": "playback page", "action": "click", ...}.

[0087] In one implementation, if there are multiple target tasks, a distribution index can be generated for each data tracking point based on the tracking information of the target tasks. This distribution index indicates the correspondence between each data tracking point and at least one target task. For example, the distribution index can be represented as: {'Data Source Name 1':["Task 1", "Task 2"], 'Data Source Name 2':["Task 3"]}, where "Data Source Name 1" and "Data Source Name 2" represent data tracking points, and "Task 1", "Task 2", and "Task 3" represent different target tasks. That is, "Data Source Name 1" corresponds to two target tasks, "Task 1" and "Task 2", while "Data Source Name 2" corresponds to one target task, "Task 3".

[0088] Then, the data tracking points are monitored, and the target task corresponding to the monitored tracking point data is determined according to the distribution index. Then, it is determined whether the monitored tracking point data meets the data filtering conditions of any target task corresponding to the data tracking point. If it does, the monitored tracking point data is used as the data to be processed for any target task.

[0089] In this way, for each data tracking point, based on the distribution index, the tracking point data only needs to be traversed once to distribute the tracking point data to its corresponding different target tasks, instead of traversing the same tracking point data separately for each target task. This avoids repeated traversal of the same data tracking point and improves processing speed.

[0090] In this step, data that meets the data filtering criteria is obtained from the data tracking points. The WHERE clause can be used to make judgments. The WHERE clause supports simple conditions and combined conditions. For example, a simple condition can be: filter data tracking points where the value of the data item "t" is 22 and use them as data to be processed. A combined condition is a combination of multiple simple conditions. When multiple simple conditions are met at the same time, the data tracking points can be used as data to be processed.

[0091] In this way, data that does not meet the data filtering criteria is filtered out, while data that does meet the criteria is treated as pending data and proceeds to the next step, reducing the amount of data that needs to be processed during subsequent aggregation and saving system resources.

[0092] S103: Aggregate the data to be processed according to the time window, obtain the aggregation result, and store the aggregation result in the data file of the target task.

[0093] In one implementation, the data to be processed is aggregated according to a time window to obtain the aggregation result, including:

[0094] The time when the task information is acquired is used as the start time to determine the deadlines for multiple time windows of the target task; when the deadline of any time window is reached, the data to be processed acquired within that time window is aggregated to obtain the aggregated result.

[0095] In other words, the aggregation of the data to be processed is based on a time window. For example, the time window can be 10 milliseconds. Then, after collecting the data to be processed for every 10 milliseconds, an aggregation is performed. This reduces the number of aggregations for the target task, saves computing resources, and at the same time, maintains the dynamic processing of the data to be processed.

[0096] In one implementation, the task information further includes a first aggregation rule to aggregate the data to be processed, obtaining an aggregation result. This includes: aggregating the data to be processed according to the first aggregation rule to obtain an aggregation result; wherein the first aggregation rule includes at least one of the following: total data count, maximum value calculation, and minimum value calculation. Aggregating the data to be processed can be done by calling a select condition statement. In the select condition statement, the count operator can be used for data count, the max operator can be used for maximum value calculation, and the min operator can be used for minimum value calculation. In this way, the data to be processed can be aggregated using different first aggregation rules, making data aggregation more flexible. Moreover, the aggregated data to be processed has a smaller volume, which is also convenient for subsequent storage and querying.

[0097] In one implementation, each piece of data to be processed includes multiple data items. The task information also includes grouped data items and aggregated data items. The grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed. That is, the data to be processed is grouped according to the grouped data items, and the aggregated data items are aggregated within each group. The aggregation result of the data to be processed includes the aggregation result corresponding to each group.

[0098] Therefore, aggregating the data to be processed to obtain the aggregation result can include: grouping the data to be processed within each time window based on the grouping data items; then aggregating the aggregated data items of the data to be processed within each group to obtain the aggregation result corresponding to that group. In other words, the data to be processed with the same grouping data items are grouped together to obtain multiple groups, and then, based on the aggregated data items, the data to be processed in each group is aggregated to obtain the aggregation result corresponding to each group.

[0099] The grouping data items can be determined by calling the groupBy statement. For example, the grouping data items can include "rpage", "block" and "rseat". Only the data to be processed with the same values ​​for these three items will be grouped together. Then, the select statement can be called to aggregate the grouped data to obtain the aggregated result for each group.

[0100] Both grouped and aggregated data items can be configured according to the business requirements of the target task. This allows for aggregation of data to be processed from different groups, making the aggregation analysis of the data to be processed more flexible and enabling more complex data aggregation processing.

[0101] In one implementation, the task information further includes a query time and a second aggregation rule. After aggregating the data to be processed according to the time window and obtaining the aggregation result, further aggregation can be performed on the aggregation result. Specifically, this may include the following steps:

[0102] The data to be queried within the query time is obtained from the aggregation results; then, the data to be queried is aggregated according to the second aggregation rule to obtain the query results. Subsequently, both the aggregation results and the query results can be stored in the data file of the target task.

[0103] The second aggregation rule is a rule that further aggregates the results based on the first aggregation rule. It can include total data count, maximum value calculation, and minimum value calculation. For example, if the first aggregation rule is to calculate the maximum value of the data to be processed within each time window, then the second aggregation rule could be to further aggregate the maximum values ​​of the data to be processed within each time window, such as calculating the sum of the maximum values ​​of the data to be processed within each time window, and so on. The specific rules are not limited.

[0104] In this way, secondary aggregation can be performed based on the aggregation results of the first aggregation rule, which is conducive to achieving extremely fast querying of the data to be processed and provides a very solid data foundation for scenarios such as mobile intelligence.

[0105] Each data point in the aggregation result has a corresponding key-value pair. According to the second aggregation rule, the data to be queried is aggregated to obtain the query result. That is, according to the second aggregation rule, the key-value pairs of the data to be queried are aggregated to obtain the query result.

[0106] It is understandable that each piece of data to be queried has a corresponding key and value. The key corresponds one-to-one with the data to be queried. Therefore, queries based on key-value pairs are more convenient and faster.

[0107] For example, you can call the statement `result = {}` to create a query dictionary and perform a query. Taking the second aggregation rule as counting the total number of data as an example, the key-value pairs of the data to be queried are represented as `ts-id` and `ts-value`. If the `ts-id` of a certain data to be queried is not in the query dictionary created by `result`, then the aggregation result is `result[ts-id] = ts-value`, that is, the total number of data remains unchanged. If the `ts-id` is in the query dictionary created by `result`, then the aggregation result is `result[ts-id] = result[ts-id] + ts-value`, that is, the total number of data needs to be increased by `ts-value`.

[0108] After obtaining the aggregation result, the aggregation result can be saved. In one implementation, the task information also includes a saving period. Storing the aggregation result as the target task's data file includes: after the time difference between the current time and the last time the aggregation result was stored in the target task's data file exceeds the saving period, storing the unstored aggregation result as the target task's data file.

[0109] For example, the storage period can be 5 minutes. If the difference between the current time and the last time the aggregation result was stored is 5 minutes, then the aggregation results within the most recent 5 minutes will be stored. In this way, by storing data according to the storage period, multiple aggregation results can be stored with each data write, thereby reducing the number of data writes and reducing the consumption of system resources.

[0110] In another implementation, the task information can also include preset operations. These preset operations trigger the storage of the aggregation results. In response to the preset operation, the system can detect whether the aggregation results of the target task have been updated. If an update has occurred, the updated aggregation results are stored as the target task's data file. For example, the preset operation could be a switching operation from the background to the foreground, etc., and the specific operation is not limited.

[0111] This allows for interaction with users, responding to their preset actions, and timely storage of aggregated results in critical scenarios or where data is prone to loss. It also reduces the number of data writes and the consumption of system resources.

[0112] In one implementation, the data file for the target task can be valid only during the program's lifecycle. When the program is closed, the data file becomes invalid, thereby reducing the consumption of system resources.

[0113] like Figure 2 The diagram shown is a schematic of the present invention. In the present invention, the cloud device includes a target task configuration backend, that is, the configuration of the target task's task information can be performed on the cloud device. Furthermore, the mobile device includes a data acquisition center, which can determine data embedding points based on the target task's task information, and then distribute the embedding point data to generate aggregated results corresponding to each target task.

[0114] As can be seen from the above, in the solution provided by the present invention, data points are embedded according to the task information and the data is aggregated to obtain the data file corresponding to the target task. If it is necessary to modify the data required by the target task, only the task information needs to be modified. There is no need to re-edit the source code and recompile the executable file. Therefore, the data acquisition method is more flexible and has fewer limitations.

[0115] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily essential to the present invention.

[0116] Reference Figure 3 The diagram illustrates a structural block diagram of a data acquisition device according to the present invention, which is applied to a mobile device. Specifically, the device may include the following modules:

[0117] The information acquisition module 201 is used to acquire task information of the target task from the cloud device. The task information includes the tracking information, data filtering conditions and time window of the target task. The cloud device is used to store the task information.

[0118] The data acquisition module 202 is used to determine the data tracking points of the target task based on the tracking point information, and to acquire tracking point data that meets the data filtering conditions from the data tracking points as data to be processed.

[0119] The aggregation module 203 is used to aggregate the data to be processed according to the time window, obtain the aggregation result, and store the aggregation result as the data file of the target task.

[0120] In one implementation, the target task includes multiple tasks, and the data acquisition module 202 is used for:

[0121] Based on the data point information of the target task, a distribution index is generated for each data point, and the distribution index is used to indicate the correspondence between each data point and at least one target task.

[0122] The data tracking points are monitored, and the target task corresponding to the monitored tracking point data is determined according to the distribution index.

[0123] Determine whether the monitored data meets the data filtering conditions of any target task corresponding to the data monitoring point. If it does, then the monitored data is used as the data to be processed for any target task.

[0124] In one implementation, the aggregation module 203 is used for:

[0125] The time when the task information is acquired is used as the start time to determine the end time of multiple time windows for the target task.

[0126] When the deadline of any time window is reached, the data to be processed obtained within that time window is aggregated to obtain the aggregated result.

[0127] In one implementation, the task information further includes a first aggregation rule, and the aggregation module 203 is used for:

[0128] According to the time window, the data to be processed is aggregated according to the first aggregation rule to obtain the aggregation result;

[0129] The first aggregation rule includes at least one of the following: data count, maximum value calculation, and minimum value calculation.

[0130] In one implementation, the task information further includes a retention period, and the aggregation module 203 is used for:

[0131] If the time difference between the current time and the time when the aggregation result was last stored in the target task's data file exceeds the storage period, the unstored aggregation result will be stored as the target task's data file.

[0132] In one implementation, the task information further includes a preset operation, which is used to trigger the storage of the aggregation result. The aggregation module 203 is used to:

[0133] In response to the preset operation, detect whether the aggregation result of the target task has been updated;

[0134] If an update occurs, the updated aggregation result is stored as the data file for the target task.

[0135] In one implementation, the task information further includes query time and a second aggregation rule; the aggregation module 203 is also used for:

[0136] Obtain the data to be queried within the specified query time from the aggregation results;

[0137] According to the second aggregation rule, the data to be queried is aggregated to obtain the query result;

[0138] The aggregation results and the query results are stored as a data file for the target task.

[0139] In one implementation, each data point in the aggregation result has a corresponding key-value pair, and the aggregation module 203 is used for:

[0140] According to the query rules, the key-value pairs of the data to be queried are aggregated to obtain the query results.

[0141] In one implementation, each piece of data to be processed includes multiple data items, and the task information further includes grouped data items and aggregated data items. The grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed.

[0142] The aggregation module 203 is used for:

[0143] Based on the grouped data items, the data to be processed is grouped;

[0144] For each group, the data to be processed within each time window is grouped;

[0145] Aggregate the aggregated data items of the data to be processed within each group to obtain the aggregation result corresponding to that group.

[0146] As can be seen from the above, in the solution provided by the present invention, by embedding data points according to task information and aggregating the data, the data file corresponding to the target task can be obtained. If it is necessary to modify the data required by the target task, only the task information needs to be modified. There is no need to re-edit the source code and recompile the executable file. Therefore, the data acquisition method is more flexible and has fewer limitations.

[0147] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0148] This invention also provides an electronic device, such as... Figure 4 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0149] Memory 703 is used to store computer programs;

[0150] When processor 701 executes a program stored in memory 703, it performs the following steps:

[0151] The task information of the target task is obtained from the cloud device. The task information includes the tracking information, data filtering conditions and time window of the target task. The cloud device is used to store the task information.

[0152] Based on the data tracking information, determine the data tracking points for the target task, and obtain the data tracking point data that meets the data filtering conditions from the data tracking points as data to be processed;

[0153] The data to be processed is aggregated according to the time window to obtain the aggregation result, and the aggregation result is stored in the data file of the target task.

[0154] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0155] The communication interface is used for communication between the aforementioned terminal and other devices.

[0156] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0157] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0158] As can be seen from the above, in the solution provided by the present invention, by embedding data points according to task information and aggregating the data, the data file corresponding to the target task can be obtained. If it is necessary to modify the data required by the target task, only the task information needs to be modified. There is no need to re-edit the source code and recompile the executable file. Therefore, the data acquisition method is more flexible and has fewer limitations.

[0159] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the data acquisition methods described in the above embodiments.

[0160] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data acquisition methods described in the above embodiments.

[0161] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0162] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0164] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data acquisition method, characterized in that, Applied to mobile devices, the method includes: The task information of the target task is obtained from the cloud device. The task information includes the tracking information of the target task, data filtering conditions, time window, grouped data items, aggregated data items, query time and second aggregation rule. The cloud device is used to configure and store the task information. Based on the data tracking information, the data tracking points for the target task are determined, and tracking data that meets the data filtering conditions is obtained from the data tracking points according to the distribution index, as data to be processed; wherein, each data to be processed includes multiple data items; the distribution index is used to indicate the correspondence between each data tracking point and at least one target task; The data to be processed is aggregated according to the time window to obtain the aggregation result, including: Based on the grouped data items and the aggregated data items, the data to be processed within each time window is grouped and aggregated to obtain the aggregation result corresponding to each group; the grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed. Based on the second aggregation rule, the data to be queried that is within the query time in the aggregation result is aggregated a second time to obtain the query result; The aggregation results and the query results are stored as a data file for the target task, and the data file is valid throughout the application's lifecycle.

2. The method according to claim 1, characterized in that, The target tasks include multiple tasks. The step of determining the data tracking points for the target tasks based on the tracking point information, and obtaining tracking point data that meets the data filtering conditions from the data tracking points according to the distribution index, as data to be processed, includes: Based on the data tracking information of the target task, generate a distribution index for each data tracking point; The data tracking points are monitored, and the target task corresponding to the monitored tracking point data is determined according to the distribution index. Determine whether the monitored data meets the data filtering conditions of any target task corresponding to the data monitoring point. If it does, then the monitored data is used as the data to be processed for any target task.

3. The method according to claim 1, characterized in that, The process of aggregating the data to be processed according to the time window to obtain the aggregation result includes: The time when the task information is acquired is used as the start time to determine the end time of multiple time windows for the target task. When the deadline of any time window is reached, the data to be processed obtained within that time window is aggregated to obtain the aggregated result.

4. The method according to claim 1, characterized in that, The task information also includes a first aggregation rule, wherein the aggregation of the data to be processed according to the time window to obtain the aggregation result includes: According to the time window, the data to be processed is aggregated according to the first aggregation rule to obtain the aggregation result; The first aggregation rule includes at least one of the following: data count, maximum value calculation, and minimum value calculation.

5. The method according to claim 1, characterized in that, The task information also includes a retention period, and storing the aggregation result as a data file for the target task includes: If the time difference between the current time and the time when the aggregation result was last stored in the target task's data file exceeds the storage period, the unstored aggregation result will be stored as the target task's data file.

6. The method according to claim 1, characterized in that, The task information also includes a preset operation, which is used to trigger the storage of the aggregation result. Storing the aggregation result as a data file for the target task includes: In response to the preset operation, detect whether the aggregation result of the target task has been updated; If an update occurs, the updated aggregation result is stored as the data file for the target task.

7. The method according to claim 1, characterized in that, After aggregating the data to be processed according to the time window to obtain the aggregation result, the method further includes: Obtain the data to be queried within the specified query time from the aggregation results; According to the second aggregation rule, the data to be queried is aggregated to obtain the query result.

8. The method according to claim 7, characterized in that, Each data point in the aggregation result has a corresponding key-value pair. The aggregation of the data to be queried according to the second aggregation rule to obtain the query result includes: According to the second aggregation rule, the key-value pairs of the data to be queried are aggregated to obtain the query result.

9. The method according to claim 1, characterized in that, The process involves grouping and aggregating the data to be processed within each time window based on grouped and aggregated data items to obtain the aggregation result corresponding to each group, including: Based on the grouped data items, the data to be processed within each time window is grouped; Aggregate the aggregated data items of the data to be processed within each group to obtain the aggregation result corresponding to that group.

10. A data acquisition device, characterized in that, The device, applied to mobile devices, includes: The information acquisition module is used to acquire task information of the target task from the cloud device. The task information includes the target task's data tracking information, data filtering conditions, time window, grouped data items, aggregated data items, query time, and second aggregation rules. The cloud device is used to configure and store the task information. The data acquisition module is used to determine the data tracking points of the target task based on the tracking point information, and to acquire tracking point data that meets the data filtering conditions from the data tracking points according to the distribution index, as data to be processed; wherein, each data to be processed includes multiple data items; the distribution index is used to indicate the correspondence between each data tracking point and at least one target task; The aggregation module is used to aggregate the data to be processed according to the time window to obtain the aggregation result, including: grouping and aggregating the data to be processed within each time window based on the grouped data items and the aggregated data items to obtain the aggregation result corresponding to each group; the grouped data items are used to group the data to be processed, and the aggregated data items are used to aggregate the data to be processed; based on the second aggregation rule, performing a second aggregation on the data to be queried within the query time in the aggregation result to obtain the query result; and storing the aggregation result and the query result as a data file for the target task, wherein the data file is valid throughout the application's lifecycle.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data acquisition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data acquisition method as described in any one of claims 1 to 9.

Images