Data processing method, device, electronic device and computer readable storage medium

Abstract

The present disclosure relates to a data processing method, device, electronic device and computer-readable storage medium. The data processing method includes: obtaining online action flow data; determining the object screening information and target field of the target acquisition task; filtering out the target data from the online action flow data according to the object screening information and the target field, wherein the object screening information is used to filter out the data of the corresponding object from the online action flow data; and storing the target data. By allowing the client to report the buried data normally and form the online action flow data, and then obtaining the target data required for the test from the online action flow data, there is no need to perform additional development on the client, which can effectively reduce the development cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computer application technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of mobile Internet, the functions that mobile apps can realize are becoming more and more diverse. After a function is launched, product managers usually need to use tracking data to view the impact of the function on app usage. In current tracking tests, the tracking data reported by the client is usually intercepted through a proxy. This solution requires additional operations on the client to make the client report data through the proxy, which will bring client development costs. Summary of the invention

[0003] The present disclosure provides a data processing method, device, electronic device and computer-readable storage medium to at least solve the problem of high development cost in the related art, and may not solve any of the above problems.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, including: acquiring online action flow data; determining object screening information and target fields of a target acquisition task, wherein the object screening information is used to screen out data of a corresponding object from the online action flow data; screening out target data from the online action flow data according to the object screening information and the target field; and storing the target data.

[0005] Optionally, the data processing method further includes: acquiring data verification information, the data verification information being related to the target acquisition task; and performing verification processing on the target data according to the data verification information.

[0006] Optionally, the verifying the target data according to the data verification information includes: obtaining client configuration information corresponding to the target data; selecting configuration verification parameters from the client configuration information according to the data verification information; and verifying the target data according to the configuration verification parameters.

[0007] Optionally, obtaining the client configuration information corresponding to the target data includes: obtaining the client configuration information corresponding to the target data from a service agent; wherein the client configuration information is obtained through the following steps: the service agent obtains the client configuration information fed back by the server for the configuration request in response to identifying a configuration request, wherein the configuration request is issued by the device that generates the target data.

[0008] Optionally, the data processing method also includes: obtaining device information of a device that generates the target data; wherein, obtaining data verification information includes: requesting a callback interface before the target acquisition task ends to obtain task parameters of the target acquisition task, wherein the task parameters include a task identifier and device information; comparing the device information corresponding to the target data with the device information in the task parameters, determining a task identifier corresponding to the same device information as the target data, recorded as a target task identifier; and obtaining the data verification information based on the target task identifier.

[0009] Optionally, the device information is a device serial number, and the target data includes a device identifier, wherein obtaining the device information of the device that generates the target data includes: querying mapping information, the mapping information recording a mapping relationship between the device serial number and the device identifier; and determining the device serial number of the device that generates the target data based on the mapping information and the device identifier in the target data.

[0010] Optionally, the target collection task is configured with at least one collection use case, the task parameters also include a use case identifier of each collection use case and an occurrence period of each collection use case, and the target data also includes time information, wherein obtaining the data verification information based on the target task identifier includes: comparing the time information of the target data with the occurrence period of each collection use case corresponding to the target task identifier, determining the use case identifier of the collection use case corresponding to the target data in time, recorded as the target use case identifier; obtaining the data verification information based on the target task identifier and the target use case identifier.

[0011] Optionally, the data processing method further includes: dynamically displaying a trend curve of the target data based on the stored target data.

[0012] Optionally, the data processing method further includes: based on the object screening information, filtering out data of the corresponding object from the online action flow data, recording the data as preliminary screening data; and storing the preliminary screening data separately from the target data.

[0013] According to a second aspect of the present disclosure, a data processing device is provided, comprising: an acquisition unit, configured to acquire online action flow data; a determination unit, configured to determine object screening information and target fields of a target acquisition task, wherein the object screening information is used to screen out data of a corresponding object from the online action flow data; a screening unit, configured to screen out target data from the online action flow data according to the object screening information and the target field; and a storage unit, configured to store the target data.

[0014] Optionally, the acquisition unit is further configured to: acquire data verification information, wherein the data verification information is related to the target acquisition task; the data processing device also includes a verification unit configured to: perform verification processing on the target data according to the data verification information.

[0015] Optionally, the verification unit is further configured to: obtain client configuration information corresponding to the target data; select configuration verification parameters from the client configuration information according to the data verification information; and perform verification processing on the target data according to the configuration verification parameters.

[0016] Optionally, the verification unit is further configured to: obtain client configuration information corresponding to the target data from the service agent; wherein the client configuration information is obtained through the following steps: the service agent, in response to identifying a configuration request, obtains the client configuration information fed back by the server for the configuration request, wherein the configuration request is issued by the device that generates the target data.

[0017] Optionally, the acquisition unit is further configured to: acquire device information of a device that generates the target data; request a callback interface before the target acquisition task is completed to acquire task parameters of the target acquisition task, wherein the task parameters include a task identifier and device information; compare the device information corresponding to the target data with the device information in the task parameters to determine a task identifier corresponding to the same device information as the target data, recorded as a target task identifier; and acquire the data verification information based on the target task identifier.

[0018] Optionally, the device information is a device serial number, the target data includes a device identifier, and the acquisition unit is further configured to: query mapping information, the mapping information records a mapping relationship between the device serial number and the device identifier; and determine the device serial number of the device that generates the target data based on the mapping information and the device identifier in the target data.

[0019] Optionally, the target collection task is configured with at least one collection use case, the task parameters also include a use case identifier of each collection use case and an occurrence period of each collection use case, the target data also includes time information, and the acquisition unit is further configured to: compare the time information of the target data with the occurrence period of each collection use case corresponding to the target task identifier, determine the use case identifier of the collection use case corresponding to the target data in time, and record it as the target use case identifier; and obtain the data verification information based on the target task identifier and the target use case identifier.

[0020] Optionally, the data processing device further includes a display unit configured to dynamically display a trend curve of the target data based on the stored target data.

[0021] Optionally, the screening unit is further configured to: screen out data of the corresponding object from the online action flow data according to the object screening information, and record it as preliminary screening data; the storage unit is further configured to: store the preliminary screening data separately from the target data.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a data processing method according to the present disclosure.

[0023] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the data processing method according to the present disclosure.

[0024] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the data processing method according to the present disclosure when executed by at least one processor.

[0025] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0026] According to the data processing method and data processing device of the embodiments of the present disclosure, by allowing the client to normally report the buried data and form online action flow data, and then obtaining the target data required for the test from the online action flow data, there is no need for additional development of the client, which can effectively reduce the development cost.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0029] Figure 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure.

[0030] Figure 2 Detailed description of the invention is given in detail below.

[0031] Figure 3 is a schematic diagram illustrating a visual representation according to an exemplary embodiment of the present disclosure.

[0032] Figure 4 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0033] Figure 5 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0036] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0037] With the development of mobile Internet, the functions that mobile apps can realize are becoming more and more diverse. After a function is launched, product managers usually need to use tracking data to view the impact of the function on the use of the app. Simply put, there are a large number of operation nodes on the app, one or some of which will have a strong correlation with the function to be tested. Tracking data is the user's operation data on these strongly associated operation nodes. Usually, the tracking process includes data reporting, data analysis, data storage, data verification, and data statistics. The tracking data is produced and reported by the client, and the reported data will be encrypted during this process. The general tracking reporting test method is to intercept the reported data within the enterprise, decrypt the intercepted data, and display the data in plain text on the web page. When QA (Quality assurance) performs tracking test, it verifies the data on the web page to ensure the correctness and integrity of the tracking data.

[0038] The launched apps often have a large number of users and many operation nodes, so it is difficult to ensure that all the acquired data are applicable to the testing of the functions to be tested. Therefore, in order to improve the efficiency of data acquisition, enterprises often use devices to register accounts on their own, and use these devices to operate the functions to be tested to generate tracking data. In the current tracking test, a common solution is to intercept and decrypt the tracking data through a proxy during the reporting process, and visualize the intercepted and decrypted data in plain text. QA then goes to the visualization platform to manually verify the data.

[0039] This solution has the following defects: 1. To implement data interception, it is necessary to configure a reporting button on the client, and trigger the reporting button through operation, so that the client reports the data through the agent, which will bring about the development cost of the client. 2. The agent is required to configure and promptly update the decryption method of the buried data, which brings about the maintenance cost of the agent. 3. The operation of intercepting data by the agent interferes with the original reporting process of the data, making it impossible to realize the original data storage of the intercepted data. If a problem is found in the buried data during verification, it can only be queried in the agent's own storage, and cannot be queried on the online data analysis platform. The proxy to implement data storage will take up additional resources, reducing the storage efficiency and usage efficiency of the data.

[0040] According to the data processing scheme of the exemplary embodiment of the present disclosure, by allowing the client to report the buried data normally and form the online action flow data, and then obtaining the target data required for the test from the online action flow data, there is no need for additional development of the client, which can effectively reduce the development cost, and the regular decryption process of the online action flow data can be executed, without the need for additional agent maintenance, which helps to control costs. In addition, all buried data generated by the client can enter the online process normally, and the buried plaintext data can be obtained without interfering with the online buried data production and consumption process. Specifically, the execution of the process of this scheme is parallel to the online process, without overlapping parts, and does not affect each other. The buried data obtained by this scheme can still be queried according to the original online query method, which ensures the storage efficiency and use efficiency of the data.

[0041] Next, we will refer to Figures 1 to 5 A data processing method and a data processing apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0042] Figure 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure. Figure 2 Detailed description of the invention is given in detail below.

[0043] Reference Figure 1 In step 101, online action flow data is obtained. Figure 2 , online action stream data is, for example, online Kafka queue data (Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on the website), client (corresponding to Figure 2 The data collection layer shown) can send out point reports through http requests. The reported point data are collected together to form online action flow data, and are handed over to the service layer to execute online consumption services. Specifically, the online action flow data can be cleaned in real time, and data decryption and data format conversion (for example, converting pb (protocol buffer) data into json (JavaScript Object Notation) data) can be completed for subsequent use. For some scenarios with large data volumes, random sampling or whitelist sampling can also be performed to reduce the amount of data. It should be understood that multiple parallel consumption services can be executed online at the same time without affecting each other. As an example, a consumption service may include recommending content that may be of interest to users based on their browsing history.

[0044] Return to reference Figure 1, in step 102, determine the object screening information and target field of the target collection task, wherein the object screening information is used to filter out the data of the corresponding object from the online action flow data. A collection task refers to a task that collects point data in order to implement a specific functional test, and a target collection task is the collection task that is currently being focused on. As mentioned earlier, in order to improve data acquisition efficiency, companies often use devices to register accounts on their own, and use these devices to operate the functions to be tested to generate point data. The object screening information is, for example, a data table or array that records the devices or accounts used by the company for testing. As an example, refer to Figure 2 , the configuration management platform inside the app can maintain a did (device identity document, used to identify a physical device) whitelist and a uid (user identity document, user identification) whitelist, i.e. object screening information. It should be understood that the device / user whitelist is different from the whitelist used for whitelist sampling in step 101.

[0045] In step 103, target data is filtered out from the online action flow data according to the object screening information and the target field. Each buried point data in the online action flow data generally contains a device identifier and a user identifier. By combining the object screening information, the data that hits the object screening information in the online action flow data can be filtered out to ensure data acquisition efficiency. At the same time, in order to meet the data requirements of different acquisition tasks, the buried point data usually contains a large number of fields, such as user data such as user gender, user age group, and user city, and device data such as device operating system and client version, and content data such as video resolution, bit rate, and frame rate. By combining the target field, the fields that need to be used can be filtered out, which helps to reduce the amount of target data, improve data pertinence, and also helps to ensure data acquisition efficiency. It should be understood that the device identifier and user identifier will generally continue to be retained in the target data for data statistics in the test.

[0046] In step 104, the target data is stored. By storing the target data obtained in step 103, QA can perform data verification, and then perform data statistics to implement functional testing. Figure 2 , the target data can be stored in the MySQL database, and the target data can be, for example, a QOS (Quality of Service) indicator.

[0047] Optionally, after step 104, the data processing method according to the exemplary embodiment of the present disclosure may further include a step of automatically verifying the target data to replace or assist manual verification and improve data processing efficiency. The step of automatically verifying the target data can be further refined into an information acquisition step and a verification processing step. Among them, the information acquisition step is to obtain data verification information, and the data verification information is related to the target acquisition task. The verification processing step is to verify the target data according to the data verification information. By obtaining the data verification information related to the target acquisition task, it is possible to collect data of multiple different target acquisition tasks at the same time, and perform automatic verification processing according to the respective data verification information, thereby improving data processing efficiency, without having to collect data of only one target acquisition task at a time and perform automatic verification according to a single data verification information.

[0048] Next, the steps of obtaining information are introduced.

[0049] Optionally, the data processing method further includes: obtaining the device information of the device that generates the target data; the information acquisition step may include: requesting a callback interface before the target acquisition task is completed to obtain the task parameters of the target acquisition task, the task parameters including the task ID of the corresponding acquisition task and the device information of the device whose data is collected when the corresponding acquisition task is executed; comparing the device information corresponding to the target data and the device information in the task parameters, determining the task ID corresponding to the same device information as the target data, and recording it as the target task ID; obtaining data verification information according to the target task ID. The target data is collected when the target acquisition task is executed, but the target acquisition task information is not added to the target data during the collection process. In order to realize the automatic verification of the target data, it is necessary to first determine the target acquisition task, so as to obtain the data verification information used. By requesting a callback interface before the target acquisition task completes data processing but ends the task, the task parameters containing the device information and the task ID can be obtained. The device information can be used as an intermediate bridge to compare with the device information corresponding to the target data, so as to find the task ID corresponding to the target data, that is, the task ID of the target acquisition task, and the data verification information can be obtained accordingly to ensure the smooth acquisition of the data verification information. Specifically, the target task identifier can be stored in the MySQL database and associated with the corresponding target data.

[0050] Optionally, device information can be used to identify a device at the system level. App platforms often manage devices through device information, such as the device serial number. The target data often includes the device identifier at the hardware level and does not directly include the device information. The step of obtaining the device information of the device that generates the target data may include: querying the mapping information, the mapping information records the mapping relationship between the device serial number and the device identifier; determining the device serial number of the device that generates the target data based on the mapping information and the device identifier in the target data. By combining the mapping information, the device information corresponding to the target data can be smoothly determined, ensuring the reliable acquisition of the target task identifier and data verification information. Figure 2 , the configuration management platform inside the app can also maintain the did mapping, that is, the mapping information, and compare the device identification and mapping information in the target data to determine the device serial number. Specifically, the device serial number can be stored in the MySQL database and associated with the corresponding target data.

[0051] Optionally, the target acquisition task is configured with at least one acquisition use case. As an example, the acquisition use case may be the operating system of the device or the client version. Different acquisition use cases of the same target acquisition task may have different configurations and thus different data verification information. The task parameters also include the use case identifier of each acquisition use case and the occurrence period of each acquisition use case, and the target data also includes time information. Accordingly, according to the target task identifier, the step of obtaining data verification information may include: comparing the time information of the target data with the occurrence period of each acquisition use case corresponding to the target task identifier, determining the use case identifier of the acquisition use case corresponding to the target data in time, and recording it as the target use case identifier; obtaining data verification information according to the target task identifier and the target use case identifier. By further comparing the generation time of the target data with the occurrence period of each acquisition use case when executing the target acquisition task, it is possible to clarify which acquisition use case the target data corresponds to, thereby determining the corresponding target use case identifier, and then obtaining the corresponding data verification information, thereby ensuring smooth and reliable acquisition of information.

[0052] Next, the verification processing steps are introduced.

[0053] Reference Figure 2 In the data processing method according to the exemplary embodiment of the present disclosure, the verification rule may include at least one of a timing verification, an integrity verification, a reference verification, and a configuration read verification.

[0054] A complete operation process has a conversation id. Different tracking points in the same process have the same conversation id, and different processes have different conversation ids. Timing verification is to verify whether the timing of occurrence and the relationship between each tracking point data on a conversation id link are normal. The data verification information can record the normal timing of occurrence and the relationship between each tracking point data. Integrity verification is to verify whether the tracking point data associated with a conversation id link is complete, such as whether the import, export, and upload tracking points exist, and whether the fields of the tracking point data are complete. The data verification information can record the tracking points of the complete link and the complete fields of the tracking point data.

[0055] For benchmark verification, the data verification information can record several benchmark fields in the buried data. If the buried data has these benchmark fields at the same time, the benchmark verification is passed.

[0056] For configuration reading verification, there is a solution in the related technology to formulate json verification rules in advance and verify and judge the acquired target data. For example, when judging the resolution buried data, you can configure the json verification rule: source.video.height=="720"&&source.video.width=="1080", that is, the video height is 720 and the width is 1080. By parsing the json verification rules and then getting the target field in the target data, you can verify whether the target data meets expectations. However, if the resolution is indeed upgraded due to changes in requirements, and the new resolution is 1080*1920, then the original verification rules are no longer applicable, and new verification rules need to be written, resulting in maintenance costs.

[0057] Optionally, the verification processing step according to the exemplary embodiment of the present disclosure may include: obtaining the client configuration information corresponding to the target data; selecting the configuration verification parameters from the client configuration information according to the data verification information; and verifying the target data in accordance with the configuration verification parameters. By actively obtaining the client configuration information corresponding to the target data and selecting the parameters that match the data verification information as the configuration verification parameters, the verification processing can be performed according to the latest configuration verification parameters, and the automatic update of the configuration verification parameters can be realized, making the verification process more flexible and helping to fully reduce the maintenance cost. As an example, for the case where the function to be tested is a video upload function, the data verification information may include the imported / exported transcoding software and hardware, bit rate, and resolution gear. Accordingly, the configuration verification parameters are the configuration values ​​of the imported / exported transcoding software and hardware, bit rate, and resolution gear, which constitute the specific transcoding strategy for importing / exporting videos when uploading videos. The verification processing is to compare the values ​​of the corresponding parameters in the target data with the configuration verification parameters to verify whether the values ​​of the corresponding parameters in the target data conform to the configuration.

[0058] Optionally, refer to Figure 2 , the step of obtaining the client configuration information corresponding to the target data includes: obtaining the client configuration information corresponding to the target data from the service agent; wherein the client configuration information is obtained by the following steps: the service agent obtains the client configuration information fed back by the server for the configuration request in response to identifying the configuration request, wherein the configuration request is issued by the device that generates the target data. Since the client configuration information is often notified to the client by the server through configuration delivery, the specific configuration strategy can be obtained by obtaining the configuration delivery content. Specifically, the client connects to the service agent, such as mitmproxy (a middleman proxy tool), and the service agent filters the http request, thereby identifying the configuration request, i.e., the configuration delivery request, and can obtain the client configuration information fed back by the server, i.e., the configuration delivery.

[0059] Optionally, according to the data processing method of the exemplary embodiment of the present disclosure, it also includes: based on the stored target data, dynamically displaying the trend curve of the target data. By using the trend curve to display the stored target data, the changing trend of the relevant target data can be intuitively displayed, which helps the staff understand the changes in the target data. It should be understood that dynamic display means that when the target data fails to pass the verification and is corrected accordingly, the trend curve can change accordingly, and the trend curve can be dynamically updated to ensure the timeliness of the transmitted information. As an example, data can be read from a database storing the target data and a data set can be created. After the data set is created, some dirty data can be filtered out through the original value filter, and a trend curve and detailed data can be created according to the display requirements.

[0060] Figure 3 is a schematic diagram illustrating a visual representation according to an exemplary embodiment of the present disclosure.

[0061] Reference Figure 3 The horizontal axis of the trend curve represents different client versions, the vertical axis represents a certain parameter, and the two curves represent two devices. Since the same device uses the same client version for a short period of time, most of the data in the figure are historical data. In one of the curves, the parameter suddenly dropped when the second client version was used, and then gradually recovered. It can be considered that the subsequent client versions have optimized this problem; however, the other curve fluctuates, indicating that the optimization effect is different on different devices.

[0062] Optionally, regarding data storage, the data processing method according to the exemplary embodiment of the present disclosure further includes: based on the object screening information, screening out the data of the corresponding object from the online action flow data, and recording it as preliminary screening data; storing the preliminary screening data and the target data separately. In combination with the previous introduction, it can be seen that the target data is the data corresponding to the target field in the preliminary screening data. By storing the complete preliminary screening data separately, it can provide a reference for data correction after data verification, ensuring the reliable implementation of data correction, and can ensure the efficient processing of the target data. Figure 2 , the initial screening data (i.e. the complete embedding data corresponding to the object screening information) can be stored in the ES (elasticsearch, a distributed multi-user full-text search engine) database to achieve convenient data search.

[0063] As an example, in general, refer to Figure 2 According to the data processing method of the exemplary embodiment of the present disclosure, it involves a data collection layer, a service layer, a storage layer, and an interaction layer.

[0064] The data collection layer is composed of multiple client devices, which can report the tracking data and other data generated during the operation of the app together to form online kafka queue data. The service layer represents the server. For the online kafka queue data, the service layer executes the consumption service in S11 to obtain the online kafka queue data, and then in S12, combined with the did whitelist or uid whitelist in the configuration management platform inside the app, the tracking data (i.e., the initial screening data) is filtered out from the obtained online kafka queue data, and in S13, the complete tracking data is stored in the ES database of the storage layer. In S14, the QOS indicator (corresponding to the target data) is filtered out from the tracking data in combination with the target field, and stored in the mysql database of the storage layer, so as to realize the acquisition, screening and storage of tracking data.

[0065] The interaction layer includes user devices and a visualization platform, which can be used for information input and output. The user device can query the interface based on the user's query input and query the corresponding information from the ES database. Before the target collection task completes data processing but ends the task, the visualization platform requests a callback interface in S21 to obtain the task parameters of the target collection task, and then compares the task parameters with the QOS indicators in the MySQL database in S22 to determine the data verification information of the target collection task, which can be used for S23 to complete the automatic verification of the QOS indicators. In addition, in S30, the visualization platform can obtain the QOS indicators in the MySQL database and perform, for example, Figure 3 The visual representation shown.

[0066] Figure 4is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0067] Reference Figure 4 The data processing device 400 includes an acquisition unit 401, a determination unit 402, a screening unit 403, and a storage unit 404.

[0068] The acquisition unit 401 may acquire online action flow data.

[0069] The determining unit 402 may determine the object screening information and the target field of the target acquisition task, wherein the object screening information is used to screen out data of a corresponding object from the online action flow data.

[0070] The screening unit 403 may screen out target data from the online action flow data according to the object screening information and the target field.

[0071] The storage unit 404 may store target data.

[0072] Optionally, the acquisition unit 401 may also acquire data verification information, which is related to the target acquisition task; the data processing device 400 also includes a verification unit, which may perform verification processing on the target data according to the data verification information.

[0073] Optionally, the verification unit may also obtain client configuration information corresponding to the target data; select configuration verification parameters from the client configuration information according to the data verification information; and perform verification processing on the target data according to the configuration verification parameters.

[0074] Optionally, the verification unit is further configured to: obtain client configuration information corresponding to the target data from the service agent; wherein the client configuration information is obtained through the following steps: the service agent obtains the client configuration information fed back by the server for the configuration request in response to identifying the configuration request, wherein the configuration request is issued by the device that generates the target data.

[0075] Optionally, the acquisition unit 401 can also acquire the device information of the device that generates the target data; request a callback interface before the target acquisition task ends to obtain the task parameters of the target acquisition task, the task parameters including the task identifier and the device information; compare the device information corresponding to the target data with the device information in the task parameters, determine the task identifier corresponding to the same device information as the target data, and record it as the target task identifier; and acquire data verification information based on the target task identifier.

[0076] Optionally, the device information is the device serial number, the target data includes the device identifier, and the acquisition unit 401 can also query mapping information, which records the mapping relationship between the device serial number and the device identifier; based on the mapping information and the device identifier in the target data, determine the device serial number of the device that generates the target data.

[0077] Optionally, the target collection task is configured with at least one collection use case, and the task parameters also include a use case identifier of each collection use case and an occurrence period of each collection use case. The target data also includes time information. The acquisition unit 401 can also compare the time information of the target data with the occurrence period of each collection use case corresponding to the target task identifier, determine the use case identifier of the collection use case corresponding to the target data in time, and record it as the target use case identifier; and obtain data verification information based on the target task identifier and the target use case identifier.

[0078] Optionally, the data processing device further includes a display unit, which can dynamically display a trend curve of the target data based on the stored target data.

[0079] Optionally, the screening unit 403 may further screen out data of a corresponding object from the online action flow data according to the object screening information, and record the data as preliminary screening data; the storage unit 404 may further store the preliminary screening data and the target data separately.

[0080] Can be combined with reference Figures 1 to 3 The described data processing method is used to understand the operations of the corresponding units in the data processing device 400.

[0081] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0082] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502, wherein the at least one memory 501 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 502, a data processing method according to an exemplary embodiment of the present disclosure is executed.

[0083] As an example, the electronic device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 500 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0084] In the electronic device 500, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0085] The processor 502 may execute instructions or codes stored in the memory 501, wherein the memory 501 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0086] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory.

[0087] In addition, the electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 500 may be connected to each other via a bus and / or a network.

[0088] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the data processing method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0089] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by at least one processor, the at least one processor is prompted to perform the data processing method according to the exemplary embodiment of the present disclosure.

[0090] According to the data processing method, device, electronic device and computer-readable storage medium of the exemplary embodiment of the present disclosure, by allowing the client to report the buried data normally and form the online action flow data, and then obtain the target data required for the test from the online action flow data, there is no need for additional development of the client, which can effectively reduce the development cost, and the regular decryption process of the online action flow data can be executed, without the need for additional agent maintenance, which helps to control costs. In addition, all buried data generated by the client can enter the online process normally, and the buried plaintext data can be obtained without interfering with the online buried data production and consumption process. Specifically, the execution of the process of this solution is parallel to the online process, without overlapping parts, and does not affect each other. The buried data obtained by this solution can still be queried according to the original online query method, which ensures the storage efficiency and use efficiency of the data.

[0091] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0092] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Get online action flow data; Determine object screening information and target fields of the target acquisition task, wherein the object screening information is used to screen out data of corresponding objects from the online action flow data; Filtering target data from the online action flow data according to the object screening information and the target field; storing the target data; Acquire device information of a device that generates the target data; Requesting a callback interface before the target acquisition task ends to obtain the task parameters of the target acquisition task, wherein the task parameters include the task identifier of the corresponding acquisition task and the device information of the device whose data is collected when executing the corresponding acquisition task; Comparing the device information corresponding to the target data with the device information in the task parameters, determining a task identifier corresponding to the same device information as the target data, and recording it as a target task identifier; According to the target task identifier, data verification information is acquired, where the data verification information is related to the target acquisition task; The target data is verified according to the data verification information.

2. The data processing method according to claim 1, characterized in that: The step of performing verification processing on the target data according to the data verification information includes: Obtaining client configuration information corresponding to the target data; According to the data verification information, selecting configuration verification parameters from the client configuration information; The target data is verified according to the configuration verification parameters.

3. The data processing method according to claim 2, characterized in that: The obtaining of the client configuration information corresponding to the target data includes: Obtaining client configuration information corresponding to the target data from a service agent; The client configuration information is obtained through the following steps: In response to identifying a configuration request, the service agent obtains the client configuration information fed back by the server in response to the configuration request, wherein the configuration request is issued by the device that generates the target data.

4. The data processing method according to claim 1, characterized in that: The device information is a device serial number, and the target data includes a device identifier, wherein the step of acquiring the device information of the device that generates the target data includes: Query mapping information, where the mapping information records the mapping relationship between the device serial number and the device identifier; The device serial number of the device that generates the target data is determined according to the mapping information and the device identification in the target data.

5. The data processing method according to claim 1, characterized in that: The target collection task is configured with at least one collection use case, the task parameters further include a use case identifier of each collection use case and an occurrence period of each collection use case, the target data further includes time information, wherein the step of obtaining the data verification information according to the target task identifier includes: Compare the time information of the target data with the occurrence period of each collection use case corresponding to the target task identifier, determine the use case identifier of the collection use case corresponding to the target data in time, and record it as the target use case identifier; The data verification information is acquired according to the target task identifier and the target use case identifier.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The data processing method further includes: Based on the stored target data, a trend curve of the target data is dynamically displayed.

7. The data processing method according to any one of claims 1 to 5, characterized in that: The data processing method further includes: According to the object screening information, data of the corresponding object is screened out from the online action flow data, and recorded as the primary screening data; The preliminary screening data and the target data are stored separately.

8. A data processing device, characterized in that: include: The acquisition unit is configured to: acquire online action flow data; A determination unit is configured to: determine object screening information and target fields of a target acquisition task; A screening unit, configured to: screen out target data from the online action flow data according to the object screening information and the target field, wherein the object screening information is used to screen out data of a corresponding object from the online action flow data; A storage unit, configured to: store the target data; The acquisition unit is further configured to: acquire device information of a device that generates the target data; request a callback interface before the target acquisition task ends to acquire task parameters of the target acquisition task, wherein the task parameters include a task identifier of the corresponding acquisition task and device information of the device whose data is collected when the corresponding acquisition task is executed; compare the device information corresponding to the target data with the device information in the task parameters, determine a task identifier corresponding to the same device information as the target data, and record it as a target task identifier; acquire data verification information according to the target task identifier, wherein the data verification information is related to the target acquisition task; The data processing device further includes a verification unit configured to perform verification processing on the target data according to the data verification information.

9. The data processing device according to claim 8, characterized in that The verification unit is further configured to: Obtaining client configuration information corresponding to the target data; According to the data verification information, selecting configuration verification parameters from the client configuration information; The target data is verified according to the configuration verification parameters.

10. The data processing device according to claim 9, characterized in that The verification unit is further configured to: obtain client configuration information corresponding to the target data from a service agent; The client configuration information is obtained through the following steps: In response to identifying a configuration request, the service agent obtains the client configuration information fed back by the server in response to the configuration request, wherein the configuration request is issued by the device that generates the target data.

11. The data processing device according to claim 8, characterized in that The device information is a device serial number, the target data includes a device identifier, and the acquisition unit is further configured to: Query mapping information, where the mapping information records the mapping relationship between the device serial number and the device identifier; The device serial number of the device that generates the target data is determined according to the mapping information and the device identification in the target data.

12. The data processing device according to claim 8, characterized in that The target acquisition task is configured with at least one acquisition use case, the task parameters further include a use case identifier of each acquisition use case and an occurrence period of each acquisition use case, the target data further includes time information, and the acquisition unit is further configured to: Compare the time information of the target data with the occurrence period of each collection use case corresponding to the target task identifier, determine the use case identifier of the collection use case corresponding to the target data in time, and record it as the target use case identifier; The data verification information is acquired according to the target task identifier and the target use case identifier.

13. The data processing device according to any one of claims 8 to 12, characterized in that The data processing device also includes a display unit configured to dynamically display a trend curve of the target data based on the stored target data.

14. The data processing device according to any one of claims 8 to 12, characterized in that The screening unit is further configured to: screen data of a corresponding object from the online action flow data according to the object screening information, and record the data as preliminary screening data; The storage unit is also configured to store the preliminary screening data and the target data separately.

15. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the data processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the data processing method according to any one of claims 1 to 7.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images