Data processing method and device, equipment and computer readable storage medium
By generating task records and data processing rules through the funnel analysis task configuration interface, the problem of long development time and high operation and maintenance costs of funnel analysis programs in the existing technology is solved, and rapid generation and efficient data analysis are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2023-09-22
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, funnel analysis programs have long development cycles, high maintenance costs, and cannot support multi-threaded funnel analysis task processing, resulting in low data analysis efficiency.
By displaying the task configuration interface for funnel analysis tasks, the system generates target funnel analysis task records, parses configuration element information, generates data processing rules, extracts data from the target data source, and calculates funnel analysis indicators based on custom aggregation functions.
It enables the rapid generation of new funnel analysis tasks, reduces development time and maintenance costs, and improves data analysis efficiency.
Smart Images
Figure CN117194523B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of big data technology, and in particular relates to a data processing method, apparatus, device and computer-readable storage medium. Background Technology
[0002] Funnel analysis is an important tool in the field of data analytics, used to explore key stages in the user journey that hinder user conversion and retention. To analyze user data and calculate funnel analysis metrics, existing technologies typically involve custom-developing funnel analysis programs tailored to specific needs.
[0003] However, under this development model, a funnel analysis program is only applicable to one funnel analysis requirement. If a new funnel analysis requirement is added, a new funnel analysis program needs to be customized and developed. Therefore, the existing technical solution involves a large amount of repetitive coding and deployment work, resulting in long development cycles, high maintenance costs, and low data analysis efficiency for funnel analysis programs. Summary of the Invention
[0004] This application provides a data processing method, apparatus, device, and computer-readable storage medium, which can reduce the development time and maintenance cost of funnel analysis programs and improve the efficiency of data analysis.
[0005] In a first aspect, embodiments of this application provide a data processing method, which includes: displaying a task configuration interface for a funnel analysis task; responding to a user's configuration operation on the task configuration interface, determining configuration element information of the target funnel analysis task to be processed; the configuration element information includes at least target data source configuration information and funnel analysis indicators corresponding to the target funnel analysis task to be processed; generating a record of the target funnel analysis task to be processed based on the configuration element information; parsing the configuration element information in the target funnel analysis task record when the record is scanned; generating data processing rules for each task node of the target funnel analysis task based on the parsed configuration element information; extracting target data from the target data source corresponding to the target data source configuration information according to the data processing rules; and performing calculations on the target data to obtain the indicator results of the funnel analysis indicators.
[0006] According to an embodiment of the first aspect of this application, the task configuration interface includes a data source configuration option. Configuration elements include a data source and funnel data filtering conditions for the data source. In response to a user's configuration operation on the task configuration interface, the configuration element information of the target funnel analysis task to be processed is determined, including: in response to a user's operation on the data source configuration option, determining target data source configuration information; searching for the target data source corresponding to the target data source configuration information from the data source configuration mapping table; determining the funnel data filtering conditions corresponding to the target data source from the data source and funnel data filtering condition configuration mapping table; and in response to a user's operation of selecting target funnel data filtering conditions from the funnel data filtering conditions corresponding to the target data source, determining the target funnel data filtering conditions for the target funnel analysis task to be processed.
[0007] According to any of the foregoing embodiments of the first aspect of this application, after determining the target funnel data filtering conditions for the target funnel analysis task to be processed, the data processing method further includes: receiving a user's target operation on the target funnel data filtering conditions, and updating the target funnel data filtering conditions for the target funnel analysis task to be processed; the target operation includes any one of the user's addition, deletion, or modification operations on the target funnel data filtering conditions.
[0008] According to any of the foregoing embodiments of the first aspect of this application, the target funnel analysis task record includes a task processing status identifier for the target funnel analysis task, and the data processing method further includes: during the process of parsing the configuration element information in the target funnel analysis task record, updating the task processing status identifier from a first identifier to a second identifier, wherein the first identifier is used to indicate that the target funnel analysis task has not been processed, and the second identifier is used to indicate that the target funnel analysis task is being processed.
[0009] According to any of the foregoing embodiments of the first aspect of this application, the target funnel analysis task record further includes a task identifier for the target funnel analysis task. After calculating and processing the target data to obtain the indicator results of the funnel analysis indicators, the data processing method further includes: storing the indicator results in the result index of the search server; and setting the task identifier as the result identifier of the indicator results during the process of storing the indicator results in the result index.
[0010] According to any of the foregoing embodiments of the first aspect of this application, after storing the indicator results in the result index of the search server, the data processing method further includes: displaying a result display interface for the funnel analysis task, the result display interface including a task identifier for the funnel analysis task; responding to a user's operation of selecting a target task identifier from the task identifier, searching for the target indicator result corresponding to the target task identifier in the result index; encapsulating the target indicator result according to a preset format; and displaying the encapsulated target indicator result.
[0011] According to any of the foregoing embodiments of the first aspect of this application, the target data includes user identifiers, user operation behavior data, and operation time of different users performing operations at different task nodes. The target data is processed to obtain the indicator results of funnel analysis, including: processing the target data based on a custom aggregation function to obtain multiple first key-value pairs with user identifiers as keys and user operation behavior data and operation time as values; merging multiple first key-value pairs with the same user identifier to obtain multiple merged second key-value pairs; and processing the second key-value pairs based on a custom aggregation function to obtain the indicator results of funnel analysis.
[0012] According to any of the foregoing embodiments of the first aspect of this application, the funnel analysis index results are obtained by performing calculations on the second key-value pairs based on a custom aggregation function, including: sorting the user operation behavior data in the second key-value pairs in order of operation time from earliest to latest to obtain a user operation behavior data sequence; filtering out target user operation behavior data that are consistent with the operation behavior data sequence corresponding to the task node from the user operation behavior data sequence; counting the number of target user operation behavior data corresponding to each task node; and calculating the funnel analysis index results based on the number of data.
[0013] Secondly, embodiments of this application provide a data processing apparatus, comprising: a first display module for displaying a task configuration interface for a funnel analysis task; a first determination module for determining configuration element information of a target funnel analysis task to be processed in response to a user's configuration operation on the task configuration interface; the configuration element information includes at least target data source configuration information and funnel analysis indicators corresponding to the target funnel analysis task to be processed; a first generation module for generating a record of the target funnel analysis task to be processed based on the configuration element information; a parsing module for parsing the configuration element information in the target funnel analysis task record when the record is scanned; a second generation module for generating data processing rules at each task node of the target funnel analysis task based on the parsed configuration element information; an extraction module for extracting target data from the target data source corresponding to the target data source configuration information according to the data processing rules; and a calculation module for performing calculations on the target data to obtain the indicator results of the funnel analysis indicators.
[0014] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the data processing method provided in the first aspect.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method provided in the first aspect.
[0016] The data processing method, apparatus, device, and computer-readable storage medium of this application embodiment can generate a target funnel analysis task record to be processed based on the user's configuration operation on the task configuration interface; generate data processing rules at each task node of the target funnel analysis task by parsing the configuration element information in the target funnel analysis task record; extract target data according to the data processing rules, perform calculation processing on the target data, and obtain the indicator results of the funnel analysis indicators. Therefore, this application embodiment can generate a new funnel analysis task by configuring the elements of the funnel analysis task, and can analyze the configured funnel analysis indicators to obtain the corresponding indicator results. Thus, this application embodiment avoids the process of re-customizing and developing a new funnel analysis program to generate a new funnel analysis task, reduces the development work of the funnel analysis program, thereby reducing the development cycle of the funnel analysis program, reducing operation and maintenance costs, and improving the efficiency of data analysis. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application;
[0019] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of this application;
[0020] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0022] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0023] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0025] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0026] Various modifications and variations can be made to this application without departing from its spirit or scope, which will be apparent to those skilled in the art. Therefore, this application is intended to cover modifications and variations falling within the scope of the corresponding claims (the claimed technical solutions) and their equivalents. It should be noted that the embodiments provided in this application can be combined with each other without contradiction.
[0027] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.
[0028] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the prior art:
[0029] As mentioned above, the inventors of this application have discovered that, under the existing development model, a funnel analysis program is only suitable for one funnel analysis requirement. With the increase in the number of users and products, the amount of data requiring funnel analysis grows exponentially. Simultaneously, the number of funnel analysis metrics to be calculated is constantly increasing, and the rules involved in these metrics are continuously being adjusted. If a new funnel analysis program must be custom-developed for each new requirement, it will result in two problems: firstly, the development cycle for funnel analysis programs is long and the maintenance costs are high; secondly, customized funnel analysis programs cannot support multi-threaded funnel analysis task processing, leading to low data analysis efficiency. Therefore, the existing technology is no longer suitable for current business needs.
[0030] To address the problems of the prior art, embodiments of this application provide a data processing method, apparatus, device, and computer-readable storage medium.
[0031] The data processing method provided in the embodiments of this application will be described below.
[0032] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 As shown, the data processing method may include the following steps S101 to S107.
[0033] S101. Displays the task configuration interface for the funnel analysis task.
[0034] S102. In response to the user's configuration operation on the task configuration interface, determine the configuration element information of the target funnel analysis task to be processed.
[0035] S103. Generate a record of the target funnel analysis task to be processed based on the configuration element information;
[0036] S104. If the target funnel analysis task record is scanned, parse the configuration element information in the target funnel analysis task record.
[0037] S105. Based on the parsed configuration element information, generate data processing rules for each task node of the target funnel analysis task.
[0038] S106. Extract target data from the target data source corresponding to the target data source configuration information according to the data processing rules.
[0039] S107. Calculate and process the target data to obtain the index results of the funnel analysis.
[0040] The specific implementation methods of the above steps will be described in detail below.
[0041] The data processing method of this application embodiment can generate a record of target funnel analysis tasks to be processed based on the user's configuration operations on the task configuration interface; generate data processing rules for each task node of the target funnel analysis task by parsing the configuration element information in the target funnel analysis task record; extract target data according to the data processing rules, perform calculations on the target data, and obtain the indicator results of the funnel analysis indicators. Therefore, this application embodiment can generate new funnel analysis tasks by configuring the elements of the funnel analysis task, and can analyze the configured funnel analysis indicators to obtain the corresponding indicator results. Thus, this application embodiment avoids the process of re-customizing and developing a new funnel analysis program to generate new funnel analysis tasks, reduces the development work of the funnel analysis program, thereby reducing the development cycle of the funnel analysis program, reducing operation and maintenance costs, and improving the efficiency of data analysis.
[0042] The specific implementation methods for each of the above steps are described below.
[0043] In S101, the task configuration interface for funnel analysis tasks can include configuration element options and funnel analysis index options. Users can select the configuration elements to be configured and the funnel analysis indexes to be analyzed on the task configuration interface according to their own funnel analysis needs.
[0044] For example, configuration elements may include the data source corresponding to the funnel analysis task, the funnel data filtering conditions corresponding to the data source, the analysis cycle of the funnel analysis task, the task processing status, the task name, etc., and funnel analysis metrics may include the number of users, the number of user clicks, the number of user page views, the user conversion rate, and the churn rate at each task node of the funnel analysis task, etc.
[0045] In S102, in response to the user's configuration operations on the task configuration interface regarding configuration element options and funnel analysis indicator options, the configuration element information of the target funnel analysis task to be processed is determined. The configuration element information includes at least the target data source configuration information and funnel analysis indicators corresponding to the target funnel analysis task to be processed.
[0046] As one implementation of S102, S102 may specifically include: in response to the user's operation on the data source configuration options, determining the target data source configuration information; searching for the target data source corresponding to the target data source configuration information from the data source configuration mapping table; determining the funnel data filtering conditions corresponding to the target data source from the data source and funnel data filtering condition configuration mapping table; and in response to the user's operation of selecting the target funnel data filtering conditions from the funnel data filtering conditions corresponding to the target data source, determining the target funnel data filtering conditions for the target funnel analysis task to be processed.
[0047] The data source configuration mapping table pre-stores data collected from different data sources through event tracking, as well as the parameter information of the data sources. Event tracking is used to capture user actions or time. The data collected by the data source through event tracking includes at least the user identifier, user action data, and action time. The parameter information of the data source can include the name of the data source. Whenever a new data source is generated, the parameter information of that data source and the data collected by that data source through event tracking are added to the data source configuration mapping table, thereby achieving standardized operation of the data source.
[0048] When a user interacts with the data source configuration options on the task configuration interface, the system responds to the user's action by determining the target data source configuration information for the target funnel analysis task to be processed. Based on the target data source configuration information, the system searches the data source configuration mapping table for the target data source corresponding to the target data source configuration information. For example, when the user selects the "Mobile Banking Tracking Data Source" data source configuration option on the task configuration interface, the system determines that the target data source name for the target funnel analysis task to be processed is "Mobile Banking Tracking Data Source," and searches the data source configuration mapping table for a data source named "Mobile Banking Tracking Data Source."
[0049] The data source and funnel data filtering condition configuration mapping table pre-stores the funnel data filtering conditions corresponding to each data source. Based on the target data source, the funnel data filtering conditions corresponding to the target data source are retrieved from the data source and funnel data filtering condition configuration mapping table. For example, when a user selects the "Mobile Banking Tracking Data Source" data source configuration option on the task configuration interface, the funnel data filtering conditions corresponding to the mobile banking tracking data source are retrieved from the data source configuration mapping table, such as: "Mobile Banking Event Name," "Mobile Banking Software Development Kit (SDK) Type," and "Mobile Banking Operating System (OS) Type," etc.
[0050] At this point, users can continue to configure the data source options on the task configuration interface. Responding to the user's selection of target funnel data filtering conditions from the funnel data filtering criteria corresponding to the target data source, the target funnel data filtering conditions for the target funnel analysis task to be processed are determined. For example, on the task configuration interface, users can select "Mobile Banking SDK Type" as the target funnel data filtering condition for the target funnel analysis task from funnel data filtering conditions such as "Mobile Banking Event Name," "Mobile Banking SDK Type," and "Mobile Banking OS Type."
[0051] In some embodiments, since the funnel data filtering conditions may specifically include the numerical values corresponding to the filtering conditions, a mapping table between the funnel data filtering conditions and the filtering condition values can be pre-established based on the numerical values corresponding to each funnel data filtering condition. The correspondence between the funnel data filtering conditions and the filtering condition values in the mapping table can be presented in the form of an operational expression. The operator expression can be, for example, greater than, less than, including, not including, empty, and not empty. The left side of the operator expression can be a specific funnel data filtering condition, such as a mobile banking SDK type, and the right side of the operator expression can be a specific numerical value, such as 5.8, 6.0, 8.0, etc. The determination of this numerical value can support automatic matching from a set of numerical values, or it can support user selection from a set of numerical values or user-defined filling of the numerical value. This application embodiment does not limit this.
[0052] In this way, after the user determines the target funnel data filtering conditions on the task configuration interface, and then operates on the values corresponding to the target funnel data filtering conditions on the task configuration interface, the system responds to the user's operation by determining the corresponding values for the target funnel data filtering conditions from the mapping table of funnel data filtering conditions and filtering condition values. For example, when the user selects the funnel data filtering condition "Mobile Banking SDK Type" on the task configuration interface, they can find a set of values containing a series of values such as 5.8 and 6.0 in the mapping table of funnel data filtering conditions and filtering condition values, and then select the corresponding value from it, or customize the value.
[0053] As another implementation of the data processing method of this application, after determining the target funnel data filtering conditions for the target funnel analysis task to be processed, the data processing method further includes: receiving the user's target operation on the target funnel data filtering conditions, and updating the target funnel data filtering conditions for the target funnel analysis task to be processed.
[0054] For example, a funnel analysis task can have multiple funnel data filtering conditions. This application embodiment supports configuration operations such as adding, deleting, and modifying funnel data filtering conditions. When a user needs to change the target funnel data filtering conditions for a target funnel analysis task, they can continue to operate on the target funnel data filtering conditions on the task configuration interface. This application embodiment can update the target funnel data filtering conditions in real time in response to the user's target operation on the target funnel data filtering conditions.
[0055] In S103, after determining the configuration element information of the target funnel analysis task to be processed, a target funnel analysis task record to be processed is generated according to the configuration element information. The target funnel analysis task record is saved in a preset format, such as JSON format, according to certain rules and stored in the funnel analysis task record. At the same time, the task processing status identifier of the target funnel analysis task is marked as the first identifier, such as "pending processing". The first identifier is used to indicate that the target funnel analysis task has not been processed.
[0056] In S104, the funnel analysis program is a program that periodically scans the progress of funnel analysis tasks. When the server scans a target funnel analysis task with a task processing status identifier of the first identifier, it parses the configuration element information in the target funnel analysis task record according to a preset format, such as JSON format, and updates the task processing status identifier of the target funnel analysis task from the first identifier to a second identifier, such as "processing". The second identifier is used to indicate that the target funnel analysis task is being processed. In this embodiment of the application, the funnel analysis program supports multi-process processing of funnel analysis tasks, and can scan and parse multiple funnel analysis tasks simultaneously to improve the efficiency of data analysis.
[0057] In S105, based on the parsed configuration element information, data processing rules for each task node of the target funnel analysis task are generated in the server's memory. For example, the data processing rules at least include target funnel data filtering conditions for each task node of the target funnel analysis task, where a task node can be understood as the node corresponding to each task stage the funnel needs to pass through in the funnel analysis task.
[0058] In S106, based on the target funnel data filtering conditions at each task node of the target funnel analysis task, the corresponding target data is extracted from the target data source. The target data includes at least the user identifiers, user action behavior data, and operation times of different users performing operations at different task nodes.
[0059] In S107, the extracted target data is processed to obtain the funnel analysis indicators. For example, the target data can be extracted and processed based on a Hadoop software framework capable of distributed processing of large amounts of data, and a Hive data warehouse tool built on the Hadoop framework.
[0060] As one implementation of S107, S107 may specifically include: processing the target data based on a custom aggregation function to obtain multiple first key-value pairs with user identifier as the key and user operation behavior data and operation time as the value; merging multiple first key-value pairs with the same user identifier to obtain multiple merged second key-value pairs; and calculating the second key-value pairs based on the custom aggregation function to obtain the indicator results of the funnel analysis indicators.
[0061] For example, the target data is processed based on a custom aggregation function, using the user identifier as the key and the user's action behavior data and action time as the value, and the target data is processed into multiple first key-value pairs. For example, multiple data entries such as (user a, behavior x, time 1), (user a, behavior y, time 4), (user a, behavior z, time 2) are converted into multiple first key-value pairs such as [key: a, value: (behavior x, time 1)], [key: a, value: (behavior y, time 4)], and [key: a, value: (behavior z, time 2)].
[0062] Since the embodiments of this application can perform data processing based on a distributed server cluster, it is necessary to merge data belonging to the same user processed by different servers in different server clusters. For example, the data belonging to the same user processed by the same server can be merged first based on an aggregation function, that is, multiple first key-value pairs with the same user identifier processed by the same server can be merged; then, the merged key-value pairs generated by different servers can be merged again based on the aggregation function, until all first key-value pairs with the same user identifier are merged into one key-value pair, resulting in multiple merged second key-value pairs.
[0063] As an example, if there are three servers in a distributed server cluster processing user identifier 'a', then firstly, the first key-value pairs processed by the first server [key: a, value: (behavior x, time 1)], [key: a, value: (behavior y, time 4)], [key: a, value: (behavior z, time 2)] are merged into the key-value pair [key: a, value: (behavior sequence x, y, z, time sequence 1, 4, 2)]. Secondly, the first key-value pairs processed by the second server [key: a, value: (behavior j, time 3)], [key: a, value: (behavior k, time 5)], [key: a, value: (behavior l, time 6)] are merged into the key-value pair [key: a, value: (behavior sequence j, k, l, time sequence 3, 5, 6)]. If the third server only processes one first key-value pair [key: a, value: (behavior t, time 8)], then no further merging is performed.
[0064] The key-value pair generated by the first server [key: a, value: (behavior sequence x, y, z, time series 1, 4, 2)], the key-value pair generated by the second server [key: a, value: (behavior sequence j, k, l, time series 3, 5, 6)], and the key-value pair processed by the third server [key: a, value: (behavior t, time 8)] are then merged again to obtain the second key-value pair [key: a, value: (behavior sequence x, y, z, j, k, l, t, time series 1, 4, 2, 3, 5, 6, 8)] corresponding to the user with user identifier a.
[0065] At this point, the second key-value pairs corresponding to multiple users are calculated and processed based on a custom aggregation function to obtain the funnel analysis metrics. It should be noted that the above example is for illustrative purposes only and is not intended to limit this application.
[0066] As one implementation of S107, the second key-value pair is processed based on a custom aggregation function to obtain the funnel analysis index results. Specifically, this may include: sorting the user operation behavior data in the second key-value pair in ascending order of operation time to obtain a user operation behavior data sequence; filtering out target user operation behavior data that matches the operation behavior data sequence corresponding to the task node from the user operation behavior data sequence; counting the number of target user operation behavior data corresponding to each task node; and calculating the funnel analysis index results based on the count.
[0067] For example, continuing with the above example, taking the second key-value pair corresponding to user 'a' as an example, the user operation behavior data in the second key-value pair is sorted according to the operation time from earliest to latest, resulting in a user operation behavior data sequence, namely the behavior sequence x, z, j, y, k, l, t. If the operation behavior data sequence corresponding to each task node of the preset target funnel analysis task is x, j, k, u, then it can be determined that only behaviors x, j, and k in the user operation behavior data sequence of user 'a' are consistent with the operation behavior data sequence corresponding to the task node. That is, user operation behavior data x corresponds to the first task node, user operation behavior data j corresponds to the second task node, and user operation behavior data k corresponds to the third task node. In other words, user 'a' is lost at the fourth task node. Therefore, it is only necessary to count the user operation behavior data of this user at the first, second, and third task nodes. According to the above embodiment, the number of target user operation behavior data corresponding to each task node can be counted, and then the funnel analysis index result can be calculated based on the counted number.
[0068] As another implementation of the data processing method in this application, such as Figure 2As shown, after step S107, the data processing method may further include the following step S201.
[0069] S201. Store the indicator results in the result index of the search server.
[0070] For example, the search server could be ElasticSearch, a Lucene-based search server capable of providing a distributed, multi-user full-text search engine based on a RESTful web interface. Since the results of funnel analysis metrics from different funnel analysis tasks can all be stored in the same search server's result index, the search server's result index needs to be standardized before storing the metric results. Statistical dimensions for the funnel analysis metrics are pre-created in the result index, such as the statistical perspective on user gender during funnel analysis metric statistics; this embodiment of the application does not limit this approach.
[0071] During the process of storing the indicator results to the result index, the funnel analysis program sets a corresponding result identifier for the indicator results of each funnel analysis indicator for each funnel analysis task. Optionally, the task identifier of the funnel analysis task can be set as the result identifier of the indicator result calculated by that funnel analysis task to distinguish the indicator results of different funnel analysis tasks and ensure the uniqueness of the indicator results of each funnel analysis task. Therefore, this embodiment of the application can also support the simultaneous storage of indicator results of multiple funnel analysis tasks in the result index of the same search server. For example, the task identifier can be the task name, funnel name, or funnel ID of the funnel analysis task, or other identifiers that can uniquely identify the funnel analysis task.
[0072] As another implementation of the data processing method in this application, such as Figure 3 As shown, after step S201, the data processing method may further include the following steps S301 to S304.
[0073] S301, Display the results of the funnel analysis task.
[0074] The results display interface for funnel analysis tasks is used to show data analysts the processing progress and key performance indicators (KPIs) of each task. The interface includes a task identifier and processing status for each task, such as "Pending," "Processing," "Completed," or "Failed." Through this interface, data analysts can view the real-time processing status of each funnel analysis task, as well as the KPIs for completed tasks, and make timely adjustments to tasks that fail.
[0075] S302. In response to the user's operation of selecting a target task identifier from the task identifiers, search for the target indicator result corresponding to the target task identifier in the result index.
[0076] Users can select the task identifier of the funnel analysis task they want to view on the results display interface, i.e., the target task identifier. In response to the user's operation, the browser on the front end will send an asynchronous query request to the data interface for querying indicator results. The data interface will then query the target indicator results with the target task identifier as the result identifier from the result index of the search server.
[0077] S303. Encapsulate the target indicator results according to the preset format.
[0078] For example, the target metric results can be encapsulated according to a preset format, such as JSON, and the encapsulated data can be sent to the front end.
[0079] S304, Display the target performance results after packaging.
[0080] For example, the front end can use different data visualization techniques to convert the target indicator results in JSON format into visual graphs or tables and display them, so that users can easily view the target indicator results on the results display interface. For example, information such as the number of users, the number of user clicks, the number of user views, the user conversion rate, and the churn rate at each task node of the funnel analysis task.
[0081] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of a data processing apparatus. Please refer to the following embodiments.
[0082] First see Figure 4 The data processing apparatus 400 provided in this application embodiment includes the following modules:
[0083] The first display module 401 is used to display the task configuration interface for the funnel analysis task.
[0084] The first determining module 402 is used to determine the configuration element information of the target funnel analysis task to be processed in response to the user's configuration operation on the task configuration interface.
[0085] The first generation module 403 is used to generate a record of the target funnel analysis task to be processed based on the configuration element information.
[0086] Parsing module 404 is used to parse the configuration element information in the target funnel analysis task record when the target funnel analysis task record is scanned;
[0087] The second generation module 405 is used to generate data processing rules for each task node of the target funnel analysis task based on the parsed configuration element information.
[0088] Extraction module 406 is used to extract target data from the target data source corresponding to the target data source configuration information according to data processing rules;
[0089] The calculation module 407 is used to perform calculations on the target data to obtain the index results of the funnel analysis indicators.
[0090] The data processing apparatus of this application embodiment can generate a record of target funnel analysis tasks to be processed based on the user's configuration operations on the task configuration interface; generate data processing rules for each task node of the target funnel analysis task by parsing the configuration element information in the target funnel analysis task record; extract target data according to the data processing rules, perform calculations on the target data, and obtain the indicator results of the funnel analysis indicators. Therefore, this application embodiment can generate new funnel analysis tasks by configuring the elements of the funnel analysis task, and can analyze the configured funnel analysis indicators to obtain the corresponding indicator results. Thus, this application embodiment avoids the process of re-customizing and developing a new funnel analysis program to generate a new funnel analysis task, reduces the development work of the funnel analysis program, thereby reducing the development cycle of the funnel analysis program, reducing operation and maintenance costs, and improving the efficiency of data analysis.
[0091] In some embodiments, the first determining module 402 is specifically configured to, in response to a user's operation on the data source configuration options, determine target data source configuration information; search for the target data source corresponding to the target data source configuration information in the data source configuration mapping table; determine the funnel data filtering conditions corresponding to the target data source in the data source and funnel data filtering condition configuration mapping table; and, in response to a user's operation of selecting target funnel data filtering conditions from the funnel data filtering conditions corresponding to the target data source, determine the target funnel data filtering conditions for the target funnel analysis task to be processed.
[0092] In some embodiments, the data processing apparatus 400 may further include a first update module, configured to receive a user's target operation on the target funnel data filtering conditions and update the target funnel data filtering conditions of the target funnel analysis task to be processed; the target operation includes any one of the user's add, delete, or modify operations on the target funnel data filtering conditions.
[0093] In some embodiments, the data processing apparatus 400 may further include a second update module, used to update the task processing status identifier from a first identifier to a second identifier during the process of parsing the configuration element information in the target funnel analysis task record. The first identifier is used to indicate that the target funnel analysis task has not been processed, and the second identifier is used to indicate that the target funnel analysis task is being processed.
[0094] In some embodiments, the data processing apparatus 400 may further include a storage module for storing the indicator results in the result index of the search server; during the process of storing the indicator results in the result index, the task identifier is set as the result identifier of the indicator results.
[0095] In some embodiments, the data processing device 400 may further include a second display module for displaying a result display interface for a funnel analysis task, the result display interface including a task identifier for the funnel analysis task; responding to a user's operation of selecting a target task identifier from the task identifier, searching for the target indicator result corresponding to the target task identifier in the result index; encapsulating the target indicator result according to a preset format; and displaying the encapsulated target indicator result.
[0096] In some embodiments, the calculation module 407 is specifically used to perform data processing on the target data based on a custom aggregation function to obtain multiple first key-value pairs with user identifier as the key and user operation behavior data and operation time as the value; merge multiple first key-value pairs with the same user identifier to obtain multiple merged second key-value pairs; and perform calculation processing on the second key-value pairs based on the custom aggregation function to obtain the indicator results of the funnel analysis index.
[0097] In some embodiments, the calculation module 407 described above can also be used to sort the user operation behavior data in the second key-value pair in order of operation time from earliest to latest to obtain a user operation behavior data sequence; filter out target user operation behavior data that are consistent with the operation behavior data sequence corresponding to the task node from the user operation behavior data sequence; count the number of target user operation behavior data corresponding to each task node; and calculate the indicator results of the funnel analysis index based on the number of data.
[0098] Figure 4 Each module in the illustrated device has the ability to implement Figure 1 The functions of each step in the process and their corresponding technical effects are described in detail here for the sake of brevity.
[0099] Based on the data processing method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Please refer to the following embodiments.
[0100] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0101] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.
[0102] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0103] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 502 may include removable or non-removable (or fixed) media, or memory 502 may be non-volatile solid-state memory. Memory 502 may be internal or external to the integrated gateway disaster recovery device.
[0104] In one example, memory 502 may be read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.
[0105] Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.
[0106] The processor 501 reads and executes computer program instructions stored in the memory 502 to achieve... Figure 1 The method / steps S101 to S107 in the illustrated embodiment are achieved. Figure 1The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.
[0107] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.
[0108] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0109] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.
[0110] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, random access memory, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, and hard disks.
[0111] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0112] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0113] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0114] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0115] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A data processing method, characterized in that, The method includes: Displays the task configuration interface for the funnel analysis task; In response to the user's configuration operation on the task configuration interface, the configuration element information of the target funnel analysis task to be processed is determined; the configuration element information includes at least the target data source configuration information and funnel analysis indicators corresponding to the target funnel analysis task to be processed. Based on the configuration element information, a record of target funnel analysis tasks to be processed is generated; wherein, the target funnel analysis task record includes a task processing status identifier of the target funnel analysis task; The funnel analysis program uses a multi-process timed scan to retrieve the target funnel analysis task records to be processed. If the target funnel analysis task record is detected, the configuration element information in the target funnel analysis task record is parsed. During the process of parsing the configuration element information in the target funnel analysis task record, the task processing status identifier is updated from a first identifier to a second identifier. The first identifier is used to indicate that the target funnel analysis task has not been processed, and the second identifier is used to indicate that the target funnel analysis task is being processed. Based on the parsed configuration element information, generate data processing rules for each task node of the target funnel analysis task; Target data is extracted from the target data source corresponding to the target data source configuration information according to the data processing rules; wherein, the target data is stored in a distributed server cluster; The target data is processed to obtain the index results of the funnel analysis. The target data includes user identifiers, user operation behavior data, and operation times of different users performing operations at different task nodes. The calculation and processing of the target data to obtain the funnel analysis indicators includes: The target data is processed based on a custom aggregation function to obtain multiple first key-value pairs with user identifier as the key and user operation behavior data and operation time as the value. Multiple first key-value pairs with the same user identifier are merged to obtain multiple merged second key-value pairs; The second key-value pair is calculated and processed using a custom aggregation function to obtain the index results of the funnel analysis.
2. The method according to claim 1, characterized in that, The task configuration interface includes data source configuration options. The configuration elements include the data source and funnel data filtering conditions for the data source. The configuration element information for determining the target funnel analysis task to be processed, in response to the user's configuration operation on the task configuration interface, includes: In response to the user's operation on the data source configuration options, determine the target data source configuration information; Find the target data source corresponding to the target data source configuration information in the data source configuration mapping table; Determine the funnel data filtering conditions corresponding to the target data source from the data source and funnel data filtering condition configuration mapping table; In response to the user's operation of selecting target funnel data filtering conditions from the funnel data filtering conditions corresponding to the target data source, the target funnel data filtering conditions for the target funnel analysis task to be processed are determined.
3. The method according to claim 2, characterized in that, After determining the target funnel data filtering criteria for the target funnel analysis task to be processed, the method further includes: Receive user's target operation on the target funnel data filtering conditions, and update the target funnel data filtering conditions for the target funnel analysis task to be processed; The target operation includes any one of the user's operations of adding, deleting, or modifying the target funnel data filtering conditions.
4. The method according to claim 1, characterized in that, The target funnel analysis task record also includes a task identifier for the target funnel analysis task. After calculating and processing the target data to obtain the indicator results of the funnel analysis indicators, the method further includes: The results of the aforementioned indicators are stored in the result index of the search server; During the process of storing the indicator results to the result index, the task identifier is set as the result identifier of the indicator results.
5. The method according to claim 4, characterized in that, After storing the indicator results in the result index of the search server, the method further includes: The results display interface for the funnel analysis task is displayed, and the results display interface includes the task identifier of the funnel analysis task. In response to the user's operation of selecting a target task identifier from the task identifiers, the target indicator result corresponding to the target task identifier is retrieved from the result index; The target indicator results are encapsulated according to a preset format; Displays the target metric results after encapsulation.
6. The method according to claim 1, characterized in that, The calculation and processing of the second key-value pair based on the custom aggregation function to obtain the funnel analysis index results include: The user operation behavior data in the second key-value pair is sorted in order of operation time from earliest to latest to obtain a user operation behavior data sequence. Filter out target user operation behavior data that matches the operation behavior data sequence corresponding to the task node from the user operation behavior data sequence; Count the number of target user action data corresponding to each task node; Based on the stated number, calculate the index results of the funnel analysis.
7. A data processing apparatus, characterized in that, The device includes: The first display module is used to display the task configuration interface for the funnel analysis task. The first determining module is used to determine the configuration element information of the target funnel analysis task to be processed in response to the user's configuration operation on the task configuration interface; the configuration element information includes at least the target data source configuration information and funnel analysis indicators corresponding to the target funnel analysis task to be processed. The first generation module is used to generate a record of target funnel analysis tasks to be processed based on the configuration element information; wherein, the target funnel analysis task record includes a task processing status identifier of the target funnel analysis task. The parsing module is used to scan the target funnel analysis task record to be processed at regular intervals based on the funnel analysis program; when the target funnel analysis task record is scanned, the configuration element information in the target funnel analysis task record is parsed. The second update module is used to update the task processing status identifier from a first identifier to a second identifier during the process of parsing the configuration element information in the target funnel analysis task record. The first identifier is used to indicate that the target funnel analysis task has not been processed, and the second identifier is used to indicate that the target funnel analysis task is being processed. The second generation module is used to generate data processing rules for each task node of the target funnel analysis task based on the parsed configuration element information. An extraction module is used to extract target data from the target data source corresponding to the target data source configuration information according to the data processing rules; wherein the target data is stored in a distributed server cluster; The calculation module is used to perform calculations on the target data to obtain the indicator results of the funnel analysis indicators; the target data includes user identifiers, user operation behavior data, and operation times of different users performing operations at different task nodes; The computing module is specifically used for: The target data is processed based on a custom aggregation function to obtain multiple first key-value pairs with user identifier as the key and user operation behavior data and operation time as the values. Multiple first key-value pairs with the same user identifier are merged to obtain multiple merged second key-value pairs; The second key-value pair is calculated and processed using a custom aggregation function to obtain the index results of the funnel analysis.
8. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 6.