Customer service data aggregation method based on ODPS and server

Abstract

The invention discloses an ODPS-based customer service data aggregation method and server, and relates to the field of electric digital data processing, and the method comprises the steps: carrying out the field classification of collected source data, and obtaining a plurality of source data sets; loading the source data sets to an ODPS platform, and performing first aggregation processing on the source data in the plurality of source data sets by using the ODPS platform to obtain a plurality of field wide tables; pulling the field wide table to a DRDS platform, and performing distributed data processing on the field wide table by using the DRDS platform to obtain a plurality of pieces of distributed data; determining a target data feature corresponding to a target report in a report request instruction according to the report request instruction sent by the user terminal; and extracting multiple pieces of target data corresponding to the target data features in the multiple pieces of distributed data, and performing second aggregation processing on the target data to generate a target report. By implementing the method, the data processing efficiency can be improved when the multi-azimuth data display of the customer service staff is carried out.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a customer service data aggregation method and server based on ODPS. Background Art

[0002] With the continuous development of modern enterprises, big data has become an important resource for enterprise decision-making and business optimization. Especially in the field of express customer service, the massive interaction data and feedback information of customer service customers provide a reference for express companies to optimize their business. How to efficiently process and analyze this data to achieve the visual data dashboard display required by customer service is becoming a major challenge for relevant staff.

[0003] In related technologies, distributed data processing and storage technology is an option for big data processing. ODPS (Open Data Processing Service) is a cloud computing-based big data processing platform that provides storage and computing-integrated big data processing services. DRDS (Distributed Relational Database Service) is a distributed database service that supports large-scale data storage and query, meeting the needs of enterprises for real-time data processing.

[0004] However, since data from different business areas are often stored in independent database systems, such as work order information, claims processing data, problem data, etc., when conducting a multi-faceted data display for a customer service staff, it will involve data retrieval and integration from multiple databases. There are also permission restrictions on each node for retrieval and integration, and the data processing efficiency is low. Summary of the invention

[0005] The present application provides a customer service data aggregation method and server based on ODPS, which are used to improve the efficiency of data processing when performing multi-faceted data display of customer service personnel.

[0006] In the first aspect, the present application provides a customer service data aggregation method based on ODPS, which is applied to a server, and the method includes: performing field classification on the collected source data to obtain multiple source data sets; loading the source data sets to the ODPS platform, and using the ODPS platform to perform a first aggregation processing on the source data in the multiple source data sets respectively, to obtain multiple field-wide tables; pulling the field-wide tables to the DRDS platform, and using the DRDS platform to perform distributed data processing on the field-wide tables to obtain multiple distributed data; determining the target data features corresponding to the target report in the report request instruction according to the report request instruction sent by the user terminal; extracting multiple target data corresponding to the target data features from the multiple distributed data, and performing a second aggregation processing on the target data to generate a target report.

[0007] In the above embodiment, the server achieves efficient customer service data aggregation by performing field classification, two-stage aggregation processing and distributed data processing on the source data. The combination of ODPS and DRDS platforms improves data processing efficiency, allowing users to obtain multi-dimensional data analysis results more quickly, that is, when performing multi-faceted data display of customer service personnel, the efficiency of data processing is improved.

[0008] In combination with some embodiments of the first aspect, in some embodiments, after extracting multiple target data corresponding to the target data features from multiple distributed data, performing a second aggregation processing on the target data, and generating a target report, the method also includes: responding to a report query request sent by a user terminal, determining query rules and display conditions corresponding to the report query request; obtaining dimension conditions and indicator conditions in the query rules, and generating a query statement in the corresponding target report; executing the query statement on the DRDS platform to obtain target display data; rendering the target display data as a query result according to the chart type and layout style in the display conditions, and sending the query result to the user terminal.

[0009] In the above embodiment, the server responds to the user's report query request and renders the target display data as the query result, so that the user can quickly obtain the required data report according to actual needs, thereby improving the efficiency and accuracy of data analysis.

[0010] In combination with some embodiments of the first aspect, in some embodiments, according to the chart type and layout style in the display conditions, the target display data is rendered as a query result, and the query result is sent to the user terminal, specifically including: obtaining the historical browsing data of the user terminal, and determining the historical interaction area of ​​the user terminal according to the historical browsing data; based on the chart type and layout style in the display conditions, determining the preset rendering result of the target display data; in the preset rendering result, setting the rendering area corresponding to the historical interaction area as the priority rendering area; rendering the priority display data corresponding to the priority rendering area in the target display data, obtaining the query result, and sending the query result to the user terminal.

[0011] In the above embodiment, the server analyzes the user's historical browsing data and preferentially renders the user's frequently used interaction areas, thereby realizing intelligent data display, reducing the user's waiting time, and optimizing the utilization efficiency of system resources.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the step of classifying the collected source data by field and obtaining multiple source data sets, the method also includes: real-time monitoring of the data update frequency of the business field where the source data is located; and determining the data upload time of the source data based on the data update frequency.

[0013] In the above embodiment, the server can ensure the real-time nature of the data by monitoring the update frequency of the source data in real time and determining the data upload time accordingly, while avoiding unnecessary data transmission and optimizing the use of network resources.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the data upload time of the source data is determined according to the data update frequency, specifically including: determining the preset upload time of the source data according to the data update frequency; adjusting the preset upload time according to the real-time network speed and the data update size of the source data to obtain the data upload time of the source data.

[0015] In the above embodiment, the server optimizes the data synchronization strategy by dynamically adjusting the data upload time based on the real-time network speed and the data update size, and can adapt to different network environments to ensure the stability and efficiency of data transmission.

[0016] In combination with some embodiments of the first aspect, in some embodiments, multiple target data corresponding to the target data characteristics are extracted from multiple distributed data, and the target data are subjected to a second aggregation process to generate a target report, which specifically includes: determining multiple target data corresponding to the target data characteristics from multiple distributed data, and determining a pulling strategy for the target data; the pulling strategy includes full pulling and incremental pulling; when the pulling strategy is incremental pulling, based on the update timestamp of the target data, the target data to be updated after the last synchronization is synchronized to the DRDS platform; the data to be updated is subjected to a second aggregation process to obtain an aggregation result; and a target report is generated based on the historical target data and the aggregation result after the last synchronization.

[0017] In the above embodiment, the server implements an efficient data update mechanism by determining the pull strategy and performing incremental pull, which can reduce the amount of data transmission, improve the data processing speed, and ensure the consistency and integrity of the data.

[0018] In combination with some embodiments of the first aspect, in some embodiments, multiple target data corresponding to the target data characteristics are determined in multiple distributed data, and a target data pulling strategy is determined, specifically including: determining multiple target data corresponding to the target data characteristics in multiple distributed data; determining a time span for pulling multiple target data; calculating the data level of the target data based on the pulling time span; when the data level is less than a preset data threshold, determining the pulling strategy to be a full pull; when the data level is not less than a preset data threshold, determining the computational complexity of the aggregated target data; when the computational complexity is lower than a preset level, determining the pulling strategy to be an incremental pull.

[0019] In the above embodiment, the server selects a full pull or incremental pull strategy by analyzing the characteristics of the target data, the pulling time span and the data level, and can select the optimal data pulling method according to the actual situation, which not only ensures the integrity of the data but also improves the efficiency of data processing.

[0020] In a second aspect, an embodiment of the present application provides a server, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the server to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a server, enables the server to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a server, enable the server to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] It is understandable that the server provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By adopting the method of classifying source data by domain, using the ODPS platform for the first aggregation processing, using the DRDS platform for distributed data processing, and performing the second aggregation processing according to the characteristics of the target data, efficient customer service data aggregation and processing are achieved. This effectively solves the problem of low efficiency and difficulty in meeting real-time query requirements when processing large-scale multi-source heterogeneous data in the existing technology, thereby achieving fast and accurate customer service data analysis and display.

[0025] 2. Since the data update frequency of the business field where the source data is located is monitored in real time and the data upload time of the source data is determined according to the frequency, effective management of data synchronization is achieved. This effectively solves the problem of resource waste caused by infrequent or infrequent data synchronization in the prior art, thereby achieving data real-time performance and optimization of resource utilization.

[0026] 3. By adopting a method to determine the pull strategy of the target data and synchronize based on the update timestamp of the target data when the pull strategy is incremental pull, an efficient data update mechanism is implemented. This effectively solves the problems of resource waste and inefficiency caused by full data pull or data integrity damage caused by incremental pull in the existing technology, thereby realizing rapid data update and optimal use of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the customer service data aggregation method based on ODPS in an embodiment of the present application; Figure 2 It is another flowchart of the customer service data aggregation method based on ODPS in an embodiment of the present application; Figure 3 It is a schematic diagram of a physical device structure of a server in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0030] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0031] STQ Express is a large national express delivery company that handles millions of packages every day, generating massive amounts of customer service data. The company's top management hopes to optimize customer service processes and improve customer satisfaction by analyzing this data. However, the data is scattered across multiple systems, including order systems, customer service hotline records, and delivery tracking systems (claims, problem items). Traditional data processing methods cannot effectively integrate and analyze such large-scale and complex data. The company needs a solution that can aggregate multi-source data, generate reports in real time, and support flexible queries so that management can gain insight into business conditions and make decisions.

[0032] In related technologies, the integration and analysis of multi-source data can be achieved by adopting traditional ETL (extraction, transformation, loading) processes and relational databases. This approach involves regularly extracting data from various source systems, standardizing the data through complex transformation processes, and then loading it into a central data warehouse for analysis and report generation. The following describes the scenarios of using the data aggregation method in related technologies.

[0033] When trying to solve the data integration problem, STQ initially adopted the traditional ETL (extract, transform, load) method. They extracted data from various systems every night and loaded it into a centralized data warehouse after a complex transformation process. However, this method has serious lags and cannot meet the needs of real-time decision-making. For example, when a large number of complaints suddenly occurred in a region, management was unable to learn about the situation and take action. In addition, as the amount of data grew, the processing time each night became longer and longer, and it was often not completed before the next day's business, affecting daily operations. When management needed to query specific data temporarily, it often took the IT department hours or even days to prepare reports, which reduced decision-making efficiency.

[0034] The ODPS-based customer service data aggregation method in the embodiment of this application is adopted. By first performing preliminary aggregation of large-scale data on the ODPS platform and then performing distributed processing and real-time query on the DRDS platform, efficient processing and rapid analysis of massive data are achieved, which not only improves the speed and flexibility of data processing, but also supports real-time or near real-time data analysis needs. The following introduces the scenario in which the ODPS-based customer service data aggregation method in this application is used.

[0035] After implementing this application plan, STQ's data processing capabilities have been significantly improved. The server first classifies data from different sources and uploads them to the ODPS platform, such as processing order data, customer service records, logistics information, etc. separately. The ODPS platform uses its powerful computing power to quickly perform preliminary aggregation of various types of data and generate wide tables in various fields. These wide tables are then transferred to the DRDS platform for further distributed processing. When management needs to view a specific report, the server can quickly extract relevant data from the DRDS platform, perform secondary aggregation, and generate the required report. For example, when there is a surge in customer complaints, the server can generate a comprehensive report containing information such as problem cause analysis, scope of impact, historical comparison, etc. within a few minutes, enabling management to respond quickly and formulate response strategies.

[0036] It can be seen that the two-stage aggregation processing method in the embodiment of the present application can not only realize the rapid processing of large-scale data, but also effectively solve the problems of delayed data update and slow query response in the traditional ETL method, thereby realizing the real-time nature of data and the high efficiency of query.

[0037] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of the customer service data aggregation method based on ODPS in an embodiment of the present application.

[0038] S101. Classify the collected source data by field to obtain multiple source data sets.

[0039] Among them, source data refers to the original data collected from various business systems, which is used to reflect various business information related to customer service. Domain classification refers to the classification of data into different categories based on the business attributes or functional characteristics of the data. The source data set refers to the set of data belonging to the same domain after domain classification. For example, the source data collected by various systems can be divided into different domain sets such as work order information, customer feedback, and problem tracking.

[0040] After receiving the source data, the server will initially organize and classify the data for subsequent processing and analysis. Specifically, the server will first analyze the attributes and characteristics of the source data, such as the source of the data, the content type of the data, the business relevance of the data, etc. Then, the server will classify data with similar attributes or belonging to the same business field together according to the preset classification rules to form multiple source data sets. This process also involves operations such as data cleaning and data standardization to ensure the accuracy and consistency of the classification.

[0041] In some embodiments, the domain classification of source data can be achieved in a variety of ways: Optionally, the server can use a machine learning algorithm for automatic classification. First, the server will train a classification model based on existing labeled data. Then, the new source data is input into the trained data classification model for prediction. Finally, the data is assigned to the corresponding domain set based on the prediction results; Optionally, first, the server can define a series of classification rules based on user settings, such as keyword matching, field value judgment, etc. Then, these rules are applied to each source data for judgment. Finally, the data is classified into the corresponding domain set based on the judgment results. It can be understood that other methods can also be used to achieve domain classification of source data, such as semi-supervised learning, cluster analysis and other methods, which are not limited here.

[0042] It should be noted that the data classification model used above is trained using labeled historical data. The input includes various features of the data (such as data source, content type, keywords, etc.), and the output is the field category to which the data belongs. The training criterion is to minimize the classification error rate while balancing the precision and recall of each category. In actual applications, the model receives new unclassified data as input and outputs the predicted data category for preliminary data classification and organization.

[0043] S102: Load the source data set to the ODPS platform, and use the ODPS platform to perform a first aggregation process on the source data in multiple source data sets to obtain multiple field wide tables.

[0044] The ODPS platform refers to the large-scale data computing service provided by Alibaba Cloud, which is used to process massive data. The first aggregation processing refers to the preliminary aggregation and calculation operations on the data. The domain-wide table refers to a large data table that contains all relevant information in a certain field, including multiple dimensions and indicators.

[0045] After completing the domain classification of the source data, the server will further process and analyze the data. Specifically, the server will first upload the classified source data set to the ODPS platform. This process involves the conversion of data formats and the optimization of data transmission. Then, the server will use the computing power of the ODPS platform to perform the first aggregation processing on each source data set. This processing process may include operations such as data deduplication, data association, and data aggregation. Through these operations, the server integrates the originally scattered data into a structured form based on domain classification, namely, a domain-wide table. Each domain-wide table contains all relevant information in the field, providing a basis for subsequent data analysis.

[0046] S103: Pull the domain wide table to the DRDS platform, and use the DRDS platform to perform distributed data processing on the domain wide table to obtain multiple distributed data.

[0047] The DRDS platform refers to the distributed relational database service provided by Alibaba Cloud, which is used to process real-time queries and transaction processing of massive data. Distributed data refers to data stored on multiple nodes after distributed processing.

[0048] After the server completes the generation of the domain-wide table on the ODPS platform, it will transfer the data to a platform that is more suitable for real-time query and processing. Specifically, the server will first establish a data transmission channel between the ODPS platform and the DRDS platform to ensure the security and efficiency of data transmission. Then, the server will pull the domain-wide table data on the ODPS platform to the DRDS platform. This process involves data format conversion and data sharding strategy formulation. The DRDS platform will perform distributed processing on the received domain-wide table. The processing process may include data sharding, index creation, data compression and other operations. Through these operations, the server converts large-scale domain-wide table data into a distributed data structure that is easier to query and analyze quickly.

[0049] In some embodiments, domain-wide table pulling and distributed data processing can be implemented in a variety of ways: Optionally, the server can use the data import tool of DRDS to pull data. First, create the corresponding table structure in DRDS. Then, use the data import command provided by DRDS to import data in batches from ODPS. Finally, the imported data is automatically sharded and indexed through the distributed execution engine of DRDS; Optionally, the server can use the database middleware to achieve real-time synchronization and processing of data. First, configure the data synchronization task between ODPS and DRDS. Then, set the rules for data conversion and distribution to achieve real-time data flow. Finally, use the distributed transaction processing capabilities of DRDS to perform parallel processing and storage optimization on real-time incoming data. It can be understood that other methods can also be used to achieve data pulling and distributed processing, such as using custom ETL tools or combining message queues to achieve asynchronous transmission and processing of data, which are not limited here.

[0050] S104: Determine, according to the report request instruction sent by the user terminal, a target data feature corresponding to the target report in the report request instruction.

[0051] The user terminal refers to the device used by the user to access the server, such as a computer, mobile phone, etc. The report request instruction refers to the command sent by the user through the terminal to obtain a specific report. The target report refers to the specific data report that the user wants to obtain. The target data feature refers to the attribute set of the data required to determine the target report.

[0052] The server will provide corresponding data reports based on the specific needs of the user. Specifically, when the server receives a report request instruction sent by the user terminal, it will first parse the instruction. The parsing process may include identifying the user identity, determining the report type, extracting query parameters, etc. Then, the server will determine the target data features corresponding to the target report based on the parsing results. These features may include the time range of the data, the business type of the data, the statistical dimension of the data, etc. By determining these features, the server can accurately locate the data set that needs to be extracted from the distributed data, and prepare for subsequent data processing and report generation.

[0053] In some embodiments, the parsing of report request instructions and the determination of target data features can be achieved in a variety of ways: Optionally, the server can use predefined report templates for matching. First, the server maintains a report template library, and each template corresponds to a specific data feature. Then, the user's request instruction is matched with the template. Finally, the target data features are determined based on the matching results; Optionally, the server can use natural language processing technology to parse the user's request. First, use the NLP model to perform semantic analysis on the user's request instruction. Then, extract key information in the instruction, such as time, dimension, indicator, etc. Finally, map the extracted information to the data features defined by the server. It can be understood that other methods can also be used to achieve the parsing of request instructions and the determination of data features, such as using a rule engine for complex conditional judgments, or predicting user intent through a machine learning model, etc., which are not limited here.

[0054] S105: extract multiple target data corresponding to the target data features from the multiple distributed data, perform second aggregation processing on the target data, and generate a target report.

[0055] After determining the characteristics of the target data, the server will extract the corresponding data from the distributed data and further process it to generate the report required by the user. Specifically, the server will first execute a distributed query on the DRDS platform based on the characteristics of the target data to extract qualified target data from multiple data nodes. This process involves SQL query optimization and data location strategy. Then, the server will perform a second aggregation process on the extracted target data. This processing process may include operations such as data merging, indicator calculation, and data pivoting to convert the original data into more representative statistical results. Finally, the server will organize the processed data into the form of a target report based on the predefined report template or user-specified display requirements, which may include operations such as generating charts and formatting data.

[0056] In some embodiments, the extraction and secondary aggregation processing of target data can be implemented in a variety of ways: Optionally, the server can use the distributed SQL engine of DRDS for data processing. First, build a complex SQL query statement based on the characteristics of the target data. Then, use the query optimizer of DRDS to optimize SQL to improve query efficiency. Finally, use the aggregation function and window function of SQL to perform secondary aggregation processing of data; Optionally, the server can use memory computing technology for real-time data processing. First, load the qualified data into memory. Then, use a high-performance memory computing framework to quickly aggregate and calculate the data. Finally, organize the calculation results into the data structure required for the report. It is understandable that other methods can also be used to implement the extraction and aggregation processing of target data, such as using a streaming computing framework for real-time data processing, or using artificial intelligence technology for intelligent data analysis, etc., which are not limited here.

[0057] In the above embodiment, the server realizes the real-time nature of data and the efficiency of query. In practical applications, the server can also dynamically adjust the data upload time according to the data update frequency, select the appropriate pull strategy according to the data characteristics, and perform the specified status update. The following introduces another implementation scenario.

[0058] With the in-depth application of the server, STQ has further optimized the data processing process. They introduced an intelligent data update frequency monitoring mechanism and adopted different update strategies for different types of data. For example, for package status information that changes frequently, the server will synchronize more frequently; while for relatively stable customer information, a lower update frequency is adopted. In addition, the server also implements intelligent report rendering based on user behavior. When management frequently views certain specific indicators, the server will give priority to calculating and displaying these data. The server further enhances the user experience, enabling STQ's management to make decisions more efficiently and improve overall operational efficiency.

[0059] After combining the above scenarios, the following is a more detailed description of the process of the method provided by this implementation. Figure 2 , which is another flow chart of the customer service data aggregation method based on ODPS in an embodiment of the present application.

[0060] S201. Classify the collected source data by field to obtain multiple source data sets.

[0061] Referring to step S101 , the server acquires multiple source data sets.

[0062] S202. Monitor in real time the data update frequency of the business field where the source data is located.

[0063] Business areas refer to different types of business activities or functional areas, such as customer service, order processing, etc. Data update frequency is used to indicate the number of times or speed at which data changes within a specific time period. For example, customer feedback data is updated every minute, while financial statement data is updated once a day.

[0064] When receiving and processing source data, the server will understand the changes in data in each business area in order to optimize the data processing strategy. Specifically, the server will set up monitoring points for each business area and continuously record data changes, including monitoring database insert, update, and delete operations, or checking the number and size of new files in the file system. The server will regularly count these changes (every second, every minute, or every hour) and calculate the data update frequency for each business area. This frequency can change over time, so the server will constantly adjust and update this statistical result.

[0065] In some embodiments, real-time monitoring of the source data update frequency can be achieved in a variety of ways: Optionally, the server can use database triggers to monitor data changes. First, create a trigger on the relevant table of the source database. Then, whenever there is a data change, the trigger will record the time and type of the change. Finally, the server regularly summarizes these records and calculates the update frequency; Optionally, the server can use log analysis methods for monitoring. First, configure the database or application server to output detailed operation logs. Then, use a log collection tool to collect these logs in real time. Finally, by analyzing the timestamps and operation types in the logs, the data update frequency is counted. It is understandable that other methods can also be used to monitor the data update frequency, such as using professional data change capture (CDC) tools, or actively querying data changes through application program interfaces (APIs), etc., which are not limited here.

[0066] S203: Determine the data upload time of the source data according to the data update frequency.

[0067] The data upload time refers to the specific time point or time interval when the source data is transferred from the original server to the data processing platform. For example, customer feedback data with a high update frequency will be uploaded every 5 minutes, while monthly report data with a low update frequency can be uploaded once every morning.

[0068] After monitoring the data update frequency of each business area, the server will formulate a reasonable data upload strategy to ensure the data quality and effective use of server resources. Specifically, the server will calculate the optimal data upload time based on the data update frequency of each business area, combined with factors such as the server's processing power and network bandwidth. For data with a high update frequency, the server can choose a more frequent upload interval to ensure the real-time nature of the data. For data with a low update frequency, the server can choose to upload during a time period with low server load to optimize resource utilization. At the same time, the server can also adopt a more precise upload strategy for critical data based on the importance and urgency of the data.

[0069] In some embodiments, the server determines the preset upload time of the source data according to the data update frequency; adjusts the preset upload time according to the real-time network speed and the data update size of the source data to obtain the data upload time of the source data.

[0070] The data update frequency refers to the number of times or speed at which data changes within a specific time period. The preset upload time refers to the data transmission time point preliminarily determined by the server. The real-time network speed is used to indicate the real-time status of the current network transmission speed. The data update size refers to the amount of new or changed data.

[0071] After monitoring the update frequency of the source data, the server will formulate a reasonable data upload strategy and dynamically adjust it according to the real-time network conditions. Specifically, the server will first calculate a preliminary upload time interval based on the data update frequency. Then, the server will monitor the network conditions in real time and obtain the current network transmission speed. At the same time, the server will also evaluate the amount of data uploaded each time. The server will adjust the preliminary upload time based on the network speed and data volume. For example, if the network speed is slow but the data volume is large, the server can extend the upload interval to avoid network congestion; if the network speed is fast and the data volume is small, the server can shorten the upload interval to improve the real-time nature of the data. Finally, the server will set the final data upload time based on the adjusted results.

[0072] In some embodiments, dynamic adjustment of the source data upload time can be achieved in a variety of ways: Optionally, the server can use an adaptive algorithm. First, establish a data update frequency prediction model based on data update frequency, network speed and data volume. Then, continuously collect real-time network speed and data volume information and input it into the model. Finally, the optimal upload time is calculated through the model, and the upload strategy is dynamically updated; Optionally, the server can adopt a rule-based method. First, set a series of adjustment rules, such as "If the network speed drops by 30%, extend the upload interval by 1.5 times." Then, monitor the changes in network speed and data volume in real time. Finally, automatically adjust the upload time according to the preset rules. It is understandable that other methods can also be used to achieve dynamic adjustment of the upload time, such as using a machine learning algorithm to predict the optimal upload time, or combining historical upload performance data for optimization, etc., which are not limited here.

[0073] It should be noted that the data update frequency prediction model is trained using historical data update records. The input includes time series data, business type, historical update mode, etc., and the output is the predicted data update frequency in the future. The training standard is to minimize the prediction error, such as the root mean square error. During the application process, real-time data update information is received as input to predict the data update frequency in the short term in the future, which is used to dynamically adjust the data upload and processing strategy.

[0074] S204, loading the source data set into the ODPS platform, and using the ODPS platform to perform a first aggregation process on the source data in the multiple source data sets respectively, to obtain multiple field wide tables.

[0075] Referring to step S102, the server generates a plurality of domain-wide tables.

[0076] S205: Pull the domain wide table to the DRDS platform, and use the DRDS platform to perform distributed data processing on the domain wide table to obtain multiple distributed data.

[0077] Referring to step S103 , the server determines a plurality of distributed data.

[0078] S206: Determine, according to the report request instruction sent by the user terminal, a target data feature corresponding to the target report in the report request instruction.

[0079] Referring to step S104, the server determines the target data characteristics.

[0080] S207: Determine multiple target data corresponding to the target data characteristics in the multiple distributed data, and determine a strategy for pulling the target data.

[0081] The pull strategy is used to guide how to obtain data from the distributed system, including full pull and incremental pull. For example, for sales data that needs to be updated in real time, the incremental pull strategy can be used, while for monthly summary reports, the full pull strategy can be used.

[0082] After receiving the user's report request, the server will accurately locate and extract the required data from the distributed data, and at the same time formulate an appropriate data pulling strategy to improve efficiency. Specifically, the server will first match and filter the distributed data according to the characteristics of the target data to determine the target data set that meets the conditions. This process involves complex distributed query optimization and data positioning technology. Then, the server will analyze the characteristics of the target data, such as data volume, update frequency, consistency requirements, etc., and determine the most suitable pulling strategy based on the current load and network conditions of the server. For situations where the amount of data is small or strong consistency needs to be guaranteed, you can choose to pull the full amount; for situations where the amount of data is large and a certain delay is allowed, you can choose to pull incrementally to improve efficiency.

[0083] In some embodiments, the determination of target data and the selection of pull strategies can be achieved in a variety of ways: Optionally, the server can use distributed indexing technology to quickly locate target data. First, a multidimensional index structure is established in a distributed system. Then, a quick search is performed in the index based on the characteristics of the target data. Finally, the location and range of the target data are determined based on the search results; Optionally, the server can use a heuristic algorithm to determine the pull strategy. First, a series of factors that affect the pull strategy are set, such as data volume, update frequency, network bandwidth, etc. Then, a weight is assigned to each factor to build a decision model. Finally, the optimal pull strategy is calculated based on the actual situation. It is understandable that other methods can also be used to determine the target data and pull strategy, such as using a pull strategy decision model to predict the best pull method, or dynamically adjusting the strategy through historical pull performance data, etc., which are not limited here.

[0084] It should be noted that the pull strategy decision model uses the performance data of historical pull operations for training. The input includes factors such as data volume, time span, and computational complexity, and the output is the optimal pull strategy (full or incremental). The training standard is to maximize data processing efficiency and system resource utilization. When used, it receives the characteristics of the current data request as input, outputs the recommended pull strategy, and guides the server on how to efficiently obtain the required data.

[0085] In some embodiments, the server determines multiple target data corresponding to the target data characteristics in multiple distributed data; determines the time span for pulling the multiple target data; calculates the data level of the target data based on the pulling time span; when the data level is less than a preset data threshold, determines the pulling strategy as full pull; when the data level is not less than the preset data threshold, determines the computational complexity of the aggregated target data; when the computational complexity is lower than the preset level, determines the pulling strategy as incremental pull.

[0086] The pull time span refers to the time range of data retrieval. The data magnitude is used to indicate the size of the data. The computational complexity refers to the computing resources and time required to perform data aggregation operations. For example, the server will first determine that it needs sales data for the past month and calculate that the data volume is about 10GB, which is less than the threshold of 20GB, so it chooses to pull the full amount.

[0087] After receiving the data request, the server will determine the most suitable data pulling strategy to balance data integrity and system performance. Specifically, the server will first locate the relevant data in the distributed storage according to the characteristics of the target data. Then, the server will analyze the time distribution of this data and determine the time range that needs to be pulled. The server will estimate the amount of data within this time range, involving metadata analysis or sampling calculation. The server will compare the estimated amount of data with the preset threshold. If the amount of data is small, the server will choose to pull the full amount to ensure the integrity of the data. If the amount of data is large, the server will further evaluate the complexity of data aggregation, and the factors involved may include the type of aggregation operation, the distribution characteristics of the data, etc. If the aggregation operation is relatively simple, the server will choose incremental pulling to improve efficiency. Through this series of judgments, the server finally determines the most appropriate pulling strategy.

[0088] S208. When the pull strategy is incremental pull, based on the update timestamp of the target data, the target data to be updated after the last synchronization is synchronized to the DRDS platform.

[0089] Among them, incremental pull means only obtaining data that has changed since the last synchronization, rather than all data. The update timestamp indicates the time when the data was last modified. Data to be updated refers to data that has been added or modified since the last synchronization. For example, if the last synchronization time is January 1, 2023, the server will only pull data updated after January 1.

[0090] After the server determines to adopt the incremental pull strategy, it will accurately identify and extract the data that has changed since the last synchronization, and transmit this data to the DRDS platform. Specifically, the server will first obtain the time point of the last synchronization, and then use this time point as a benchmark to find all data records in the source data whose update timestamp is greater than this benchmark time. This process involves distributed query optimization to improve the retrieval efficiency in large-scale data. The server will perform necessary conversion and cleaning on the data to be updated to ensure that the data format is compatible with the DRDS platform. Finally, the server will synchronize the data to be updated to the DRDS platform through batch insertion or real-time streaming. During the synchronization process, the server will also handle possible network anomalies, data conflicts and other problems to ensure data integrity and consistency.

[0091] In some embodiments, timestamp-based incremental data synchronization can be achieved in a variety of ways: Optionally, the server can use the change data capture (CDC) function of the database. First, enable the CDC function in the source database to record all data changes. Then, the server periodically checks the CDC log and extracts all change records since the last synchronization. Finally, these change records are applied to the DRDS platform to achieve data synchronization; Optionally, the server can use log parsing methods for incremental synchronization. First, configure the source database to output detailed operation logs. Then, use log parsing tools to analyze these logs in real time and extract data change information. Finally, convert the extracted change information into SQL statements executable by the DRDS platform, and execute these statements on DRDS to complete data synchronization. It is understandable that other methods can also be used to achieve incremental data synchronization, such as using professional data integration tools, or through customized ETL processes and other methods, which are not limited here.

[0092] S209: Perform a second aggregation process on the data to be updated to obtain an aggregation result.

[0093] The data to be updated refers to the data set that is newly synchronized from the source server to the DRDS platform and needs to be processed. The aggregation result refers to the data set obtained after the second aggregation process, which contains statistical information in multiple dimensions. For example, for the newly synchronized sales data, it will be summarized according to dimensions such as region and product category to calculate indicators such as sales volume and profit margin.

[0094] After synchronizing the data to be updated to the DRDS platform, the server will further process the new data to meet the needs of report generation. Specifically, the server will first analyze the structure and content of the data to be updated and determine the aggregation operations that need to be performed, including data grouping, statistical calculations, and correlation analysis. Then, the server will use the distributed computing capabilities of the DRDS platform to execute the predefined aggregation processing logic. This process involves complex SQL queries, custom aggregation functions, or specific data processing algorithms. During the processing, the server will also involve the balance between data consistency and computing efficiency, and can use parallel computing or incremental computing and other technologies to increase processing speed. Finally, the server will organize the processing results into a predefined format to form an aggregated result, which contains statistical data in multiple dimensions and provides a basis for subsequent report generation.

[0095] S210. Generate a target report based on the historical target data and aggregation results after the last synchronization.

[0096] The historical target data refers to the processed data set that existed during the last synchronization.

[0097] After the server completes the aggregation of new data, it will integrate the old and new data and generate the final target report in a predefined format. Specifically, the server will first merge the historical target data with the new aggregation results. This process involves comparing, updating, and supplementing data to ensure that the latest and complete data set is used in the end. Then, the server will further calculate and process the merged data according to the definition requirements of the report, such as calculating the month-on-month growth rate and generating a pivot table. The server will convert the processed data into various visual forms, such as line charts, bar charts, pie charts, etc., according to the preset report template or user-defined layout. In this process, the server will also involve the display effect of the data, and can perform operations such as data scaling and color mapping. Finally, the server will integrate all charts and data into a complete report, and make final format adjustments and beautification.

[0098] In some embodiments, the generation of target reports can be achieved in a variety of ways: Optionally, the server can use a professional report generation tool. First, import historical data and aggregation results into the data source of the reporting tool. Then, use the drag-and-drop interface or scripting language provided by the tool to define the structure and style of the report. Finally, the data is converted into a visual report format through the rendering engine of the tool; Optionally, the server can use a customized report generation framework. First, develop a flexible report template system that supports various data display forms and layouts. Then, write a data processing script to convert historical data and aggregation results into the format required by the template. Finally, the data is rendered into a predefined template through the template engine to generate the final report. It can be understood that other methods can also be used to achieve the generation of target reports, such as using web front-end technology to dynamically generate interactive reports, or combining artificial intelligence technology to automatically generate report interpretations and suggestions, etc., which are not limited here.

[0099] S211. In response to a report query request sent by a user terminal, determine a query rule and a display condition corresponding to the report query request.

[0100] The query rule is a set of conditions and logic that define how to filter and extract the required information from the data. The display condition is used to specify how the report is presented, including chart type, layout style, etc. For example, a user can send a request to view the sales volume of each region in the past 30 days and display it in the form of a bar chart.

[0101] After receiving the user's report query request, the server will parse the request content and determine the corresponding data processing and display methods. Specifically, the server will first parse the received query request and extract key information, such as time range, data dimension, query index, etc. Then, the server will map this information to the predefined query rule set to determine the specific data screening and calculation logic. This process involves complex rule matching and parameter replacement. At the same time, the server will also analyze the display requirements in the request, such as chart type, color scheme, sorting method, etc., and convert these requirements into display conditions that the server can understand. In this process, the server will involve the user's permission level to ensure that the user can only access the data and report types for which he is authorized. Finally, the server will save the determined query rules and display conditions to prepare for subsequent data processing and report generation.

[0102] In some embodiments, the parsing and rule determination of report query requests can be implemented in a variety of ways: Optionally, the server can use predefined query templates. First, maintain a template library containing commonly used query scenarios. Then, match the user's query request with these templates. Finally, based on the matching results, select the most appropriate template, and fill in the variables in the template with the parameters provided by the user to obtain specific query rules and display conditions; Optionally, the server can use natural language processing technology to parse user requests. First, use the NLP model to perform semantic analysis on the user's query request. Then, extract key information in the request, such as time range, data dimension, query indicator, etc. Finally, convert the extracted information into query rules and display conditions defined by the server. It can be understood that other methods can also be used to implement the parsing and rule determination of query requests, such as using a decision tree algorithm to dynamically generate query rules, or using a machine learning model to predict the user's query intention, etc., which are not limited here.

[0103] S212: Obtain dimension conditions and indicator conditions in the query rule, and generate a query statement in the corresponding target report.

[0104] Dimension conditions refer to the criteria used for data grouping and filtering, such as time, region, product category, etc. Indicator conditions refer to the specific values ​​that need to be calculated, such as sales, profit margin, etc. Query statements refer to the structured language used to retrieve specific data from the database, which is an SQL statement. For example, a dimension condition can be "group by month and region", and an indicator condition can be "calculate total sales and average order amount".

[0105] After determining the query rules, the server converts these rules into specific query statements that can be executed in the database. Specifically, the server will first parse the query rules and extract the dimension conditions and indicator conditions. This process involves complex syntax analysis and semantic understanding. Then, the server will combine the preset aggregation rules to determine the data operations that need to be performed, such as grouping, aggregation, sorting, etc. The preset aggregation rules here can contain some complex calculation logic, such as rolling average, year-on-year growth rate, etc. The server will construct the optimal SQL query structure based on the characteristics and performance optimization principles of the DRDS platform. This involves the use of subqueries, optimization of join operations, and the use of indexes. During the construction process, the server will also involve data security and permission control, and some filtering conditions can be added to limit the access scope of the data. Finally, the server will generate a complete SQL query statement to ensure that its syntax is correct and can be executed efficiently.

[0106] In some embodiments, the generation of query statements can be achieved in a variety of ways: Optionally, the server can use a query builder tool. First, the dimension conditions and indicator conditions are input into the query builder. Then, use the graphical interface or API provided by the builder to gradually build the various parts of the query, such as the SELECT clause, FROM clause, WHERE clause, etc. Finally, the optimized SQL query statement is generated through the optimization engine of the builder; Optionally, the server can use a templated approach to generate query statements. First, a series of SQL templates are predefined, each template corresponding to a different query scenario. Then, according to the specific dimension conditions and indicator conditions, a suitable template is selected. Finally, the conditional parameters are filled into the template to generate the final SQL query statement. It is understandable that other methods can also be used to generate query statements, such as using a rule engine to dynamically combine SQL fragments, or automatically generating and optimizing SQL queries through machine learning models, etc., which are not limited here.

[0107] S213. Execute the query statement on the DRDS platform to obtain target display data.

[0108] The target display data refers to the data set that meets the query conditions and is used to generate the final report. For example, executing the SQL statement to query sales data will obtain a result set containing the monthly sales of each region.

[0109] After generating query statements, the server will execute these statements on the DRDS platform and obtain the query results. Specifically, the server will first send the generated SQL query statements to the DRDS platform. The DRDS platform will parse and optimize the received query statements, involving the generation of query plans, the selection of execution paths, etc. Then, the DRDS platform will execute queries in parallel on its distributed architecture, which involves complex operations such as data sharding and data aggregation across nodes. During the query execution process, the server will monitor the progress and resource consumption of the query, and intervene or optimize when necessary. After the execution is completed, the DRDS platform will return the query results to the server. After receiving the results, the server will also perform some post-processing, such as data format conversion, temporary storage, etc., and finally form a target data set that can be used for display.

[0110] S214 . Rendering the target display data as the query result according to the chart type and layout style in the display condition, and sending the query result to the user terminal.

[0111] Among them, display conditions refer to a set of parameters that define how the report is presented, including chart type, color scheme, layout style, etc. Chart type represents the specific form of data visualization, such as bar chart, line chart, pie chart, etc. Layout style refers to the arrangement and organization of report elements. Rendering refers to the process of converting raw data into a visual chart. Query results refer to the final report content after processing and visualization. For example, sales data is rendered into an interactive report containing a bar chart and a data table.

[0112] After obtaining the target display data, the server will convert the data into a visual form that is easy for users to understand and analyze. Specifically, the server will first parse the display conditions to determine the chart type and layout style to be generated. Then, the server will perform necessary conversions and organizations on the target display data according to different chart types, such as data grouping, sorting, calculating percentages, etc. The server will use a visual rendering engine to convert the processed data into various chart forms. This process involves complex graphics drawing algorithms, color mapping, coordinate axis settings, etc. At the same time, the server will also reasonably arrange the positions of various charts and data tables according to the layout style to ensure the beauty and readability of the overall report. During the rendering process, the server will also involve the implementation of interactivity, such as adding data filters, floating prompts and other functions. Finally, the server will package the rendered report, which can include HTML, JavaScript, CSS and other content, and send it to the user's terminal device through the network.

[0113] In some embodiments, data rendering and result sending can be achieved in a variety of ways: Optionally, the server can use a professional data visualization library. First, select a suitable visualization library, such as ECharts, D3.js, etc. Then, convert the target display data into the data format required by the library. Next, use the API provided by the library to create various charts and layouts. Finally, embed the generated visualization content into an HTML page and send it to the user terminal; Optionally, the server can use server-side rendering. First, use the graphics processing library on the server to generate static chart images. Then, combine these images with other report elements into a complete report page. Finally, send the generated HTML page directly to the user terminal to reduce the computing burden on the client. It is understandable that other methods can also be used to achieve data rendering and result sending, such as using WebGL technology to achieve 3D visualization of large-scale data, or optimizing the display performance of large reports through progressive loading technology, etc., which are not limited here.

[0114] In some embodiments, the server obtains historical browsing data of the user terminal, determines the historical interaction area of ​​the user terminal based on the historical browsing data; determines the preset rendering results of the target display data based on the chart type and layout style in the display conditions; in the preset rendering results, sets the rendering area corresponding to the historical interaction area as the priority rendering area; renders the priority display data corresponding to the priority rendering area in the target display data, obtains the query results, and sends the query results to the user terminal.

[0115] Among them, historical browsing data refers to the record of the user's past interactions with the server. The historical interaction area represents the part of the report that the user frequently pays attention to or operates. Display conditions refer to a set of parameters that define how the report is presented. Chart type refers to the specific form of data visualization, such as bar charts, line charts, etc. Layout style is used to represent the arrangement and organization of various elements of the report. Preset rendering results refer to the report visualization effect initially generated by the server. Priority rendering area refers to the part of the report that the server decides to prioritize processing and display. Priority display data refers to the data that needs to be presented in the priority rendering area. For example, the server determines that users often view the regional sales ranking part of the sales report, so it prioritizes processing and displaying this part when rendering.

[0116] When the server generates a report, it will consider the user's usage habits and provide a personalized data display experience. Specifically, the server will first analyze the user's historical browsing history and identify the report areas that the user often pays attention to. This involves statistics and pattern recognition of user behavior data. Then, the server will preliminarily design the rendering plan of the report based on the current display conditions, such as the required chart type and overall layout. The server will compare and integrate the identified historical interaction areas with the preliminary rendering plan to determine which parts should be rendered first. During the rendering process, the server will give priority to the data and charts of these important areas, involving the adjustment of the rendering order, the optimization of resource allocation, etc. Finally, the server will generate a complete report, but ensure that the parts that the user cares about most can be presented as quickly and clearly as possible. After the rendering is completed, the server will transmit the results to the user's terminal device via the network.

[0117] In some embodiments, priority rendering based on user habits can be achieved in a variety of ways: Optionally, the server can use heat map analysis. First, generate a report interaction heat map based on the user's historical browsing data. Then, map the heat map with the current report layout to identify high-frequency interaction areas. Finally, give priority to the data and charts in these high-frequency areas during the rendering process; Optionally, the server can use a progressive rendering method. First, determine the overall structure and layout of the report. Then, divide the areas with frequent historical interactions into multiple priority levels. Finally, render the report content step by step according to priority to ensure that high-priority areas are rendered first and displayed to the user. It is understandable that other methods can also be used to achieve priority rendering, such as using preloading technology to prepare the data required by the user in advance, or by dynamically adjusting the report structure to place important content in a prominent position, etc., which are not limited here.

[0118] In the embodiments of the present application, due to the adoption of an ODPS-based customer service data aggregation method, combined with intelligent data update frequency monitoring, dynamic pull strategy selection, and user behavior-aware priority rendering mechanism, it is possible to efficiently process massive multi-source heterogeneous data, achieve rapid data aggregation, flexible query, and personalized display, and effectively solve the problems of data update lag, low processing efficiency, slow query response, and poor user experience in traditional ETL methods, thereby achieving high efficiency in data processing, real-time query, accuracy in analysis, and personalization in display.

[0119] The following describes the server in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a server in an embodiment of the present application.

[0120] It should be noted that Figure 3 The structure of the server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0121] like Figure 3As shown, the server includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0122] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0123] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0124] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0126] Specifically, the server of this embodiment includes a processor and a memory. A computer program is stored in the memory. When the computer program is executed by the processor, the customer service data aggregation method based on ODPS provided in the above embodiment is implemented.

[0127] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the server described in the above embodiment; or may exist independently without being assembled into the server. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the server, the server implements the customer service data aggregation method based on ODPS provided in the above embodiment.

[0128] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0129] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0130] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A customer service data aggregation method based on ODPS, characterized in that: Applied to a server, the method comprises: Classify the collected source data by domain to obtain multiple source data sets; Loading the source data set to the ODPS platform, and using the ODPS platform to perform first aggregation processing on the source data in the multiple source data sets respectively, to obtain multiple field wide tables; Pull the domain wide table to the DRDS platform, and use the DRDS platform to perform distributed data processing on the domain wide table to obtain multiple distributed data; Determining, according to the report request instruction sent by the user terminal, a target data feature corresponding to the target report in the report request instruction; Extract a plurality of target data corresponding to the target data features from the plurality of distributed data, perform a second aggregation process on the target data, and generate the target report.

2. The method according to claim 1, characterized in that After the step of extracting a plurality of target data corresponding to the target data features from the plurality of distributed data, performing a second aggregation process on the target data, and generating the target report, the method further includes: In response to a report query request sent by a user terminal, determining a query rule and a display condition corresponding to the report query request; Obtain the dimension conditions and indicator conditions in the query rule, and generate a query statement in the corresponding target report; Execute the query statement on the DRDS platform to obtain target display data; According to the chart type and layout style in the display condition, the target display data is rendered as a query result, and the query result is sent to the user terminal.

3. The method according to claim 2, characterized in that The rendering of the target display data as a query result according to the chart type and layout style in the display condition, and sending the query result to the user terminal specifically includes: Acquire historical browsing data of the user terminal, and determine a historical interaction area of ​​the user terminal according to the historical browsing data; Determining a preset rendering result of the target display data based on the chart type and layout style in the display condition; In the preset rendering result, the rendering area corresponding to the historical interaction area is set as a priority rendering area; Render the priority display data corresponding to the priority rendering area in the target display data, obtain a query result, and send the query result to the user terminal.

4. The method according to claim 1, characterized in that: After the step of classifying the collected source data by fields to obtain multiple source data sets, the method further includes: Real-time monitoring of the data update frequency of the business field where the source data is located; The data upload time of the source data is determined according to the data update frequency.

5. The method according to claim 4, characterized in that Determining the data upload time of the source data according to the data update frequency specifically includes: Determining a preset upload time of the source data according to the data update frequency; The preset upload time is adjusted according to the real-time network speed and the data update size of the source data to obtain the data upload time of the source data.

6. The method according to claim 1, characterized in that The extracting a plurality of target data corresponding to the target data features from the plurality of distributed data, performing a second aggregation process on the target data, and generating the target report specifically includes: Determine a plurality of target data corresponding to the target data characteristics among the plurality of distributed data, and determine a pulling strategy for the target data; the pulling strategy includes full pulling and incremental pulling; When the pull strategy is incremental pull, based on the update timestamp of the target data, the target data to be updated after the last synchronization is synchronized to the DRDS platform; Performing a second aggregation process on the data to be updated to obtain an aggregation result; Generate a target report based on the historical target data after the last synchronization and the aggregation result.

7. The method according to claim 6, characterized in that The determining of a plurality of target data corresponding to the target data features in the plurality of distributed data and determining a strategy for pulling the target data specifically includes: Determine a plurality of target data corresponding to the target data feature among the plurality of distributed data; Determining a time span for pulling the plurality of target data; Calculating the data magnitude of the target data based on the pulling time span; When the data level is less than a preset data threshold, determining the pulling strategy to be full pull; When the data magnitude is not less than the preset data threshold, determining the computational complexity of aggregating the target data; When the computational complexity is lower than a preset level, the pulling strategy is determined to be incremental pulling.

8. A server, characterized in that: The server includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the server to execute the method described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a server, the server is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a server, the server is caused to execute the method according to any one of claims 1 to 7.

Images