Data processing method and device, server and storage medium

By building keyword tables and generating aggregated data tables, the problem of high pressure on data query in business systems is solved, and query efficiency and server performance are improved.

CN119066073APending Publication Date: 2024-12-03PING AN BANK CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411202463.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Due to the inconsistent time of business data in the business system, the data stored in the database gradually increases. All data indicators and data values ​​need to be traversed during querying, resulting in excessive pressure on server storage and query.

Method used

By obtaining business data for preset time periods from the database, extracting relevant parameters to build a keyword table, aggregating data generates an aggregated data table with keywords as the primary key index, and deleting the original data, receiving query instructions with keywords as the primary key index, and receiving query instructions Query the aggregated data table and feedback the results.

Benefits of technology

Reduces the number of data strips during data query, improves query efficiency, reduces storage and query pressure, and improves server performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066073A_ABST
    Figure CN119066073A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, and provides a data processing method which comprises the following steps: acquiring business data in a preset time period from a database, and extracting relevant parameters of preset business from the business data; constructing a keyword table according to related parameters of a preset service, extracting a service data record corresponding to each keyword in the keyword table from the service data, aggregating all the service data records corresponding to each keyword, and generating an aggregated data table taking the keyword as a main key index; and storing the aggregated data table into a database, and deleting the business data in the preset time period from the database. According to the data processing method, the number of data pieces corresponding to the keywords during data query is reduced, the data query efficiency is improved, the storage data volume is reduced, the storage pressure and query pressure of the server where the database is located are reduced, and the performance of the server is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method, device, server and storage medium. Background Art

[0002] Since the business data of the business system is generated at inconsistent times, business data is usually stored in the database one by one. As time goes by, the business data stored in the database gradually increases, resulting in increased server storage pressure. When users query business data corresponding to some indicators, the server needs to traverse and search all data indicators and data values ​​of all business data in the database, which puts a lot of query pressure. Summary of the invention

[0003] The present invention provides a data processing method, device, server and storage medium, the main purpose of which is to provide a method for reducing the amount of data stored in a database and improving the efficiency of database query.

[0004] To achieve the above object, the present invention provides a data processing method, the method comprising:

[0005] Acquire business data for a preset time period from a database, and extract relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time;

[0006] Construct a keyword table according to the relevant parameters of the preset business, extract the business data record corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index;

[0007] The aggregate data table is stored in the database, and the business data of the preset time period is deleted from the database.

[0008] Optionally, the data processing method further includes:

[0009] Receiving a data query instruction from a client to the database, and extracting a keyword field and a time field in the data query instruction;

[0010] The business data corresponding to the time field is queried from the aggregate data table stored in the database using the keyword contained in the keyword field as the primary key index, and fed back to the client.

[0011] Optionally, the data processing method further includes:

[0012] If the business data corresponding to the time field is not found in the aggregate data table, determining whether the time corresponding to the time field is within the preset time period;

[0013] If not, the corresponding business data is queried from the business data outside the aggregated data table in the database according to the keyword field and the time field, and fed back to the client.

[0014] Optionally, constructing a keyword table according to relevant parameters of the preset service includes:

[0015] Obtaining the fields of business indicators and business scenarios in the relevant parameters of the preset business;

[0016] The fields of the business indicators and the fields of the business scenarios are used as keywords to construct a keyword table.

[0017] Optionally, after the fields of the business indicator and the fields of the business scenario are used as keywords to construct a keyword table, the method further includes:

[0018] Acquire newly added fields of business indicators and newly added fields of business scenarios in the relevant parameters of the preset business;

[0019] The newly added fields of the business indicator and the newly added fields of the business scenario are updated to the keyword table as newly added keywords.

[0020] Optionally, constructing a keyword table according to relevant parameters of the preset service includes:

[0021] Obtaining historical query data of the database;

[0022] Counting the number of occurrences of each query field in the historical query data to obtain a high-frequency query field;

[0023] A keyword table is constructed by taking high-frequency query fields of relevant parameters belonging to the preset business as keywords.

[0024] Optionally, aggregating all business data records corresponding to each keyword to generate an aggregated data table indexed by the keyword as a primary key includes:

[0025] All business data records corresponding to each keyword are counted according to the time when the business data was generated to obtain the date and month;

[0026] Aggregate the salesperson codes of business data records with the same business data generation time to obtain a salesperson code collection;

[0027] Aggregate the business operator codes and indicator data volumes of business data records with the same business data generation time to obtain a business data sequence;

[0028] Based on the keyword, the salesperson code collection, the business data sequence, the date and the month, an aggregate data table with the keyword as the primary key index is generated.

[0029] In order to solve the above problem, the present invention further provides a data processing device, the data processing device comprising:

[0030] A data acquisition module is used to acquire business data of a preset time period from a database, and extract relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time;

[0031] A data aggregation module is used to construct a keyword table according to the relevant parameters of the preset business, extract the business data records corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index;

[0032] A data updating module is used to store the aggregated data table in the database and delete the business data of the preset time period from the database.

[0033] In order to solve the above problem, the present invention further provides a server, the server comprising:

[0034] at least one processor; and,

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the data processing method as described above.

[0037] In order to solve the above problem, the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the data processing method described above is implemented.

[0038] The present invention takes business indicators and business scenarios with high query frequency in the business system as keywords, and uses the keywords as primary key indexes to aggregate business data and construct an aggregated data table, thereby reducing the number of data items corresponding to the keywords during data query and improving data query efficiency. Compared with unprocessed business data, the aggregated data table reduces the amount of stored data, reduces the storage pressure and query pressure of the server where the database is located, and improves server performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of an embodiment of a data processing method of the present invention;

[0040] Figure 2 A schematic diagram of an embodiment of a server of the present invention;

[0041] Figure 3 It is a structural schematic diagram of an embodiment of a data processing device of the present invention.

[0042] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clear, the principle and spirit of the present invention will be described below with reference to several specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0045] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0046] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" that appear in the differences in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0047] like Figure 1 FIG. 1 is a flow chart of an embodiment of a data processing method of the present invention. The data processing method is applied to a server and includes steps S1-S3.

[0048] S1. Obtain business data for a preset time period from a database, and extract relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time.

[0049] In this embodiment, the database is a database for storing business data of a business system, and the business system may be a banking business system or a business system related to other business scenarios, such as an insurance business system, a communication business system, a smart home business system, and the like.

[0050] In one embodiment, the business data of the banking business system includes but is not limited to the card issuance data of the bank card (such as the number of cards issued, the ID of the card issuing clerk, the location of the card issuance, the time of the card issuance, etc.), and the new user data of the bank account (such as the number of new users, the ID of the account opening clerk, the location of the account opening, the time of the account opening, etc.). Correspondingly, in the relevant parameters of the business data, the clerk code is the card issuing clerk ID and the account opening clerk ID, the business indicators are the number of cards issued and the number of new users, the indicator data volume is the number of cards issued and the number of new users, the business scenario is the location of the card issuance and the location of the account opening, and the business generation time is the card issuance time and the account opening time.

[0051] In other embodiments, the business data of the insurance business system includes but is not limited to insurance data (such as the insured quantity, insurance agent ID, insurance type, insurance time, insurance location, etc.). Correspondingly, the agent code in the relevant parameters of the business data is the insurance agent ID, the business indicator is the insurance type, the indicator data volume is the insured quantity, the business scenario is the insurance location, and the business generation time is the insurance time.

[0052] In other embodiments, the business data of the communication business system includes but is not limited to user data of communication services (such as communication items, number of communication users, communication operator ID, communication location, activation time, etc.), maintenance data of communication services (such as maintenance items, maintenance quantity, maintenance operator ID, maintenance location, maintenance time, etc.). Correspondingly, in the relevant parameters of the business data, the operator code is the communication operator ID and the maintenance operator ID, the business indicators are communication projects and maintenance items, the indicator data volume is the number of communication users and the number of maintenance, the business scenarios are the communication location and the maintenance location, and the business generation time is the activation time and the maintenance time.

[0053] In other embodiments, the business data of the smart home business system includes but is not limited to smart home service data (such as smart home type, customer service ID, number of paid services, service location, service time, etc.). Correspondingly, in the relevant parameters of the business data, the salesperson code is the customer service ID, the business indicator is the smart home type, the indicator data volume is the number of paid services, the business scenario is the service location, and the business generation time is the service time.

[0054] In one embodiment, the database is an Es (Elasticsearch) database. Elasticsearch can provide near real-time search and analysis for all types of data, whether it is structured or unstructured text, digital data or geospatial data, Elasticsearch can efficiently store and index data in a way that supports fast search, making it easy for users to go beyond simple data retrieval and aggregate information to discover trends and patterns in the data. And as the amount of data storage and query volume increases, the distributed nature of Elasticsearch enables database deployment to grow seamlessly.

[0055] Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed, and explored. Taking full advantage of the horizontal scalability of Elasticsearch can make data more valuable in a production environment. The implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then the corresponding sentence is segmented through the word segmentation controller, and its weight and word segmentation results are stored in the data. When the user searches for data, the results are ranked and scored according to the weight, and then the returned results are presented to the user.

[0056] Elasticsearch is often integrated into a database solution called the "Elastic Stack" along with a data collection and log parsing engine called Logstash and an analysis and visualization platform called Kibana. Elasticsearch can be used to search for all kinds of documents. It provides scalable search, has near real-time search, and supports multi-tenancy. Elasticsearch is distributed, which means that the index can be divided into shards, and each shard can have 0 or more replicas. Each node hosts one or more shards and acts as a coordinator to delegate operations to the correct shards. Related data is usually stored in the same index, which consists of one or more primary shards and zero or more replica shards. Once an index is created, the number of primary shards cannot be changed. Elasticsearch uses Lucene and tries to provide all its features through JSON and Java APIs. It supports facetting and percolating, which is very useful for notifications if a new document matches a registered query. Another feature is called "gateway", which handles the long-term persistence of the index; for example, in the event of a server crash, the index can be restored from the gateway. Elasticsearch supports real-time GET requests and is suitable as a NoSQL data store, but lacks distributed transactions.

[0057] In one embodiment, acquiring the service data of a preset time period from a database includes: acquiring service data of a service generated within the preset time period from the database.

[0058] For example, the business data entered into the database between 0-24 hours every day generates the business generation time within 24 hours after the end of the day, that is, the data of day n will all generate the business generation time on day n+2. When the current time of obtaining business data is day t, the corresponding preset time period is day t-2. That is, when the current time is May 14, the corresponding preset time period is May 12, and the data of May 12 is obtained from the database as business data, thereby ensuring that the business data contains the business generation time.

[0059] In one embodiment, the parameters related to the preset service include a service provider code, service indicator, indicator data volume, service scenario, and service generation time, for example:

[0060]

[0061]

[0062] In this embodiment, since business data that is close to the data processing time may not have generated the corresponding business data generation time, when obtaining business data from the database, only business data within the preset time period that includes the business data generation time is obtained for processing, and business data that has not yet generated the corresponding business data generation time is left for processing at the data processing time point after the business data generation time is generated, thereby ensuring that the processed data will not be missed when performing data queries containing time fields.

[0063] S2. Construct a keyword table according to the relevant parameters of the preset business, extract the business data records corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table indexed by the keyword as the primary key.

[0064] In one embodiment, constructing a keyword table according to the relevant parameters of the preset business includes: obtaining the fields of business indicators and the fields of business scenarios in the relevant parameters of the preset business; and constructing the keyword table using the fields of the business indicators and the fields of the business scenarios as keywords.

[0065] For example, the business indicator fields in the business data include the number of cards issued and the number of new users, the business scenario fields include RT-Mart and Lcm Plaza, and the keywords in the keyword table constructed based on the business data are number of cards issued, number of new users, RT-Mart and Lcm Plaza.

[0066] In this embodiment, when the banking business system obtains the original business data records, the business staff usually inputs them into the database of the business system one by one using the business staff code as an identifier. Since the banking business system usually performs data query based on business indicators and business scenarios when performing business data query, the fields of business indicators and business scenarios are used as primary key indexes to aggregate the original business data records, and aggregated data records with fewer data records are obtained, which reduces the storage pressure of the server where the database is located, and can also improve the query efficiency of data query based on business indicators and business scenarios.

[0067] Specifically, after constructing a keyword table using the fields of the business indicators and the fields of the business scenarios as keywords, it also includes: obtaining the newly added fields of the business indicators and the newly added fields of the business scenarios in the relevant parameters of the preset business; and updating the newly added fields of the business indicators and the newly added fields of the business scenarios as newly added keywords to the keyword table.

[0068] In another embodiment, a keyword table is constructed according to relevant parameters of the preset business, including: obtaining historical query data of the database; counting the number of occurrences of each query field in the historical query data to obtain high-frequency query fields; and using the high-frequency query fields of the relevant parameters of the preset business as keywords to construct a keyword table.

[0069] Specifically, the number of occurrences of each query field in the historical query data is counted to obtain high-frequency query fields, including: counting the number of occurrences of each query field in the historical query data; sorting each query field from high to low according to the number of occurrences; obtaining query fields whose number of occurrences is higher than a preset threshold, or obtaining a preset number or a preset proportion of query fields in order from high to low as high-frequency query fields.

[0070] Specifically, after constructing a keyword table using the high-frequency query fields of the relevant parameters of the preset business as keywords, it also includes: obtaining the newly added high-frequency query fields in the historical query records, and determining whether the newly added high-frequency query fields belong to the relevant parameters of the preset business; and updating the newly added high-frequency query fields of the relevant parameters of the preset business as newly added keywords to the keyword table.

[0071] For example: the historical query data of the business database includes the following query fields: "Card issuance volume", "New user volume", "XX Plaza", "YY Shopping Center" and multiple date fields in the year-month-day format. Among them, the four fields "Card issuance volume", "New user volume", "XX Plaza" and "YY Shopping Center" are related parameters of the preset business and appear frequently. The above four high-frequency query fields are matched in the keyword table. The keyword table already contains the keywords "Card issuance volume", "New user volume" and "XX Plaza", but does not contain the keyword "YY Shopping Center". In this case, "YY Shopping Center" will be updated to the keyword table as a new keyword.

[0072] In one embodiment, all business data records corresponding to each keyword are aggregated to generate an aggregated data table with the keyword as the primary key index, including: counting all business data records corresponding to each keyword according to the time when the business data was generated to obtain the date and month; aggregating the salesperson codes of the business data records with the same business data generation time to obtain a salesperson code collection; aggregating the salesperson codes and indicator data volumes of the business data records with the same business data generation time to obtain a business data sequence; and generating an aggregated data table with the keyword as the primary key index based on the keyword, the salesperson code collection, the business data sequence, the date and the month.

[0073] Among them, counting all business data records corresponding to each keyword according to the business data generation time to obtain the date and month includes: classifying and counting all business data records corresponding to each keyword according to the business data generation time to obtain the business data records corresponding to each keyword sorted by month and date.

[0074] Aggregating the salesperson code and indicator data volume of business data records with the same business data generation time to obtain a business data sequence includes: combining the salesperson code and the corresponding indicator data volume in the business data record in the format of "salesperson code-indicator data volume", and then aggregating the "salesperson code-indicator data volume" with the same business data generation time into a data sequence separated by commas.

[0075] In this embodiment, by performing aggregate calculation on the data, the existing business data records are pre-calculated (pre-processed) for further calling. The database can call the data in the aggregate table through data linkage, query and selection of data.

[0076] For example: Get business data from the database as follows

[0077]

[0078] Get the business data records containing the keyword "card issuance amount" as

[0079]

[0080] Get the business data records containing the keyword "new user volume" as

[0081]

[0082] Get the business data records containing the keyword "Darunfa" as

[0083]

[0084] Get the business data records containing the keyword "Lcm Square" as

[0085]

[0086] Aggregate all business data records corresponding to each keyword to generate an aggregate data table with the keyword as the primary key index:

[0087]

[0088] In this embodiment, business indicators and locations with high query frequency in the business system are used as keywords, and keywords are used as primary key indexes to aggregate business data and construct an aggregated data table. This can effectively reduce the number of data items corresponding to the keywords during data query, thereby improving the data query efficiency of the database.

[0089] S3. Store the aggregate data table in the database, and delete the business data of the preset time period from the database.

[0090] In one embodiment, the data processing method also includes: receiving a data query instruction from a client to the database, extracting a keyword field and a time field in the data query instruction; using the keyword contained in the keyword field as a primary key index to query the business data corresponding to the time field from the aggregated data table stored in the database, and feeding back the data to the client.

[0091] Specifically, if the business data corresponding to the time field is not found from the aggregate data table, it is determined whether the time corresponding to the time field is within the preset time period; if not, the corresponding business data is queried from the business data outside the aggregate data table in the database based on the keyword field and the time field, and the data is fed back to the client.

[0092] For example: if the current time is day t, if the time in the data query instruction is day t-2 or before day t-2, the corresponding business data is queried from the aggregated data based on the keyword and time for display; if the time in the data query instruction is day t-1, since the business data of day t-1 has not been aggregated on day t, the corresponding business data is queried from the business data in the database based on the keyword for display.

[0093] Specifically, the business data corresponding to the time field is queried from the aggregate data table stored in the database using the keyword contained in the keyword field as the primary key index, including: searching for the data table record corresponding to the keyword in the aggregate data table, matching the corresponding business data record from the data table record corresponding to the keyword according to the time field; and performing statistics on the business data sequence in the matched business data record to obtain the business data corresponding to the time field.

[0094] For example: the aggregate data table of the database is

[0095]

[0096] When the user queries the card issuance volume on 20230512, the data table record corresponding to the card issuance volume is found based on the keyword "card issuance volume" in the keyword field.

[0097]

[0098] Match the second row of business data records in this area based on the time field "20230512"

[0099] Keywords: salesperson code collection business data sequence date month

[0100] Card issuance volume A12353 A12353-12,1113 20230512 202305

[0101] By adding up the values ​​of all the business data in the business data sequence of the business data record, we can get the card issuance quantity of 20230512, which is 1125.

[0102] This embodiment uses business indicators and business scenarios with high query frequency in the business system as keywords, and uses the keywords as primary key indexes to aggregate business data to build an aggregated data table, thereby reducing the number of data items corresponding to the keywords during data query and improving data query efficiency. Compared with unprocessed business data, the aggregated data table reduces the amount of stored data, reduces the storage pressure and query pressure of the server where the database is located, and improves server performance.

[0103] In one embodiment, the data processing method also includes: receiving a data query instruction from a client to the database; when it is detected that the data query instruction contains a preset field associated with comparative analysis, extracting a keyword field in the data query instruction, and identifying multiple keywords that need to be compared and analyzed in the keyword field through natural language analysis; extracting a time field in the data query instruction; using the multiple keywords as primary key indexes to query the business data corresponding to the time field from the aggregated data table stored in the database, performing matching calculations on the business data queried by the multiple keywords, and generating a comparative data table based on the calculation results of the matching calculations and feeding it back to the client.

[0104] For example: the aggregate data table of the database is

[0105]

[0106] When the user queries the comparative analysis of the number of cards issued and the number of new users on 20230512, the data table record corresponding to the time field "20230512" is found based on the keywords "number of cards issued" and "number of new users" in the keyword field.

[0107]

[0108] By adding up the values ​​of all the business data in the business data sequence with the keyword "card issuance volume" in the business data record, we can get that the card issuance volume on 20230512 is 1125, and according to the values ​​of the business data in the business data sequence with the keyword "new user volume" in the business data record, we can get that the new user volume on 20230512 is 111, and the comparison data table is generated as follows:

[0109] Keyword 20230512 Index value

[0110] Card issuance volume: 1125

[0111] Number of new users: 111

[0112] In other embodiments, the data processing method also includes: calling a preset data analysis algorithm according to the client's data query instruction to the database, and extracting the keyword field and time field in the data query instruction; using the keyword contained in the keyword field as the primary key index to query the business data corresponding to the time field from the aggregated data table stored in the database, performing calculation and analysis on the queried business data through a preset data analysis algorithm, and feeding back the results of the calculation and analysis to the client.

[0113] The preset data analysis algorithms include but are not limited to data table matching algorithms and funnel analysis algorithms.

[0114] Common data table matching algorithms mainly involve MySQL's table join query, which supports multiple join algorithms, among which BlockNested Loop (block-based nested loop join algorithm) is an optimization algorithm that reduces the number of full table scans. In addition, there is the VLOOKUP formula used in Excel to quickly match data, which is a common matching technology in database and table processing. MySQL supports multiple join algorithms, including but not limited to Simple Nested Loop Join and BlockNested Loop (block-based nested loop join). The latter improves query efficiency by reducing the number of full table scans. When executing the EXPLAIN command, you can view the execution plan of the query, including the join algorithm used and optimization suggestions. VLOOKUP is a function that quickly finds and matches data in Excel. It can find a specific value in the first column of a range or table array and return the value of any cell in that row. The basic syntax of the VLOOKUP formula is: =VLOOKUP(lookup_value,table_array,col_index_num,[range_lookup]), where lookup_value is the value to be searched, table_array is the cell range containing data, col_index_num is the column number where the matching value is expected to be returned, and range_lookup is a logical value that specifies whether an approximate match is required. The VLOOKUP formula is very useful when processing large amounts of data, especially when data needs to be matched from one worksheet to another. In summary, the data table matching algorithm not only involves database query optimization technology, such as MySQL connection algorithm selection, but also includes matching functions in spreadsheet software such as Excel, such as VLOOKUP. The data table matching algorithm can improve the data processing efficiency and accuracy of database tables.

[0115] Funnel analysis is a data analysis method used to track user behavior and break down user paths into a series of key steps to identify problems in the conversion process and optimize conversion rates. This method helps understand user behavior and needs by visualizing the entire conversion process, thereby optimizing products and services. The core steps of funnel analysis include determining the research object, selecting the analysis dimension, determining the start and end time of the event, breaking down the user path to identify key nodes, and specifying key indicators to comprehensively characterize the business and guide business optimization. Funnel analysis is widely used in the e-commerce field. For example, in the user growth strategy of the shopping system, funnel analysis is used to analyze the entire process from user exposure to information to the final purchase, including user acquisition, user activation, retention, user payment, and user communication. Through this analysis, those subtle factors can be located to improve conversion performance, thereby optimizing products and services12. There are two types of funnel models: unordered funnel and ordered funnel. Unordered funnels do not limit the order of events between multiple steps in the funnel during the funnel cycle, while ordered funnels strictly limit the order of events between each step. These two models each have their own application scenarios. For example, funnel analysis plays an important role in locating the specific reasons for user loss, detecting the effectiveness of a special event, and comparing the conversion rates of different versions.

[0116] For example: The original business data of the banking system is recorded as follows

[0117]

[0118] After data processing, the aggregated data table is

[0119]

[0120] Among them, the business indicators "card issuance volume" and "new user volume" have a conversion correlation relationship, the business scenarios "RT-Mart" and "Lcm Plaza" have a comparative correlation relationship, and the relevant business data records are often used for funnel calculations. According to the aggregated data table, the corresponding business data records can be queried more quickly and matching calculations and funnel analysis can be performed.

[0121] It can be seen from the above embodiments that the embodiments of the present invention use business indicators and business scenarios with high query frequency in the business system as keywords, and use the keywords as primary key indexes to aggregate business data to construct an aggregated data table, thereby reducing the number of data items corresponding to the keywords during data query and improving data query efficiency. Compared with unprocessed business data, the aggregated data table reduces the amount of stored data, reduces the storage pressure and query pressure of the server where the database is located, and improves server performance.

[0122] like Figure 2FIG. 1 is a schematic diagram of an embodiment of a server of the present invention. Server 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The server 1 can be a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets.

[0123] In this embodiment, the server 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a data processing program 10 that can be executed by the processor 12. Figure 2 Only the server 1 having components 11-13 and the data processing program 10 is shown. It can be understood by those skilled in the art that Figure 2 The structure shown does not constitute a limitation on the server 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0124] The memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the server 1; the readable storage medium may be volatile or non-volatile. Specifically, the readable storage medium may be a storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the server 1, such as a hard disk of the server 1; in other embodiments, the storage medium may also be an external storage device of the server 1, such as a plug-in hard disk equipped on the server 1, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 mainly includes a program storage area and a data storage area, wherein the program storage area is generally used to store an operating system and various application software installed on the server 1, such as storing the code of the data processing program 10 in one embodiment of the present invention; the data storage area can store data created according to the use of the blockchain node, such as various types of data that have been output or are to be output.

[0125] The processor 12 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 12 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the programs or modules stored in the memory 11 (for example, executing resource dynamic scheduling programs, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data. The processor 12 is generally used to control the overall operation of the server 1, such as executing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the data processing program 10, etc.

[0126] The network interface 13 may include a wireless network interface or a wired network interface.

[0127] Optionally, the server 1 may further include a user interface, which may include a display, an input unit such as a keyboard, and an optional user interface may further include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (OLED) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the server 1 and to display a visual user interface.

[0128] In one embodiment of the present invention, when the data processing program 10 is executed by the processor 12, the following data processing method is implemented, including steps S1-S3:

[0129] S1. Obtaining business data for a preset time period from a database, and extracting relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time;

[0130] S2. Construct a keyword table according to the relevant parameters of the preset business, extract the business data record corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index;

[0131] S3. Store the aggregate data table in the database, and delete the business data of the preset time period from the database.

[0132] The specific operation steps implemented by the above steps S1-S3 are substantially the same as the steps S1-S3 of the first embodiment of the data processing method of the present invention, and will not be repeated here.

[0133] In other embodiments, the data processing program 10 can also be divided into one or more modules, one or more modules are stored in the memory 11, and are executed by one or more processors (processor 12 in this embodiment) to complete the present invention. The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, and is used to describe the execution process of the data processing program 10 in the server 1.

[0134] like Figure 3 , which is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention.

[0135] In the first embodiment of the present invention, the data processing device 100 includes a data acquisition module 110, a data aggregation module 120 and a data update module 130. Exemplarily:

[0136] The data acquisition module 110 is used to acquire business data of a preset time period from a database, and extract relevant parameters of the preset business from the business data, wherein the relevant parameters of the preset business include business operator code, business indicator, indicator data volume, business scenario and business generation time;

[0137] The data aggregation module 120 is used to construct a keyword table according to the relevant parameters of the preset business, extract the business data records corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index;

[0138] The data updating module 130 is used to store the aggregated data table in the database and delete the business data of the preset time period from the database.

[0139] The specific operation steps implemented when the data acquisition module 110, the data aggregation module 120 and the data update module 130 are executed are substantially the same as those in the above embodiment and will not be repeated here.

[0140] In addition, the embodiment of the present invention further proposes a computer-readable storage medium, which may be volatile or non-volatile. Specifically, the computer-readable storage medium may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium stores a data processing program.

[0141] The specific software function implementation of the computer-readable storage medium of the present invention is substantially the same as the specific software implementation in the above-mentioned server 1, and will not be described in detail here.

[0142] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0143] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0145] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire business data for a preset time period from a database, and extract relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time; Construct a keyword table according to the relevant parameters of the preset business, extract the business data record corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index; The aggregate data table is stored in the database, and the business data of the preset time period is deleted from the database.

2. The data processing method according to claim 1, characterized in that: The data processing method further includes: Receiving a data query instruction from a client to the database, and extracting a keyword field and a time field in the data query instruction; The business data corresponding to the time field is queried from the aggregate data table stored in the database using the keyword contained in the keyword field as the primary key index, and the business data is fed back to the client.

3. The data processing method according to claim 2, characterized in that: The data processing method further includes: If the business data corresponding to the time field is not found in the aggregate data table, determining whether the time corresponding to the time field is within the preset time period; If not, the corresponding business data is queried from the business data outside the aggregated data table in the database according to the keyword field and the time field, and fed back to the client.

4. The data processing method according to claim 1, characterized in that: The step of constructing a keyword table according to the relevant parameters of the preset service includes: Obtaining the fields of business indicators and business scenarios in the relevant parameters of the preset business; The fields of the business indicators and the fields of the business scenarios are used as keywords to construct a keyword table.

5. The data processing method according to claim 4, characterized in that: After the fields of the business indicator and the fields of the business scenario are used as keywords to construct a keyword table, the following further includes: Acquire newly added fields of business indicators and newly added fields of business scenarios in the relevant parameters of the preset business; The newly added fields of the business indicator and the newly added fields of the business scenario are updated to the keyword table as newly added keywords.

6. The data processing method according to claim 1, characterized in that: The step of constructing a keyword table according to the relevant parameters of the preset service includes: Obtaining historical query data of the database; Counting the number of occurrences of each query field in the historical query data to obtain a high-frequency query field; A keyword table is constructed by taking high-frequency query fields of relevant parameters belonging to the preset business as keywords.

7. The data processing method according to claim 1, characterized in that: The step of aggregating all business data records corresponding to each keyword to generate an aggregate data table with the keyword as the primary key index includes: All business data records corresponding to each keyword are counted according to the time when the business data was generated to obtain the date and month; Aggregate the salesperson codes of business data records with the same business data generation time to obtain a salesperson code collection; Aggregate the business operator codes and indicator data volumes of business data records with the same business data generation time to obtain a business data sequence; Based on the keyword, the salesperson code collection, the business data sequence, the date and the month, an aggregate data table with the keyword as the primary key index is generated.

8. A data processing device, characterized in that: The data processing device comprises: A data acquisition module is used to acquire business data of a preset time period from a database, and extract relevant parameters of a preset business from the business data, wherein the relevant parameters of the preset business include a business operator code, business indicators, indicator data volume, business scenario, and business generation time; A data aggregation module is used to construct a keyword table according to the relevant parameters of the preset business, extract the business data records corresponding to each keyword in the keyword table from the business data, aggregate all business data records corresponding to each keyword, and generate an aggregated data table with the keyword as the primary key index; A data updating module is used to store the aggregated data table in the database and delete the business data of the preset time period from the database.

9. A server, characterized in that: The server comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Index management method, related equipment and storage medium

    CN112800066A

  • Dispatch link fault determination method and device, equipment and storage medium

    CN113806360A

  • HBase-based trillion-level real-time data association method and device and HBase-based trillion-level real-time data retrieval method and device

    CN115934882A

  • Structured large-field data query method and device and electronic equipment

    CN118132550A