Data statistics method, device, equipment, storage medium and program product

Abstract

The embodiments of the present application disclose a data statistics method, device, equipment, storage medium and program product, which relate to the field of data processing technology. The method includes: obtaining a log file of a database, parsing the log file, obtaining statistical configuration information, combining target data with historical statistical data stored in a first storage space based on the statistical configuration information to obtain target statistical data, and updating and storing the target statistical data in the first storage space, where the first storage space is used to store statistical data for query. By directly obtaining the target data by parsing the log file of the database, performing statistics on the target data according to the statistical configuration information, and pre-storing the statistical results, it is possible to perform statistics only on data that has query requirements, thereby saving computer storage resources, improving query efficiency, and reducing query latency by pre-storing statistical results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data statistics method, apparatus, device, storage medium, and program product. Background Art

[0002] In live broadcast scenarios, a large amount of data is generated every moment based on object behavior, and accordingly, there is a need for statistical queries on this data.

[0003] In related technologies, real-time flow data is stored in a database. When the server receives a query request from the business party, it splices out the database query language according to the query request, and then requests the database to query based on the spliced ​​database query language. Finally, the statistical results obtained from the query are fed back to the business party.

[0004] However, when the amount of data is large and frequent queries are required, the above method has a large impact on database performance and a long query response time, which cannot meet the query requirements for massive data. Summary of the Invention

[0005] The present invention provides a data statistics method, apparatus, device, storage medium, and program product. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a data statistics method is provided, the method comprising:

[0007] Obtaining a log file of a database, wherein the log file is used to record data updates of the database;

[0008] Parsing the log file to obtain target data in the log file, where the target data is updated data in the log file;

[0009] Obtaining statistical configuration information, wherein the statistical configuration information is a pre-set data statistical plan;

[0010] combining the target data with historical statistical data stored in the first storage space based on the statistical configuration information to obtain target statistical data;

[0011] The target statistical data is updated and stored in the first storage space, where the first storage space is used to store statistical data for query.

[0012] According to one aspect of an embodiment of the present application, a data statistics device is provided, the device comprising:

[0013] An acquisition module is used to acquire a log file of a database, wherein the log file is used to record data updates of the database;

[0014] A parsing module, configured to parse the log file and obtain target data in the log file, where the target data is updated data in the log file;

[0015] The acquisition module is further configured to acquire statistical configuration information, wherein the statistical configuration information is a pre-set data statistics scheme;

[0016] a processing module, configured to combine the target data with historical statistical data stored in the first storage space based on the statistical configuration information to obtain target statistical data;

[0017] The processing module is further configured to update and store the target statistical data in the first storage space, where the first storage space is configured to store statistical data for query.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data statistics method described in any of the above embodiments.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is loaded and executed by a processor to implement the data statistics method described in any one of the above embodiments.

[0020] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned data statistics method.

[0021] The technical solutions provided by the embodiments of the present application include at least the following beneficial effects:

[0022] By parsing the database log files to directly obtain the target data, the target data is counted according to the statistical configuration information, and the statistical results are pre-stored. This allows statistics to be performed only on data that has query requirements, saving computer storage resources, improving query efficiency, and reducing query latency by pre-storing statistical results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0025] Figure 2 This is a flow chart of a data statistics method provided by an embodiment of the present application;

[0026] Figure 3 is a flow chart of a data statistics method provided by another embodiment of the present application;

[0027] Figure 4 is a flow chart of a data statistics method provided by another embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of a non-periodic statistical data structure provided by an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of a periodic statistical data structure provided by an embodiment of the present application;

[0030] Figure 7 This is a technical principle diagram provided by an embodiment of the present application;

[0031] Figure 8 This is a block diagram of a data statistics device provided by one embodiment of the present application;

[0032] Figure 9 It is a block diagram of a data statistics device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of this application more clear, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0034] In this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on the quantity and execution order.

[0035] The traditional data statistics query technology solution is to splice out a database language according to the needs of the business party, request the database to query according to the spliced ​​database language, and then feed back the query results to the business party. This solution has a large performance loss on the database when the amount of data is relatively large and frequent queries are required, and the query response time is long, which cannot meet the query needs of massive data. The embodiment of the present application provides a data statistics method, which writes flow data into a database, obtains updated flow data, that is, target data, by parsing the log file of the database, and counts the target data according to the statistical configuration information, and writes the statistical results into the storage to complete the pre-statistics of the data. The embodiment of the present application realizes on-demand statistics of data by setting the statistical configuration information to personalize the data that needs to be counted, which can save computer storage space and improve query efficiency; at the same time, the embodiment of the present application calculates the statistical results in advance and writes them into the storage, so that the query delay is reduced, and it can cope with higher frequency query scenarios.

[0036] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment of an embodiment of the present application. Schematically, the implementation environment includes a first terminal 100, a second terminal 110, a server 120 and a communication network 130. Optionally, the implementation environment includes multiple different first terminals 100 and second terminals 110, Figure 1 Only a first terminal 100 and a second terminal 110 are taken as an example for description.

[0037] In some optional embodiments, a target application that can generate flow data in real time is installed and run in the first terminal 100. Schematically, the target application includes a live broadcast application, a social application, a game application, a shopping application, etc., which is not limited in the embodiments of the present application.

[0038] When the first terminal 100 runs the target application, it will upload the flow data to the server 120 in real time. Optionally, after receiving the flow data, the server 120 will first write the flow data into the database. During the writing process of the flow data, the database will generate a corresponding log file to record the data update of the database. The server 120 will monitor the log file to obtain the target data. For example, when new flow data is written to the database, the log file will record the update action of the database. The server 120 will monitor the update action and parse the update action to obtain the target data. After obtaining the target data, the server 120 will perform statistical processing on the target data according to the pre-configured statistical configuration information and write the statistical results into the storage space.

[0039] In some optional embodiments, a query system is running in the second terminal 110, and the statistical results can be queried. The second terminal 110 sends a statistical query request to the server 120. In response to the statistical query request, the server 120 queries the storage space and feeds back the statistical results to the second terminal 110.

[0040] In some optional embodiments, the first terminal 100 and the second terminal 110 are, but are not limited to, smartphones, tablet computers, laptop computers, desktop computers, smart home appliances, smart car terminals, smart speakers, etc. Optionally, the first terminal 100 and the second terminal 110 can be two independent devices or the same device, which is not limited in this application.

[0041] In some optional embodiments, the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, and cloud computing. The server 120 may also be implemented as a node in a blockchain system, which is not limited in this embodiment.

[0042] The first terminal 100, the second terminal 110 and the server 120 are connected via a communication network 130. In some optional embodiments, the communication network 130 may be a wired network or a wireless network, which is not limited here.

[0043] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the transaction data involved in this application was obtained with full authorization.

[0044] The data statistics method provided in the embodiments of the present application can be applied to at least the following application scenarios:

[0045] In live broadcast applications, based on the behavior of objects (for example, recharging, gift giving, etc.), a large amount of data is generated every moment. After the object is authorized, this data can be obtained and stored in the database. The data statistics method provided in the embodiment of the present application does not require the deployment of a complex data engine. By monitoring the log files of the database, real-time flow data can be obtained by parsing. These data can be pre-statisticed through standardized configuration and data processing. Among them, technicians configure statistical configuration files according to actual needs, saving computer storage resources. At the same time, since a pre-statistical solution is adopted, the live broadcast product strategy can directly return the cached statistical value when querying, greatly reducing query latency.

[0046] It is worth noting that the above application scenarios are only illustrative examples, and the embodiments of the present application do not limit other application scenarios of the data statistics method.

[0047] Combined with the above introduction and implementation environment, Figure 2 This is a flow chart of a data statistics method provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0048] Step 201: Obtain the log file of the database.

[0049] Log files are used to record data updates in the database.

[0050] Optionally, the database is a relational database management system (MySQL). Schematically, the log file corresponding to MySQL is a binary log (Binary log, Binlog). The Binlog log can be directly enabled in the MySQL configuration file. When the data in MySQL is updated (for example: data addition, data deletion, data change, etc.), the Binlog log is automatically updated.

[0051] Optionally, the following steps should be included before obtaining the database log file:

[0052] Obtain transaction data; write transaction data to the database. This transaction data refers to real-time data generated by the subject when using the application. For example, in a live streaming application, the subject can give virtual gifts to the host in the live streaming room. Each gift corresponds to a transaction data entry. The subject can authorize data management in the settings interface of the live streaming application. Agreeing to authorization means agreeing to upload the real-time transaction data to the server.

[0053] In some optional embodiments, the database includes a transaction database corresponding to at least two business servers. Schematically, the use scenarios of the transaction database corresponding to at least two business servers include at least one of the following scenarios:

[0054] Scenario 1: Writing transaction data into different transaction databases. Schematically, each transaction database is connected to the same / different number of business terminals, and the same / different applications are installed in the business terminals. The transaction data generated in each business terminal will be written into the transaction database connected to it; or, each transaction database is connected to different modules of the application, and the transaction data generated in each module will be written into the transaction database connected to it. For example, when the application is implemented as a shopping application, order information for which resource exchange has been implemented and information about items added to the shopping cart can be written into different transaction databases respectively.

[0055] Scenario 2: Determine a mainstream water database among multiple water flow databases. Except for the mainstream water database, other water flow databases are subordinate water flow databases. The water flow data are all written into the mainstream water database. When the water flow data needs to be queried, the subordinate water flow database is read, thereby realizing read-write separation.

[0056] Step 202: parse the log file to obtain target data in the log file.

[0057] The target data is the updated data in the log file. For example, if a piece of transaction data is written into the database and the database is updated, then this piece of transaction data is the target data.

[0058] In some optional embodiments, the process of acquiring target data may further include the following steps:

[0059] Monitor the log file; in response to the log file having content updated, parse the log file to obtain the target data in the log file.

[0060] For example, when monitoring log files implemented as Binlogs, the Binlogs may optionally record MySQL execution statements. When a transaction is written to MySQL, the written statement is added to the Binlog. Upon detecting that a statement has been added to the Binlog, the Binlog parser parses the statement to retrieve the target data.

[0061] Optionally, after acquiring the target data, the target data is written into a message system to wait for processing. Schematically, the target data can be written into a message queue of a distributed message system (eg, Apache Kafka) for caching.

[0062] Optionally, the target data includes an object identifier. Before processing the target data, the target data may be diverted. The diversion process refers to diverting target data with the same object identifier to the same queue or downstream node. Optionally, the target data may be diverted according to the object identifier based on a hash algorithm, and target data with the same object identifier may be processed serially.

[0063] Step 203: Obtain statistical configuration information.

[0064] Statistical configuration information is a pre-set data statistics plan.

[0065] Optionally, the statistical configuration information may be information pre-configured by the business party in a statistical configuration file. Schematically, the business party may be a technical person related to the development of the target application.

[0066] Optionally, the data statistics scheme is used to instruct the server to perform statistics on the data in the database according to the rules for the target data, for example:

[0067] When the target application is implemented as a host application, the object behavior corresponding to the target data is that the object is currently following a certain live broadcast room. The corresponding data statistics plan is to count and query the total number of live broadcast rooms followed by the object within 7 days. In this case, the total number of live broadcast rooms followed by the object in the past 7 days from the current time point is queried in the database.

[0068] Optionally, the data statistics scheme is further used to indicate whether the server needs to perform statistics on the target data. Schematically, the manner of managing data statistics requirements through the data statistics scheme includes at least one of the following manners:

[0069] 1. Add a statistical demand switch in the statistical configuration information. Optionally, the statistical configuration information includes specific statistical categories of the data (for example, the number of live broadcast rooms followed by the object, the length of time the object watches the live broadcast, etc.), the account category of the object in the target application (for example, anchor account, audience account, etc.), etc. The business party can add a statistical demand switch based on these categories, for example, configuring the statistical demand for "the number of live broadcast rooms followed by the audience account within 7 days"; the statistical configuration information also includes object identification, and the business party can also manage the statistical demand for a single object data in the statistical configuration information, for example, prohibiting the statistics of all data of a certain object account, or prohibiting the statistics of all data of a certain object account within a specific time period.

[0070] Optionally, the demand switch can be set to a real switch button form. When the demand switch is set to the "on" state, it indicates that the target data needs to be counted, and when the demand switch is set to the "off" state, it indicates that the target data does not need to be counted; the demand switch can also be set to a simple 0 or 1. When the demand switch is set to "0", it indicates that the target data does not need to be counted, and when the demand switch is set to "1", it indicates that the target data needs to be counted.

[0071] 2. Optionally, the target data includes an object identifier and a statistical category. The corresponding statistical configuration information is queried in the statistical configuration file according to the object identifier and the statistical category. If there is corresponding statistical configuration information in the statistical configuration file, it means that the business party has statistical requirements for the target data; if the corresponding statistical configuration information cannot be queried in the statistical configuration file, it means that the business party has no statistical requirements for the target data.

[0072] The above-mentioned method of managing statistical requirements is merely an illustrative example and is not limited to this embodiment of the present application.

[0073] Step 204 : Based on the statistical configuration information, the target data is combined with the historical statistical data stored in the first storage space to obtain target statistical data.

[0074] Optionally, the historical statistical data refers to statistical data stored in the first storage space within a historical time period.

[0075] Optionally, in response to the data statistics scheme in the statistics requirement information requiring statistics on the target data, the processing of the target data includes at least one of the following situations:

[0076] 1. Directly combine the target data with the historical statistical data stored in the first storage space to obtain the target statistical data.

[0077] 2. According to the data statistics plan of the target data, obtain statistical results from the database, and the results are the target statistical data; or, according to the data statistics plan of the target data, directly store the target data in the first storage space as the target statistical data.

[0078] Optionally, in response to the data statistics solution in the statistics requirement information that does not require statistics to be performed on the target data, the target data is discarded, that is, no data statistics operation is performed on the target data.

[0079] Step 205: Update and store the target statistical data in the first storage space.

[0080] The first storage space is used to store statistical data for query.

[0081] Illustratively, the first storage space may be implemented as a non-relational database (eg, a Redis cache).

[0082] Optionally, target statistical data is obtained from a database according to statistical configuration information and stored in the first storage space, or target data is combined with historical statistical data to obtain target statistical data and the historical statistical data in the first storage space is updated to target statistical data.

[0083] To sum up, the embodiment of the present application directly obtains the target data by parsing the log files of the database, counts the target data according to the statistical configuration information, and pre-stores the statistical results. It can achieve statistics only on the data that has query requirements, saves computer storage resources, improves query efficiency, and reduces query latency by pre-storing statistical results.

[0084] Figure 3 is a flow chart of a data statistics method provided by another exemplary embodiment of the present application. Figure 3 As shown, the method includes:

[0085] Step 301: Obtain the log file of the database.

[0086] Log files are used to record data updates in the database.

[0087] The process of obtaining the log files of the database has been described in step 201 and will not be repeated here.

[0088] Step 302: parse the log file to obtain target data in the log file.

[0089] The target data is the updated data in the log file.

[0090] The process of obtaining the target data in the log file has been described in step 202 and will not be repeated here.

[0091] Step 303: Match the target data with the historical statistical data of the first storage space.

[0092] In some optional embodiments, the target data corresponds to a first object identifier and first statistical item information, and the historical statistical data corresponds to a second object identifier and second statistical item information.

[0093] Optionally, the object logged in in the target application has a unique object identifier. Schematically, the object identifier can be implemented as an account number in the target application.

[0094] Optionally, the first statistical item information is associated with the business behavior contained in the target data, and the second statistical item information is associated with the business behavior contained in the historical statistical data. For example, one of the statistical types corresponding to the behavior of the object following the live broadcast room is the number of live broadcast rooms followed by the object.

[0095] In some optional embodiments, the first object identifier is matched with the second object identifier; and the first statistical item information is matched with the second statistical item information.

[0096] Optionally, the first object identifier and the second object identifier are matched first. If the match fails, there is no need to match the first statistical item information with the second statistical item information. If the match succeeds, the first statistical item information is matched with the second statistical item information.

[0097] Optionally, the first statistical item information is matched with the second statistical item information first. If the match fails, there is no need to match the first object identifier with the second object identifier; if the match succeeds, the first object identifier is matched with the second object identifier.

[0098] Step 304 : In response to a failure in matching the target data with the historical statistical data, obtain statistical configuration information.

[0099] In some optional embodiments, the historical statistical data corresponds to status information, and the status information is used to indicate the validity of the historical statistical data.

[0100] In some optional embodiments, the method for obtaining statistical configuration information includes at least one of the following methods:

[0101] 1. In response to a failure in matching the first object identifier with the second object identifier; or a failure in matching the first statistical item information with the second statistical item information, obtain statistical configuration information.

[0102] Indicatively, the situations where matching fails in method 1 include the following:

[0103] (1) Optionally, the first object identifier and the second object identifier refer to object account numbers. If there is no second object identifier identical to the first object identifier in the first storage space, the first object identifier and the second object identifier fail to match;

[0104] (2) Optionally, the first statistical item information and the second statistical item information refer to statistical content. In the first storage space, the statistical content corresponding to the first statistical item information and the second statistical item information is inconsistent, and the first statistical item information fails to match the second statistical item information.

[0105] Illustratively, if the first statistical item information refers to the number of live broadcast rooms that the subject follows, and the second statistical item information refers to the total number of live broadcast rooms that the subject follows within a period of time, the first statistical item information and the second statistical item information are matched; if the first statistical item information refers to the number of live broadcast rooms that the subject follows, and the second statistical item information refers to the total length of time the subject watches the live broadcast within a period of time, the first statistical item information and the second statistical item information are mismatched.

[0106] As long as at least one of the above two situations exists, the target data will fail to match the historical statistical data.

[0107] 2. The first object identifier successfully matches the second object identifier and the first statistical item information successfully matches the second statistical item information. In response to the status information corresponding to the historical statistical data indicating that the historical statistical data is invalid, obtain statistical configuration information.

[0108] Indicatively, the target data is the data generated when the object follows a certain live broadcast room at the current time, and the historical statistical data refers to the total number of live broadcast rooms followed by the object in the past 7 days. However, if the status information of the historical statistical data is set to invalid, the target data fails to match the historical statistical data.

[0109] Step 305 : Based on the statistical configuration information, the target data is combined with the historical statistical data stored in the first storage space to obtain target statistical data.

[0110] The process of obtaining target statistical data based on the statistical configuration information has been described in step 204 and will not be repeated here.

[0111] In some optional embodiments, in response to the target data successfully matching the historical statistical data, the target data is combined with the historical statistical data in a specified combination manner to obtain the target statistical data.

[0112] It is worth noting that since the target data successfully matches the historical statistical data, it is assumed that the target data needs to be counted. Generally, it can be assumed that the target data has statistical requirements. Optionally, the business party can change the statistical requirements for the target data at any time. If the statistical requirements for the target data change from the original need for statistics to no need for statistics, then the target data needs to be discarded and the historical statistical data associated with it needs to be deleted from the first storage space; or, the target data needs to be discarded but the historical statistical data associated with it is not deleted from the first storage space.

[0113] Optionally, the successful matching of the target data with the historical statistical data means that the first object identifier and the second object identifier are matched successfully, the first statistical item information and the second statistical item information are matched successfully, and the status information corresponding to the historical statistical data is used to indicate that the historical statistical data is normal.

[0114] Optionally, the target data includes statistical values. For example, if the target object currently follows a live broadcast room, the statistical value is 1; if the target object currently stops following a live broadcast room, the statistical value is -1.

[0115] Optionally, the specified combination method includes at least one of the following methods:

[0116] 1. Optionally, the historical statistical data includes a presidential count value and a total number of records. For example, if the number of times the target object follows the live broadcast room within 7 days is 1000 (excluding the number of unfollowings), then the presidential count value is 1000. If the target object follows the live broadcast room a total of 10 times within 7 days (including the unfollowing behavior), then the total number of records is 10.

[0117] Specifying the combination method means accumulating the statistical value of the target data and the presidential count value in the historical statistical data, and updating the total number of records in the historical statistical data.

[0118] 2. The designated combination method refers to associating the target data with the historical statistical data, but does not change the statistical values ​​in the target data or the presidential count values ​​in the historical statistical data. In principle, the data generated by the subject following the live broadcast room within 7 days are connected to form the target statistical data.

[0119] Step 306: Update and store the target statistical data in the first storage space.

[0120] The first storage space is used to store statistical data for query.

[0121] The method for storing target statistical data has been described in step 205 and will not be repeated here.

[0122] To sum up, the embodiment of the present application directly obtains the target data by parsing the log files of the database, counts the target data according to the statistical configuration information, and pre-stores the statistical results. It can achieve statistics only on the data that has query requirements, saves computer storage resources, improves query efficiency, and reduces query latency by pre-storing statistical results.

[0123] The method provided in the embodiment of the present application matches the target data with the historical statistical data. If the match is successful, the target data and the historical statistical data can be directly combined to obtain the target statistical data without querying the database, thereby reducing the performance loss of the database; if the match fails, only the target data required for statistics is counted according to the statistical configuration information, thereby saving the computer's storage space and computing resources.

[0124] Figure 4is a flow chart of a data statistics method provided by another exemplary embodiment of the present application. Figure 4 As shown, the method includes:

[0125] Step 401: Obtain the log file of the database.

[0126] Log files are used to record data updates in the database.

[0127] The process of obtaining the log files of the database has been described in step 201 and will not be repeated here.

[0128] Step 402: parse the log file to obtain target data in the log file.

[0129] The target data is the updated data in the log file.

[0130] The process of obtaining the target data in the log file has been described in step 202 and will not be repeated here.

[0131] Step 403: Write the target data into the data statistics write service.

[0132] The data statistics write service is used to cache the target data before statistics are collected.

[0133] Optionally, after the target data is written into the data statistics write service, the server also needs to receive a response result of this writing, where the response result is used to indicate whether the target data is successfully written into the data statistics write service.

[0134] Optionally, during the process of writing target data to the data statistics write service, if network or other system abnormalities may cause target data loss, you need to roll back the target data. The following are the specific steps for rolling back the target data:

[0135] In response to a failure in writing target data to the data statistics write service, the target data is written to a rollback queue; the data to be rolled back from the rollback queue is rolled back to the data statistics write service in sequence; in response to a successful rollback of the target data, the target data is deleted from the rollback queue.

[0136] The data to be rolled back is the target data for which the write service fails.

[0137] The situation where the target data fails to be written into the data statistics writing service includes at least one of the following situations:

[0138] 1. If no response result is received from the target data writing service, it means that the target data writing service has failed.

[0139] 2. Receive a response result from the target data writing data statistics writing service, but the response result indicates that the target data writing data statistics writing service is unsuccessful. Repeat the write operation multiple times. If after multiple repetitions, you still receive a response result indicating that the target data writing data statistics writing service is unsuccessful, it means that the target data writing data statistics writing service has failed.

[0140] Optionally, the target data rollback success means that the data statistics write service receives the target data, that is, the target data has been successfully written into the cache in the data statistics write service.

[0141] Step 404: Obtain statistical configuration information.

[0142] Statistical configuration information is a pre-set data statistics plan.

[0143] The process of obtaining statistical configuration information has been described in step 203 and will not be repeated here.

[0144] Step 405 : Based on the statistical configuration information, the target data is combined with the historical statistical data stored in the first storage space to obtain target statistical data.

[0145] Optionally, the target data includes a timestamp, which is used to indicate the generation time of the target data.

[0146] Optionally, the historical statistical data includes status information, and the status information is used to indicate the validity of the historical statistical data.

[0147] The process of obtaining target statistical data based on target data that has not been rolled back has been described in step 204 and will not be repeated here.

[0148] Optionally, the storage structure of historical statistical data in the cache includes the following two implementation methods:

[0149] 1. Non-periodic historical statistical data structure

[0150] In some optional embodiments, the historical statistical data includes at least one historical statistical data node corresponding to at least one historical statistical sub-data, and the historical statistical data node corresponds to a node level.

[0151] Optionally, the historical statistical sub-data refers to statistical data recorded within a historical time period. The historical statistical sub-data includes a timestamp, and each historical statistical sub-data has a different timestamp.

[0152] Optionally, the historical statistical data node corresponds to a node level, and the historical statistical data node can be used to store historical statistical sub-data.

[0153] Optionally, the historical statistical sub-data with a timestamp closest to the current timestamp is stored in a historical statistical data node with a higher node level.

[0154] For illustration, please refer to Figure 5 There are three levels of historical statistical data nodes in the historical statistical data 500, among which the node level rule is level 3>level 2>level 1. The latest statistical data is always stored in the historical statistical data node represented by level 3, and different historical statistical sub-data are stored in the three historical statistical data nodes of different levels.

[0155] Each historical statistical sub-data contains: num (total number of records), value (president count), ts (last timestamp), and status (data status). The total number of records refers to the total number of target data counted from the first count to the last timestamp; the presidential count refers to the total statistical value from the first count to the last timestamp. For example, if the target object has followed 1,000 live broadcast rooms from the beginning to the current time (excluding those that have been unfollowed), the presidential count is 1,000; the last timestamp refers to the timestamp of the most recent update of the historical statistical sub-data. Specifically, it can be the timestamp corresponding to the target data when the most recent target data statistics were accumulated; the data status is used to indicate whether the historical statistical sub-data is valid or invalid.

[0156] In some optional embodiments, historical statistical sub-data in historical statistical data nodes with high node levels are periodically saved to historical statistical data nodes with low node levels. This process may include the steps of: obtaining the first historical statistical sub-data at the i-th node level; obtaining the second historical statistical sub-data at the i+1-th node level, where the i-th node level is higher than the i+1-th node level; i is a positive integer greater than 1; and updating the first historical statistical sub-data at the i-th node level to the i-1-th node level and updating the second historical statistical sub-data at the i+1-th node level to the i-th node level according to a preset period.

[0157] For illustration, please refer to Figure 5 Every 10 minutes, the historical statistical sub-data in the level 2 historical statistical data node is updated to the historical statistical sub-data in the level 3 historical statistical data node; the historical statistical sub-data in the level 1 historical statistical data node is updated to the historical statistical sub-data in the level 2 historical statistical data node.

[0158] 2. Periodic historical statistical data structure

[0159] Optionally, the historical statistical data structure includes a header and at least one window node.

[0160] For illustration, please refer to Figure 6 The head contains: step_type (step type), time_zone (statistical time zone), node_num (number of window nodes), last_ts (latest timestamp), and top_idx (top node sequence number). The step type indicates the statistical period of each node (window node). For example, if the step type is monthly, then one window node will count the flow data generated in one month. The statistical time zone is used to indicate the world time zone. The number of window nodes refers to the number of historical statistical data that can be stored in the historical statistical data structure, which is used to determine the space size of the historical statistical data structure. The latest timestamp refers to the timestamp corresponding to the most recently updated historical statistical data. The top node sequence number indicates the window node represented by the historical statistical data structure. For example, if the top node sequence number is 3, then the window node represented by the historical statistical data structure is node3. By changing the top node sequence number, the window in the historical statistical data structure can be slid.

[0161] Each window node contains: num (total number of records), value (count of presidents), and status (data status). When the step type is implemented as monthly, the total number of records refers to the number of target data counted in a month; the count of presidents refers to the total statistical value for a month; and the data status indicates whether the window node stores valid or invalid data.

[0162] Optionally, before combining the target data with the historical statistical data stored in the first storage space to obtain the target statistical data, it is necessary to query the first storage space for historical statistical data associated with the target data.

[0163] Optionally, in response to querying historical statistical data associated with target data, when the target data is data that has been rolled back, it is necessary to perform rollback processing on the historical statistical data.

[0164] Optionally, the rollback process refers to setting the status information corresponding to the historical statistical data to invalid in response to a corresponding timestamp in the historical statistical data being greater than a timestamp of the current target data.

[0165] Indicatively, (1) Figure 5 As shown in FIG, when the historical statistical data structure is implemented as a non-periodic historical statistical data structure, the data status of the historical statistical sub-data in the level 3 historical statistical data node is first set to invalid; then the timestamps of the historical statistical sub-data in the level 2 and level 1 historical statistical data nodes are checked in sequence. If the last timestamp is greater than the timestamp of the current target data, the data status of the historical statistical sub-data in the level 2 and level 1 historical statistical data nodes is set to invalid. (2) Figure 6As shown, when the historical statistical data structure is implemented as a periodic historical statistical data structure, if the latest timestamp stored in the header is greater than the timestamp of the current target data, the window node is found according to the top node sequence number, and the data status in the window node is set to invalid.

[0166] Step 406: Update and store the target statistical data in the first storage space.

[0167] The method for storing target statistical data has been described in step 205 and will not be repeated here.

[0168] The first storage space is used to store statistical data for query.

[0169] In some optional embodiments, the query requester may send a statistical query request to query statistical data, and the step of querying statistical data may include at least one of the following situations:

[0170] 1. Receive a statistical data query request; perform a data query on a first storage space based on the statistical data query request; and provide feedback on the statistical data in response to statistical data corresponding to the statistical data query request being found in the first storage space.

[0171] 2. Receive a statistical data query request; perform a data query on the first storage space based on the statistical data query request; in response to not finding statistical data corresponding to the statistical data query request in the first storage space, obtain statistical results from the database according to the data query request, write them into the cache, and provide feedback on the statistical results.

[0172] 3. Receive a statistical data query request; perform a data query on the first storage space based on the statistical data query request; in response to statistical data corresponding to the statistical data query request being found in the first storage space, but status information in the statistical data indicates that the statistical data is invalid, obtain statistical results from the statistical query in the database according to the data query request, write them into the cache, and provide feedback on the statistical results.

[0173] Optionally, invalid statistical data also needs to be repaired. Schematically, when the statistical data structure is implemented as a non-periodic statistical data structure, when repairing the statistical data, the statistical results are statistically queried in the database according to the data query request, and the statistical results are written to the highest-level statistical data node in the non-periodic statistical data structure to complete the repair of the statistical data.

[0174] To sum up, the embodiment of the present application directly obtains the target data by parsing the log files of the database, counts the target data according to the statistical configuration information, and pre-stores the statistical results. It can achieve statistics only on the data that has query requirements, saves computer storage resources, improves query efficiency, and reduces query latency by pre-storing statistical results.

[0175] The embodiment of the present application designs a data structure of multi-level nodes. When an exception occurs in the push of target data, the problem of duplicate statistics is avoided through strict incremental checking and rollback operations of timestamps, thereby saving storage resources and improving work efficiency.

[0176] Figure 7 This is a working principle diagram of a data statistics method provided by an exemplary embodiment of the present application. Figure 7 The working principle is explained in Figure 7 As shown:

[0177] Step 1: Write the transaction data generated by the business terminal into the transaction database through the business background.

[0178] Figure 7 There are three transaction databases shown in the figure, among which the business terminal 711 is connected to the transaction database 712, that is, the transaction data generated in the business terminal 711 will be written into the transaction database 712, the business terminal 713 is connected to the transaction database 714, that is, the transaction data generated in the business terminal 713 will be written into the transaction database 714, and the business terminal 715 is connected to the transaction database 716, that is, the transaction data generated in the business terminal 715 will be written into the transaction database 716.

[0179] Step 2: Binlog parser 717 obtains target data in real time by directly monitoring and parsing the Binlog logs in flow database 712, flow database 714 and flow database 716.

[0180] Step 3: The Binlog parser 717 writes the obtained target data into the Apache Kafka message queue for caching.

[0181] Step 4: The real-time stream consumption service 718 reads the target data by subscribing to Apache Kafka, serially processes the target data with the same object identifier according to hash routing, and distributes the target data to the data statistics writing service 719 for statistical processing.

[0182] Step 5: After receiving the target data pushed by the real-time flow consumption service 718, the data statistics writing service 719 first queries the corresponding historical statistical data from the statistical data cache 720. If the historical statistical data exists and is in normal status, the value of the current target data is added to the historical statistical data to obtain the target statistical data, and then the target statistical data is written back to the statistical data cache 720. If no historical statistical data is found or the found historical statistical data is invalid, it is necessary to handle it in two cases according to the statistical demand information in the business statistics configuration 721:

[0183] Case 1: The statistical requirement information indicates that statistics on the target data are not required, and the current target data is directly discarded;

[0184] Case 2: The statistical requirement information indicates that statistics need to be collected on the target data. According to the business statistics configuration 721 , the statistical query results are obtained from the flow database and written into the statistical data cache 720 .

[0185] It is worth noting that if the current historical statistical data status is normal, but the timestamp of the target data is earlier than the timestamp saved in the historical statistical data, it may be because duplicate target data has been pushed. In this case, the current historical statistical data status needs to be set to invalid.

[0186] Steps 1 to 5 above are the basic statistical processing process for the flow data generated by the business terminal when running the target application, that is, the pre-statistical process of the target data. During the pre-statistical process of the target data, the data statistics reading service 722 is responsible for processing the statistical query request of the business party.

[0187] After receiving the business statistics query request, the data statistics read service 722 will first try to read the historical statistical data from the statistical data cache 720 according to the request parameters. If the historical statistical data is read and the status of the historical statistical data is normal, the result will be directly returned to the requester; if the historical statistical data is not read, the data statistics write service 719 will be requested to complete the first statistics. At this time, the data statistics write service 719 needs to query the results from the flow database according to the business statistics configuration 721 and write them into the statistical data cache 720. Then the data statistics read service 722 will return the result of the first statistics to the requester; if the historical statistical data read is invalid, the data statistics write service 719 will be requested to repair the historical statistical data and return the repaired statistical data to the requester.

[0188] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0189] Please refer to Figure 8, which shows a block diagram of a data statistics device provided by an exemplary embodiment of the present application. The device has the function of implementing the above-mentioned data statistics method, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 800 may include:

[0190] An acquisition module 810 is configured to acquire a log file of a database, wherein the log file is configured to record data updates of the database;

[0191] A parsing module 820 is configured to parse the log file and obtain target data in the log file, where the target data is updated data in the log file;

[0192] The acquisition module 810 is further configured to acquire statistical configuration information, wherein the statistical configuration information is a pre-set data statistics scheme;

[0193] A processing module 830 is configured to combine the target data with the historical statistical data stored in the first storage space based on the statistical configuration information to obtain target statistical data;

[0194] The processing module 830 is further configured to update and store the target statistical data in the first storage space, where the first storage space is configured to pre-store statistical data for query.

[0195] Please refer to Figure 9 In an exemplary embodiment, the acquisition module 810 includes:

[0196] a matching submodule 811, configured to match the target data with the historical statistical data of the first storage space;

[0197] The acquisition module 810 is further configured to acquire the statistical configuration information in response to a failure in matching the target data with the historical statistical data.

[0198] In an exemplary embodiment, the target data corresponds to a first object identifier and first statistical item information, and the historical statistical data corresponds to a second object identifier and second statistical item information; the matching submodule 811 is also used to match the first object identifier with the second object identifier; and match the first statistical item information with the second statistical item information.

[0199] Optionally, the acquisition module 810 is further configured to acquire the statistical configuration information in response to a failure in matching between the first object identifier and the second object identifier; or a failure in matching between the first statistical item information and the second statistical item information.

[0200] In an exemplary embodiment, the historical statistical data corresponds to status information, and the status information is used to indicate the validity of the historical statistical data; the acquisition module 810 is also used to obtain the statistical configuration information in response to the status information corresponding to the historical statistical data indicating that the historical statistical data is invalid.

[0201] In an exemplary embodiment, the processing module 830 is further configured to combine the target data with the historical statistical data in a specified combination manner to obtain the target statistical data in response to a successful match between the target data and the historical statistical data.

[0202] In an exemplary embodiment, the historical statistical data includes at least one historical statistical data node corresponding to at least one historical statistical sub-data, and the historical statistical data node corresponds to a node level; the acquisition module 810 is also used to obtain the first historical statistical sub-data at the i-th node level; obtain the second historical statistical sub-data at the i+1-th node level, and the i-th node level is higher than the i+1-th node level; i is a positive integer greater than 1.

[0203] Optionally, the apparatus 800 further includes:

[0204] The updating module 840 is configured to update the first historical statistical sub-data of the i-th node level to the i-1-th node level and update the second historical statistical sub-data of the i+1-th node level to the i-th node level according to a preset period.

[0205] In an exemplary embodiment, the apparatus 800 further includes:

[0206] The writing module 850 is used to write the target data into the data statistics writing service, and the data statistics writing service is used to cache the target data before statistics are performed.

[0207] In an exemplary embodiment, the writing module 850 is further configured to write the target data into a rollback queue in response to a failure in writing the target data into the data statistics writing service.

[0208] Optionally, the writing module 850 includes:

[0209] The rollback submodule 851 is used to roll back the to-be-rolled-back data from the rollback queue to the data statistics writing service in sequence;

[0210] The deletion submodule 852 is configured to delete the target data from the rollback queue in response to the target data being rolled back successfully.

[0211] In an exemplary embodiment, the acquisition module 810 further includes:

[0212] A monitoring submodule 812 is used to monitor the log file;

[0213] The processing submodule 813 is configured to parse the log file in response to a content update in the log file to obtain the target data in the log file.

[0214] In an exemplary embodiment, the database includes a flow database corresponding to at least two business servers.

[0215] In an exemplary embodiment, the apparatus 800 further includes:

[0216] Receiving module 860, configured to receive a statistical data query request;

[0217] A query module 870 is configured to perform a data query on the first storage space based on the statistical data query request;

[0218] The feedback module 880 is configured to provide feedback on the statistical data in response to finding statistical data corresponding to the statistical data query request in the first storage space.

[0219] To sum up, the embodiment of the present application directly obtains the target data by parsing the log files of the database, counts the target data according to the statistical configuration information, and pre-stores the statistical results. It can achieve statistics only on the data that has query requirements, saves computer storage resources, improves query efficiency, and reduces query latency by pre-storing statistical results.

[0220] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data statistics apparatus and the data statistics method embodiment provided in the above embodiments are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0221] In an exemplary embodiment, a computer device is further provided. The computer device includes a processor and a memory. A computer program is stored in the memory. The computer program is loaded and executed by the processor to implement the above-mentioned data statistics method.

[0222] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein the storage medium stores a computer program, which is loaded and executed by a processor to implement the above-mentioned data statistics method. Optionally, the above-mentioned computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, optical data storage device, etc.

[0223] In an exemplary embodiment, a computer program product is further provided. The computer program product includes computer instructions. The computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-mentioned data statistics method.

[0224] It should be understood that the "multiple" mentioned in this article refers to two or more. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution order between the steps. In some other embodiments, the above steps may also be executed in a non-numbered order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the opposite order to that shown in the figure. This embodiment of the application is not limited to this.

[0225] The above are merely exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data statistics method, characterized in that: The method comprises: Obtaining a log file of a database, wherein the log file is used to record data updates of the database; Parsing the log file to obtain target data in the log file, where the target data is updated data in the log file; matching the target data with historical statistical data of the first storage space; In response to a failure in matching the target data with the historical statistical data, obtaining statistical configuration information, the statistical configuration information being a pre-set data statistical scheme, the statistical configuration information being used to indicate statistical requirements for the target data; combining the target data with historical statistical data stored in the first storage space based on the statistical configuration information to obtain target statistical data; and updating and storing the target statistical data in the first storage space, the first storage space being used to store statistical data for query; In response to a successful match between the target data and the historical statistical data, when the target data is data that has been rolled back and the timestamp of the historical statistical data is greater than the timestamp of the target data, setting the status information corresponding to the historical statistical data to invalid; the timestamp of the target data indicates the generation time of the target data, and the timestamp of the historical statistical data indicates the latest update time of the historical statistical data; receiving a statistical data query request; and performing a data query on the first storage space based on the statistical data query request; In response to finding statistical data corresponding to the statistical data query request in the first storage space and the statistical data being invalid, statistical results are queried in the database according to the statistical data query request, the statistical data are repaired based on the statistical results, and feedback is provided on the statistical results.

2. The method according to claim 1, characterized in that The target data corresponds to a first object identifier and first statistical item information, and the historical statistical data corresponds to a second object identifier and second statistical item information; Matching the target data with historical statistical data of the first storage space includes: Matching the first object identifier with the second object identifier; matching the first statistical item information with the second statistical item information; The acquiring of the statistical configuration information in response to a failure in matching the target data with the historical statistical data includes: In response to a failure in matching the first object identifier with the second object identifier; or a failure in matching the first statistical item information with the second statistical item information, the statistical configuration information is acquired.

3. The method according to claim 1, characterized in that The historical statistical data corresponds to status information, and the status information is used to indicate the validity of the historical statistical data; The acquiring of the statistical configuration information in response to a failure in matching the target data with the historical statistical data includes: In response to the status information corresponding to the historical statistical data indicating that the historical statistical data is invalid, the statistical configuration information is acquired.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to a successful match between the target data and the historical statistical data, the target data is combined with the historical statistical data in a specified combination manner to obtain the target statistical data.

5. The method according to any one of claims 1 to 3, characterized in that: The historical statistical data includes at least one historical statistical data node corresponding to at least one historical statistical sub-data, and the historical statistical data node corresponds to a node level; The method further comprises: Obtaining first historical statistical sub-data at the i-th node level; obtaining second historical statistical sub-data at the i+1-th node level, wherein the i-th node level is higher than the i+1-th node level; i is a positive integer greater than 1; According to a preset period, the first historical statistical sub-data of the i-th node level is updated to the i-1-th node level, and the second historical statistical sub-data of the i+1-th node level is updated to the i-th node level.

6. The method according to any one of claims 1 to 3, characterized in that: After parsing the log file to obtain the target data in the log file, the method further includes: The target data is written into a data statistics writing service, where the data statistics writing service is used to cache the target data before statistics are taken.

7. The method according to claim 6, characterized in that Writing the target data into the data statistics writing service includes: In response to a failure in writing the target data into the data statistics write service, writing the target data into a rollback queue; Rolling back the data to be rolled back from the rollback queue to the data statistics writing service in sequence; In response to the target data being rolled back successfully, the target data is deleted from the rollback queue.

8. The method according to any one of claims 1 to 3, characterized in that: Parsing the log file to obtain target data in the log file includes: Monitoring the log file; In response to the log file having content updated, the log file is parsed to obtain the target data in the log file.

9. The method according to any one of claims 1 to 3, characterized in that: The database includes a flow database corresponding to at least two business servers.

10. A data statistics device, characterized in that: The device comprises: An acquisition module is used to acquire a log file of a database, wherein the log file is used to record data updates of the database; A parsing module, configured to parse the log file and obtain target data in the log file, where the target data is updated data in the log file; The acquisition module is further configured to match the target data with historical statistical data of the first storage space; in response to a failure of matching the target data with the historical statistical data, acquire statistical configuration information, wherein the statistical configuration information is a pre-set data statistical scheme and is used to indicate statistical requirements for the target data; a processing module configured to combine the target data with historical statistical data stored in a first storage space based on the statistical configuration information to obtain target statistical data; and update and store the target statistical data in the first storage space, where the first storage space is used to store statistical data for query; The acquisition module is further configured to, in response to a successful match between the target data and the historical statistical data, set the status information corresponding to the historical statistical data to invalid when the target data is rolled back and the timestamp of the historical statistical data is greater than the timestamp of the target data; the timestamp of the target data indicates the generation time of the target data, and the timestamp of the historical statistical data indicates the latest update time of the historical statistical data; A receiving module, configured to receive a statistical data query request; A query module, configured to perform data query on the first storage space based on the statistical data query request; A feedback module is used to, in response to finding statistical data corresponding to the statistical data query request in the first storage space, and the statistical data is invalid, statistically query the statistical results in the database according to the statistical data query request, repair the statistical data based on the statistical results, and provide feedback on the statistical results.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the data statistics method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the data statistics method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the data statistics method according to any one of claims 1 to 9 when executed by a processor.

Images