Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: based on an account identifier of a target account, determining a predicted data volume which needs to be stored by the target account; according to the predicted data volume, determining a storage space allocated to the target account, the storage space being located in a cache database, and the storage space being used for storing data generated by the target account within a predetermined time period; receiving and storing data generated by the target account; under the condition that the state of the storage space meets a preset rule, performing statistics on the data stored in the storage space to obtain a statistical result; and storing the statistical result in a memory database. The technical problem of low data processing efficiency in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and information technology, and specifically, to a data processing method, device, non-volatile storage medium and electronic device. Background Art

[0002] As the financial technology field continues to deepen, it also brings huge amounts of transaction data. Traditional transaction data processing methods rely on frequent reading and writing of the database, which not only increases the database load and leads to performance bottlenecks, but also because the data storage adopts a relatively redundant format, resulting in large storage space consumption and low data statistics efficiency. The method of directly storing each transaction data in the database not only causes a high frequency of database write operations, but also requires a large amount of data retrieval and processing when performing transaction volume statistics, which is inefficient and occupies a lot of computing resources. In addition, for the storage of transaction status, the traditional method does not fully utilize the characteristics of binary bits, resulting in a waste of storage space, especially when processing massive data, the storage cost increases significantly, and there are technical problems of low data processing efficiency in related technologies.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, non-volatile storage medium and electronic device to at least solve the technical problem of low data processing efficiency existing in the related art.

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, including: determining a predicted amount of data that needs to be stored in a target account based on an account identifier of the target account; determining a storage space to be allocated to the target account according to the predicted amount of data, wherein the storage space is in a cache database, and the storage space is used to store data generated by the target account within a predetermined time period; receiving and storing the data generated by the target account; when a state of the storage space satisfies a predetermined rule, performing statistics on the data stored in the storage space to obtain statistical results; and storing the statistical results in a memory database.

[0006] Optionally, the storage space is a bitmap structure using a binary bit storage method, the data generated by the target account is transaction data, and the receiving and storing of the data generated by the target account includes: determining an identifier of a transaction status corresponding to the transaction data; and storing the identifier in a bit in the storage space according to the order in which the transaction data is received.

[0007] Optionally, the method further includes: checking the amount of stored data in the storage space according to a predetermined period, wherein the predetermined time period includes at least one of the predetermined periods; and adjusting the storage space based on the amount of stored data.

[0008] Optionally, adjusting the storage space based on the amount of stored data includes: when the amount of stored data is greater than or equal to a first predetermined threshold, increasing the storage space, and / or reducing the predetermined time period.

[0009] Optionally, the method further includes: when the amount of stored data is less than a second predetermined threshold, reducing the storage space and / or increasing the predetermined time period, wherein the first predetermined threshold is greater than the second predetermined threshold.

[0010] Optionally, determining the storage space allocated to the target account according to the predicted data volume includes: in case there is a storage overflow record for the target account, determining the overflow frequency based on the storage overflow record; determining an initial space allocation strategy corresponding to the target account based on the predicted data volume; adjusting the initial space allocation strategy according to the overflow frequency to obtain a target space allocation strategy; and determining the storage space based on the target space allocation strategy.

[0011] Optionally, the method also includes: obtaining a query request for the target account; when the query time range indicated in the query request exceeds the predetermined time period, based on a first time period in the query time range that coincides with the predetermined time period, performing statistics on the data stored in the storage space to obtain a first query result; based on a second time period in the query time range that exceeds the predetermined time period, querying the historical statistical results stored in the in-memory database to obtain a second query result; and obtaining a total query result for the query request based on the first query result and the second query result.

[0012] According to another aspect of an embodiment of the present application, a data processing device is provided, including: a data volume determination module, used to determine the predicted data volume that needs to be stored in the target account based on the account identifier of the target account; a storage space allocation module, used to determine the storage space allocated to the target account according to the predicted data volume, wherein the storage space is in a cache database, and the storage space is used to store data generated by the target account within a predetermined time period; a data receiving module, used to receive and store the data generated by the target account; an acquisition result module, used to count the data stored in the storage space to obtain statistical results when the state of the storage space meets a predetermined rule; and a database storage module, used to store the statistical results in a memory database.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any one of the data processing methods described above.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods described.

[0015] In the embodiment of the present application, the predicted amount of data that the target account needs to store is determined based on the account identifier of the target account; according to the predicted amount of data, the storage space allocated to the target account is determined, wherein the storage space is in a cache database, and the storage space is used to store the data generated by the target account within a predetermined time period; the data generated by the target account is received and stored; when the state of the storage space meets the predetermined rules, the data stored in the storage space is counted to obtain the statistical results; the statistical results are stored in the memory database. The purpose of efficient storage and rapid statistics of account data is achieved, and the technical effect of improving data processing efficiency is achieved, thereby solving the technical problem of low data processing efficiency existing in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a flowchart of an optional data processing method provided according to an embodiment of the present application;

[0018] Figure 2 is a first schematic diagram of an optional data processing method provided according to an embodiment of the present application;

[0019] Figure 3 is a second schematic diagram of an optional data processing method provided according to an embodiment of the present application;

[0020] Figure 4 is a third schematic diagram of an optional data processing method provided according to an embodiment of the present application;

[0021] Figure 5 is a fourth schematic diagram of an optional data processing method provided according to an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of an optional data processing device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0026] Redis (Remote Dictionary Server), or remote dictionary service, is a high-performance open source data structure server that supports network communication and can be used as an in-memory data storage system. It also has the ability of persistent storage, supports network, memory-based and persistent log-type, Key-Value (key-value pair) databases, and provides APIs (Application Programming Interface) in multiple languages. It can be stored in volatile memory to achieve high read and write speeds, and can also be saved to the hard disk regularly to prevent accidental data loss. In the field of financial technology, using Redis for real-time statistics of transaction data can quickly respond to business needs and reduce data latency. At the same time, storing transaction status through bitmaps can greatly save storage space and reduce storage costs.

[0027] A bitmap is a collection of bit-oriented operations on a string type. A string type is a binary object with a maximum length of 512MB (megabytes).

[0028] bitcount is an important command for bitmap data types in Redis. It is mainly used to count the total number of bits set to 1 in a given key. When a large amount of transaction information is stored in a bitmap, the bitcount command can quickly count the number of successful transactions within a certain time range.

[0029] According to an embodiment of the present application, an embodiment of a method for data processing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flowchart of an optional data processing method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S102, determining the amount of predicted data that needs to be stored in the target account based on the account identifier of the target account;

[0032] It can be understood that based on the account identifier of the target account, such as the merchant ID (identification), the system will pre-evaluate and determine the predicted data volume expected to be generated by the account in a specific time period. By analyzing historical transaction data, combined with the account's transaction scale, transaction frequency and other characteristics, intelligent estimation is performed to customize the most appropriate storage template for each account. By accurately predicting the transaction data volume of the target account and reasonably allocating storage resources, waste or shortage of resources is avoided.

[0033] Step S104, determining the storage space allocated to the target account according to the predicted data volume, wherein the storage space is in a cache database, and the storage space is used to store the data generated by the target account within a predetermined time period;

[0034] It can be understood that appropriate storage space is determined and allocated based on the predicted amount of data that the target account needs to store. The storage space is located in the cache database and is used to store the data generated by the target account within a predetermined time period. By allocating storage space to the account in the cache database, the response speed of data processing is improved and the delay of data storage is reduced.

[0035] Optionally, there can be multiple ways to allocate storage space. For example, the storage process includes allocating space, storage rules, overflow checking and storage. The storage process allocates storage space, stores transaction data, checks overflow and stores data according to the storage template of the merchant (i.e., the target account). When a new merchant joins the system, the system will evaluate the merchant's transaction volume based on its transaction volume estimate and determine its storage template. Initialize or update the storage template configuration based on the merchant scale. Allocate the corresponding storage space according to the template configuration. For each transaction data, success is recorded as 1 and failure is recorded as 0, and stored in the bitmap. Check whether the storage space is about to overflow. If the overflow frequency is high, update the template configuration, increase the memory or reduce the validity period. When the storage space is full or reaches the validity period, the stored bitmap data is counted and stored, and then the storage space is reallocated. Determine whether the validity period set in the storage template has been reached. Complete the data processing of the current storage cycle.

[0036] Optionally, the allocated space is configured according to the template, and the corresponding storage space is allocated for storage. The storage rule uses the merchant number as the key, and then according to each transaction, a successful transaction is recorded as 1, and a failed transaction is recorded as 0. The highest bit of the allocated storage block is the status bit, which can store up to 2 32 -1 bit. Overflow check performs binary AND judgment based on the allocated storage space to determine whether overflow is about to occur. If the overflow frequency becomes higher, the merchant storage template table is updated, a larger memory is reallocated or the effective time of the storage memory block is reduced, and then storage is restarted. For storage blocks that are full or have reached their effective time, the data in them is dropped from the warehouse, and then new storage blocks are reallocated for data storage.

[0037] Optionally, the memory size is in bits, with a minimum of 2. 3 bit, maximum 2 32 -1bit, the unit is bit, which means the basic unit of storage space is binary bit, not byte or larger unit, which makes the storage very compact, especially when processing binary attribute information such as transaction data (transaction success or failure), it can greatly save storage resources. Minimum is 2 3 bit, that is, 8 bits, can represent 2 3 = 8 states or 7 independent binary bits. This provides the lowest storage configuration for accounts with small transaction volumes, meeting data recording requirements while avoiding resource waste. Maximum 2 32 -1bit, that is, 4294967295 bits, minus the highest bit as the status bit, the actual number of available bits is 4294967294 bits. A very large storage space can be allocated to accounts with high transaction volumes, which is sufficient to cope with large-scale transaction data statistics needs, while also ensuring the continuity and integrity of data records.

[0038] Optionally, there can be multiple ways of recording time. For example, the creation time is calculated according to year, month, day, hour, minute, and second. Each record will record a timestamp accurate to the second when it is created. This precise time recording method is to facilitate subsequent data management and query, ensuring that the system can accurately track and manage the time points of each operation. The time format of year, month, day, hour, minute, and second also ensures the uniformity and readability of time records, which is convenient for manual review and automatic processing by the system.

[0039] In an optional embodiment, the storage space allocated to the target account is determined according to the predicted data volume, including: if there is a storage overflow record for the target account, determining the overflow frequency based on the storage overflow record; determining the initial space allocation strategy corresponding to the target account based on the predicted data volume; adjusting the initial space allocation strategy according to the overflow frequency to obtain the target space allocation strategy; and determining the storage space based on the target space allocation strategy.

[0040] It can be understood that when there are storage overflow records in the target account, the overflow frequency is analyzed based on these overflow records, that is, the number of times the storage space reaches the upper limit within a predetermined time period. Subsequently, based on the prediction of the amount of account transaction data, an initial space allocation strategy is preset, which specifies the number of storage bits and storage cycle of the account. The initial space allocation strategy is intelligently adjusted to form a more optimized target space allocation strategy. If the overflow frequency is high, it indicates that the initial strategy may underestimate the data generation rate of the account or overestimate the effectiveness of the storage cycle. According to the overflow frequency, the number of storage bits is increased or the storage cycle is shortened to ensure that the storage space can adapt to the amount of transaction data of the account, avoid frequent overflows, realize dynamic optimization of storage resources, and ensure efficient use of storage space and data integrity.

[0041] Optionally, there can be multiple overflow checking methods, for example, performing binary AND based on the allocated storage space to determine whether an overflow is about to occur, and if the overflow frequency becomes higher, updating the merchant storage template table, reallocating a larger memory or reducing the effective usage time of the storage memory block, and then restarting storage.

[0042] Step S106, receiving and storing data generated by the target account;

[0043] It can be understood that receiving and storing the data generated by the target account within a predetermined time period and efficiently and quickly receiving and storing the transaction data of the target account not only reduces the storage cost but also improves the speed and efficiency of data processing.

[0044] In an optional embodiment, the storage space is a bitmap structure using a binary bit storage method, the data generated by the target account is transaction data, and receiving and storing the data generated by the target account includes: determining an identifier of a transaction status corresponding to the transaction data; and storing the identifier in a bit in the storage space according to the order in which the transaction data is received.

[0045] It can be understood that the specific implementation form of the storage space is a bitmap structure in a binary bit storage mode, and the object of storage is transaction data. When the transaction data of the target account is received, the status identifier of the transaction is determined, that is, whether the transaction is successful or failed. Subsequently, according to the order in which the transaction data is received, the corresponding status identifier is stored in the bit of the bitmap structure in the form of binary bits, where a successful transaction corresponds to 1 and a failed transaction corresponds to 0. The bitmap structure uses binary bits to store data status, which reduces the storage space requirements, especially when processing massive transaction data, it can significantly reduce storage costs.

[0046] Optionally, the bitmap structure can be of multiple types, for example, a Redis bitmap. The bitmap structure uses a single bit (1 or 0) to store information, which is very suitable for fast query and update of large-scale data. When implemented based on the Redis bitmap, the data of each transaction will be mapped to a bit in the bitmap, with successful transactions marked as 1 and failed transactions marked as 0. Through commands such as bitcount provided by Redis, bit setting and statistical operations can be performed efficiently, and extremely low latency can be maintained even when processing massive amounts of data. Due to the compactness of the bitmap structure, a large amount of transaction information can be stored with minimal space consumption. In addition, Redis's high concurrency performance and fast response capabilities enable real-time data statistics and analysis in high-traffic transaction scenarios, thereby supporting real-time detection and policy adjustments.

[0047] In an optional embodiment, the method further includes: checking the amount of stored data in the storage space according to a predetermined period, wherein the predetermined time period includes at least one predetermined period; and adjusting the storage space based on the amount of stored data.

[0048] It is understandable that the amount of data stored in the storage space is checked regularly according to a predetermined cycle to ensure that the usage of the storage space can be obtained in a timely manner at any time. Based on the amount of stored data, the storage space is adjusted. Dynamic adjustment of storage space can avoid the waste or shortage of resources that may be caused by fixed strategies, and real-time detection and intelligent management of storage resources can effectively handle transaction data with large volatility.

[0049] In an optional embodiment, adjusting the storage space based on the amount of stored data includes: when the amount of stored data is greater than or equal to a first predetermined threshold, increasing the storage space, and / or reducing the predetermined time period.

[0050] It can be understood that if it is found during the detection process that the amount of stored data has reached or exceeded the first predetermined threshold, it means that the current storage configuration may not be sufficient to cope with the amount of transaction data generated by the target account. The system will automatically respond by increasing the storage space and reducing the predetermined time period to adjust the storage strategy, thereby ensuring the dynamic balance of storage resources and effectively responding to fluctuations in the amount of transaction data.

[0051] In an optional embodiment, the method further includes: when the amount of stored data is less than a second predetermined threshold, reducing the storage space, and / or increasing the predetermined time period, wherein the first predetermined threshold is greater than the second predetermined threshold.

[0052] It can be understood that during the detection process, if the amount of stored data is found to be lower than the second predetermined threshold, measures are automatically taken to optimize the storage strategy by reducing the storage space and increasing the predetermined time period. In this process, the first predetermined threshold is greater than the second predetermined threshold. By adjusting the storage space and storage period when the amount of stored data is lower than the predetermined threshold, increasing the predetermined time period means that data can be accumulated over a longer period of time, reducing the frequency of data falling into the database, thereby reducing the burden on the database.

[0053] Step S108, when the state of the storage space satisfies a predetermined rule, statistics are collected on the data stored in the storage space to obtain statistical results;

[0054] It can be understood that when the status of the storage space meets the predetermined rules, it automatically triggers statistical analysis of the data stored in the space to obtain statistical results about the transaction status of the target account in a specific time period, ensuring the timeliness and accuracy of data statistics and analysis.

[0055] In an optional embodiment, the method further includes: obtaining a query request for the target account; in a case where the query time range indicated in the query request exceeds a predetermined time period, based on a first time period in the query time range that coincides with the predetermined time period, performing statistics on data stored in the storage space to obtain a first query result; based on a second time period in the query time range that exceeds the predetermined time period, querying historical statistical results stored in the in-memory database to obtain a second query result; and obtaining a total query result for the query request based on the first query result and the second query result.

[0056] It can be understood that when a query request for a target account is received, the time range in the request is parsed. If this time range exceeds the current scheduled time period, the time range is divided into two parts: one part coincides with the current scheduled time period, namely the first time period, and the other part exceeds the current scheduled time period, namely the second time period. For the first time period that coincides with the current scheduled time period, the data in the corresponding time period in the storage space is directly counted to obtain the first query result. For the second time period that exceeds the current scheduled time period, the historical statistical results stored in the memory database are turned to for query to obtain the second query result. Based on the first query result and the second query result, the total query result for the query request is obtained. By means of segmented query and result aggregation, the problem of cross-time period data query is solved and the query efficiency is improved.

[0057] Optionally, there may be multiple transaction volume query rules, for example, determining whether the time range to be queried is completely within the current storage validity period, and if so, directly returning it through bitcount calculation; if it is archived historical data, statistics are performed through statistical transaction volume statistics table; if the time range is included in the current storage block validity period, the two blocks of content are summarized and counted.

[0058] Step S110, storing the statistical results in the memory database.

[0059] It can be understood that storing the obtained statistical results in the memory database ensures the persistence and accessibility of the statistical results and improves the security and reliability of the data.

[0060] Through the above step S102, based on the account identifier of the target account, the predicted amount of data that the target account needs to store is determined; step S104, according to the predicted amount of data, the storage space allocated to the target account is determined, wherein the storage space is in a cache database, and the storage space is used to store the data generated by the target account within a predetermined time period; step S106, receiving and storing the data generated by the target account; step S108, when the state of the storage space meets the predetermined rules, the data stored in the storage space is counted to obtain statistical results; step S110, storing the statistical results in the memory database, can achieve the purpose of efficient storage and rapid statistics of account data, and achieve the technical effect of improving data processing efficiency, thereby solving the technical problem of low data processing efficiency existing in related technologies.

[0061] Based on the above embodiments and optional embodiments, the present application proposes an optional implementation method, including: merchant storage template, overflow check, template table, transaction volume statistics table, and data statistics.

[0062] Step S1, merchant template configuration rules, when the merchant accesses, first conduct a preliminary assessment of the merchant's transaction volume, configure the template according to the assessed transaction volume and allocate the corresponding storage memory, set the storage time period, which represents how long the allocated memory can be used to store the merchant's transaction volume, and the configured merchant storage template is stored in the database. Table 1 is a merchant template table field provided according to an embodiment of the present application, as shown in Table 1. The creation time is calculated according to year, month, day, hour, minute, and second. The unit of allocated memory size is bit, with a minimum of 2 3 bit, maximum 2 32 -1 bit.

[0063] Table 1

[0064]

[0065] Take a store with id 1 as an example. The expected transaction volume is 1 million transactions per day, and the corresponding allocated storage space size is 2 22 bit, the minimum setting range of storage space is 2 3 bit, maximum 2 32 -1bit, used for storage time period of 3 days. The optional time interval pre-stored values ​​are 1 day, 3 days, 7 days, 30 days, and 180 days. The judgment is made based on the transaction scale of the merchant. If the transaction volume is small, 7 days of transaction data can be stored at one time and then saved in the database. The other optional time periods are handled in the same way.

[0066] The creation time is recorded in a custom format such as year, month, day, hour, minute, and second (indicated by "000000" in Table 1), and each record will be recorded with a timestamp accurate to the second when it is created. This precise time recording method is to facilitate subsequent data management and query, ensuring that the system can accurately track and manage the time points of each operation. The time format of year, month, day, hour, minute, and second also ensures the uniformity and readability of time records, which is convenient for manual review and automatic processing by the system.

[0067] The unit of allocated memory size is bit, the minimum is 2 3 bit, maximum 2 32 -1bit, the unit is bit, which means the basic unit of storage space is binary bit, not byte or larger unit, which makes the storage very compact, especially when processing binary attribute information such as transaction data (transaction success or failure), it can greatly save storage resources. Minimum is 2 3 bit, that is, 8 bits, can represent 2 3 = 8 states or 7 independent binary bits. This provides the lowest storage configuration for accounts with small transaction volumes, meeting data recording requirements while avoiding resource waste. Maximum 2 32-1bit, that is, 4294967295 bits, minus the highest bit as the status bit, the actual number of available bits is 4294967294 bits. A very large storage space can be allocated to accounts with high transaction volumes, which is sufficient to cope with large-scale transaction data statistics needs, while also ensuring the continuity and integrity of data records.

[0068] The bitmap structure can be a Redis bitmap. The bitmap structure uses a single bit (1 or 0) to store information, which is very suitable for fast query and update of large-scale data. When implemented based on the Redis bitmap, the data of each transaction will be mapped to a bit in the bitmap, with successful transactions marked as 1 and failed transactions marked as 0. Through commands such as bitcount provided by Redis, bit setting and statistical operations can be performed efficiently, and extremely low latency can be maintained even when processing massive amounts of data. Due to the compactness of the bitmap structure, a large amount of transaction information can be stored with minimal space consumption. In addition, Redis's high concurrency performance and fast response capabilities enable real-time data statistics and analysis in high-traffic transaction scenarios, thereby supporting real-time detection and policy adjustment.

[0069] Step S2, the storage process includes allocating space, storing rules, overflow checking and storage. Figure 2 is a first schematic diagram of an optional data processing method provided according to an embodiment of the present application, such as Figure 2 As shown, Figure 2 It is a storage process. The storage process allocates storage space, stores transaction data, checks overflow, and stores data according to the merchant's storage template. When a new merchant joins the system, the system will evaluate the merchant's transaction volume based on its transaction volume estimate and determine its storage template. Initialize or update the storage template configuration according to the merchant scale. Allocate the corresponding storage space according to the template configuration. For each transaction data, success is recorded as 1 and failure is recorded as 0, and stored in the bitmap. Check whether the storage space is about to overflow. If the overflow frequency is high, update the template configuration, increase the memory or reduce the validity period. When the storage space is full or reaches the validity period, the stored bitmap data is counted and stored, and then the storage space is reallocated. Determine whether the validity period set in the storage template has been reached. Complete the data processing of the current storage cycle.

[0070] Allocate space According to the template configuration, allocate the corresponding storage space for storage. The storage rule uses the merchant number as the key, and then according to each transaction, a successful transaction is recorded as 1, and a failed transaction is recorded as 0. The highest bit of the allocated storage block is the status bit, which can store up to 2 32-1 bit. Overflow check performs binary AND judgment based on the allocated storage space to determine whether overflow is about to occur. If the overflow frequency becomes higher, the merchant storage template table is updated, a larger memory is reallocated or the effective time of the storage memory block is reduced, and then storage is restarted. For storage blocks that are full or have reached their effective time, the data in them is dropped from the warehouse, and then new storage blocks are reallocated for data storage.

[0071] Step S3, the recording rules for transaction volume storage are formulated. When the storage block expires or is full, the corresponding transaction data statistics are stored in the database. The successful transaction volume set to 1 is calculated using bitcount, and the failed transaction volume is calculated by subtracting the successful transaction volume from the number of storage block bits. If the storage block expires, the highest status bit of the storage block and the last bit to be stored are recorded as 1 to facilitate the calculation of storage. Figure 3 is a second schematic diagram of an optional data processing method provided according to an embodiment of the present application, such as Figure 3 The figure shows the flow chart of transaction volume storage and recording. Successful transaction: recorded as 1 in the current position, failed transaction: recorded as 0 in the current position, successful transaction volume: counted by bitcount command, failed transaction volume: currently used storage position minus successful transaction volume position. It should be noted that the highest position is currently allocated as the status position, which is recorded as 0 by default. If the storage validity period expires and the storage is to be put into storage, the highest position status position is recorded as 1 as a mark.

[0072] Figure 4 is a third schematic diagram of an optional data processing method provided according to an embodiment of the present application, such as Figure 4 As shown in the figure, the recording process focuses on how transaction data is counted. The process starts, determines whether the highest bit is 1, and checks the highest bit status of the storage block. The successful transaction volume is counted using the bitcount command, and then the currently used storage bit is reduced by 1. The failed transaction volume is the current storage bit minus the successful transaction volume. The statistical results are stored in the database. The storage block data is completed and the validity period ends.

[0073] Step S4, transaction volume query rules, determine whether the time range to be queried is completely within the current storage validity period. If so, it is directly returned through bitcount calculation; if it is archived historical data, statistics are performed through statistical transaction volume statistics table; if the time range is included in the validity period of the current storage block, the two blocks of content are summarized and counted. Figure 5 is a fourth schematic diagram of an optional data processing method provided according to an embodiment of the present application, such as Figure 5 As shown, Figure 5It is a query flow chart. Start the process. Enter the query time, and the user specifies the data time period to be queried. Determine whether it is completely within the new storage time period, and check whether the query time is completely covered by the currently valid storage time period. If it is not completely within the new storage time period, further determine whether the query time is between the start time of the latest storage time period. If it is completely within the new storage time period, query the archived transaction volume statistics table during the query, and count the returned data. If it is between the start time of the latest storage time period, directly call the bitcount command to calculate and return the data in the storage time period. If it is completely within the new storage time period, call the bitcount command, count the return, and summarize the query results.

[0074] Step S5, Table 2 is a field of a warehouse statistics table provided according to an embodiment of the present application. As shown in Table 2, the transaction volume statistics table performs warehouse statistics based on merchants and storage validity periods, and records whether the warehouse is full and the highest bit status.

[0075] Table 2

[0076]

[0077] As can be seen from Table 2, the storage is recorded according to the storage time period on the template from the creation time. If it is three days, it is all the transactions from the 7th to the 9th. 2805616 means 2805616 successful transactions in these three days, and 1156 means 2805616 failed transactions in these three days. When the full flag is 1, it is determined that the allocated storage block (i.e. storage space) is full, otherwise the flag is recorded as 0. The highest bit state defaults to 0. It should be noted that if the storage time period expires and is not full, it is recorded as 1.

[0078] The above optional implementation method achieves at least the following effects: Redis is used for cache storage to greatly reduce the access to the database and reduce the database pressure; for massive data, the transaction data of the target merchant within a certain period of time can be quickly counted, including the success rate, failure rate, and the increase and decrease of transaction volume; bits are used for storage, and the corresponding Redis commands can be used to summarize and then store in the database, which can effectively reduce storage consumption; customized configuration based on the merchant dimension ensures that dynamic adjustments can be made according to the merchant scale in actual production.

[0079] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0080] In this embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0081] According to an embodiment of the present application, a device embodiment for implementing a data processing method is also provided. Figure 6 is a schematic diagram of a data processing device according to an embodiment of the present application, such as Figure 6 As shown, the above-mentioned data processing device includes a data volume determination module 602, a storage space allocation module 604, a data receiving module 606, a result acquisition module 608, and a database storage module 610. The device is described below.

[0082] A data volume determination module 602 is used to determine the predicted data volume that needs to be stored in the target account based on the account identifier of the target account;

[0083] The storage space allocation module 604 is connected to the data volume determination module 602 and is used to determine the storage space allocated to the target account according to the predicted data volume, wherein the storage space is in a cache database and is used to store the data generated by the target account within a predetermined time period;

[0084] A data receiving module 606, connected to the storage space allocation module 604, is used to receive and store data generated by the target account;

[0085] The result acquisition module 608 is connected to the data receiving module 606 and is used to perform statistics on the data stored in the storage space to obtain statistical results when the state of the storage space meets the predetermined rules;

[0086] The storage database module 610 is connected to the result acquisition module 608 and is used to store the statistical results in the memory database.

[0087] In a data processing device provided by an embodiment of the present application, a data volume determination module 602 is set to determine the predicted data volume that the target account needs to store based on the account identifier of the target account; a storage space allocation module 604 is connected to the data volume determination module 602, and is used to determine the storage space allocated to the target account according to the predicted data volume, wherein the storage space is in a cache database, and the storage space is used to store the data generated by the target account within a predetermined time period; a data receiving module 606 is connected to the storage space allocation module 604, and is used to receive and store the data generated by the target account; a result acquisition module 608 is connected to the data receiving module 606, and is used to count the data stored in the storage space when the state of the storage space meets the predetermined rules to obtain the statistical results; a database storage module 610 is connected to the result acquisition module 608, and is used to store the statistical results in the memory database. The purpose of efficiently storing and quickly counting account data is achieved, and the technical effect of improving data processing efficiency is achieved, thereby solving the technical problem of low data processing efficiency existing in the related technology.

[0088] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0089] It should be noted that the above-mentioned data volume determination module 602, storage space allocation module 604, data reception module 606, result acquisition module 608, and database storage module 610 correspond to steps S102 to S110 in the embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules as part of the device can be run in a computer account.

[0090] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0091] The above-mentioned data processing device may also include a processor and a memory. The data volume determination module 602, the storage space allocation module 604, the data receiving module 606, the result acquisition module 608, the storage database storage module 610, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0092] The processor includes a kernel, which retrieves the corresponding program unit from the memory. There can be one or more kernels. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0093] An embodiment of the present application provides a non-volatile storage medium on which a program is stored, and when the program is executed by a processor, a data processing method is implemented.

[0094] The embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program stored in the memory and can be run on the processor. When the processor executes the program, the following steps are implemented: based on the account identifier of the target account, determine the predicted amount of data that the target account needs to store; according to the predicted amount of data, determine the storage space allocated to the target account, wherein the storage space is in a cache database, and the storage space is used to store the data generated by the target account within a predetermined time period; receive and store the data generated by the target account; when the state of the storage space meets the predetermined rules, count the data stored in the storage space to obtain statistical results; store the statistical results in the memory database. The device in this article can be a server, a PC, etc.

[0095] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: based on the account identifier of the target account, determining the predicted amount of data that the target account needs to store; according to the predicted amount of data, determining the storage space allocated to the target account, wherein the storage space is in a cache database, and the storage space is used to store data generated by the target account within a predetermined time period; receiving and storing the data generated by the target account; when the state of the storage space meets the predetermined rules, counting the data stored in the storage space to obtain statistical results; and storing the statistical results in a memory database.

[0096] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0097] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0100] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0101] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0102] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0105] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Determining, based on the account identifier of the target account, the amount of predicted data that needs to be stored in the target account; Determine, according to the predicted data volume, a storage space allocated to the target account, wherein the storage space is in a cache database, and the storage space is used to store data generated by the target account within a predetermined time period; Receiving and storing data generated by the target account; When the state of the storage space satisfies a predetermined rule, statistics are performed on the data stored in the storage space to obtain a statistical result; The statistical results are stored in a memory database.

2. The method according to claim 1, characterized in that The storage space is a bitmap structure using a binary bit storage method, the data generated by the target account is transaction data, and the receiving and storing of the data generated by the target account includes: Determining an identifier of a transaction status corresponding to the transaction data; The identifiers are stored in bits in the storage space according to the order in which the transaction data are received.

3. The method according to claim 1, characterized in that The method further comprises: Checking the amount of stored data in the storage space according to a predetermined period, wherein the predetermined time period includes at least one of the predetermined periods; Based on the amount of stored data, the storage space is adjusted.

4. The method according to claim 3, characterized in that The adjusting the storage space based on the amount of stored data includes: When the amount of stored data is greater than or equal to a first predetermined threshold, the storage space is increased and / or the predetermined time period is reduced.

5. The method according to claim 4, characterized in that The method further comprises: When the amount of stored data is less than a second predetermined threshold, the storage space is reduced, and / or the predetermined time period is increased, wherein the first predetermined threshold is greater than the second predetermined threshold.

6. The method according to claim 1, characterized in that The step of determining the storage space allocated to the target account according to the predicted data volume includes: In the case where the target account has a storage overflow record, determining an overflow frequency based on the storage overflow record; Based on the predicted data volume, determining an initial space allocation strategy corresponding to the target account; According to the overflow frequency, the initial space allocation strategy is adjusted to obtain a target space allocation strategy; Based on the target space allocation strategy, the storage space is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining a query request for the target account; In the case where the query time range indicated in the query request exceeds the predetermined time period, based on a first time period in the query time range that coincides with the predetermined time period, statistics are collected on the data stored in the storage space to obtain a first query result; Based on a second time period in the query time range that exceeds the predetermined time period, query the historical statistical results stored in the memory database to obtain a second query result; A total query result for the query request is obtained according to the first query result and the second query result.

8. A data processing device, characterized in that: include: A data volume determination module, used to determine the predicted data volume that needs to be stored in the target account based on the account identifier of the target account; a storage space allocation module, configured to determine the storage space allocated to the target account according to the predicted data volume, wherein the storage space is in a cache database and is used to store data generated by the target account within a predetermined time period; A data receiving module, used for receiving and storing data generated by the target account; An acquisition result module, configured to perform statistics on the data stored in the storage space to obtain a statistical result when the state of the storage space satisfies a predetermined rule; The storage database module is used to store the statistical results in the memory database.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the data processing method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method described in any one of claims 1 to 7.

Images