Data processing methods, apparatus, equipment and storage media

Abstract

This application discloses a data processing method, apparatus, device, and storage medium. The method includes: acquiring real-time business data stored in a message queue; obtaining a total data volume corresponding to target dimension information based on a target data volume stored in memory and the real-time business data, wherein the target data volume is the data volume of historical business data corresponding to the target dimension information acquired from a storage device; and updating the data volume of the historical business data corresponding to the target dimension information in memory based on the total data volume corresponding to the target dimension information. Using this method can effectively improve the efficiency and convenience of data processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, with the continuous development of internet technology, the requirements for retrospective calculation of real-time business data are also constantly increasing. Users typically need to retrospectively analyze the amount of certain business data (e.g., the number of cases of a certain disease, sales data of a certain product, login count of a certain business platform, and click count of a certain advertisement) over a period of time. Since new data is constantly generated during the retrospective process, existing technologies store historical and new business data in a message queue according to their generation time, and then use real-time streaming computing to calculate the total amount of business data to be retrospectively analyzed. However, existing technologies suffer from low data processing efficiency and complex operations when the amount of data to be retrospectively analyzed is too large. Summary of the Invention

[0003] In view of this, embodiments of this application propose a data processing method, apparatus, device, and storage medium that can effectively improve the efficiency and convenience of data processing.

[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising: acquiring real-time business data stored in a message queue; obtaining a total amount of data corresponding to target dimension information based on a target amount of data stored in memory and the real-time business data, wherein the target amount of data is a pre-acquired amount of historical business data corresponding to the target dimension information; and updating the amount of historical business data corresponding to the target dimension information in memory based on the total amount of data corresponding to the target dimension information.

[0005] Secondly, embodiments of this application provide a data processing apparatus, comprising: a data acquisition module, a total data volume acquisition module, and a data volume update module. The data acquisition module is used to acquire real-time business data stored in a message queue; the total data volume acquisition module is used to obtain a total data volume corresponding to target dimension information based on a target data volume stored in memory and the real-time business data, wherein the target data volume is a pre-acquired data volume of historical business data corresponding to the target dimension information; the data volume update module is used to update the data volume of the historical business data corresponding to the target dimension information in memory based on the total data volume corresponding to the target dimension information.

[0006] In one possible implementation, the data processing device further includes a data volume acquisition module and a first storage module. The data volume acquisition module is used to acquire the data volume of historical business data corresponding to each dimension based on the total historical business data. The first storage module is used to store the data volume of historical business data corresponding to each dimension to a storage device, wherein the data volume of historical business data corresponding to each dimension includes the data volume of historical business data corresponding to the target dimension, and the target data volume stored in memory is the data volume of historical business data corresponding to the target dimension, which is pre-acquired from the storage device.

[0007] In one possible implementation, the data volume acquisition module is also used to perform batch processing calculations on the full historical business data to obtain the data volume of the historical business data corresponding to each dimension of information.

[0008] In one possible implementation, the first storage module is further configured to store the amount of historical business data corresponding to each dimension of information in the form of key-value pairs in the distributed storage device.

[0009] In one possible implementation, the historical full-volume business data is business data collected before a specified time. The data acquisition module is also used to acquire specified business data stored in the message queue as real-time business data, wherein the data acquisition time of the specified business data is after the specified time.

[0010] In one possible implementation, the data processing device further includes a second storage module, which is used to retrieve the amount of historical business data corresponding to the target dimension information from the storage device and store it in the memory if the amount of historical business data corresponding to the target dimension information is not stored in the memory.

[0011] In one possible implementation, the data processing device further includes a confirmation module, which is used to confirm that there is no amount of historical business data corresponding to the target dimension information stored in the memory when the tag value corresponding to the target dimension information stored in the memory is not a first tag value.

[0012] In one possible implementation, the total data acquisition module includes a data acquisition unit and a data fusion unit. The data acquisition unit is used to obtain the data volume of real-time business data corresponding to the target dimension information based on the real-time business data; the data fusion unit is used to fuse the data volume of real-time business data corresponding to the target dimension information with the data volume of historical business data corresponding to the target dimension information to obtain the total data volume corresponding to the target dimension information.

[0013] In one possible implementation, the data processing device further includes a first sending module, configured to send the total amount of data corresponding to the target dimension information to a business storage system associated with the memory for querying.

[0014] In one possible implementation, the data processing device further includes a second sending module, configured to send the total amount of data corresponding to the target dimension information to the storage device, so that when the storage device receives the total amount of data corresponding to the target dimension information, it saves the total amount of data and the time of receiving the total amount of data.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0017] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device retrieves the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0018] This application provides a data processing method, apparatus, device, and storage medium. The method includes: acquiring real-time business data stored in a message queue; obtaining a total data volume corresponding to target dimension information based on a target data volume stored in memory and the real-time business data; wherein the target data volume is the data volume of historical business data corresponding to the target dimension information acquired from a storage device. Through this method, real-time streaming processing can be performed only on real-time business data during data backtracking, thereby effectively improving the efficiency and convenience of data processing. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This diagram illustrates data processing using the Lambda architecture.

[0021] Figure 2 This diagram illustrates data processing using the Kappa architecture.

[0022] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application is shown;

[0023] Figure 4 A schematic diagram illustrating data processing using the data processing method of this application is shown;

[0024] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this application is shown;

[0025] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application is shown;

[0026] Figure 7 A flowchart illustrating another data processing method provided in an embodiment of this application is shown;

[0027] Figure 8 This paper shows a connection block diagram of a data processing apparatus provided in an embodiment of this application;

[0028] Figure 9 A structural block diagram of an electronic device for performing the methods of embodiments of this application is shown. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] With the development of computer technology, more and more fields are using backtracking of business data over a period of time for data analysis. Examples include backtracking the number of cases and recoveries of a certain disease over a period of time, backtracking login volume or pageviews of specific data on a platform over a period of time, and backtracking sales and return volumes of a certain product. Currently, there are two common methods for backtracking business data over a period of time: one is using the Lambda architecture for data processing, and the other is using the Kappa architecture.

[0031] like Figure 1The diagram illustrates the use of the Lambda architecture for data processing to trace the volume of business data over a period of time. When using the Lambda architecture for data processing, the message queue transmits the received data to both the real-time stream computing framework and the batch processing framework. The real-time stream computing framework calculates the volume of business data for the day, while the batch processing framework calculates the volume of historical business data. After the calculations are completed, the data from both frameworks is transmitted to the storage system to obtain the final result. In the Lambda architecture, when calculating the total volume of business data for day T, since batch processing is performed daily, if it starts at 9:00 AM, it may not be completed until 1:00 PM. Therefore, the volume of business data from the morning perspective = (T-2) the volume obtained from batch processing + (T-1) the daily increment from real-time stream computing + (T) the daily increment from real-time stream computing. Conversely, the volume of business data from the afternoon perspective = (T-1) the historical total volume from batch processing + (T) the daily increment from real-time stream computing. Based on the above analysis, it is evident that the two sets of code used for batch processing and real-time stream processing may be maintained and developed by multiple individuals, which can easily lead to inconsistent computational results. While batch processing can handle complex data processing logic, real-time stream processing typically handles simpler computational logic. However, technical alignment is difficult, resulting in discrepancies in computational results and inefficient, complex computational processes.

[0032] like Figure 2 The diagram illustrates the data volume of business data processed using the Kappa architecture to trace back a period of time. In the Kappa architecture, during data processing, historical business data and new data are stored in a message queue in chronological order of their generation to obtain input data (Input_topic) for the real-time stream computing framework. This allows the message queue to send data to the framework in batches (e.g., batching the input data into Job_version_n and Job_version_n+1). The real-time stream computing method is then used to sequentially calculate the batch data input to the real-time stream computing framework, obtaining the calculation results (i.e., obtaining the output_table_n corresponding to the input data Job_version_n and the calculation result Output_table_n+1 corresponding to the input data Job_version_n+1). The calculation results obtained by the real-time stream computing framework for the batch data are then sequentially sent to the storage system, ultimately yielding the total amount of real-time business data to be traced back. During this calculation process, if the amount of historical business data is large, the message queue requires a significant amount of memory, and the computational efficiency is low when storing data in the message queue for calculation.

[0033] Based on the above analysis, it can be seen that the existing technology has problems such as low computational efficiency, complex calculation process, or consumption of a lot of message queue resources when performing backtracking calculations on business data.

[0034] In view of this, the inventors provide a data processing method, the method comprising: acquiring real-time business data stored in a message queue; obtaining the total amount of data corresponding to target dimension information based on the target amount of data stored in memory and the real-time business data, wherein the target amount of data is the amount of historical business data corresponding to the target dimension information acquired from the storage device; and updating the amount of historical business data corresponding to the target dimension information in memory based on the total amount of data corresponding to the target dimension information.

[0035] By employing the method of this application, during computation, only the real-time stream computing framework is needed to calculate the real-time business data in the message queue. That is, the process of obtaining the total data volume corresponding to the target dimension information based on the target data volume stored in memory and the real-time business data is performed by the real-time stream computing framework (running with real-time stream code). Therefore, compared with the existing technology that uses the Lambda architecture for data processing, this application does not need to consider the consistency of the logic of the batch processing calculation code and the real-time stream computing code, nor does it need to maintain and create two sets of code simultaneously. Thus, the labor cost can be reduced by about 50%. Moreover, the data processing method of this application is applicable to all scenarios that require real-time stream computing of historical total business data. In addition, since this application does not use batch processing data to replace real-time stream computing data, there will be no data jitter during the data processing process, so the final total data volume is more reliable.

[0036] Furthermore, compared to using the Kappa architecture for data processing, this application does not require the use of valuable message queue resources to store historical business data. The message queue only needs to store data for a few days. Since the storage of message queues is much more valuable than the storage resources of batch processing, this application can save a lot of message queue resources.

[0037] As described above, this application only requires real-time stream computing on the real-time business data stored in the message queue, and does not need to consider the consistency of the two sets of code logic: batch processing code and real-time stream computing code. Therefore, the data processing process of this application is simple, computationally efficient, and does not require excessive message queue resources.

[0038] It should be understood that the above-described data processing method can run on a server or terminal device. As an optional implementation, the server or terminal device can also serve as a node in a blockchain. The target data volume and real-time business data used in the data processing method disclosed in this application can be stored in the node, and the data processing method can be specifically executed by one or more nodes in the blockchain.

[0039] Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and to generate the next block.

[0040] The following section will first introduce the technical terms used in this application.

[0041] A message queue is a container that stores messages during transmission. A message refers to the data being transmitted; it can be a simple text string or a custom, complex format (as long as it can be parsed according to a predefined format). It's a first-in, first-out (FIFO) data structure. Messages are enqueued from the rear and dequeued from the front; enqueuing is the process of sending a message, and dequeuing is the process of receiving a message. The main purpose of a queue is to provide routing and ensure message delivery. Generally, if the receiver is unavailable when a message is sent, the message queue will retain the message until it can be successfully delivered. The data stored in a message queue is limited; typically, a message queue can store several days' worth of data, meaning it's usually only used to store a few days' worth of real-time business data.

[0042] Memory, also known as internal memory or main memory, is a crucial component of a computer. It temporarily stores data used for CPU operations and data exchanged with external storage devices such as hard drives. It acts as a bridge between external storage and the CPU. All programs in a computer run within memory, and the performance of the memory directly impacts the overall performance of the computer. Once the computer starts running, the operating system retrieves the necessary data from memory to the CPU for processing. After the processing is complete, the CPU sends the result back to the CPU.

[0043] Dimensional information refers to the criteria used to categorize business data based on type, region, or generation method. For example, when it's necessary to trace the number of patients with different types of cancer over a period of time, the dimensional information could include liver cancer, lung cancer, and lymphoma, etc.; similarly, when it's necessary to trace the purchase volume of different types of goods on a shopping platform over a period of time, the dimensional information could include daily necessities, food, and clothing, etc.; furthermore, when it's necessary to trace the number of patients with a certain type of influenza in different provinces over a period of time, the dimensional information could include Sichuan, Hunan, Beijing, and Chongqing, etc.; and when it's necessary to trace the interaction methods with a product advertisement over a period of time, the dimensional information could include clicks, likes, favorites, and purchases, etc.

[0044] Target dimension information refers to at least one dimension selected from multiple dimensions based on user or statistical needs. For example, when it's necessary to trace the number of liver cancer cases over a period of time, the target dimension information would specifically be liver cancer. Similarly, when it's necessary to trace the number of influenza cases in Sichuan and Hunan provinces over a period of time, the target dimension information would be Sichuan and Hunan. Furthermore, when it's necessary to trace clicks and likes on a product advertisement over a period of time, the target dimension information would include clicks and likes.

[0045] Business data refers to data generated by business systems (such as medical systems, shopping platform systems, or advertising platform systems), including the time of data generation and business information. For example, when the business system is a medical system, if it is necessary to trace the number of patients with different types of cancer, the business data generated by the business system will include the time of data generation and the type of disease. Similarly, when the business system is a shopping platform system, if it is necessary to trace the purchase volume of different types of goods, the business data generated by the business system will include the time of data generation and the type of goods purchased; the time of data generation can be the time of goods purchase.

[0046] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0047] Please see Figure 3 , Figure 3 The diagram shown is a flowchart of a data processing method applicable to an electronic device according to an embodiment of this application. The electronic device can be a server or a monitoring device. The method includes:

[0048] Step S110: Obtain the real-time business data stored in the message queue.

[0049] The business data stored in the message queue can include business data received within a certain period of time (e.g., within a few days or a few hours). Typically, business data can be obtained by business systems (e.g., game business systems, shopping platforms, and medical systems) and then sequentially transmitted to the message queue. Therefore, the business data stored in the message queue is stored sequentially according to the time when the business data was generated. Correspondingly, the message queue also stores the business data and the time when the business data was generated.

[0050] It should be understood that different business data may correspond to different dimensional information. For example, when the business system is a game business system, if the dimensional information includes game launch and game logout, the business data may specifically include the user's online time (the time when the business data is generated) and game launch information, and the business data may also include the user's offline time (the time when the business data is generated) and game logout information, etc. When the business system is a shopping platform, if the dimensional information includes daily necessities, food, and clothing, the business data may include the item type and the item purchase time (the time when the business data is generated), such as the purchase information of daily necessities, food, and clothing. When the business system is a medical system, if the dimensional information includes the province where the patient is located, the business data may include the province where the influenza patient is located and the time when the business data is generated.

[0051] Retrieving real-time business data stored in the message queue can be done in two ways: starting with the earliest data stored in the message queue and retrieving the data sequentially as real-time business data; or starting with the business data corresponding to a specified time in the message queue and retrieving the data sequentially as real-time business data.

[0052] In one possible implementation, step S110 includes: sequentially reading business data generated after a specified time from the message queue as real-time business data according to the order in which the business data was generated.

[0053] In this implementation, the specific method for obtaining real-time business data can be as follows: adjust the data reading position in the message queue to the position of the business data generated at the specified time, and start reading data from the business data at the specified time to obtain the real-time business data generated after the specified time in the message queue.

[0054] The specified time mentioned above can be a time within a day or a specific day (such as the start time of a day (00:00), 8:00 AM or 12:00 PM, etc.), a time within a week, or a time within an hour.

[0055] Step S120: Based on the target data volume stored in memory and the real-time business data, obtain the total data volume corresponding to the target dimension information. The target data volume is the amount of historical business data corresponding to the target dimension information that has been obtained in advance.

[0056] It should be understood that the data generation time of historical business data and real-time business data should be different. That is, if real-time business data is business data generated by the business system after a specified time, then the corresponding historical business data should be data generated by the business system before the specified time.

[0057] The target data volume can be one or multiple. It should be understood that when there are multiple target data volumes, there should also be multiple target dimension information, and each target dimension information corresponds to one target data volume.

[0058] For example, when there are multiple target data volumes, multiple threads are also created in memory. Each thread corresponds to a target data volume, and each target data volume corresponds to a dimension information. Each thread can execute the following: based on the target data volume stored in memory corresponding to the thread and the real-time business data, obtain the total amount of data corresponding to the target dimension information to which the target data volume belongs.

[0059] The amount of historical business data corresponding to the target dimension information stored in memory can be obtained in advance from the memory in the electronic device and stored in memory, or it can be obtained in advance from the storage device or cloud platform associated with the electronic device and stored in memory. It should be understood that the memory in the electronic device refers to the device with data storage function set in the electronic device other than memory and message queue.

[0060] Specifically, the memory or storage device can store the amount of historical business data corresponding to different dimensional information. The specific method for obtaining the amount of business data corresponding to different dimensional information can be: the processor in the electronic device classifies and statistically analyzes the total amount of historical business data generated by the business system before a specified time according to the dimensional information, and obtains the amount of business data corresponding to different dimensional information.

[0061] It should be understood that the aforementioned historical full-volume business data may include historical business data corresponding to different dimensions of information. This historical full-volume business data can be all business data obtained by the business system before a specified time, or business data obtained within a time period before the specified time.

[0062] To enable the data processing method of this application to trace back the total amount of business data within different time periods (such as within one month or one year before a specified time), the aforementioned historical full business data can be all historical business data generated by the business system within a preset time range before the specified time. Similarly, to enable the data processing method of this application to trace back the total amount of business data after a preset time (such as after the start time of the first day of a month), the aforementioned historical full business data can also be all historical business data generated by the business system within the time period between the specified time and the preset time.

[0063] There are several ways to obtain the total amount of data corresponding to the target dimension information, depending on the amount of target data stored in memory and real-time business data.

[0064] In one implementation, the real-time business data can be divided according to dimensional information to obtain the data volume of real-time business data corresponding to each dimensional information. The data volume of real-time business data corresponding to the target dimensional information is then merged with the data volume of historical business data corresponding to the target dimensional information to obtain the total data volume corresponding to the target dimensional information.

[0065] In another implementation, it can be: obtaining the amount of real-time business data corresponding to the target dimension information based on real-time business data; and merging the amount of real-time business data corresponding to the target dimension information with the amount of historical business data corresponding to the target dimension information to obtain the total amount of data corresponding to the target dimension information.

[0066] The aforementioned fusion can be achieved by summing the amount of real-time business data corresponding to the target dimension information and the amount of historical business data corresponding to the target dimension information.

[0067] Step S130: Update the amount of historical business data in memory corresponding to the target dimension information according to the total amount of data corresponding to the target dimension information.

[0068] Specifically, the method for updating the amount of historical business data in memory corresponding to the target dimension information based on the total amount of data corresponding to the target dimension information can be as follows: take the amount of current business data corresponding to the target dimension information as the amount of new historical business data corresponding to the target dimension information and store it in memory.

[0069] By updating the amount of historical business data in memory corresponding to the target dimension information, when new real-time business data transmitted by the business system is received in the message queue, the new real-time business data stored in the message queue can be obtained, and the new total amount of data corresponding to the target dimension information can be obtained based on the amount of this real-time business data and the amount of updated historical business data stored in memory.

[0070] To enable the total amount of business data obtained by backtracking each time data processing is performed during subsequent query calls, in this embodiment, the method further includes: sending the total amount of data corresponding to the target dimension information to the storage device, so that when the storage device receives the total amount of data corresponding to the target dimension information, it saves the total amount of data and the time of receiving the total amount of data.

[0071] To facilitate users' observation of the amount of data generated by the business system backtracking in real time or at intervals, in this embodiment, the method further includes: sending the total amount of data corresponding to the target dimension information to the business storage system associated with memory for querying.

[0072] As one possible implementation, the above method of sending the total amount of data corresponding to the target dimension information to the business storage system associated with memory can be to obtain the amount of data corresponding to the target dimension information in memory at preset intervals and send the amount of data corresponding to the target dimension information to the business storage system associated with memory.

[0073] This application provides a data processing method that obtains real-time business data stored in a message queue; based on the target data volume stored in memory and the real-time business data, it obtains the total data volume corresponding to the target dimension information, where the target data volume is the volume of historical business data corresponding to the target dimension information obtained from the storage device. This method enables data backtracking by using real-time stream computation on the real-time business data stored in the message queue, and obtaining the total data volume corresponding to the target dimension information based on the result of the real-time stream computation and the pre-obtained volume of historical business data corresponding to the target dimension information. This simplifies the data processing process, increases computational efficiency, and minimizes the use of message queue resources.

[0074] For details, please refer to the following: Figure 4 As shown, by adopting the data processing method of this application, during the data processing process, the real-time business data corresponding to different dimension information obtained from the real-time business data stored in the message queue is calculated by the processor in the electronic device calling the real-time stream computing framework in memory. The amount of historical business data corresponding to the target dimension information is pre-acquired and stored in memory. That is, the data processing method of this application does not need to consider the consistency of the two code logics of batch processing calculation code and real-time stream computing code. This makes the data processing method of this application more versatile and suitable for all scenarios that need to backtrack the total amount of historical business data. In the process of backtracking the total amount of historical business data, there is no need to use the message queue to store historical business data. Therefore, the data processing process is simple and computationally efficient, and does not require too much message queue resources.

[0075] Please see Figure 5This application provides a data processing method applicable to electronic devices, the method comprising:

[0076] Step S210: Obtain the amount of historical business data corresponding to each dimension of information based on the historical full business data.

[0077] Historical full-volume business data can refer to all historical business data generated by the business system before a specified time. It can also refer to historical business data generated by the business system between a specified time and a preset time, where the preset time is before the specified time. Alternatively, it can refer to historical business data generated by the business system within a preset time period before the specified time.

[0078] Considering that the business data generated by the business system has different dimensional information corresponding to different dimensions, and that the amount of business data generated by the business system is usually large, the above method of obtaining the amount of historical business data corresponding to each dimension information based on the full amount of business data can be: performing batch processing calculation on the full amount of historical business data to obtain the amount of historical business data corresponding to each dimension information.

[0079] Batch processing refers to the batch processing of a specific object (historical full-volume business data). It is generally considered a simplified scripting language used in DOS and Windows systems. Batch processing can be understood as a series of related tasks executed sequentially or in parallel, one after another. The input to batch processing is data collected over a period of time. The output of each batch processing iteration can become the input for the next.

[0080] In this implementation method, the historical full-volume business data can be divided in batches according to multiple dimensions of information to obtain the historical business data corresponding to each dimension of information, and the data volume of the historical business data corresponding to each dimension of information can be counted.

[0081] Step S220: Store the amount of historical business data corresponding to each dimension of information to the storage device.

[0082] The amount of historical business data corresponding to each dimension includes the amount of historical business data corresponding to the target dimension, and the amount of target data stored in memory is the amount of historical business data corresponding to the target dimension that is obtained in advance from the storage device.

[0083] There are several ways to store the amount of historical business data corresponding to each dimension of information in the storage device. For example, the amount of historical business data corresponding to each dimension of information can be stored in the storage device in a sequential storage manner, or in a linked storage manner, or in a key-value pair manner, or in a hash storage manner.

[0084] In one implementation, the amount of historical business data corresponding to each dimension of information can be stored in a distributed storage device in the form of key-value pairs.

[0085] Key-value pairs are the simplest form of database organization. A key is used to distinguish different objects (i.e., dimension information), while the value is the actual content of each object (i.e., the amount of historical business data). Because the key distinguishes objects, it is used to search for a specific element.

[0086] Step S230: Obtain the real-time business data stored in the message queue.

[0087] In one possible implementation, if the historical full-volume business data is business data collected before a specified time, then the above step S230 can be: obtaining the specified business data stored in the message queue as real-time business data, wherein the data collection time of the specified business data is after the specified time.

[0088] Step S240: Based on the target data volume stored in memory and the real-time business data, obtain the total data volume corresponding to the target dimension information. The target data volume is the amount of historical business data corresponding to the target dimension information that has been pre-obtained from the storage device and stored in memory.

[0089] Step S250: Update the amount of historical business data in memory corresponding to the target dimension information according to the total amount of data corresponding to the target dimension information.

[0090] This application provides a data processing method that obtains the amount of historical business data corresponding to each dimension based on historical full-volume business data, and stores the amount of historical business data corresponding to each dimension in a storage device. This allows for the determination of the total amount of data corresponding to the target dimension during data backtracking by simply using the amount of historical business data corresponding to each dimension stored in the storage device and the real-time business data stored in the message queue. This method efficiently recovers the total amount of data corresponding to the target dimension during data backtracking, and does not occupy the storage space of the message queue during the backtracking process.

[0091] Please see Figure 6This application also provides a data processing method applicable to electronic devices, the method comprising:

[0092] Step S310: Obtain the real-time business data stored in the message queue.

[0093] Step S320: Detect whether there is a certain amount of historical business data corresponding to the target dimension information stored in memory.

[0094] In one possible implementation, the method for detecting whether the memory stores the amount of historical business data corresponding to the target dimension information can be: detecting whether the memory stores the target dimension information stored in key-value pairs and the amount of historical business data corresponding to the target dimension information. Specifically, this method can involve detecting whether the memory stores the key (target dimension information) in the key-value pairs. If it does, the corresponding value (the amount of historical business data corresponding to the target dimension information) is found based on the key in the key-value pair. It should be understood that if no key is detected in the key-value pair or no value is found corresponding to the key, then the memory stores the amount of historical business data corresponding to the target dimension information.

[0095] In another possible implementation, the method to detect whether the memory stores the amount of historical business data corresponding to the target dimension information can be: different storage spaces are allocated in the memory for different dimension information, and by detecting whether the storage space corresponding to the target dimension information stores data, it can be confirmed whether the memory stores the amount of historical business data corresponding to the target dimension information.

[0096] In another possible implementation, the method for detecting whether the amount of historical business data corresponding to the target dimension information is stored in memory can be as follows: based on the marker value stored in memory corresponding to the target dimension information, it can be determined whether the amount of historical business data corresponding to the target dimension information is stored in memory. Specifically, a marker value of the first marker value indicates that the amount of historical business data corresponding to the target dimension information is stored in memory. That is, if the marker value stored in memory corresponding to the target dimension information is not the first marker value, it can be confirmed that the amount of historical business data corresponding to the target dimension information is not stored in memory.

[0097] The first flag value can be any constant such as 0, 1, or 2. For example, when the first flag value is 1, that is, when the flag value corresponding to the target dimension information in memory is 1, it represents the amount of historical business data corresponding to the target dimension information stored in memory.

[0098] If not stored, proceed to step S330: retrieve the amount of historical business data corresponding to the target dimension information from the storage device and store it in memory.

[0099] The storage device can be a memory in a server or computing device associated with an electronic device, a cloud platform or memory associated with an electronic device, or a device with data storage function other than memory and message queues installed in an electronic device.

[0100] It should be understood that the amount of historical business data corresponding to the target dimension information stored in the storage device can be obtained by using steps S210-S220 in the above embodiments. Therefore, for an explanation of the amount of historical business data corresponding to the target dimension information, please refer to the description of steps S210-S220 and the specific description of step S120 above. They will not be repeated here.

[0101] If the target dimension information contains the amount of historical business data, or if the target dimension information contains the amount of historical business data, and the data is stored in memory, then step S340 is executed: based on the target data stored in memory and the real-time business data, the total amount of data corresponding to the target dimension information is obtained.

[0102] Step S350: Update the amount of historical business data in memory corresponding to the target dimension information according to the total amount of data corresponding to the target dimension information.

[0103] This application provides a data processing method that obtains real-time business data stored in a message queue and checks whether the amount of historical business data corresponding to the target dimension information is stored in memory. If not, the method retrieves the amount of historical business data corresponding to the target dimension information from the storage device and stores it in memory. If the amount of historical business data corresponding to the target dimension information is stored, or if the amount of historical business data corresponding to the target dimension information is stored in memory, the method obtains the total amount of data corresponding to the target dimension information based on the amount of target data stored in memory and the real-time business data. This method simplifies the data processing process, increases computational efficiency, and minimizes the need to consume excessive message queue resources.

[0104] Please refer to the following: Figure 7 This application also provides a data processing method applicable to electronic devices, the method comprising:

[0105] Step S410: Perform batch processing calculations on all historical business data to obtain the data volume of historical business data corresponding to each dimension of information.

[0106] Specifically, if the current day is day T, step S410 can be as follows: All business data generated by the business system before the start time of day T-1 can be obtained as historical full-volume business data. Batch processing calculations can then be performed on this historical full-volume business data to obtain the data volume (value) of the historical business data corresponding to each dimension (Key). Since batch processing resources are relatively abundant, performing batch processing calculations using historical full-volume business data can improve the efficiency of obtaining the data volume of historical business data corresponding to each dimension.

[0107] Step S420: Store the amount of historical business data corresponding to each dimension of information in the form of key-value pairs on a distributed storage device.

[0108] Considering that batch processing takes a long time, step S420 above can specifically be: a scheduled task periodically writes the data volume (value) of the historical business data corresponding to each dimension information (Key) into the distributed storage system (key-value storage system).

[0109] Step S430: Obtain the real-time business data stored in the message queue.

[0110] Specifically, if the historical full-volume business data refers to all business data generated by the business system before the start time of day T-1, then the real-time business data stored in the message queue is obtained by sequentially obtaining the business data generated by the business system from the start time of day T-1 onwards. That is, during the acquisition process, the start time of day T-1 will be used as the consumption time point of the message queue.

[0111] Step S440: Obtain the amount of real-time business data corresponding to the target dimension information based on the real-time business data.

[0112] Step S450: Detect whether there is a certain amount of historical business data corresponding to the target dimension information stored in memory.

[0113] Specifically, step S440 can be: based on the tag value stored in memory corresponding to the target dimension information, confirm whether there is a data volume of historical business data corresponding to the target dimension information stored in memory, wherein when the tag value is the first tag value, it indicates that there is a data volume of historical business data corresponding to the target dimension information stored in memory.

[0114] For example, the tag value corresponding to the target dimension information may include a first tag value and a second tag value. When the tag value corresponding to the target dimension information is the first tag value, such as 1, it represents the amount of historical business data corresponding to the target dimension information stored in memory, that is, the amount of historical business data corresponding to the target dimension information that does not need to be loaded from the storage device. When the tag value corresponding to the target dimension information is the second tag value, such as 0, it represents the amount of historical business data corresponding to the target dimension information that is not stored in memory, that is, the amount of historical business data corresponding to the target dimension information that needs to be loaded from the storage device.

[0115] If not stored, proceed to step S460: retrieve the amount of historical business data corresponding to the target dimension information from the storage device and store it in memory.

[0116] If the data volume of historical business data corresponding to the target dimension information is stored, or after storing the data volume of historical business data corresponding to the target dimension information into memory, then step S470 is executed: the data volume of real-time business data corresponding to the target dimension information is merged with the data volume of historical business data corresponding to the target dimension information in memory to obtain the total data volume corresponding to the target dimension information.

[0117] Step S480: Update the amount of historical business data in memory corresponding to the target dimension information based on the total amount of data corresponding to the target dimension information.

[0118] After the update is completed, the method further includes: step S190: sending the total amount of data corresponding to the target dimension information to the business storage system associated with the memory for querying.

[0119] It should be understood that when there are multiple target dimension information, all or part of the steps in steps S410-S480 above should be executed for each dimension information to complete the acquisition and update of the total amount of data corresponding to each target dimension information.

[0120] For example, taking the current day as day T, this example uses electronic devices to backtrack the number of influenza cases in each province based on the influenza patient information (business data) generated in the medical system, categorized by different provinces (dimensional information). The influenza patient information includes user information, the province corresponding to that user, and the time the medical system generated the patient information. The specific data processing procedure is as follows:

[0121] Electronic devices can acquire all historical influenza patient information generated in the medical system up to day T-1 as historical full-volume business data, and then divide the historical full-volume business data into batches according to province information to obtain the corresponding historical influenza patient data volume for each province. The corresponding historical influenza patient data volume for each province is stored in the storage device in the form of key-value pairs.

[0122] Because the medical system generates influenza patient information in real time, electronic devices invoke real-time stream computing programs to run the program in memory. When tracing the amount of influenza patient information from different provinces across different time periods, the real-time stream computing program must be restarted. Therefore, when an electronic device needs to trace the amount of influenza patient information from a target province across different time periods, it checks whether the data is stored in memory. If not, it retrieves the data from storage and stores it in memory. This allows it to calculate the total amount of influenza patient information for the target province up to the current moment, obtaining the real-time influenza patient information data for the target province based on real-time business data. Finally, it adds the real-time influenza patient information data for the target province to the historical influenza patient information data for the target province corresponding to the target dimension information to obtain the total influenza patient information data for the target province. After a calculation is completed, the total amount of influenza patient information for the target province can be used as the amount of historical influenza patient information for the new target identity and stored in memory. This allows for the generation of a new total amount of influenza patient information for the target province when real-time influenza disease information is retrieved from the message queue. This process is repeated cyclically, generating the amount of influenza patient information for the target province in milliseconds. Finally, a timer can be used to write the result data for this key to the business's storage system for querying.

[0123] By adopting the method described in this application, the adaptability of real-time stream computing applications can be greatly improved after adjustments to business computing logic (i.e., when it is necessary to backtrack business data corresponding to different time periods or different target dimensions). Since it avoids the need to use real-time stream computing to calculate all historical full data up to day T-1 (inclusive) after adjustments to business computing logic, the data backtracking time is reduced from tens of days to the few hours required by this invention. Furthermore, as described above, batch processing and real-time stream computing can be executed step-by-step. Therefore, the data processing process only requires maintaining one copy of the real-time stream computing code, without needing to consider the consistency between the batch processing code and the real-time stream computing code. This can save approximately 50% of labor costs and has good versatility, suitable for all scenarios requiring real-time stream computing of historical business data. Since the consistency between the batch processing code and the real-time stream computing code does not need to be considered, errors caused by data fluctuations will not occur. Furthermore, during the calculation process, there is no need to store historical business data in the message queue. Therefore, there is no need to use valuable message queue resources to save historical business data. The message queue only needs to store data for a few days. Since the storage of the message queue is much more valuable than the storage resources of batch processing, message queue resources can be saved.

[0124] Please see Figure 8 This application provides a data processing device 500 for use in electronic devices. The device 500 includes: a data acquisition module 510, a total data acquisition module 520, and a data update module 530.

[0125] The data acquisition module 510 is used to acquire real-time business data stored in the message queue.

[0126] The total data acquisition module 520 is used to obtain the total data volume corresponding to the target dimension information based on the target data volume stored in memory and real-time business data. The target data volume is the amount of historical business data corresponding to the target dimension information that has been acquired in advance.

[0127] The data volume acquisition module 520 includes a data volume acquisition unit and a data fusion unit.

[0128] The data volume acquisition unit is used to obtain the data volume of real-time business data corresponding to the target dimension information based on real-time business data.

[0129] The data fusion unit is used to merge the amount of real-time business data corresponding to the target dimension information with the amount of historical business data corresponding to the target dimension information to obtain the total amount of data corresponding to the target dimension information.

[0130] The data volume update module 530 is used to update the data volume of historical business data in memory corresponding to the target dimension information based on the total data volume corresponding to the target dimension information.

[0131] In one possible implementation, the data processing device 500 further includes a data acquisition module and a first storage module.

[0132] The data volume acquisition module is used to obtain the data volume of historical business data corresponding to each dimension of information based on the historical full volume of business data.

[0133] The data volume acquisition module is also used to perform batch processing calculations on all historical business data to obtain the data volume of historical business data corresponding to each dimension of information.

[0134] The first storage module is used to store the amount of historical business data corresponding to each dimension information to the storage device. The amount of historical business data corresponding to each dimension information includes the amount of historical business data corresponding to the target dimension information. The amount of target data stored in memory is the amount of historical business data corresponding to the target dimension information that has been obtained in advance from the storage device.

[0135] The first storage module is also used to store the amount of historical business data corresponding to each dimension of information in the form of key-value pairs on a distributed storage device.

[0136] In one possible implementation, the historical full-volume business data is the business data collected before a specified time. The data acquisition module 510 is also used to acquire the specified business data stored in the message queue as real-time business data. The data acquisition time of the specified business data is after the specified time.

[0137] In one possible implementation, the data processing device 500 further includes a second storage module.

[0138] The second storage module is used to retrieve the amount of historical business data corresponding to the target dimension information from the storage device and store it in memory when the amount of historical business data corresponding to the target dimension information is not stored in memory.

[0139] In one possible implementation, the data processing apparatus 500 further includes a confirmation module. The confirmation module is used to confirm, when the tag value corresponding to the target dimension information stored in memory is not a first tag value, the amount of historical business data corresponding to the target dimension information not stored in memory.

[0140] In one possible implementation, the data processing device 500 further includes a first sending module, used to send the total amount of data corresponding to the target dimension information to the business storage system associated with memory for querying.

[0141] In one possible implementation, the data processing device 500 further includes a second sending module, configured to send the total amount of data corresponding to the target dimension information to the storage device, so that when the storage device receives the total amount of data corresponding to the target dimension information, it saves the total amount of data and the time of receiving the total amount of data.

[0142] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0143] The following will combine Figure 8 This application provides a description of an electronic device 100.

[0144] Please see Figure 9 Based on the data processing method provided in the above embodiments, this application also provides another electronic device 100 including a processor 102 capable of executing the aforementioned method. The electronic device 100 may be a server 10 or a terminal device, and the terminal device may be a smartphone, tablet computer, computer or portable computer or other devices.

[0145] The electronic device 100 also includes a memory 104. The memory 104 stores a program that can execute the contents of the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104.

[0146] The processor 102 may include one or more cores for data processing and message matrix units. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0147] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data acquired by the electronic device 100 during use (e.g., dimension information, the total amount of data corresponding to the dimension information, and the amount of historical business data corresponding to the dimension information, etc.).

[0148] The electronic device 100 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and facilitate data interaction.

[0149] In some embodiments, the electronic device 100 may further include a peripheral interface 106 and at least one peripheral device. The processor 102, memory 104, and peripheral interface 106 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118, and a power supply 122.

[0150] Peripheral interface 106 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 102 and memory 104. In some embodiments, processor 102, memory 104 and peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 102, memory 104 and peripheral interface 106 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0151] The radio frequency (RF) component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF component 108 communicates with communication networks and other communication devices via electromagnetic signals. The RF component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF component 108 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF component 108 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0152] Positioning component 112 is used to locate the current geographic location of an electronic device to enable navigation or LBS (Location Based Service). Positioning component 112 can be a positioning component based on the US GPS (Global Positioning System), BeiDou system, or Galileo system.

[0153] Camera 114 is used to capture images or videos. Optionally, camera 114 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 100, and the rear-facing camera is located on the back of the electronic device 100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera 114 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0154] Audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to processor 102 for processing, or input to radio frequency component 108 for voice communication. For stereo acquisition or noise reduction purposes, there may be multiple microphones, each located at a different part of electronic device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 102 or radio frequency component 108 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as ranging. In some embodiments, audio component 114 may also include a headphone jack.

[0155] Display screen 118 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 118 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 102 for processing. In this case, display screen 118 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 118, which serves as the front panel of electronic device 100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of electronic device 100 or in a folded design; in still other embodiments, display screen 118 may be a flexible display screen, disposed on a curved or folded surface of electronic device 100. Furthermore, display screen 118 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 118 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0156] Power supply 122 is used to supply power to various components in electronic device 100. Power supply 122 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0157] This application also provides a computer-readable storage medium. This computer-readable medium stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0158] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0159] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various optional implementations above.

[0160] In summary, the data processing method, apparatus, device, and storage medium provided in this application acquire real-time business data stored in a message queue; based on the target data volume stored in memory and the real-time business data, the total data volume corresponding to the target dimension information is obtained, where the target data volume is the volume of historical business data corresponding to the target dimension information acquired from the storage device. This enables data backtracking by using real-time stream computation on the real-time business data stored in the message queue, and obtaining the total data volume corresponding to the target dimension information based on the result of the real-time stream computation and the pre-acquired volume of historical business data corresponding to the target dimension information. This simplifies the data processing process, increases computational efficiency, and minimizes the use of message queue resources.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring real-time business data stored in a message queue; obtaining a total amount of data corresponding to target dimension information according to a target data amount stored in a memory and the real-time business data, the target data amount being a data amount of historical business data corresponding to the target dimension information acquired in advance; updating the data amount of the historical business data corresponding to the target dimension information in the memory according to the total amount of data corresponding to the target dimension information; before the step of obtaining the total amount of data corresponding to the target dimension information according to the target data amount stored in the memory and the real-time business data, the method further comprises: if a mark value corresponding to the target dimension information stored in the memory is not a first mark value, confirming that the memory does not store the data amount of the historical business data corresponding to the target dimension information; if the memory does not store the data amount of the historical business data corresponding to the target dimension information, acquiring the data amount of the historical business data corresponding to the target dimension information from a storage device and storing the data amount in the memory.

2. The data processing method according to claim 1, characterized in that, before the step of obtaining the total amount of data corresponding to the target dimension information according to the target data amount stored in the memory and the real-time business data, the method further comprises: obtaining the data amount of the historical business data corresponding to each dimension information according to historical full-amount business data; storing the data amount of the historical business data corresponding to each dimension information in a storage device, wherein the data amount of the historical business data corresponding to each dimension information includes the data amount of the historical business data corresponding to the target dimension information, and the target data amount stored in the memory is the data amount of the historical business data corresponding to the target dimension information acquired in advance from the storage device.

3. The data processing method according to claim 2, characterized in that, the step of obtaining the data amount of the historical business data corresponding to each dimension information according to the historical full-amount business data comprises: performing batch processing calculation on the historical full-amount business data to obtain the data amount of the historical business data corresponding to each dimension information respectively.

4. The data processing method according to claim 2, characterized in that, the storage device comprises a distributed storage device that stores data in the form of key-value pairs, and the step of storing the data amount of the historical business data corresponding to each dimension information in the storage device comprises: storing the data amount of the historical business data corresponding to each dimension information in the distributed storage device in the form of key-value pairs.

5. The data processing method according to claim 2, characterized in that, the historical full-amount business data is business data collected before a specified time, and the step of acquiring real-time business data stored in a message queue comprises: acquiring specified business data stored in a message queue as real-time business data, the data collection time of the specified business data being after the specified time.

6. The data processing method according to any one of claims 1 to 5, characterized in that, the step of obtaining a total amount of data corresponding to target dimension information according to a target data amount stored in a memory and the real-time business data comprises: obtaining a data amount of real-time business data corresponding to the target dimension information according to the real-time business data; fusing the data amount of the real-time business data corresponding to the target dimension information and the data amount of the historical business data corresponding to the target dimension information to obtain the total amount of data corresponding to the target dimension information.

7. The data processing method according to any one of claims 1 to 5, characterized in that, After obtaining the total data amount corresponding to the target dimension information according to the target data amount stored in the memory and the real-time business data, the method further comprises: sending the total data amount corresponding to the target dimension information to a business storage system associated with the memory for querying.

8. The data processing method according to any one of claims 1 to 5, characterized in that, After obtaining the total data amount corresponding to the target dimension information according to the target data amount stored in the memory and the real-time business data, the method further comprises: sending the total data amount corresponding to the target dimension information to a storage device, so that the storage device saves the total data amount and a receiving time when the total data amount is received.

9. A data processing apparatus, characterized by, The apparatus comprises: a data obtaining module configured to obtain real-time business data stored in a message queue; a total data amount obtaining module configured to obtain a total data amount corresponding to target dimension information according to a target data amount stored in a memory and the real-time business data, the target data amount being a data amount of historical business data corresponding to the target dimension information obtained in advance; a data amount updating module configured to update the data amount of the historical business data corresponding to the target dimension information in the memory according to the total data amount corresponding to the target dimension information; The data processing apparatus further comprises: a confirming module configured to confirm that the memory does not store the data amount of the historical business data corresponding to the target dimension information if a mark value corresponding to the target dimension information stored in the memory is not a first mark value; a second storage module configured to obtain the data amount of the historical business data corresponding to the target dimension information from a storage device and store the data amount in the memory if the memory does not store the data amount of the historical business data corresponding to the target dimension information.

10. The apparatus of claim 9, wherein, The data processing apparatus further comprises: a data amount obtaining module configured to obtain a data amount of historical business data corresponding to each dimension information according to historical full-amount business data; a first storage module configured to store the data amount of the historical business data corresponding to each dimension information in a storage device, wherein the data amount of the historical business data corresponding to each dimension information includes the data amount of the historical business data corresponding to the target dimension information, and the target data amount stored in the memory is a data amount of the historical business data corresponding to the target dimension information obtained in advance from the storage device.

11. The apparatus of claim 10, wherein, The data amount obtaining module is further configured to perform batch processing calculation on the historical full-amount business data to obtain the data amount of the historical business data corresponding to each dimension information respectively.

12. The apparatus of claim 10, wherein, The storage device comprises a distributed storage device configured to store data in the form of key-value pairs, and the first storage module is further configured to store the data amount of the historical business data corresponding to each dimension information in the distributed storage device in the form of key-value pairs.

13. The apparatus of claim 10, wherein, The historical full-amount business data is business data collected before a specified time, and the data obtaining module is further configured to obtain specified business data stored in a message queue as real-time business data, the data collection time of the specified business data being after the specified time.

14. The apparatus of any one of claims 9-13, wherein, The total data amount obtaining module comprises: a data volume obtaining unit, configured to obtain a data volume of real-time service data corresponding to the target dimension information according to the real-time service data; a data fusion unit, configured to fuse the data volume of real-time service data corresponding to the target dimension information and a data volume of historical service data corresponding to the target dimension information, to obtain a total data volume corresponding to the target dimension information.

15. The apparatus of any one of claims 9-13, wherein, The data processing apparatus further comprises: a first sending module, configured to send the total data volume corresponding to the target dimension information to a service storage system associated with the memory, for querying.

16. The apparatus of any one of claims 9-13, wherein, The data processing apparatus further comprises: a second sending module, configured to send the total data volume corresponding to the target dimension information to a storage device, so that the storage device saves the total data volume and a receiving time when the total data volume is received.

17. An electronic device, comprising: The computer readable storage medium stores a program code, wherein the program code is run by a processor to implement the method in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program code, wherein the program code is run by a processor to implement the method in any one of claims 1-8.

19. A computer program product, characterised in that, The computer readable storage medium stores a program code, wherein the program code is run by a processor to implement the method in any one of claims 1-8.

Images