Transaction data preprocessing method and device, storage medium and electronic device

Abstract

Embodiments of the present application provide a transaction data preprocessing method and device, a storage medium and an electronic device. The method comprises: obtaining transaction subjects, transaction volumes and transaction timestamps of each transaction data; counting and recording the transaction volume data of each transaction subject in each time slice, wherein the transaction timestamps of each transaction data in the transaction volume data in each time slice are within the corresponding time slice; obtaining a transaction analysis task of an accelerator, the transaction analysis task comprising at least a target data source channel and a target timestamp; from the recorded transaction volume data, summarizing the transaction volume data of each transaction subject in the target timestamp in the target data source channel as the summary transaction volume data of the target data source channel; and delivering the summary transaction volume data of the target data source channel to the accelerator. In this way, the waste of space and time can be reduced, and the utilization rate of the storage resources of the data processing device can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a transaction data preprocessing method and apparatus, storage medium and electronic device thereof. Background Technology

[0002] Nowadays, in big data processing systems, such as securities firms in the financial sector, risk control judgments are made to check whether stock trading orders have been intercepted by the exchange based on market data. For example, trading institutions use data processing equipment to obtain real-time trading data issued by financial trading centers and perform risk control analysis based on the trading data.

[0003] Traditionally, trading institutions can configure general-purpose processors based on instruction sets in their data processing equipment. However, such processors are easily limited by the programming language used in their development. Therefore, current trading institutions use accelerators such as FPGAs (Field Programmable Gate Arrays, a type of programmable logic device) in their data processing equipment for risk control calculations. These accelerators can perform high-performance parallel data processing, improving the processing capabilities of financial risk control.

[0004] In existing transaction data processing solutions, real-time transaction volume, transaction amount, latest transaction price, and other transaction market data and order data are obtained from multiple data source channels. Based on this data, the reference transaction task data of relatively optimal multi-level buy or sell orders is calculated. The data processing equipment then performs risk control analysis based on this reference transaction task data. However, the transaction task data obtained in this way is not conducive to making full use of the parallel processing capabilities of the accelerator in the data processing equipment. Furthermore, in the process of filtering out the relatively optimal reference transaction task data based on real-time transaction data and order data, a certain amount of transaction task data is filtered out, resulting in a waste of the storage resources of the data processing equipment. Summary of the Invention

[0005] In view of this, embodiments of this application provide a transaction data preprocessing method and apparatus, storage medium and electronic device thereof, to solve or alleviate the problems existing in the prior art.

[0006] According to a first aspect of the embodiments of this application, a transaction data preprocessing method is provided, comprising: acquiring the transaction entity, transaction volume, and transaction timestamp of each transaction data; statistically recording the transaction volume data of each transaction entity in each time slice, wherein the transaction timestamp of each transaction data in the transaction volume data in each time slice is within the corresponding time slice; acquiring the transaction analysis task of the accelerator, the transaction analysis task including at least a target data source channel and a target timestamp; summarizing the transaction volume data of each transaction entity in the target data source channel within the target timestamp from the recorded transaction volume data, as the summarized transaction volume data of the target data source channel; and sending the summarized transaction volume data of the target data source channel to the accelerator.

[0007] In some embodiments of this application, obtaining the transaction subject, transaction volume, and transaction timestamp of each transaction data includes: reading each transaction data from the transaction message queue; parsing the transaction data to obtain the transaction subject, transaction volume, and transaction timestamp of each transaction data.

[0008] In some embodiments of this application, before obtaining the transaction subject, transaction volume, and transaction timestamp of the transaction data, the method further includes: obtaining the transaction timestamp and data source channel of the current transaction data; if the transaction timestamp of the current transaction data is less than the transaction timestamp of the previously obtained transaction data of the data source channel, then discarding the current transaction data; if the transaction timestamp of the current transaction data is greater than or equal to the transaction timestamp of the previously obtained transaction data of the data source channel, then adding the current transaction data to the transaction message queue.

[0009] In some embodiments of this application, the transaction volume data of each transaction entity in each time slice is statistically recorded, including: creating a cache space for transaction volume data based on the transaction entity and time slice as cache dimensions; and accumulating each transaction data into the cache dimension of the transaction entity of each transaction data and the cache dimension of the time slice where the transaction timestamp of each transaction data is located.

[0010] In some embodiments of this application, before creating the cache space for transaction volume data, the method further includes: creating a first whitelist and a second whitelist respectively, and verifying the transaction subject of the current transaction data through the first whitelist and verifying the transaction subject of the current snapshot data through the second whitelist, wherein the first whitelist is used to store the transaction subject of the transaction data and the second whitelist is used to store the transaction subject of the snapshot data; creating a snapshot message queue and a transaction message queue respectively, and transmitting snapshot data through the snapshot message queue and transmitting transaction data through the transaction message queue.

[0011] In some embodiments of this application, the transaction subject of the current transaction data is verified by a first whitelist, and the transaction subject of the current snapshot data is verified by a second whitelist. This includes: determining whether the transaction subject of the current transaction data is in the first whitelist and whether the transaction subject of the current snapshot data is in the second whitelist; if the transaction subject of the current transaction data is in the first whitelist, then the data source channel of the current transaction data is obtained; if the transaction subject of the current snapshot data is in the second whitelist, then the transaction price corresponding to the transaction subject of the current snapshot data is sent to the accelerator through the snapshot message queue.

[0012] In some embodiments of this application, obtaining the transaction analysis task of the accelerator includes: obtaining the timer interval and the current timestamp of the current data source channel; obtaining the transaction analysis task of the accelerator when the timer interval is greater than or equal to a first preset time interval, or when the current timestamp of the current data source channel changes.

[0013] According to a second aspect of the embodiments of this application, a transaction data preprocessing device is provided, comprising: a first acquisition module, configured to acquire the transaction entity, transaction volume, and transaction timestamp of each transaction data; a statistics module, configured to count and record the transaction volume data of each transaction entity in each time slice, wherein the transaction timestamp of each transaction data in the transaction volume data in each time slice is within the corresponding time slice; a second acquisition module, configured to acquire the transaction analysis task of the accelerator, the transaction analysis task including at least a target data source channel and a target timestamp; a summarization module, configured to summarize the transaction volume data of each transaction entity in the target data source channel within the target timestamp from the recorded transaction volume data, as the summarized transaction volume data of the target data source channel; and a distribution module, configured to distribute the summarized transaction volume data of the target data source channel to the accelerator.

[0014] According to a third aspect of the present application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the transaction data preprocessing method of any of the above embodiments.

[0015] According to a fourth aspect of the present application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the transaction data preprocessing method as described in any of the above embodiments.

[0016] According to the transaction data preprocessing method, device, storage medium, and electronic device provided in the embodiments of this application, the transaction volume data of each transaction entity in each time slice in each transaction data is statistically analyzed. Based on the target data source channel and target timestamp of the accelerator's transaction analysis task, the summarized transaction volume data of each transaction entity in the target data source channel within the target timestamp is obtained from the recorded transaction volume data. Then, the summarized transaction volume data is sent to the accelerator for data calculation, thereby quickly determining the task data to be processed by the accelerator. This method is more suitable for the accelerator's data calculation and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thus improving the utilization rate of the storage resources of the data processing device. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart illustrating a transaction data preprocessing method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of a transaction data preprocessing system according to an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating a transaction data preprocessing method according to another embodiment of this application;

[0021] Figure 4 This is a structural block diagram of a transaction data preprocessing apparatus according to an embodiment of this application;

[0022] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0024] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a transaction data preprocessing method according to an embodiment of this application.

[0026] like Figure 1 As shown, the transaction data preprocessing methods include:

[0027] S11: Obtain the transaction entity, transaction volume, and transaction timestamp for each transaction.

[0028] Tick-by-tick transaction data refers to the transaction data, including each trade and order, issued by the financial trading center and acquired by trading institutions through data processing equipment. This data includes the trading entity, trading volume, data source channel, and transaction timestamp. The trading entity refers to the company represented by the stock traded in each tick. The data source channel refers to the different channels used by the financial trading center to forward transaction data from different trading entities. There is a corresponding relationship between trading entities and data source channels; that is, the same data source channel corresponds to at least one trading entity, and each trading entity corresponds to one data source channel. The transaction timestamp is obtained by updating the transaction time of each tick with an accuracy of 10ms.

[0029] S12: Statistically record the transaction volume data of each trading entity in each time slice. The transaction timestamps of each transaction in the transaction volume data of each time slice are all within the corresponding time slice.

[0030] A time slice is a slice obtained by dividing a time period into 1-second cycles. For example, 1 second can be divided into 100 time slices, with each time slice being 10ms. The numerical division of the time slice can be adjusted according to the accuracy of the trading timestamp of the trading institution.

[0031] It is understandable that there are 100 time slices within each second. These 100 time slices can be sorted sequentially. Therefore, after obtaining the trading entity, trading volume, and trading timestamp, the sequence number of the time slice corresponding to the trading timestamp can be determined, and then the trading volume data of the time slice corresponding to the trading entity can be counted.

[0032] Because trading institutions' data processing equipment seeks faster processing speeds when handling trading data, it typically aggregates data from the same time slice across all trading entities for batch processing. Therefore, by finely dividing the trading volume data corresponding to each trading entity into time slices, it is beneficial for the trading institutions' data processing equipment to aggregate and batch process the trading volume data of all trading entities corresponding to the same time slice, thereby improving processing efficiency.

[0033] S13: Obtain the accelerator's transaction analysis task. The transaction analysis task includes at least the target data source channel and the target timestamp.

[0034] The accelerator can be an FPGA, which has parallel computing capabilities and can quickly process large amounts of data. The target data source channel refers to the channel where the task needs to be completed. This target data source channel contains transaction volume data from different trading entities, and the data within the same channel is arranged sequentially according to a certain order. The target timestamp refers to the specific moment when the task needs to be completed.

[0035] S14: From the recorded transaction volume data, summarize the transaction volume data of each trading entity in the target data source channel within the target timestamp, and use it as the summary transaction volume data of the target data source channel.

[0036] It is understandable that when there are transaction analysis tasks in the current queue, it is necessary to obtain the transaction analysis tasks of the accelerator, parse them to obtain the target data source channel and the target timestamp. Since there is a corresponding relationship between the transaction entity and the data source channel, there is at least one transaction entity in the same data source channel. Therefore, based on the target data source channel, we can obtain each transaction entity corresponding to the channel, and based on the target timestamp, we can obtain the corresponding time slice. Thus, in the recorded transaction volume data, we can summarize the transaction volume data of each transaction entity in the target data source channel in each time slice within the target timestamp to obtain the summarized transaction volume data.

[0037] S15: Send the aggregated transaction volume data of the target data source channel to the accelerator.

[0038] Furthermore, the aggregated transaction volume data of the target data source channel is sent to the accelerator through the interaction module that interacts with the accelerator for risk control calculation. The accelerator is set up in the data processing equipment to accelerate the risk control calculation.

[0039] By aggregating and distributing transaction volume data within the target timestamp, the problem of high computational and spatial consumption and low computational efficiency of the accelerator due to real-time distribution of transaction volume data can be avoided. On the other hand, it can save PCIe (Peripheral Component Interconnect Express, a computer bus standard) bus interaction, save computation time, and improve the resource utilization of the accelerator.

[0040] According to the transaction data preprocessing method in this application, the transaction volume data of each transaction entity in each time slice of the transaction data is statistically analyzed. Based on the target data source channel and target timestamp of the accelerator's transaction analysis task, the summarized transaction volume data of each transaction entity in the target data source channel within the target timestamp is obtained from the recorded transaction volume data. Then, the summarized transaction volume data is sent to the accelerator for data calculation, thereby quickly determining the task data to be processed by the accelerator. This method is more suitable for the accelerator's data calculation and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thus improving the utilization rate of the storage resources of the data processing equipment.

[0041] Figure 2 This is a schematic diagram of the structure of a transaction data preprocessing system according to an embodiment of this application, as shown below. Figure 2 As shown, the transaction data preprocessing system 200 includes an external interface 201, a server 202, and an accelerator 203. The server 202 includes a receiving module 2021, a transaction-by-transaction message queue 2022, a snapshot message queue 2023, a management module 2024, a task distribution queue 2025, a timer 2026, and a caching module 2027.

[0042] like Figure 2 As shown, the initialization phase includes the following steps:

[0043] A1: The server 202 creates a first whitelist consisting of transaction subjects of transaction data and a second whitelist consisting of transaction subjects of snapshot data through the management module 2024, and stores the first whitelist and the second whitelist in memory in the form of a hash table data structure;

[0044] A2: Server 202 creates a transaction-by-transaction message queue and a snapshot message queue through management module 2024. Transaction-by-transaction data is transmitted through the transaction-by-transaction message queue, and snapshot data is transmitted through the snapshot message queue. Management module 2024 obtains and stores information such as market type and number of data source channels from the configuration file of server 202. The market type can be Shanghai Stock Exchange, Shenzhen Stock Exchange, STAR Market, etc.

[0045] A3: The server 202 sends the first whitelist and the second whitelist to the receiving module 2021 through the management module 2024, and sends the data types issued by the transaction message queue and the snapshot message queue to the receiving module 2021, so that the receiving module 2021 can cache the first whitelist, the second whitelist, and the data types issued by the transaction message queue and the snapshot message queue in memory.

[0046] A4: The server 202 sends the number of transaction entities and the preset number of time slices to the cache module 2027 through the management module 2024, so that the cache module 2027 can request dynamic memory according to the number of transaction entities and the number of time slices;

[0047] The specific operation of the cache module 2027 in requesting dynamic memory is as follows: an index table is created in the memory of the cache module 2027. Each row of the index represents a transaction entity. For example, the first row represents transaction entity 1. The various time slices of transaction entity 1 are arranged in order from left to right to obtain t11, t12, t13, t14, t15, t16, t17, t18, t19 and other time slices. The transaction volume data of the transaction entity in each time slice is stored. The second row represents transaction entity 2. The various time slices of transaction entity 2 are arranged in order from left to right to obtain t21, t22, t23, t24, t25, t26, t27, t28, t29 and other time slices. And so on. Each transaction entity corresponds to its own cache space.

[0048] The cache module 2027 presets a maximum number of time slices for each trading entity (one time slice corresponds to one 10ms in 1 second). These 100 time slices have corresponding transaction volume data (each transaction volume data occupies 8 bytes). In extreme cases, storing 2 time slices for each trading entity in the cache module 2027 can meet the requirements.

[0049] It should be noted that each transaction entity corresponds to an index in the accelerator. The index starts from 0 and increments, consistent with the number of transaction entities. In other words, the index of the transaction entity stored in the memory of the cache module 2027 and the index of the transaction entity stored in the accelerator are mutually mapped. Each transaction entity has its own index. The storage location of the same transaction entity in the accelerator can be found from the index of the cache module 2027. Therefore, when sending, the storage location of the transaction entity can be directly told to the accelerator.

[0050] A5: Server 202 sends a pre-set first preset time interval to timer 2026 through management module 2024, so that timer 2026 periodically inserts transaction analysis tasks into task queue 2025. The first preset time interval can be 10ms, 9ms, 8ms, 7ms, or other values ​​less than or equal to 10ms.

[0051] Continue to refer to Figure 2 After the initialization phase is completed, the data processing phase begins, which includes the following steps:

[0052] B1: The server 202 receives the current transaction data and current snapshot data from the external interface 201 through the receiving module 2021, and parses the current transaction data and current snapshot data through a preset protocol to verify them;

[0053] The specific operation of the receiving module 2021 in verifying the current transaction data and the current snapshot data is as follows: obtain the transaction subject of the current transaction data and the current snapshot data, determine whether the transaction subject of the current transaction data is in the first whitelist, and whether the transaction subject of the current snapshot data is in the second whitelist; if the transaction subject of the current transaction data is in the first whitelist, obtain the data source channel of the current transaction data, and determine the corresponding transaction timestamp based on the data source channel;

[0054] If the timestamp of the current transaction data is less than the timestamp of the previously acquired transaction data from the data source channel, the current transaction data is determined to be out-of-order data and is therefore discarded. If the timestamp of the current transaction data is greater than or equal to the timestamp of the previously acquired transaction data from the data source channel, the current transaction data is determined to have passed verification. The current transaction data is then converted from an external format to an internal format and added to the transaction message queue 2022. The fields in the external format of the current transaction data are larger than those in the internal format. This conversion reduces the amount of data processed.

[0055] If the trading entity in the current snapshot data is within the second whitelist, the current snapshot data is deemed to have passed verification. The current snapshot data is then converted from an external format to an internal format. At this point, the management module 2024 sends the transaction price corresponding to the trading entity in the snapshot data to the accelerator 203 via the snapshot message queue 2023. Since the second whitelist pre-stores the trading entities in the snapshot data and is used to update the latest transaction price for the latest trading entity, when the trading entity in the snapshot data is within the second whitelist, the corresponding transaction price can be sent to the accelerator 203 so that the accelerator 203 can obtain the latest transaction price for the latest trading entity.

[0056] B2: Server 202 retrieves a transaction data from transaction message queue 2022 through management module 2024, and queries the index of the transaction subject in the dynamic memory pre-allocated by cache module 2027 using the transaction subject and transaction timestamp in the transaction data, and records the transaction volume data of each transaction subject in the transaction data in each time slice.

[0057] The management module 2024 obtains the transaction volume data of each transaction entity in each time slice of the transaction data as follows: The management module 2024 determines the index of transaction entity 1 in the dynamic memory pre-allocated by the cache module 2027, then determines the corresponding at least one time slice based on the transaction timestamp, then queries the index for the transaction volume data of the transaction entity in at least one time slice, and finally, through the insertion function of the cache module 2027, summarizes the transaction volume data of transaction entity 1 in at least one time slice into the cache position in the cache module 2027 for accumulation, and so on, to obtain and record the transaction volume data of each transaction entity in each time slice of the transaction data.

[0058] It should be noted that when aggregating the transaction volume data of each trading entity for at least one time slice into the cache location in the cache module 2027 for accumulation, if the trading entity has not received data from any data source channel, it can be assumed that the transaction volume data of that trading entity has not been aggregated. Therefore, the trading entity is inserted into the aggregation array of its corresponding data source channel so that the transaction volume data aggregated in the aggregation array can be sent to the accelerator in the subsequent task distribution phase. Specifically, as follows... Figure 3 As shown:

[0059] S30: Beginning;

[0060] S31: Insert the transaction volume data of each trading entity in each time slice into the cache module for recording;

[0061] S32: Determine whether the transaction entity is in the data source channel list. If yes, proceed to S34; otherwise, proceed to S33. The data source channel list is used to determine whether the transaction entity is appearing for the first time.

[0062] S33: Insert the data source channel where the transaction entity is located into the list of data source channels, and establish the correspondence between the transaction entity and its data source channel;

[0063] S34: Determine whether the latest timestamp of the data source channel has changed. If yes, proceed to S35; otherwise, proceed to S36.

[0064] S35: Insert the identifier ID of the data source channel and the previous timestamp of the data source channel into the task queue;

[0065] S36: End.

[0066] Continue to refer to Figure 2 After the data processing phase is completed, the task distribution and processing phase begins, which includes the following steps:

[0067] C1: Server 202 obtains the time interval of timer 2026 and the current timestamp of the current data source channel;

[0068] C2: When the time interval of timer 2026 is greater than or equal to the first preset time interval, or when the current timestamp of the current data source channel changes, it is determined that there is a transaction analysis task in the task queue 2025. At this time, the transaction analysis task of accelerator 203 is obtained. The timer obtains the target data source channel and the target timestamp every first preset time interval and sends them to the task queue 2025, so that there is a transaction analysis task of accelerator 203 in the task queue.

[0069] C3: Server 202 obtains the target data source channel and target timestamp from the task queue 2025 through management module 2024 and sends it to cache module 2027. It also summarizes the transaction volume data of each trading entity in the target data source channel within the target timestamp from the transaction volume data recorded in B2, and uses it as the summary transaction volume data of the target data source channel, and sends it to accelerator 203.

[0070] The process of obtaining aggregated transaction volume data is as follows: Based on the transaction volume data recorded in B2 from the target data source channel, all trading entities corresponding to that target data source channel are retrieved. Then, based on the target timestamp, the corresponding target time slices are determined. Next, the transaction volume data for each trading entity's target time slices is retrieved from the cache space in cache module 2027. Finally, the transaction volume data for each trading entity is read using the read function of cache module 2027, and the read transaction volume data for each trading entity is aggregated to obtain the aggregated transaction volume data for the target data source channel. This process continues until the transaction volume data for all trading entities in the target data source channel within the target timestamp has been aggregated. The aggregated transaction volume data can then be sent to the accelerator for data calculation. Specifically, this aggregated transaction volume data represents the summation of the transaction volume data for all trading entities in the target data source channel within the same time slice.

[0071] In this way, quickly determining the task data that the accelerator needs to process is more suitable for the accelerator's data computation and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thereby improving the utilization rate of the storage resources of the data processing equipment.

[0072] Furthermore, in some embodiments, obtaining the transaction subject, transaction volume, and transaction timestamp of each transaction data includes: reading each transaction data from the transaction message queue; parsing the transaction data to obtain the transaction subject, transaction volume, and transaction timestamp of each transaction data. That is... Figure 2In the middle, the management module 2024 retrieves a transaction data from the transaction message queue 2022, parses it to obtain the transaction subject, transaction volume and transaction timestamp of the transaction data, which helps to obtain the transaction volume data of each transaction subject in each time slice based on the transaction subject and transaction timestamp.

[0073] Furthermore, in some embodiments, before obtaining the transaction subject and transaction timestamp of the transaction data, the method further includes: obtaining the transaction timestamp and data source channel of the current transaction data; if the transaction timestamp of the current transaction data is less than the transaction timestamp of the previously obtained transaction data of the data source channel, then discarding the current transaction data; if the transaction timestamp of the current transaction data is greater than or equal to the transaction timestamp of the previously obtained transaction data of the data source channel, then adding the current transaction data to the transaction message queue.

[0074] As can be understood, referring to B1, the server 202 receives the current transaction data from the external interface 201 through the receiving module 2021, and parses the current transaction data through a preset protocol to verify it. Only after the verification is passed will the transaction subject, transaction volume and transaction timestamp of the transaction data be obtained. The receiving module 2021 receives the current transaction data in the same channel from the external interface 201 and sends it sequentially according to the order of the transaction timestamps.

[0075] For example, the transaction data 1 and transaction data 2 stored in channel A are arranged sequentially according to the transaction timestamp, that is, the transaction timestamp of transaction data 2 is greater than the transaction timestamp of transaction data 1. Under normal circumstances, the receiving module 2021 receives the transaction data from channel A in the order of 1 and 2, that is, it receives the transaction data 1 sent by channel A first, and then receives the transaction data 2. At this time, since the transaction timestamp of the current transaction data 2 is greater than the transaction timestamp of the previously obtained transaction data 1, it is considered that the receiving module 2021 has completed receiving the transaction data 1, and adds the transaction data 1 to the transaction message queue 2022 so as to obtain the transaction subject, transaction volume and transaction timestamp of the transaction data 1 later.

[0076] If the receiving module 2021 receives transaction data 1 again from channel A (the order of the received transaction data is 121), it can determine that the transaction data 1 sent by channel A is out of order because the transaction timestamp of the current transaction data 1 is less than the transaction timestamp of the previously acquired transaction data 2, and will discard the currently sent transaction data 1.

[0077] Thus, by determining whether the data packets of the current transaction data are sent out of order, and discarding the out-of-order transaction data if so, the risk of abnormal data is eliminated. It should be noted that after the receiving module 2021 adds transaction data 1 to the transaction message queue 2022, if transaction data 3 is received again from channel A, the above operation will continue to be performed based on the transaction timestamps of transaction data 3 and transaction data 2 to add transaction data 2 to the transaction message queue 2022, and so on. Furthermore, if the receiving module 2021 does not receive transaction data 3 from channel A, timer 2026 will add transaction data 2 to the transaction message queue 2022.

[0078] Furthermore, in some embodiments, the transaction volume data of each transaction entity in each time slice is statistically recorded, including: creating a cache space for transaction volume data based on the transaction entity and time slice as cache dimensions; accumulating each transaction data to the cache dimension of the transaction entity of each transaction data and the cache dimension of the time slice where the transaction timestamp of each transaction data is located.

[0079] In other words, the caching module 2027 creates sequentially stored transaction volume data for each time slice of each transaction entity based on the transaction entity data and the number of time slices. It allocates corresponding cache space for each transaction entity. Then, by performing simple calculations on the transaction timestamps in the transaction data and combining them with the corresponding transaction entity, the transaction volume data for each time slice of each transaction entity can be quickly retrieved from the dynamic memory of the caching module 2027. This allows the transaction volume data to be accumulated in the cache dimension of the transaction entity and the cache dimension of the time slice, thus improving the efficiency of finding the time slice location by statistically analyzing the transaction volume data of each transaction entity in each time slice.

[0080] It should be noted that the transaction data accumulated in the cache space based on the transaction subject and time slice will be sent to the accelerator for data calculation along with the task queue. Therefore, the transaction data accumulated in the cache space will be updated, making the cache space reusable. This solves the problem of resource waste caused by repeatedly allocating and releasing memory, and improves resource utilization.

[0081] Furthermore, in some embodiments, before creating the cache space for transaction volume data, the method further includes: creating a first whitelist and a second whitelist respectively, and verifying the transaction subject of the current transaction data through the first whitelist and verifying the transaction subject of the current snapshot data through the second whitelist, wherein the first whitelist is used to store the transaction subject of the transaction data and the second whitelist is used to store the transaction subject of the snapshot data; creating a snapshot message queue and a transaction message queue respectively, and transmitting snapshot data through the snapshot message queue and transmitting transaction data through the transaction message queue.

[0082] Snapshot data is data compiled from transaction data over a preset period, including the highest and lowest transaction prices and transaction volumes. Thus, by creating a first whitelist and a second whitelist to verify the parsed current transaction data and the current snapshot data, the security of data transmission is ensured. Furthermore, by creating separate transaction message queues and snapshot message queues for transmitting transaction data and snapshot data respectively, data transmission chaos is avoided.

[0083] Furthermore, in some embodiments, verifying the transaction subject of the current transaction data through a first whitelist and verifying the transaction subject of the current snapshot data through a second whitelist includes: determining whether the transaction subject of the current transaction data is in the first whitelist and whether the transaction subject of the current snapshot data is in the second whitelist; if the transaction subject of the current transaction data is in the first whitelist, then obtaining the data source channel of the current transaction data; if the transaction subject of the current snapshot data is in the second whitelist, then sending the transaction price corresponding to the transaction subject of the current snapshot data to the accelerator through the snapshot message queue.

[0084] In other words, such as Figure 2 The receiving module 2021 receives current transaction data and current snapshot data from external interface 201, and parses the current transaction data and current snapshot data through a preset protocol to verify the parsed data, ensure the security of data transmission, and also obtain the latest transaction price in the current snapshot data.

[0085] In some embodiments, obtaining the accelerator's transaction analysis task includes: obtaining the timer's interval and the current timestamp of the current data source channel; obtaining the accelerator's transaction analysis task when the timer's interval is greater than or equal to a first preset time interval, or when the current timestamp of the current data source channel changes. That is, when the timer has elapsed the first preset time interval or the current timestamp changes, a task is generated and a task queue for the accelerator's transaction analysis task exists.

[0086] The transaction data preprocessing method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0087] Figure 4 This is a structural block diagram of a transaction data preprocessing device according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the transaction data preprocessing device.

[0088] like Figure 4 As shown, the transaction data preprocessing device 400 includes: a first acquisition module 402, used to acquire the transaction entity, transaction volume, and transaction timestamp of each transaction data; a statistics module 403, used to count and record the transaction volume data of each transaction entity in each time slice, wherein the transaction timestamp of each transaction data in the transaction volume data in each time slice is within the corresponding time slice; a second acquisition module 404, used to acquire the transaction analysis task of the accelerator, the transaction analysis task including at least the target data source channel and the target timestamp; a summary module 405, used to summarize the transaction volume data of each transaction entity in the target data source channel within the target timestamp from the recorded transaction volume data, as the summary transaction volume data of the target data source channel; and a distribution module 406, used to distribute the summary transaction volume data of the target data source channel to the accelerator.

[0089] In some embodiments of this application, the first acquisition module 402 is specifically used to: read transaction data from the transaction message queue; parse the transaction data to obtain the transaction subject, transaction volume and transaction timestamp of the transaction data.

[0090] In some embodiments of this application, before obtaining the transaction subject, transaction volume, and transaction timestamp of the transaction data, the first acquisition module 402 is further configured to: acquire the transaction timestamp and data source channel of the current transaction data; if the transaction timestamp of the current transaction data is less than the transaction timestamp of the previously acquired transaction data of the data source channel, then discard the current transaction data; if the transaction timestamp of the current transaction data is greater than or equal to the transaction timestamp of the previously acquired transaction data of the data source channel, then add the current transaction data to the transaction message queue.

[0091] In some embodiments of this application, the statistics module 403 is specifically used to: create a cache space for transaction volume data based on the transaction subject and time slice as cache dimensions; and accumulate each transaction data to the cache dimension of the transaction subject of the transaction data and the cache dimension of the time slice where the transaction timestamp of the transaction data is located.

[0092] In some embodiments of this application, before creating the cache space for transaction volume data, the creation module (not shown in the figure) is used to: create a first whitelist and a second whitelist respectively, and verify the transaction subject of the current transaction data through the first whitelist and verify the transaction subject of the current snapshot data through the second whitelist, wherein the first whitelist is used to store the transaction subject of the transaction data and the second whitelist is used to store the transaction subject of the snapshot data; create a snapshot message queue and a transaction message queue respectively, and transmit snapshot data through the snapshot message queue and transmit transaction data through the transaction message queue.

[0093] In some embodiments of this application, the creation module is specifically used to: determine whether the transaction subject of the current transaction data is in the first whitelist and whether the transaction subject of the current snapshot data is in the second whitelist; if the transaction subject of the current transaction data is in the first whitelist, then obtain the data source channel of the current transaction data; if the transaction subject of the current snapshot data is in the second whitelist, then send the transaction price corresponding to the transaction subject of the current snapshot data to the accelerator through the snapshot message queue.

[0094] In some embodiments of this application, the second acquisition module 404 is specifically used to: acquire the timer interval and the current timestamp of the current data source channel; and acquire the accelerator's transaction analysis task when the timer interval is greater than or equal to the first preset time interval, or when the current timestamp of the current data source channel changes.

[0095] According to the transaction data preprocessing device provided in this application embodiment, by statistically analyzing the transaction volume data of each transaction entity in each time slice of the transaction data, and based on the target data source channel and target timestamp of the accelerator's transaction analysis task, the device summarizes the transaction volume data of each transaction entity in the target data source channel within the target timestamp from the recorded transaction volume data, and then sends the summarized transaction volume data to the accelerator for data calculation, thereby quickly determining the task data to be processed by the accelerator. This is more suitable for the data calculation of the accelerator and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thus improving the utilization rate of the storage resources of the data processing device.

[0096] The transaction data preprocessing device of this embodiment is used to implement the corresponding transaction data preprocessing methods in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the transaction data preprocessing device of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0097] Figure 5This is a structural block diagram of an electronic device according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0098] like Figure 5 As shown, the electronic device 500 includes a processor 502, a memory 503, a communications interface 504, and a communications bus 505. The processor 502, memory 503, and communications interface 504 communicate with each other via the communications bus 505. The memory 503 stores at least one executable instruction, which causes the processor 502 to perform the operation corresponding to the transaction data preprocessing method of any of the above embodiments.

[0099] The processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0100] The memory 503 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0102] According to the electronic device provided in the embodiments of this application, by statistically analyzing the transaction volume data of each transaction entity in each time slice of the transaction data, and based on the target data source channel and target timestamp of the accelerator's transaction analysis task, the summarized transaction volume data of each transaction entity in the target data source channel within the target timestamp is obtained from the recorded transaction volume data. Then, the summarized transaction volume data is sent to the accelerator for data calculation, thereby quickly determining the task data that the accelerator needs to process. This is more suitable for the accelerator's data calculation and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thus improving the utilization rate of the storage resources of the data processing device.

[0103] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements the transaction data preprocessing method as described in any of the above embodiments.

[0104] According to the computer storage medium provided in the embodiments of this application, by statistically analyzing the transaction volume data of each transaction entity in each time slice of the transaction data, and based on the target data source channel and target timestamp of the accelerator's transaction analysis task, the aggregated transaction volume data of each transaction entity in the target data source channel within the target timestamp is obtained from the recorded transaction volume data. Then, the aggregated transaction volume data is sent to the accelerator for data calculation, thereby quickly determining the task data that the accelerator needs to process. This is more suitable for the accelerator's data calculation and avoids the problem of wasting space and time due to too much data being sent to the accelerator, thus improving the utilization rate of the storage resources of the data processing device.

[0105] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0106] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the transaction data preprocessing methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the transaction data preprocessing methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the transaction data preprocessing methods shown herein.

[0107] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0108] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A transaction data preprocessing method, characterized in that, include: Obtain transaction data for each transaction, including the transaction entity, transaction volume, and transaction timestamp. The transaction volume data of each trading entity in each time slice is statistically analyzed and recorded. The transaction timestamps of each transaction data in the transaction volume data of each time slice are all within the corresponding time slice. The transaction analysis task of the accelerator is obtained, and the transaction analysis task includes at least the target data source channel and the target timestamp; From the recorded transaction volume data, the transaction volume data of each trading entity in the target data source channel within the target timestamp is aggregated as the aggregated transaction volume data of the target data source channel; The aggregated transaction volume data of the target data source channel is sent to the accelerator; The process of statistically recording the transaction volume data of each transaction entity in each time slice includes: creating a cache space for transaction volume data based on the transaction entity and time slice as cache dimensions; and accumulating the transaction data into the cache dimension of the transaction entity of the transaction data and the cache dimension of the time slice where the transaction timestamp of the transaction data is located. Before creating the cache space for transaction volume data, the method further includes: creating a first whitelist and a second whitelist respectively, and verifying the transaction subject of the current transaction data through the first whitelist and verifying the transaction subject of the current snapshot through the second whitelist, wherein the first whitelist is used to store the transaction subject of the transaction data and the second whitelist is used to store the transaction subject of the snapshot; creating a snapshot message queue and a transaction message queue respectively, and transmitting snapshot data through the snapshot message queue and transmitting transaction data through the transaction message queue; The step of verifying the transaction entity of the current transaction data through the first whitelist and verifying the transaction entity of the current snapshot data through the second whitelist includes: determining whether the transaction entity of the current transaction data is in the first whitelist and whether the transaction entity of the current snapshot data is in the second whitelist; if the transaction entity of the current transaction data is in the first whitelist, then obtaining the data source channel of the current transaction data; if the transaction entity of the current snapshot data is in the second whitelist, then sending the transaction price corresponding to the transaction entity of the current snapshot data to the accelerator through the snapshot message queue.

2. The method according to claim 1, characterized in that, The process of obtaining transaction data for each transaction includes the transaction entity, transaction volume, and transaction timestamp, among other things: Read transaction data from the transaction message queue; The transaction data is parsed to obtain the transaction entity, transaction volume, and transaction timestamp for each transaction.

3. The method according to claim 2, characterized in that, Before obtaining the transaction entity, transaction volume, and transaction timestamp of each transaction, the method further includes: Obtain the transaction timestamp and data source channel for each current transaction; If the transaction timestamp of the current transaction data is less than the transaction timestamp of the previously acquired transaction data from the data source channel, then the current transaction data is discarded. If the transaction timestamp of the current transaction data is greater than or equal to the transaction timestamp of the previously acquired transaction data from the data source channel, then the current transaction data is added to the transaction message queue.

4. The method according to claim 1, characterized in that, The transaction analysis task for acquiring the accelerator includes: Get the timer interval and the current timestamp of the current data source channel; The accelerator's transaction analysis task is acquired when the timer's interval is greater than or equal to a first preset time interval, or when the current timestamp of the current data source channel changes.

5. A transaction data preprocessing device, characterized in that, include: The first acquisition module is used to acquire the transaction entity, transaction volume and transaction timestamp of each transaction data; The statistics module is used to count and record the transaction volume data of each trading entity in each time slice. The transaction timestamp of each transaction data in the transaction volume data of each time slice is within the corresponding time slice. The second acquisition module is used to acquire the transaction analysis task of the accelerator, wherein the transaction analysis task includes at least the target data source channel and the target timestamp. The aggregation module is used to aggregate the transaction volume data of each trading entity in the target data source channel within the target timestamp from the recorded transaction volume data, and use it as the aggregated transaction volume data of the target data source channel. The distribution module is used to distribute the aggregated transaction volume data of the target data source channel to the accelerator; The statistics module is also used to create a cache space for transaction volume data based on the transaction subject and time slice as cache dimensions; and to accumulate the transaction data into the cache dimension of the transaction subject and the cache dimension of the time slice where the transaction timestamp of the transaction data is located. A creation module is used to create a first whitelist and a second whitelist before creating a cache space for transaction volume data. The first whitelist is used to verify the transaction subject of the current transaction data, and the second whitelist is used to verify the transaction subject of the current snapshot. The first whitelist is used to store the transaction subjects of the transaction data, and the second whitelist is used to store the transaction subjects of the snapshot. A snapshot message queue and a transaction message queue are created respectively, and snapshot data is transmitted through the snapshot message queue, and transaction data is transmitted through the transaction message queue. The creation module is specifically used to determine whether the transaction subject of the current transaction data is in the first whitelist and whether the transaction subject of the current snapshot data is in the second whitelist; if the transaction subject of the current transaction data is in the first whitelist, then the data source channel of the current transaction data is obtained; if the transaction subject of the current snapshot data is in the second whitelist, then the transaction price corresponding to the transaction subject of the current snapshot data is sent to the accelerator through the snapshot message queue.

6. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the transaction data preprocessing method as described in any one of claims 1-4.

7. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the transaction data preprocessing method as described in any one of claims 1-4.

Images