Transaction data low-delay credible acquisition method, device and equipment

Abstract

The invention provides a transaction data low-delay credible acquisition method, device and equipment, and relates to the technical field of data acquisition, and the method comprises the steps: building a plurality of first data channels for collecting data of a plurality of transaction data sources, and obtaining multi-channel data; preprocessing the multi-channel data to obtain preprocessed data; dynamically adjusting bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain a plurality of second data channels; distributing the loads of the plurality of second data channels to obtain the loads of the plurality of second data channels; and transmitting the preprocessed data according to the plurality of second data channel loads. The technical problem that data transmission in a high-frequency transaction environment has a bottleneck effect and a large amount of data cannot be processed in time due to the fact that data transmission depends on a single channel or a limited parallel channel can be solved, the technical target of efficient parallel processing of multi-source transaction data is achieved, and the data transmission efficiency is improved. The technical effects of reducing the data transmission delay and improving the system resource utilization rate are achieved.

Description

Technical Field

[0001] The present application relates to the field of data collection technology, and in particular to a method, device and equipment for low-latency and reliable collection of transaction data. Background Art

[0002] In modern financial markets, the ability to quickly collect and process transaction data is crucial to the performance and competitiveness of trading systems. With the rise of high-frequency trading and real-time data analysis, how to achieve high-speed collection of transaction data under low latency conditions has become a key challenge in the industry. However, existing transaction data collection technologies have many defects in speed and latency control and cannot fully meet the market's demand for efficient data collection. In order to solve this problem, a new method is urgently needed to increase the speed of transaction data collection while reducing latency.

[0003] At present, existing technologies usually rely on single channels or limited parallel channels for data transmission when processing multi-source transaction data. Although this method can meet the needs of general data processing, in a high-frequency trading environment, due to the large amount of transaction data and frequent transmission, single channels or limited parallel channels are prone to cause bottleneck effects in data transmission, and cannot process large amounts of data in a timely manner, ultimately leading to increased latency. In addition, the traditional single-channel data transmission method has a low utilization rate of system resources, which is prone to bandwidth waste or uneven resource allocation, thereby affecting the operating efficiency of the entire system.

[0004] In summary, the prior art has technical problems in that, due to reliance on a single channel or limited parallel channels for data transmission, a bottleneck effect occurs in data transmission under a high-frequency trading environment, and a large amount of data cannot be processed in a timely manner, further affecting transaction efficiency and the overall performance of system operation. Summary of the invention

[0005] The purpose of this application is to provide a low-latency trusted collection method, device and equipment for transaction data, so as to solve the technical problem in the prior art that due to reliance on a single channel or limited parallel channels for data transmission, data transmission has a bottleneck effect in a high-frequency trading environment, and a large amount of data cannot be processed in time, further affecting transaction efficiency and the overall performance of system operation.

[0006] In view of the above problems, in a first aspect, the present application provides a low-latency trusted collection method for transaction data, including: establishing multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data; preprocessing the multi-channel data to obtain preprocessed data; dynamically adjusting the bandwidth allocation of the multiple first data channels according to the importance of the preprocessed data to obtain multiple second data channels; allocating the loads of the multiple second data channels to obtain multiple second data channel loads; and transmitting the preprocessed data according to the multiple second data channel loads.

[0007] In a possible implementation, before obtaining multi-channel data, the following processing is performed: the integrity of the multi-channel data is monitored through a dynamic data acquisition feedback system; when there is data missing in the monitoring results, a data compensation mechanism is activated; and the missing data is supplemented by the data compensation mechanism to obtain the multi-channel data.

[0008] In a possible implementation, the multi-channel data is preprocessed to obtain preprocessed data, and the following processing is performed: redundancy removal is performed on the multi-channel data to obtain first preprocessed data; the first preprocessed data is compressed using a data compression algorithm to obtain second preprocessed data; different priorities are assigned to the importance of the second preprocessed data to obtain the preprocessed data.

[0009] In a possible implementation, the bandwidth allocation of the multiple first data channels is dynamically adjusted according to the importance of the preprocessed data to obtain multiple second data channels, and the following processing is performed: extracting the priority of the preprocessed data to obtain multiple priorities; allocating the bandwidth of the multiple first data channels according to the multiple priorities to obtain multiple allocated bandwidths; obtaining multiple transmission orders according to the multiple allocated bandwidths and the multiple priorities; and adding the multiple transmission orders to the multiple second data channels.

[0010] In a possible implementation, the bandwidth allocation of the multiple first data channels is dynamically adjusted according to the importance of the preprocessed data to obtain multiple second data channels, and the following processing is also performed: obtaining the data flow of the multiple second data channels; when it is detected that the data flow of any second data channel is overloaded, automatically adjusting the bandwidth to obtain the multiple second data channels.

[0011] In a possible implementation, the loads of the multiple second data channels are allocated to obtain the multiple second data channel loads, and the following processing is performed: deep learning training is performed on the historical transaction data using an intelligent analysis model; the collection trend and load of the preprocessed data are predicted by the intelligent analysis model; the loads of the multiple second data channels are allocated according to the prediction results to obtain the multiple second data channel loads.

[0012] In a possible implementation, after the pre-processed data is transmitted according to the multiple second data channel loads, the following processing is performed: configuring a multi-level cache mechanism; determining whether the cache data of the multi-level cache mechanism is saturated, and when saturated, automatically cleaning the data and saving high-frequency identification data.

[0013] In a possible implementation, high-frequency identification data is obtained and the following processing is performed: a high-frequency access threshold is preset; the preprocessed data is traversed and compared with the high-frequency access threshold to obtain screening data; the screening data is frequency-identified according to the number of accesses to obtain the high-frequency identification data.

[0014] In a second aspect of the present application, a low-latency trusted collection device for transaction data is provided, the device comprising:

[0015] The collection module is used to establish multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data.

[0016] The preprocessing module is used to preprocess the multi-channel data to obtain preprocessed data.

[0017] The bandwidth allocation module is used to dynamically adjust the bandwidth allocation of the multiple first data channels according to the importance of the pre-processed data to obtain multiple second data channels.

[0018] The load distribution module is used to distribute the loads of the multiple second data channels to obtain multiple second data channel loads.

[0019] A data transmission module is used to transmit the pre-processed data according to the multiple second data channel loads.

[0020] In a third aspect of the present application, an electronic device is provided. The electronic device comprises at least one processor; and a memory in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present application.

[0021] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0022] The method collects data from multiple transaction data sources by establishing multiple first data channels to obtain multi-channel data; pre-processes the multi-channel data to obtain pre-processed data; dynamically adjusts the bandwidth allocation of the multiple first data channels according to the importance of the pre-processed data to obtain multiple second data channels; allocates the loads of the multiple second data channels to obtain multiple second data channel loads; transmits the pre-processed data according to the multiple second data channel loads, that is, by creating multiple initial data channels to collect information from different transaction data sources, a multi-channel data stream is obtained; preliminarily processes the multi-channel data to obtain processed data; dynamically adjusts the bandwidth allocation of multiple initial data channels according to the importance of the processed data to form multiple new data channels; loads the new data channels to determine their respective load conditions; finally, transmits the processed data according to the load conditions of the new data channels, so as to achieve the technical goal of efficient parallel processing of multi-source transaction data and achieve the technical effect of reducing data transmission delay and improving system resource utilization.

[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0025] Figure 1 A flowchart of a low-latency and reliable collection method of transaction data for this application;

[0026] Figure 2 This is a schematic diagram of the process of obtaining pre-processed data in a low-latency trusted collection method for transaction data in this application;

[0027] Figure 3 A block diagram of a low-latency trusted acquisition device for transaction data for this application;

[0028] Figure 4 A block diagram of an exemplary electronic device capable of implementing the embodiments of the present application;

[0029] Among them, 400 is an electronic device, 401 is a computing unit, 402 is a ROM, 403 is a RAM, 404 is a bus, 405 is an I / O interface, 406 is an input unit, 407 is an output unit, 408 is a storage unit, and 409 is a communication unit. DETAILED DESCRIPTION

[0030] This application provides a low-latency and reliable collection method for transaction data, which solves the technical problem in the prior art that data transmission relies on a single channel or limited parallel channels, resulting in a bottleneck effect in data transmission under high-frequency trading environments, and inability to process large amounts of data in a timely manner, further affecting transaction efficiency and the overall performance of system operation. The technical goal of efficiently processing multi-source transaction data in parallel is achieved, achieving the technical effect of reducing data transmission delays and improving system resource utilization.

[0031] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0032] Please see attached Figure 1 The present application provides a low-latency and reliable collection method for transaction data, which specifically includes the following steps:

[0033] S101. Establish multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data.

[0034] Specifically, by establishing multiple first data channels, data is collected from multiple transaction data sources at the same time, thereby improving the concurrency and efficiency of data collection. Then, each channel is independently responsible for the collection of a part of the transaction data, thereby avoiding the data bottleneck problem that may be caused by a single channel. For example, in a high-frequency trading system, each channel collects data from different markets or trading platforms to ensure that all data can be synchronized and quickly acquired, and further obtain multi-channel data for subsequent processing and analysis. At the same time, the setting of multiple channels can also distribute different data volumes through load balancing, optimize data transmission efficiency, reduce latency and improve the overall performance of the system.

[0035] S102: preprocess the multi-channel data to obtain preprocessed data.

[0036] Specifically, preprocessing of multi-channel data is to remove redundant information and noise, and improve data quality and processing efficiency. Through the preprocessing step, invalid data is cleaned up and parts that are of no value to subsequent analysis are filtered out, thereby reducing the amount of data. For example, in the data collection of multiple trading platforms, there may be duplicate or useless data. Preprocessing can effectively filter out key trading information. Then, the preprocessed data is more streamlined and efficient, which is convenient for further data analysis and processing, which not only speeds up the processing speed of the system, but also improves the accuracy and consistency of the data. Therefore, the process of preprocessing data is crucial to optimizing system performance.

[0037] S103 . Dynamically adjust bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain a plurality of second data channels.

[0038] Specifically, based on the importance of the pre-processed data, the bandwidth allocation of multiple first data channels is dynamically adjusted to ensure that critical data obtains more transmission resources. High-priority data will occupy bandwidth first, thereby speeding up its transmission, while low-priority data may be allocated less bandwidth resources. Then, these bandwidth allocations are adjusted in real time based on the actual data load and network conditions. For example, in a financial trading system, urgent transaction data will receive more bandwidth support to ensure timely transmission, while less urgent data will be processed later. Finally, this process of dynamic bandwidth adjustment generates multiple second data channels, which are more flexible and efficient when processing data of different priorities, optimizing the overall data transmission performance.

[0039] S104. Allocate the loads of the multiple second data channels to obtain multiple second data channel loads.

[0040] Specifically, the load of multiple second data channels is distributed to ensure that the workload of each channel is balanced to avoid a channel being overloaded and affecting system performance. Through reasonable load distribution, the load of different channels is dynamically adjusted according to the current processing capacity and data volume of each channel. For example, in financial data transmission, if the amount of data in a certain channel suddenly increases, part of the data will be transferred to other channels to ensure that the load of all channels is within an acceptable range. Then, this load distribution process can improve the efficiency of data transmission and the stability of the system, and finally obtain multiple second data channel loads, which can still run smoothly when processing high data volumes.

[0041] S105. Transmit the pre-processed data according to the multiple second data channel loads.

[0042] Specifically, according to the load conditions of multiple second data channels, the transmission mode of pre-processed data is optimized to ensure that each channel can reasonably utilize bandwidth and resources. In this way, channels with high loads may reduce their transmission volume, and channels with low loads can take on more data transmission tasks, thereby achieving balanced resource allocation. For example, in a financial trading system, if a channel processes a large amount of data, part of the pre-processed data will be transmitted through other channels with lighter loads to avoid bottlenecks. Then, load balancing can effectively improve transmission efficiency and ensure that all data can be transmitted in a timely manner without affecting system performance.

[0043] The low-latency and reliable collection method for transaction data can achieve the technical goal of efficient and parallel processing of multi-source transaction data, and achieve the technical effect of reducing data transmission delays and improving system resource utilization.

[0044] Furthermore, the present application also includes:

[0045] The integrity of the multi-channel data is monitored through a dynamic data acquisition feedback system; when there is data missing in the monitoring result, a data compensation mechanism is activated; the missing data is supplemented by using the data compensation mechanism to obtain the multi-channel data.

[0046] Specifically, the dynamic data collection feedback system can monitor the integrity of multi-channel data in real time to ensure that the collected data is not lost or damaged. Through this monitoring mechanism, the data flow of each channel is continuously analyzed to determine whether there are abnormalities or incomplete situations. For example, in transaction data collection, if the data transmission of a certain channel is affected by network delays or failures, the feedback system can promptly detect data incompleteness.

[0047] Then, when the monitoring results show that there is data missing, the data compensation mechanism is automatically started. At this time, the role of the data compensation mechanism is to respond quickly and find or re-request the missing data to ensure the accuracy and completeness of data collection. For example, if a channel fails to transmit part of the data due to bandwidth problems, the compensation mechanism resends the part of the data to avoid information loss.

[0048] Then, the data compensation mechanism will complete the lost data to restore the entire multi-channel data to a complete state. In this way, the lost data is retrieved from multiple channels to ensure that the data of all channels is complete and meets the requirements. For example, when collecting financial market data, any data loss may lead to biased analysis results, so the completion mechanism can effectively avoid this situation.

[0049] The integrity of multi-channel data is monitored through a dynamic data collection feedback system. When data missing is found, the data compensation mechanism is activated. The compensation mechanism is used to complete the lost data and obtain complete multi-channel data, ensuring the accuracy and continuity of the data. Especially in a complex transaction data environment, the reliability and stability of the system can be guaranteed.

[0050] Further, if Figure 2 As shown, this application also includes:

[0051] S201 , performing redundancy removal processing on the multi-channel data to obtain first pre-processed data.

[0052] Specifically, redundancy processing is performed on multi-channel data to reduce the amount of duplicate or invalid data and improve data processing efficiency. By removing redundancy processing, data can be made more concise and optimized, thus avoiding unnecessary information interference. For example, in the data collection of multiple trading platforms, there may be the same transaction records or useless information. These data can be filtered out through redundancy processing, leaving data that is meaningful for analysis.

[0053] S202: compress the first preprocessed data using a data compression algorithm to obtain second preprocessed data.

[0054] The first preprocessed data is compressed using a data compression algorithm to further reduce the data volume and make data transmission and storage more efficient. The compression algorithm can maintain the integrity of the data while significantly reducing the amount of data that needs to be transmitted. For example, after compression processing, the data transmission time will be greatly shortened, thereby reducing latency in a high-frequency trading environment and improving the response speed of the system.

[0055] S203: Assign different priorities to the importance of the second preprocessed data to obtain the preprocessed data.

[0056] The importance of the second pre-processed data is classified and assigned different priorities. This process can prioritize key data based on the time sensitivity or value of the data to ensure that important information can be transmitted and processed quickly. For example, urgent high-value transaction data will be given a higher priority for rapid transmission, while secondary data can be processed when the system is idle.

[0057] By removing redundancy from multi-channel data, the first pre-processed data is obtained, and then a compression algorithm is used to reduce the data volume. Priority classification is then used to ensure that important data is processed first. The entire process can effectively improve the efficiency of data processing and transmission.

[0058] Furthermore, the present application also includes:

[0059] Extract the priority of the preprocessed data to obtain multiple priorities; allocate bandwidths of the multiple first data channels according to the multiple priorities to obtain multiple allocated bandwidths; obtain multiple transmission orders according to the multiple allocated bandwidths and the multiple priorities; and add the multiple transmission orders to the multiple second data channels.

[0060] Specifically, the priority of extracting pre-processed data means that the system classifies different priority levels according to the importance and timeliness of the data. Through this classification, it is clear which data needs to be processed first and which data can be processed later. For example, in a trading system, urgent high-frequency trading data will be marked as high priority, while non-critical historical data may be given a lower priority.

[0061] Next, bandwidth allocation for the first data channel is performed based on multiple priorities, ensuring that high-priority data obtains more bandwidth resources to speed up transmission and avoid delays or packet loss due to insufficient bandwidth during transmission of important data. For example, real-time transaction data will occupy a larger bandwidth to ensure fast processing, while low-priority data will only occupy less bandwidth.

[0062] Then, multiple transmission orders are generated according to the allocated bandwidth and the corresponding priority. During the transmission process, the data will be transmitted in the order determined by the priority and bandwidth resources. For example, high-priority data will be in the front and transmitted first, while lower-priority data will be in the back.

[0063] Finally, multiple transmission orders will be added to the second data channel to ensure that different channels can smoothly perform transmission tasks according to the importance of data and bandwidth allocation. For example, each channel will arrange the transmission order according to the priority of the data to ensure that critical data is delivered in time.

[0064] By extracting the priority of pre-processed data, allocating bandwidth, generating a transmission order and applying it to the second data channel, it ensures that important data can be transmitted first, optimizes bandwidth utilization and data transmission efficiency, which is particularly critical in application scenarios that process real-time data.

[0065] Furthermore, the present application also includes:

[0066] The data flow of the plurality of second data channels is obtained; when it is detected that the data flow of any second data channel is overloaded, the bandwidth is automatically adjusted to obtain the plurality of second data channels.

[0067] Specifically, obtaining the data flow of multiple second data channels refers to real-time monitoring of the amount of data currently processed by each channel to ensure accurate understanding of the load status of each channel. For example, in a high-frequency trading system, each channel may process different data flows, and it is necessary to continuously obtain these data to determine the load status of the channel.

[0068] Then, when the data traffic of any second data channel is detected to be overloaded, the bandwidth adjustment mechanism is automatically triggered to avoid transmission delays or data loss caused by traffic overload. For example, if the transaction data of a channel suddenly surges, the bandwidth is adjusted to balance the load of the channel to prevent crashes or delays.

[0069] Then, the bandwidth is reallocated through a dynamic adjustment mechanism to ensure that all channels can maintain stable operation when traffic is overloaded, and to ensure that even when data traffic changes greatly, multiple channels can still work in coordination and the overall transmission performance will not be affected by the overload of individual channels.

[0070] By acquiring the data flow of multiple second data channels, detecting overload conditions, and automatically adjusting the bandwidth to cope with load changes, the stability and efficiency of the system can be effectively guaranteed, especially in a high-load network environment, to prevent transmission bottlenecks and ensure smooth data transmission of each channel.

[0071] Furthermore, the present application also includes:

[0072] Perform deep learning training on historical transaction data using an intelligent analysis model; predict the collection trend and load of the preprocessed data using the intelligent analysis model; and allocate the loads of the multiple second data channels according to the prediction results to obtain the loads of the multiple second data channels.

[0073] Specifically, the deep learning training of historical transaction data using intelligent analysis models is to enable the intelligent analysis models to understand and identify the patterns and trends in the data. This training continuously optimizes the model through a large amount of historical data, so that the intelligent analysis model can better predict future data changes. For example, the peak and low peak periods in financial transaction data can be effectively learned through intelligent analysis models, thus laying the foundation for subsequent predictions.

[0074] Next, the intelligent analysis model is used to predict the collection trend and load of pre-processed data, understand the upcoming data traffic in advance, and make corresponding adjustments when the load suddenly increases, so as to avoid insufficient resources or improper bandwidth allocation. For example, the intelligent analysis model can predict that the transaction volume will surge in a certain period of time, and adjust resource allocation in advance to ensure smooth data processing.

[0075] Then, based on the prediction results, the loads of multiple second data channels are allocated to ensure that the loads of each channel are balanced. In this way, each channel can reasonably allocate data according to its predicted load, thereby avoiding overloading of some channels or waste of other channel resources. For example, when it is predicted that a channel is heavily loaded, part of the load is transferred to other channels in time to ensure smooth operation of each channel.

[0076] Through deep learning training of historical data through intelligent analysis models, future data collection trends and loads can be predicted, and the load of the second data channel can be reasonably allocated based on the predictions. This can greatly improve the flexibility and response capabilities of the system, ensuring that the system can still operate stably and efficiently in a complex data environment.

[0077] Furthermore, the present application also includes:

[0078] Configure a multi-level cache mechanism; determine whether the cache data of the multi-level cache mechanism is saturated, and when saturated, automatically clean up the data and save the high-frequency identification data.

[0079] Specifically, the multi-level cache mechanism is configured to perform hierarchical caching according to different data priorities, ensuring that important data can be processed first in the cache, and the cache mechanism can be dynamically adjusted according to the importance of the data. For example, critical transaction data will be stored in a cache layer with a higher priority, while ordinary data may be stored in a lower cache layer for quick access when needed.

[0080] Next, determine whether the cached data in the multi-level cache mechanism is saturated. When the cache space is close to saturation, check the current cache status to ensure that there is enough space to accommodate new data. For example, when multiple transaction data channels transmit data to the cache at the same time, the cache may reach its capacity limit, and then automatically determine the cache status.

[0081] Then, when the cache is detected to be saturated, the data cleaning mechanism is automatically triggered, and high-frequency identification data is retained first, low-priority or infrequently used data is cleared, and space is freed up for important data with high access frequency. For example, real-time transaction data in the financial market is considered high-frequency data, and this data is retained first to ensure the smooth progress of subsequent transactions.

[0082] By configuring a multi-level cache mechanism, intelligent judgment is made based on the saturation level of the cached data, and low-priority data is automatically cleaned up when the cache space is insufficient, retaining high-frequency identification data, thereby optimizing data storage and transmission efficiency and ensuring smooth operation under high-load environments.

[0083] Furthermore, the present application also includes:

[0084] Preset a high-frequency access threshold; traverse the preprocessed data and compare it with the high-frequency access threshold to obtain screening data; perform frequency identification on the screening data according to the number of accesses to obtain the high-frequency identification data.

[0085] Specifically, the preset high-frequency access threshold is to determine the data that needs to be processed and saved first. By setting a standard, the data with high access frequency can be identified and distinguished from other data. For example, real-time transaction data has a high access frequency, so a lower threshold can be set for it to make it easier to meet the high-frequency access standard.

[0086] Next, the pre-processed data is traversed and compared with the preset high-frequency access threshold to ensure that which data reaches the threshold is identified, so as to filter out the data that needs to be processed. For example, when the access frequency of a certain transaction data exceeds the set threshold, it is marked as filtered data and used as the focus of subsequent processing.

[0087] Then, the filtered data is frequency-labeled according to its access times, which can further refine the data priority. By recording the access frequency of each data, high-frequency identification data is generated. For example, although some data exceeds the high-frequency access threshold, its access times are relatively small, and a lower frequency identification is assigned, while data with a higher access times is assigned a higher frequency identification for subsequent processing.

[0088] By presetting high-frequency access thresholds, traversing and comparing pre-processed data to screen out important data, and then marking the frequency according to the number of accesses, it is possible to effectively identify and prioritize data with higher access frequencies, ensuring that critical data can receive higher processing priority and improving the efficiency of the overall system.

[0089] In summary, the low-latency and reliable collection method of transaction data provided by this application has the following technical effects:

[0090] The method collects data from multiple transaction data sources by establishing multiple first data channels to obtain multi-channel data; pre-processes the multi-channel data to obtain pre-processed data; dynamically adjusts the bandwidth allocation of the multiple first data channels according to the importance of the pre-processed data to obtain multiple second data channels; allocates the loads of the multiple second data channels to obtain multiple second data channel loads; transmits the pre-processed data according to the multiple second data channel loads, that is, by creating multiple initial data channels to collect information from different transaction data sources, a multi-channel data stream is obtained; preliminarily processes the multi-channel data to obtain processed data; dynamically adjusts the bandwidth allocation of multiple initial data channels according to the importance of the processed data to form multiple new data channels; loads the new data channels to determine their respective load conditions; finally, transmits the processed data according to the load conditions of the new data channels, so as to achieve the technical goal of efficient parallel processing of multi-source transaction data and achieve the technical effect of reducing data transmission delay and improving system resource utilization.

[0091] Please see attached Figure 3 The present application provides a low-latency trusted transaction data collection device 300, which specifically includes the following modules:

[0092] The collection module 310 is used to establish multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data.

[0093] The preprocessing module 320 is used to preprocess the multi-channel data to obtain preprocessed data.

[0094] The bandwidth allocation module 330 is used to dynamically adjust the bandwidth allocation of the multiple first data channels according to the importance of the pre-processed data to obtain multiple second data channels.

[0095] The load distribution module 340 is used to distribute the loads of the multiple second data channels to obtain multiple second data channel loads.

[0096] The data transmission module 350 is used to transmit the pre-processed data according to the multiple second data channel loads.

[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0098] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0099] According to an embodiment of the present invention, the present invention further provides an electronic device.

[0100] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0101] The electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0102] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0103] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as methods S101 to S105. For example, in some embodiments, methods S101 to S105 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the methods S101 to S105 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute methods S101 to S105 in any other appropriate manner (eg, by means of firmware).

[0104] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0108] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0109] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0110] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0111] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. A low-latency and reliable collection method for transaction data, characterized in that: include: Establish multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data; Preprocessing the multi-channel data to obtain preprocessed data; Dynamically adjusting bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain a plurality of second data channels; Allocating the loads of the plurality of second data channels to obtain a plurality of second data channel loads; The pre-processed data is transmitted according to the plurality of second data channel loads.

2. A low-latency and reliable transaction data collection method as claimed in claim 1, characterized in that: The obtaining of multi-channel data previously includes: Monitoring the integrity of the multi-channel data through a dynamic data acquisition feedback system; When there is data missing in the monitoring results, the data compensation mechanism is activated; The data compensation mechanism is used to supplement the lost data to obtain the multi-channel data.

3. A low-latency and reliable transaction data collection method as claimed in claim 1, characterized in that: The preprocessing of the multi-channel data to obtain preprocessed data includes: Performing redundancy removal processing on the multi-channel data to obtain first pre-processed data; Compressing the first preprocessed data using a data compression algorithm to obtain second preprocessed data; The importance of the second pre-processed data is assigned different priorities to obtain the pre-processed data.

4. A low-latency and reliable transaction data collection method as claimed in claim 3, characterized in that: The step of dynamically adjusting bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain a plurality of second data channels comprises: Extracting the priority of the preprocessed data to obtain multiple priorities; Allocate bandwidths of the plurality of first data channels according to the plurality of priorities to obtain a plurality of allocated bandwidths; Obtaining multiple transmission orders according to the multiple allocated bandwidths and the multiple priorities; The plurality of transmission sequences are added to the plurality of second data channels.

5. A low-latency and reliable transaction data collection method as claimed in claim 1, characterized in that: The method of dynamically adjusting the bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain the plurality of second data channels further includes: Obtaining data flows of the plurality of second data channels; When it is detected that the data flow of any second data channel is overloaded, the bandwidth is automatically adjusted to obtain the multiple second data channels.

6. A low-latency and reliable transaction data collection method as claimed in claim 1, characterized in that: The allocating the loads of the plurality of second data channels to obtain the plurality of second data channel loads comprises: Use intelligent analysis models to conduct deep learning training on historical transaction data; Predicting the collection trend and load of the preprocessed data by the intelligent analysis model; The loads of the plurality of second data channels are distributed according to the prediction result to obtain the plurality of second data channel loads.

7. A low-latency and reliable transaction data collection method as claimed in claim 1, characterized in that: The step of transmitting the pre-processed data according to the plurality of second data channel loads comprises: Configure a multi-level cache mechanism; It is determined whether the cache data of the multi-level cache mechanism is saturated. When saturated, data cleaning is automatically performed and high-frequency identification data is saved.

8. A low-latency and reliable transaction data collection method as claimed in claim 7, characterized in that: Obtain high-frequency identification data, including: Preset high-frequency access threshold; Traversing the pre-processed data and comparing it with the high-frequency access threshold to obtain screening data; The screening data is frequency-labeled according to the number of accesses to obtain the high-frequency labeled data.

9. A low-latency and reliable transaction data collection device, characterized in that: include: A collection module, used to establish multiple first data channels to collect data from multiple transaction data sources to obtain multi-channel data; A preprocessing module, used for preprocessing the multi-channel data to obtain preprocessed data; A bandwidth allocation module, configured to dynamically adjust bandwidth allocation of the plurality of first data channels according to the importance of the preprocessed data to obtain a plurality of second data channels; A load distribution module, used to distribute the loads of the plurality of second data channels to obtain a plurality of second data channel loads; A data transmission module is used to transmit the pre-processed data according to the multiple second data channel loads.

10. An electronic device comprising at least one processor; and A memory communicatively connected to the at least one processor; characterized in that The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images