A real-time distribution method and device for quantitative trading data based on cloud transmission

Abstract

The present invention discloses a method and device for real-time distribution of quantitative trading data based on cloud transmission. When compressing, the present invention performs cluster analysis on the real-time trading data stream, thereby dividing the real-time trading data stream into multiple data clusters; then, the individual difference data between each cluster center and the remaining trading data in the corresponding data cluster is determined, and the multiple data clusters are simplified into a representation form of cluster center plus individual difference data; then, the cluster center and the individual difference data are compressed to obtain a compressed data stream corresponding to the real-time trading data stream; finally, the compressed data stream is sent to each trading device to complete the real-time distribution of the trading data; thus, the present invention only compresses the common part and the difference part in the trading data, thereby greatly reducing the amount of compressed data. Based on this, the compression efficiency can be improved, and the transmission delay caused by excessively long compression time can be reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and in particular relates to a method and device for real-time distribution of quantitative trading data based on cloud transmission. Background Art

[0002] With the rapid development of Internet technology, big data processing technology is needed in more and more fields, especially the financial industry. Since there are many categories of stocks, funds, bonds, and wealth management in financial transactions, in order to enable customers to grasp market information in real time, it is necessary to send market transaction data of many categories to the client in real time. However, due to the high timeliness requirements of data transmission and the rapid growth of the number of products in each category, the amount of market transaction data required to be sent at a time is getting larger and larger. Traditional technologies directly send large batches of data to the client. This data transmission method faces problems such as bandwidth bottlenecks and delays. Effective data compression algorithms can significantly reduce data volume, alleviate these problems, and improve transmission efficiency and resource utilization.

[0003] At present, traditional lossless and lossy compression technologies both directly encode all data. Although they can achieve data compression and reduce transmission delays, the amount of market trading data that needs to be sent at a time is increasing, which makes the compression time longer and longer. This will still increase the transmission delay of trading data, thereby affecting the real-time performance of data transmission. Therefore, how to provide a real-time distribution method for quantitative trading data with high compression efficiency to ensure real-time data transmission has become an urgent problem to be solved. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and device for real-time distribution of quantitative trading data based on cloud transmission, so as to solve the problem that the existing technology directly encodes all data, resulting in excessively long compression time and thus increased data transmission delay.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, a method for real-time distribution of quantitative trading data based on cloud transmission is provided, comprising:

[0007] Get real-time transaction data streams;

[0008] Clustering each transaction data in the real-time transaction data stream to obtain at least one data cluster;

[0009] Determine the cluster center of each data cluster, and determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data;

[0010] Performing data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center;

[0011] Utilizing each compressed cluster center, the compressed individual difference data corresponding to each cluster center, and the difference position corresponding to each individual difference data, a compressed data stream corresponding to the real-time transaction data stream is formed;

[0012] The compressed data stream is distributed to each transaction device to complete the real-time distribution of the real-time transaction data stream.

[0013] Based on the above disclosed content, after obtaining the real-time transaction data stream, the present invention first clusters the transaction data in the real-time transaction data stream to obtain at least one data cluster; then, the cluster center of each data cluster is determined, and the individual difference data between each cluster center and the remaining transaction data in the corresponding data cluster and the difference position corresponding to each individual difference data are determined; then, each cluster center and the individual difference data corresponding to each cluster center are compressed to obtain each compressed cluster center and compressed individual difference data; then, the difference position of the aforementioned individual difference data, the compressed cluster center and the compressed individual difference data can be used to form a compressed data stream; finally, the compressed data stream is sent to each transaction device to complete the real-time distribution of the real-time transaction data stream.

[0014] Through the above design, the present invention performs cluster analysis on the real-time transaction data stream during compression, thereby dividing the real-time transaction data stream into multiple data clusters; then, the individual difference data between each cluster center and the remaining transaction data in the corresponding data cluster is determined, and the multiple data clusters are simplified to a representation of the cluster center plus the individual difference data; then, the cluster center and the individual difference data are compressed to obtain a compressed data stream corresponding to the real-time transaction data stream; finally, the compressed data stream is sent to each transaction device to complete the real-time distribution of the transaction data; thus, the present invention only compresses the common part and the difference part in the transaction data, thereby greatly reducing the amount of compressed data. Based on this, the compression efficiency can be improved, and the transmission delay caused by the excessively long compression time is reduced. Therefore, the present invention is very suitable for large-scale application and promotion in the field of data transmission.

[0015] In one possible design, each transaction data in the real-time transaction data stream is clustered to obtain at least one data cluster, including:

[0016] Calculating the data distance between each transaction data in the real-time transaction data stream;

[0017] Determine multiple initial cluster centers based on the data distance between each transaction data;

[0018] Optimizing the multiple initial cluster centers using a cluster optimization algorithm to obtain multiple optimal initial cluster centers of the real-time transaction data stream;

[0019] Based on multiple optimal initial clustering centers, cluster the real-time transaction data stream to obtain multiple initial data clusters;

[0020] Cluster correction processing is performed on the multiple initial data clusters to obtain the at least one data cluster.

[0021] In a possible design, multiple initial cluster centers are determined based on the data distances between each transaction data, including:

[0022] Using the data distances between each transaction data, a distance matrix is ​​constructed, and the data distances in the distance matrix are sorted from small to large to obtain a distance sequence;

[0023] Based on the distance sequence, a distance threshold is determined;

[0024] For any transaction data in the real-time transaction data stream, calculate the local density of the transaction data based on the distance threshold and the data distance between the transaction data and each designated data in the designated data set, and obtain the local density of each transaction data after polling all transaction data in the real-time transaction data stream, wherein each designated data in the designated data set is each transaction data in the real-time transaction data stream excluding the transaction data;

[0025] For any of the transaction data, at least one target data is selected from the real-time transaction data stream, wherein a local density of any of the target data is greater than a local density of any of the transaction data;

[0026] Filtering the minimum data distance from the data distances between each target data and any transaction data to serve as the calibration distance corresponding to the any transaction data, and calculating the calibration distance corresponding to each transaction data after polling all transaction data in the real-time transaction data stream;

[0027] By using the local density and calibration distance of each transaction data, multiple initial cluster centers are determined from each transaction data.

[0028] In one possible design, a cluster optimization algorithm is used to optimize multiple initial cluster centers to obtain multiple optimal initial cluster centers of the real-time transaction data stream, including:

[0029] Based on multiple initial cluster centers, a bat population is constructed, wherein each individual bat in the bat population corresponds to an initial position, an initial velocity, an initial pulse loudness, and an initial pulse emission frequency, and the initial position of any individual bat corresponds to an initial cluster center;

[0030] Initialize the number of optimization searches t, and cluster the real-time transaction data stream based on the positions of each bat individual at the t-th optimization search to obtain the clusters at the t-th optimization search, where each bat individual corresponds to a cluster, the initial value of t is 1, and when t is 1, the position of any bat individual at the t-th optimization search is the initial position of any bat individual;

[0031] Based on the clustering clusters at the t-th optimization search, the fitness of each bat individual at the t-th optimization search is calculated, wherein the greater the fitness of any bat individual, the higher the clustering accuracy of the cluster corresponding to any bat individual;

[0032] Determine whether the fitness of each bat individual at the tth optimization search is greater than the historical optimal fitness of each bat individual;

[0033] If not, determine the maximum fitness, search weight factor and pulse frequency during the t-th optimization search;

[0034] According to the maximum fitness, search weight factor and pulse frequency, the speed of each bat individual at the t-th optimization search is updated to obtain the updated speed of each bat individual, and the position of the bat individual at the t-th optimization search is updated using the updated speed of each bat individual to obtain the updated position of each bat individual, wherein when t is 1, the speed of any bat individual at the t-th optimization search is the initial speed corresponding to the bat individual;

[0035] The pulse emission frequency of each bat individual during the t-th optimization is used to perturb the updated position of each bat individual to obtain the perturbed position of each bat individual. When t is 1, the pulse emission frequency of any bat individual during the t-th optimization is the initial pulse emission frequency of the bat individual.

[0036] Based on the pulse loudness of each bat individual at the t-th optimization, determine whether to retain the disturbance position corresponding to each bat individual. When t is 1, the pulse loudness of any bat individual at the t-th optimization is the initial pulse loudness of any bat individual.

[0037] If so, then t is incremented by 1, and the disturbance position corresponding to each bat individual is used as the position of each bat individual at the t-th optimization search. The real-time transaction data stream is clustered again based on the position of each bat individual at the t-th optimization search until the fitness of each bat individual at the t-th optimization search is greater than the historical optimal fitness of each bat individual. Based on the position of each bat individual at the t-th optimization search, multiple optimal initial clustering centers of the real-time transaction data stream are determined.

[0038] In one possible design, the search weight factor for the tth optimization search is determined, including:

[0039] According to the following formula (1), the search weight factor for the tth optimization is determined;

[0040] (1)

[0041] In the above formula (1), Represents the search weight factor during the t-th optimization search, represents the maximum number of optimization attempts, denotes the maximum search weight and the minimum search weight, respectively, and represents the weight index;

[0042] Correspondingly, the pulse emission frequency of each bat individual during the t-th optimization is used to perturb the updated position corresponding to each bat individual to obtain the perturbed position corresponding to each bat individual, which includes:

[0043] For any individual bat, generate a first random number, and determine whether the first random number is greater than the pulse emission frequency of any individual bat during the t-th optimization search;

[0044] If so, obtain the maximum disturbance factor, the minimum disturbance factor, and the total number of individuals in the bat population;

[0045] According to the maximum disturbance factor, the minimum disturbance factor and the total number of individuals, the disturbance coefficient of the t-th optimization search is calculated;

[0046] The perturbation coefficient during the t-th optimization search is used to perform perturbation processing on the updated position corresponding to any bat individual to obtain the perturbation position corresponding to any bat individual.

[0047] In a possible design, the perturbation coefficient at the tth optimization search is calculated based on the maximum perturbation factor, the minimum perturbation factor, and the total number of individuals, including:

[0048] According to the following formula (2), the disturbance coefficient of the t-th optimization search is calculated;

[0049] (2)

[0050] In the above formula (2), represents the perturbation coefficient during the t-th optimization search, represent the maximum disturbance factor and the minimum disturbance factor respectively, Represents the total number of individuals.

[0051] In one possible design, performing cluster correction processing on a plurality of initial data clusters to obtain the at least one data cluster includes:

[0052] filtering out misclassified data from the real-time transaction data stream based on each initial data cluster;

[0053] All the incorrectly classified data that have been screened out are used to form a set to be classified;

[0054] Remove all misclassified data from the real-time transaction data stream and use the remaining transaction data to form a correctly classified set;

[0055] For any misclassified data in the set to be classified, determine the transaction data closest to the misclassified data from the correct classification set as the nearest data to the misclassified data;

[0056] According to the nearest data of any of the erroneously classified data, the erroneously classified data is reclassified, and after all the erroneously classified data in the set to be classified are polled, the clustering correction processing of multiple initial data clusters is completed to obtain the at least one data cluster.

[0057] In a possible design, data compression processing is performed on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center, including:

[0058] Performing sampling compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain the sampling compressed data corresponding to each cluster center and the sampling compressed individual difference data corresponding to each cluster center;

[0059] The sampled compressed data corresponding to each cluster center and the sampled compressed individual difference data corresponding to each cluster center are subjected to lossless compression processing, so as to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center after the lossless compression processing.

[0060] In a second aspect, a real-time distribution device for quantitative trading data based on cloud transmission is provided, comprising:

[0061] An acquisition unit, used to acquire real-time transaction data streams;

[0062] A clustering unit, configured to perform clustering processing on each transaction data in the real-time transaction data stream to obtain at least one data cluster;

[0063] A compression unit, configured to determine the cluster center of each data cluster, and to determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data;

[0064] A compression unit is used to perform data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center;

[0065] The compression unit is further configured to compose a compressed data stream corresponding to the real-time transaction data stream using the compressed cluster centers, the compressed individual difference data corresponding to the cluster centers, and the difference positions corresponding to the individual difference data;

[0066] The distribution unit is used to distribute the compressed data stream to each transaction device to complete the real-time distribution of the real-time transaction data stream.

[0067] In a third aspect, another device for real-time distribution of quantitative trading data based on cloud transmission is provided. Taking the device as an electronic device as an example, it includes a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the real-time distribution method of quantitative trading data based on cloud transmission as described in the first aspect or any possible design of the first aspect.

[0068] In a fourth aspect, a storage medium is provided, on which instructions are stored. When the instructions are executed on a computer, the real-time distribution method of quantitative trading data based on cloud transmission is executed as described in the first aspect or any possible design of the first aspect.

[0069] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute the cloud-based transmission-based real-time distribution method for quantitative trading data as described in the first aspect or any possible design of the first aspect.

[0070] Beneficial effects:

[0071] When performing compression, the present invention performs cluster analysis on the real-time transaction data stream, thereby dividing the real-time transaction data stream into multiple data clusters; then, the individual difference data between each cluster center and the remaining transaction data in the corresponding data cluster is determined, and the multiple data clusters are simplified to a representation form of the cluster center plus the individual difference data; then, the cluster center and the individual difference data are compressed to obtain a compressed data stream corresponding to the real-time transaction data stream; finally, the compressed data stream is sent to each transaction device to complete the real-time distribution of the transaction data; thus, the present invention only compresses the common part and the difference part in the transaction data, thereby greatly reducing the amount of compressed data, based on this, the compression efficiency can be improved, and the transmission delay caused by the excessively long compression time is reduced. Therefore, the present invention is very suitable for large-scale application and promotion in the field of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of the steps of a real-time distribution method for quantitative trading data based on cloud transmission provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of a real-time distribution device for quantitative trading data based on cloud transmission provided by an embodiment of the present invention;

[0074] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0076] It should be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments of the present invention.

[0077] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may indicate three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this document describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B may indicate two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0078] Example:

[0079] See also Figure 1 As shown, the real-time distribution method of quantitative trading data based on cloud transmission provided by this embodiment performs cluster analysis on the real-time trading data stream during compression, and determines the individual difference data between the data clusters obtained by clustering and the remaining trading data in the clusters. In this way, the real-time trading data stream can be simplified to a representation of the cluster center plus the individual difference data; then, the aforementioned cluster center and individual difference data are compressed to obtain a compressed data stream corresponding to the real-time trading data stream. Finally, the compressed data stream is sent to each trading device to complete the real-time distribution of the trading data; thus, the method only compresses the common part and the difference part in the trading data, thereby greatly reducing the amount of compressed data. Based on this, the compression efficiency can be improved, and the transmission delay caused by the long compression time is reduced. Therefore, the method is very suitable for large-scale application and promotion in the field of data transmission; among them, for example, the method can be, but is not limited to, running on the cloud server side. It can be understood that the aforementioned execution subject does not constitute a limitation on the embodiment of the present application. Accordingly, the running steps of the method can be, but are not limited to, as shown in the following steps S1 to S6.

[0080] S1. Obtaining a real-time transaction data stream. In this embodiment, the real-time transaction data stream may include, but is not limited to, market transaction data for a variety of categories, such as stocks, funds, bonds, and wealth management products. The real-time transaction data is crawled by the cloud server from various financial transaction websites or platforms at preset time intervals. This ensures the real-time nature of the transaction data. Of course, the foregoing examples are for illustrative purposes only, and the types of transaction data are not limited to these examples.

[0081] After acquiring the real-time trading data stream, in order to transmit the collected real-time trading data stream to each node (i.e., trading device) in the trading system with low latency, thereby ensuring that market data and trading signals can be distributed to each trading device (i.e., trading terminal) in real time and stably, this embodiment provides an improved real-time compression method to reduce the compression time of the real-time trading data stream, thereby alleviating the problem that traditional compression methods compress all data, resulting in excessive compression time and increased transmission delay.

[0082] The compression process of the real-time transaction data stream may be, but is not limited to, steps S2 to S5 as shown below.

[0083] S2. Clustering each transaction data in the real-time transaction data stream to obtain at least one data cluster. In this embodiment, cluster analysis is performed on the real-time transaction data stream to cluster similar transaction data in the real-time transaction data stream into a cluster. Then, individual difference data between the cluster center of each cluster and the remaining transaction data in the corresponding cluster is determined. Finally, only the cluster center and the individual difference data are compressed to reduce the amount of compressed data and thereby achieve the purpose of reducing compression time.

[0084] Optionally, the clustering process of the real-time transaction data stream may be, but is not limited to, steps S21 to S25 as shown below.

[0085] S21. Calculate the data distance between each transaction data in the real-time transaction data stream. In this embodiment, each transaction data in the real-time transaction data stream can be encoded as a vector (e.g., using one-hot encoding), and then the data distance between each transaction data can be calculated using a vector distance formula. Simultaneously, for any transaction data in the real-time transaction data stream, the data distance between the transaction data and each remaining transaction data in the real-time transaction data stream is calculated. In this manner, after all transaction data are polled, the data distance between each transaction data in the real-time transaction data stream can be obtained.

[0086] S22. Determine multiple initial cluster centers based on the data distances between the transaction data. In this embodiment, the local density of the data points is used to determine the multiple initial cluster centers. The process may be, but is not limited to, steps S22a to S22f below.

[0087] S22a. A distance matrix is ​​constructed using the data distances between each transaction data item, and the data distances in the distance matrix are sorted in ascending order to obtain a distance sequence. In this embodiment, any row in the distance matrix represents the data distance between a transaction data item and each remaining transaction data item in the real-time transaction data stream. For example, if the first row in the distance matrix corresponds to the first transaction data item, then that row represents the data distance between the first transaction data item and each remaining transaction data item in the real-time transaction data stream. After constructing the distance matrix in the aforementioned manner, the elements in the distance matrix can be sorted in ascending order to obtain a distance sequence. A distance threshold can then be determined based on the distance sequence, so that the local density of each transaction data item can be subsequently calculated based on the distance threshold. The process of determining the distance threshold is shown in the following step S22b.

[0088] S22b. Determine a distance threshold based on the distance sequence. In this embodiment, the distances of the 2% of data in the sorted distance sequence are screened out, and then the average is calculated to serve as the distance threshold.

[0089] After the distance threshold is determined, the local density of each transaction data can be calculated based on it, and the process is shown in the following step S22c.

[0090] S22c. For any transaction data in the real-time transaction data stream, calculate the local density of the any transaction data based on the distance threshold and the data distance between the any transaction data and each designated data in the designated data set, and obtain the local density of each transaction data after all transaction data in the real-time transaction data stream are polled, wherein each designated data in the designated data set is each transaction data in the real-time transaction data stream excluding the any transaction data. In specific applications, the local density of the aforementioned any transaction data can be calculated by, for example but not limited to, the following formula (3).

[0091] (3)

[0092] In the above formula (3), represents the local density of any transaction data, Indicates that any transaction data and the first The data distance between the specified data, represents the distance threshold, Indicates the total number of specified data, where is the local density function, and when When it is less than 0, the local density function takes the value of 1. When it is greater than or equal to 0, the local density function takes the value of 0.

[0093] Thus, based on the aforementioned formula (3), the local density of each transaction data in the real-time transaction data stream can be calculated. Based on this, the initial cluster center can be preliminarily screened based on the local density of the data points. The process is shown in the following steps S22d to S22f.

[0094] S22d. For any of the transaction data, at least one target data is screened out from the real-time transaction data stream, wherein the local density of any target data is greater than the local density of any of the transaction data. In specific implementation, for any of the transaction data, this is equivalent to screening out transaction data having a local density greater than the local density of the transaction data, and then using it as the target data. Finally, the initial cluster center can be preliminarily selected with the help of the target data determined above, and the process is shown in the following steps S22e to S22f.

[0095] S22e. From the data distances between each target data and any of the transaction data, the minimum data distance is selected as the calibration distance corresponding to the any of the transaction data. After all the transaction data in the real-time transaction data stream are polled, the calibration distance corresponding to each of the transaction data is selected. In this embodiment, this step is to select the high-density data point closest to the any of the transaction data. Then, the distance between the high-density data point and the aforementioned any of the transaction data, that is, the high-density distance, is used as the calibration distance. Finally, multiple initial cluster centers can be determined based on the calibration distance and the local density. The process is shown in the following step S22f.

[0096] S22f. Utilize the local density and calibration distance of each transaction data to determine multiple initial cluster centers from each transaction data. In this embodiment, the standard deviation of the calibration distances corresponding to all transaction data in the real-time transaction data stream and the density mean of the local density corresponding to all transaction data are first calculated. Then, based on the standard deviation, a calibration threshold is determined (for example, the calibration threshold is 2 times the standard deviation). Next, from each transaction data in the real-time transaction data stream, transaction data having a calibration distance greater than or equal to the calibration threshold are screened out as preselected initial cluster centers. Finally, from each preselected initial cluster center, preselected initial cluster centers having a local density greater than or equal to the density mean are screened out, and the screened-out preselected initial cluster centers are used as the initial cluster centers.

[0097] In this way, this embodiment is equivalent to using the density of data points in the real-time transaction data stream (i.e., local density) to objectively select the initial cluster centers, that is, the selected cluster centers should have a large distance between them. Therefore, the high-density distance of other data points in the cluster (i.e., the aforementioned calibration distance) should be less than or equal to the aforementioned calibration threshold. Based on this, transaction data with a calibration distance greater than the calibration threshold can be used as the pre-selected initial cluster centers; at the same time, in actual use, there may be noise data points in the data set with a large high-density distance but low local density. Once such noise data points are selected, they are very likely to interfere with the selection of other normal cluster centers. Therefore, it is also necessary to use local density to denoise the pre-selected initial cluster centers, so that the initial cluster centers can be obtained after denoising.

[0098] Therefore, through the aforementioned steps S22a to S22f, the selection of the initial cluster center can be completed, and then the initial cluster center can be optimized to obtain at least one optimal initial cluster center; the reason for optimizing the initial cluster center is that the clustering algorithm is sensitive to the initial cluster center. If the initial cluster center is not selected reasonably, it is easy for the cluster to fall into the local optimum and the convergence speed is slow; therefore, this embodiment provides a cluster optimization algorithm to avoid the problem of clustering falling into the local optimum due to unreasonable selection of the initial cluster center.

[0099] The optimization process of the initial cluster center may be, but is not limited to, the process shown in the following step S23.

[0100] S23. Utilize a cluster optimization algorithm to optimize the multiple initial cluster centers to obtain multiple optimal initial cluster centers for the real-time transaction data stream. In specific applications, this embodiment provides an improved bat algorithm to optimize the initial cluster centers, and the process can be, but is not limited to, steps S23a to S23i as shown below.

[0101] S23a. A bat population is constructed based on the multiple initial cluster centers, wherein each individual bat in the bat population corresponds to an initial position, an initial velocity, an initial pulse loudness, and an initial pulse emission frequency, and the initial position of any individual bat corresponds to one initial cluster center. In this embodiment, the number of individual bats is the same as the number of initial cluster centers, and there is a one-to-one correspondence between them. Therefore, the initial cluster center corresponding to each individual bat serves as its initial position. Of course, the aforementioned initial velocity, initial pulse loudness, and initial pulse emission frequency are preset values, and the initial velocity, initial pulse loudness, and initial pulse emission frequency corresponding to each individual bat are different and can be specifically set according to actual use.

[0102] After the bat population is constructed, the bat sonar detection of individual bats can be used to optimize the initial cluster center, and the process is shown in the following steps S23b to S23i.

[0103] S23b. Initialize the number of optimization searches t, and cluster the real-time transaction data stream based on the position of each bat individual at the t-th optimization search, so as to obtain the clustering cluster at the t-th optimization search, wherein each bat individual corresponds to a clustering cluster, the initial value of t is 1, and when t is 1, the position of any bat individual at the t-th optimization search is the initial position of any bat individual; in specific applications, it is equivalent to determining the clustering center of the real-time transaction data stream at the t-th optimization search based on the position of each bat individual at the t-th optimization search, and then clustering the real-time transaction data stream according to the clustering center at the t-th optimization search, so as to obtain As for the clustering clusters during the t-th optimization search, as explained above, the initial position of the individual bat corresponds to an initial clustering center. Therefore, at the beginning of the optimization search (i.e., the first optimization search), the multiple initial clustering centers determined in the aforementioned step S22 are used to cluster the real-time transaction data stream to obtain the clustering clusters during the first optimization search; then, the positions of the individual bats are updated to obtain the clustering centers during the second optimization search, and then, the real-time transaction data stream is clustered using the clustering centers during the second optimization search. In this way, according to this principle, until the end of the optimization search, the optimal initial clustering center can be determined based on the position of the individual bats.

[0104] Furthermore, in this embodiment, the K-means clustering algorithm can be used, for example but not limited to, to perform clustering processing on the real-time transaction data stream. Of course, K-means clustering is a commonly used method for data clustering, and its principle will not be repeated here.

[0105] After obtaining the clustering clusters during the t-th optimization search, the fitness of each bat individual during the t-th optimization search is calculated based on this, so that the optimization search can be stopped based on the fitness. The fitness calculation process is shown in the following step S23c.

[0106] S23c. Based on the clustering clusters at the t-th optimization search, the fitness of each bat individual at the t-th optimization search is calculated, wherein the greater the fitness of any bat individual, the higher the clustering accuracy of the clustering cluster corresponding to the any bat individual; in specific implementation, the fitness of any bat individual at the t-th optimization search is used to characterize the dissimilarity between each transaction data and the clustering center in the clustering cluster corresponding to the any bat individual at the t-th optimization search, and the smaller the dissimilarity, the greater the fitness; at the same time, in this embodiment, one bat individual corresponds to one clustering cluster, therefore, taking any bat individual as an example, the fitness calculation process is explained, namely: first calculate the distance between the clustering center of the clustering cluster corresponding to the any bat individual and the rest of the data in the clustering cluster; then, sum the distances between the clustering center and the data, and take the inverse of the sum, to obtain the fitness of the any bat individual at the t-th optimization search.

[0107] For example, assuming that the cluster center of any bat individual during the t-th optimization search is transaction data C, and its corresponding cluster cluster is cluster 1, then the distance between each transaction data in cluster 1 and transaction data C is calculated, and then the sum of the distances is calculated and the inverse is taken to obtain the fitness of any bat individual during the t-th optimization search; thus, the above fitness calculation process can conclude that: the more similar the data in the cluster are to the cluster center, the greater the fitness, that is, the smaller the dissimilarity, the greater the fitness.

[0108] Based on this, after the fitness of each bat individual during the t-th optimization search is calculated based on step S23, a decision on whether to stop the optimization search iteration can be made, and the process is shown in the following step S23d.

[0109] S23d. Determine whether the fitness of each bat individual at the t-th optimization search is greater than the historical optimal fitness of each bat individual; in this embodiment, the historical optimal fitness of any bat individual is the maximum fitness obtained by any bat individual in the process before the t-th optimization search; of course, at the beginning of the optimization search, such as at the first optimization search, the fitness of each bat individual at the first optimization search can be directly set as the corresponding historical optimal fitness, and at this time, the optimization search needs to be continued; in this way, when the second optimization search is completed, each bat individual calculates a fitness again, and at this time, the fitness of each bat individual at the second optimization search can be compared with the corresponding historical optimal fitness. If it is greater than, the historical optimal fitness is updated to the fitness at the second optimization search, otherwise, it is not updated; based on this, the optimization is continuously performed, and the historical optimal fitness can be continuously updated; of course, in the aforementioned step S23d, if the iteration stop condition cannot be met, the optimization search needs to be continued, and the process is shown in the following steps S23e to S23i.

[0110] S23e. If not, the maximum fitness, search weight factor, and pulse frequency during the t-th optimization search are determined. In this embodiment, the traditional bat algorithm does not add any weight when performing speed update, resulting in a slow search speed or premature convergence during the search. Therefore, this embodiment balances the global search and local search of the algorithm by adding a search weight factor. For example, but not limited to, the search weight factor during the t-th optimization search can be determined according to the following formula (1).

[0111] (1)

[0112] In the above formula (1), Represents the search weight factor during the t-th optimization search, represents the maximum number of optimization attempts, denotes the maximum search weight and the minimum search weight, respectively, and Represents the weight index; in this embodiment, the maximum search weight, the minimum search weight and the weight index can be set to, but not limited to, 0.9, 0.2 and 2 respectively; of course, the specific numerical settings can be set according to actual use and are not limited to the above examples.

[0113] Thus, from the above formula (1), we can know that in the early stage of optimization, the search weight factor is large, and as the number of optimizations increases, the search weight factor gradually becomes smaller. The larger the value of the search weight factor, the stronger the global search capability, and vice versa. Therefore, after adding the search weight factor, we can have a stronger global search capability in the early stage to improve the convergence speed, and a higher local search in the later stage to improve the search accuracy.

[0114] After calculating the search weight factor during the t-th optimization search, the pulse frequency during the t-th optimization search can be calculated so that the speed of the individual bat can be updated based on the pulse frequency.

[0115] In this embodiment, each of the aforementioned bat individuals corresponds to a pulse frequency during the t-th optimization search. Taking any bat individual as an example, the following formula (4) is used to calculate its corresponding pulse frequency during the t-th optimization search.

[0116] (4)

[0117] In the above formula (4), represents the pulse frequency of any bat individual during the t-th optimization, represent the maximum pulse frequency and the minimum pulse frequency respectively, and It represents a random number on [0,1].

[0118] Thus, according to the above formula (4), due to The pulse frequency of each bat individual during the t-th optimization search is different; after calculating the search weight factor and pulse frequency during the t-th optimization search, the speed and position can be updated, and the process is shown in the following step S23f.

[0119] S23f. Based on the maximum fitness, search weight factor and pulse frequency, the speed of each bat individual at the t-th optimization search is updated to obtain the updated speed corresponding to each bat individual, and the position of the bat individual at the t-th optimization search is updated using the updated speed corresponding to each bat individual to obtain the updated position corresponding to each bat individual, wherein when t is 1, the speed of any bat individual at the t-th optimization search is the initial speed corresponding to any bat individual.

[0120] In a specific implementation, for example, but not limited to, first determining the optimal bat position at the t-th optimization search based on the maximum fitness at the t-th optimization search (i.e., the position of the bat individual corresponding to the maximum fitness at the t-th optimization search is used as the optimal bat position at the t-th optimization search); then, for any bat individual, based on the optimal bat position, the search weight factor, and the pulse frequency, and using the following formula (5), calculate the updated speed corresponding to the any bat individual.

[0121] (5)

[0122] In the above formula (5), represents the updated speed corresponding to any bat individual, represents the speed of any bat individual during the t-th optimization search, represents the position of any bat individual at the tth optimization time, represents the optimal bat position, represents the pulse frequency of any bat individual during the t-th optimization, Represents the search weight factor during the t-th optimization search.

[0123] In this way, based on the above formula (5), the speed update of any bat individual can be completed; then, the position update can be performed. Taking any bat individual as an example, the following is explained. First, a random vector is generated for any bat individual (in this embodiment, the elements in the random vector have values ​​between [-1, 1], and the length is the same as the length of the position vector corresponding to the bat individual); then, based on the updated speed corresponding to any bat individual and the random vector, the updated position corresponding to any bat individual is determined.

[0124] Optionally, the following formula (6) is used to calculate the updated position corresponding to any bat individual.

[0125] (6)

[0126] The above formula (6) represents the updated position corresponding to any bat individual, represents a random vector, represents the norm of the vector; based on the above formula (6), it can be seen that this embodiment introduces random variables to increase the ability of individual bat positions to change, thereby improving the diversity of the population.

[0127] In this way, based on the above formulas (5) and (6), the speed and position of each individual bat can be updated; then, a local search of the individual bat can be performed, that is, position disturbance, and the process is shown in the following step S23g.

[0128] S23g. Use the pulse emission frequency of each bat individual during the t-th optimization search to perturb the updated position corresponding to each bat individual to obtain the perturbed position corresponding to each bat individual, wherein when t is 1, the pulse emission frequency of any bat individual during the t-th optimization search is the initial pulse emission frequency of any bat individual.

[0129] In specific applications, the search step size during local search in the traditional bat algorithm is a fixed value (usually between [-1,1]), which is obviously unable to adapt to the changes in the algorithm during operation. At the same time, a larger step size is beneficial to improving the global exploration ability of the algorithm, and a smaller step size is beneficial to the local development ability of the algorithm and improves the optimization accuracy. Therefore, this embodiment provides a step size with adaptive adjustment to improve the adaptability of the algorithm.

[0130] In this embodiment, any individual bat is taken as an example to illustrate the position disturbance step, and the process may be but is not limited to the following steps S23g1 to S23g4.

[0131] S23g1. For any individual bat, generate a first random number, and determine whether the first random number is greater than the pulse emission frequency of any individual bat during the t-th optimization search; in this embodiment, when t is 1, the pulse emission frequency of any individual bat during the t-th optimization search is the initial pulse emission frequency of any individual bat; wherein, when the first random number is greater than the pulse emission frequency of any individual bat during the t-th optimization search, a local search can be performed, and the process is shown in the following steps S23g2 to S23g4.

[0132] If so, the maximum disturbance factor, the minimum disturbance factor, and the total number of individuals in the bat population are obtained. In this embodiment, the maximum and minimum frequency factors are set values ​​and are not specifically limited here. After obtaining the maximum disturbance factor, the minimum disturbance factor, and the total number of individuals in the bat population, the disturbance coefficient for the t-th optimization search can be calculated. The process is shown in the following step S23g3.

[0133] S23g3. Calculate the disturbance coefficient for the t-th optimization search based on the maximum disturbance factor, the minimum disturbance factor, and the total number of individuals. In this embodiment, the disturbance coefficient for the t-th optimization search may be calculated, for example but not limited to, according to the following formula (2).

[0134] (2)

[0135] In the above formula (2), represents the perturbation coefficient during the t-th optimization search, represent the maximum disturbance factor and the minimum disturbance factor respectively, Represents the total number of individuals.

[0136] Thus, based on formula (2), it can be seen that this embodiment uses an exponential decreasing factor to replace the fixed step size (ie, the aforementioned perturbation coefficient) in the traditional technology, thereby improving the optimization accuracy.

[0137] After calculating the perturbation coefficient during the t-th optimization search, the updated position corresponding to any individual bat can be disturbed based on the perturbation coefficient, and the process is shown in the following step S23g4.

[0138] S23g4. Using the perturbation coefficient from the t-th optimization search, perform perturbation processing on the updated position corresponding to any individual bat to obtain the perturbed position corresponding to any individual bat. In this embodiment, the perturbation of the updated position of any individual bat may be performed using, for example but not limited to, the following formula (7).

[0139] (7)

[0140] In the above formula (7), represents the disturbance position corresponding to any bat individual, represents the average pulse loudness of all individual bats in the bat population during the t-th optimization search. In this embodiment, when t is 1, the pulse loudness of any individual bat during the t-th optimization search is the initial pulse loudness of that individual bat. Thus, during the first optimization search, the average pulse loudness of all individual bats is the mean of the initial pulse loudnesses of all individual bats. Of course, the pulse loudness and pulse emission frequency are continuously updated during each iteration, and the update process is described in detail below.

[0141] In this way, based on the aforementioned steps S23g1 to S23g4, the position disturbance of each individual bat can be completed, that is, the local search of the position of each individual bat can be achieved; then, it is necessary to determine whether the position disturbance is acceptable, and the process is shown in the following step S23h.

[0142] S23h. Based on the impulse loudness of each individual bat during the t-th optimization search, determine whether to retain the perturbed position corresponding to each individual bat. When t is 1, the impulse loudness of any individual bat during the t-th optimization search is the initial impulse loudness of the individual bat. In a specific application, for any individual bat, a second random number is generated. Then, determine whether the second random number is less than the impulse loudness of the individual bat during the t-th optimization search. If so, calculate the fitness of the individual bat before and after the position perturbation to obtain a new fitness and an original fitness, respectively. Then, determine whether the new fitness is greater than the original fitness. If it is greater than the original fitness, accept the position perturbation, i.e., use the perturbed position corresponding to the individual bat as its position during the next iteration. Assuming that this is the first optimization search, the corresponding perturbed position is the position of the individual bat during the second optimization search.

[0143] Of course, if the above conditions are not met, that is, the second random number is greater than or equal to the pulse loudness of any bat individual during the t-th optimization, or the new fitness is less than or equal to the original fitness, then the position disturbance is not accepted, that is, the updated position corresponding to any bat individual is directly used as the position of the next iteration.

[0144] At the same time, the following disclosure of the update process of the pulse loudness and pulse emission frequency of any individual bat is also explained using any individual bat as an example:

[0145] Among them, for example, but not limited to, the following formulas (8) and (9) can be used to update the pulse loudness and the pulse transmission frequency.

[0146] (8)

[0147] (9)

[0148] In the above formula (8), represents the impulse loudness of any bat individual at the t+1th optimization time, represents the impulse loudness of any bat individual during the t-th optimization search, is the attenuation coefficient, which takes values ​​in [-1,1].

[0149] In the above formula (9), represents the pulse emission frequency of any bat individual during the t-th optimization, represents the frequency adjustment coefficient, greater than 0, represents the initial pulse emission frequency of any bat individual.

[0150] In this way, after the determination of whether the disturbance position is to be retained is completed based on the aforementioned step S23h, further iterations may be performed, and the process is shown in the following step S23i.

[0151] S23i. If so, t is incremented by 1, and the disturbance position corresponding to each bat individual is used as the position of each bat individual at the t-th optimization search. The real-time transaction data stream is clustered again based on the position of each bat individual at the t-th optimization search until the fitness of each bat individual at the t-th optimization search is greater than the historical optimal fitness of each bat individual. Then, based on the position of each bat individual at the t-th optimization search, multiple optimal initial cluster centers of the real-time transaction data stream are determined. In this embodiment, when t is 2, it is equivalent to using the disturbance position corresponding to each bat individual at the first optimization search as the position of each bat individual at the second optimization search. Then, the aforementioned steps S23b to S23i are re-executed until the aforementioned iterative stopping condition is met. At this time, the position of each bat individual when the iterative stopping condition is met can be used as the optimal initial cluster center of the real-time transaction data stream. In addition, in this embodiment, if the optimal initial cluster center obtained by optimization does not have corresponding transaction data in the real-time transaction data stream, then the transaction data with the smallest distance to the optimal initial cluster center can be used as the optimal initial cluster center.

[0152] Of course, in the aforementioned step S23g1, for any bat individual, if the first random number generated is less than or equal to the pulse emission frequency of any bat individual during the t-th optimization, then no local search will be performed on the bat individual, and step S23i can be directly executed, that is, its updated position will be used as the position for the next iteration.

[0153] In this way, through the aforementioned steps S23a to S23i, the optimization process of the initial cluster center can be completed, and then, the real-time transaction data stream can be clustered based on the optimal initial cluster center, and the process is shown in the following step S24.

[0154] S24. Based on multiple optimal initial clustering centers, the real-time transaction data stream is clustered to obtain multiple initial data clusters. In this embodiment, based on multiple optimal initial clustering centers, the real-time transaction data stream is clustered using the K-means clustering algorithm to obtain multiple initial data clusters.

[0155] Since the K-means algorithm may cause some data objects in the clustering result to be incorrectly classified when clustering data, this embodiment further includes a cluster correction step, the process of which is shown in the following step S25.

[0156] S25. Perform cluster correction processing on the multiple initial data clusters to obtain the at least one data cluster. In this embodiment, cluster correction can be performed by, for example but not limited to, the following steps S25a to S25e.

[0157] S25a. Filter out misclassified data from the real-time transaction data stream based on each initial data cluster. In a specific application, for any transaction data in the real-time transaction data stream, the distance between the any transaction data and the cluster center in each initial data cluster can be first calculated, and the calculated distances can be sorted in ascending order to obtain a sorted sequence. Then, the first two distances in the sorted sequence are filtered out, and it is determined whether the absolute value of the difference between the first two distances is less than a preset threshold. If so, the any transaction data is treated as misclassified data, and when all transaction data in the real-time transaction data stream are polled, all misclassified data are filtered out from the real-time transaction data stream.

[0158] In this embodiment, if the absolute value of the distance difference between the data object and the two nearest class centers is less than a preset threshold, it is classified as misclassified data. In this way, after all misclassified data are filtered out from the real-time transaction data stream, a set to be classified can be formed. The process is shown in the following step S25b.

[0159] S25b. All incorrectly classified data that has been filtered out is used to form a set to be classified. After the set to be classified is obtained, the incorrectly classified data can be deleted from the real-time transaction data stream to obtain correctly classified data. The correctly classified data can then be used to form a correctly classified set. The process is shown in step S25c below.

[0160] S25c. Delete all incorrectly classified data from the real-time transaction data stream and use the remaining transaction data to form a correctly classified set. After the correctly classified set is formed based on this step, the incorrectly classified data in the set to be classified can be reclassified based on this. This process is shown in steps S25d and S25e below.

[0161] S25d. For any erroneously classified data in the set to be classified, determine the transaction data closest to the erroneously classified data from the correctly classified set as the nearest data to the erroneously classified data. In this embodiment, this is equivalent to selecting the correctly classified transaction data most similar to the erroneously classified data from the correctly classified set as the nearest data to the erroneously classified data. Then, the erroneously classified data can be reclassified based on the nearest data. The reclassification process is shown in step S25e below.

[0162] S25e. Reclassify any erroneously classified data according to its nearest data, and after polling all erroneously classified data in the set to be classified, complete clustering correction of multiple initial data clusters to obtain at least one data cluster. In this embodiment, any erroneously classified data is directly classified into the initial data cluster corresponding to its nearest data. For example, assuming that any erroneously classified data originally belongs to initial data cluster 3, and the initial data cluster where its nearest data belongs is cluster 2, then any erroneously classified data is divided into cluster 2. Thus, clustering correction of any erroneously classified data can be completed.

[0163] In this way, through the aforementioned steps S25a to S25e, this embodiment finds the nearest neighbors of each data object in the set to be classified, and with the help of the nearest neighbor idea, classifies the data objects in the set to be classified into the category of their nearest neighbors. In this way, the data objects that are misclassified in the initial clustering results can be corrected back to the correct category, thereby improving the accuracy of clustering.

[0164] Therefore, through the aforementioned steps S21 to S25, the clustering processing of the real-time transaction data stream can be completed, so that similar data in the real-time transaction data stream can be classified into one category; then, the difference between the cluster center and the remaining data in each data cluster can be determined, so that the data cluster can be represented as the cluster center + individual difference representation; finally, only the cluster center and individual difference data are compressed to obtain the compressed data stream of the real-time transaction data stream.

[0165] The process of determining the individual difference data is shown in the following step S3.

[0166] S3. Determine the cluster center of each data cluster, and determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data; in this embodiment, when using the K-means clustering algorithm, the cluster center of each data cluster will be determined, and then, the cluster center and each transaction data in the data cluster can be compared to obtain the individual difference data between the cluster center and each remaining transaction data in the corresponding data cluster.

[0167] Optionally, an example is used below to illustrate:

[0168] Assume that the cluster center of data cluster 3 is: CGTGTACTGTGATACGTG (starting at position 0 and ending at position 17), and the transaction data contained in the cluster are Q1 and Q2, where Q1 is: CGACTGTACTATGATACG and Q2 is: CGCGTACTGATTGATACGTG. Then, by comparing the cluster center with Q1, we can see that the difference between the two is: inserting AC after the first position of the cluster center, replacing G at the eighth position with A, and deleting the 15th and 16th positions, we can get transaction data Q1. Similarly, by replacing the second digit of the cluster center with C and inserting AT after the eighth digit, the transaction data Q2 can be obtained. Based on this, the individual difference data between the cluster center and Q1 are AC, A and GT, and the corresponding difference positions are the first, eighth and fifteenth to sixteenth digits, and the corresponding difference operations are insertion, replacement and deletion. Of course, the principles for obtaining the individual difference data and difference positions between the cluster center and the transaction data Q2 are the same as those in the above examples and will not be repeated here. Of course, the above examples are only for illustration, and the data content of the examples is not regarded as real transaction data.

[0169] In this way, after determining the individual difference data and difference positions of each cluster center and the remaining transaction data in the corresponding data cluster, compression processing of the cluster center and the individual difference data can be performed, and the process is shown in the following step S4.

[0170] S4. Perform data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center; in specific applications, first perform sampling compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain the sampled compressed data corresponding to each cluster center and the sampled compressed individual difference data corresponding to each cluster center; then, perform lossless compression processing on the sampled compressed data corresponding to each cluster center and the sampled compressed individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center after the lossless compression processing.

[0171] Optionally, when performing sampling compression processing, any cluster center and the individual difference data corresponding to the any cluster center are encoded into a column vector, and then a 128×256 observation matrix is ​​used to perform matrix multiplication operation with the column vector to obtain a set of 128-dimensional column vectors as the compression result after compressed sampling (that is, the sampled compressed data and the sampled compressed individual difference data are actually in one column vector); then, the LZW algorithm (a lossless data compression algorithm based on a dynamic dictionary) is used to losslessly compress the aforementioned sampled compressed data and the sampled compressed individual difference data, thereby obtaining the compressed any cluster center and the compressed individual difference data corresponding to the any cluster center; of course, the compression process of the remaining cluster centers and their corresponding individual difference data is the same, and will not be repeated here.

[0172] Furthermore, the aforementioned LZW algorithm is a commonly used technology for lossless compression, and its principle will not be described in detail.

[0173] Thus, after the data compression is completed based on the aforementioned step S4, the difference positions of the aforementioned individual difference data can be combined to form a compressed data stream, and the process is shown in the following step S5.

[0174] S5. Utilize the compressed cluster centers, the compressed individual difference data corresponding to the cluster centers, and the difference positions corresponding to the individual difference data to form a compressed data stream corresponding to the real-time transaction data stream; in this embodiment, it is still necessary to associate and record the difference operations corresponding to the difference positions to obtain a compressed data stream, so as to subsequently decompress the data based on the difference operations and positions; and after obtaining the compressed data stream corresponding to the real-time transaction data stream, the data can be distributed, and the process is shown in the following step S6.

[0175] S6. Distribute the compressed data stream to each transaction device to complete the real-time distribution of the real-time transaction data stream.

[0176] Therefore, according to the real-time distribution method of quantitative transaction data based on cloud transmission described in detail in the aforementioned steps S1 to S6, the present invention performs cluster analysis on the real-time transaction data stream during compression, and determines the individual difference data between the data clusters obtained by clustering and the remaining transaction data in the clusters. In this way, the real-time transaction data stream can be simplified to a representation of the cluster center plus the individual difference data; then, the aforementioned cluster center and individual difference data are compressed to obtain a compressed data stream corresponding to the real-time transaction data stream; finally, the compressed data stream is sent to each transaction device to complete the real-time distribution of the transaction data; thus, the present invention only compresses the common part and the difference part in the transaction data, thereby greatly reducing the amount of compressed data. Based on this, the compression efficiency can be improved, and the transmission delay caused by the excessively long compression time is reduced. Therefore, the present invention is very suitable for large-scale application and promotion in the field of data transmission.

[0177] like Figure 2 As shown, the second aspect of this embodiment provides a hardware device for implementing the real-time distribution method of quantitative trading data based on cloud transmission described in the first aspect of the embodiment, including:

[0178] The acquisition unit is used to obtain real-time transaction data streams.

[0179] The clustering unit is used to perform clustering processing on each transaction data in the real-time transaction data stream to obtain at least one data cluster.

[0180] The compression unit is used to determine the cluster center of each data cluster, and determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data.

[0181] The compression unit is used to perform data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center.

[0182] The compression unit is further used to use the compressed cluster centers, the compressed individual difference data corresponding to each cluster center, and the difference positions corresponding to each individual difference data to form a compressed data stream corresponding to the real-time transaction data stream.

[0183] The distribution unit is used to distribute the compressed data stream to each transaction device to complete the real-time distribution of the real-time transaction data stream.

[0184] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0185] like Figure 3As shown, the third aspect of this embodiment provides another real-time distribution device for quantitative trading data based on cloud transmission. Taking the device as an electronic device as an example, it includes: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the real-time distribution method for quantitative trading data based on cloud transmission as described in the first aspect of the embodiment.

[0186] For example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in first-out memory (FIFO), and / or first-in last-out memory (FILO). Specifically, the processor may include one or more processing cores, such as a quad-core processor or an octal-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Furthermore, the processor may include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state.

[0187] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. For example, the processor may be, but is not limited to, a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU). The transceiver may be, but is not limited to, a Wireless Fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0188] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0189] The fourth aspect of this embodiment provides a storage medium storing instructions for the real-time distribution method for quantitative trading data based on cloud transmission as described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the real-time distribution method for quantitative trading data based on cloud transmission as described in the first aspect of the embodiment is executed.

[0190] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0191] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0192] A fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method for real-time distribution of quantitative trading data based on cloud transmission as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0193] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A real-time distribution method for quantitative trading data based on cloud transmission, characterized in that: include: Get real-time transaction data streams; Clustering each transaction data in the real-time transaction data stream to obtain at least one data cluster; Determine the cluster center of each data cluster, and determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data, wherein the difference position corresponds to a difference operation, and the difference operation includes an insert operation, a replace operation, and a delete operation; Performing data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center; Utilizing each compressed cluster center, the compressed individual difference data corresponding to each cluster center, and the difference position corresponding to each individual difference data, a compressed data stream corresponding to the real-time transaction data stream is formed; Distributing the compressed data stream to each transaction device to complete the real-time distribution of the real-time transaction data stream; Clustering is performed on each transaction data in the real-time transaction data stream to obtain at least one data cluster, including: Calculating the data distance between each transaction data in the real-time transaction data stream; Determine multiple initial cluster centers based on the data distance between each transaction data; Optimizing the multiple initial cluster centers using a cluster optimization algorithm to obtain multiple optimal initial cluster centers of the real-time transaction data stream; Based on multiple optimal initial clustering centers, cluster the real-time transaction data stream to obtain multiple initial data clusters; Performing cluster correction processing on the multiple initial data clusters to obtain the at least one data cluster; Based on the data distance between each transaction data, multiple initial cluster centers are determined, including: Using the data distances between each transaction data, a distance matrix is ​​constructed, and the data distances in the distance matrix are sorted from small to large to obtain a distance sequence; Based on the distance sequence, a distance threshold is determined; For any transaction data in the real-time transaction data stream, calculate the local density of the transaction data based on the distance threshold and the data distance between the transaction data and each designated data in the designated data set, and obtain the local density of each transaction data after polling all transaction data in the real-time transaction data stream, wherein each designated data in the designated data set is each transaction data in the real-time transaction data stream excluding the transaction data; For any of the transaction data, at least one target data is selected from the real-time transaction data stream, wherein a local density of any of the target data is greater than a local density of any of the transaction data; Filtering the minimum data distance from the data distances between each target data and any transaction data to serve as the calibration distance corresponding to the any transaction data, and calculating the calibration distance corresponding to each transaction data after polling all transaction data in the real-time transaction data stream; Using the local density and calibration distance of each transaction data, multiple initial cluster centers are determined from each transaction data; Among them, using the local density and calibration distance of each transaction data, multiple initial cluster centers are determined from each transaction data, including: Calculating the standard deviation of the calibrated distances corresponding to all transaction data in the real-time transaction data stream and the density mean of the local densities corresponding to all transaction data; Based on the standard deviation, the calibration threshold is determined, and the calibration threshold is 2 times the standard deviation; Filtering, from each transaction data in the real-time transaction data stream, transaction data with a calibration distance greater than or equal to the calibration threshold as preselected initial cluster centers; Screening out preselected initial cluster centers whose local density is greater than or equal to the density mean from each preselected initial cluster center, and using the screened out preselected initial cluster centers as the initial cluster centers; A cluster optimization algorithm is used to optimize the multiple initial cluster centers to obtain multiple optimal initial cluster centers of the real-time transaction data stream, including: Based on multiple initial cluster centers, a bat population is constructed, wherein each individual bat in the bat population corresponds to an initial position, an initial velocity, an initial pulse loudness, and an initial pulse emission frequency, and the initial position of any individual bat corresponds to an initial cluster center; Initialize the number of optimization searches t, and cluster the real-time transaction data stream based on the positions of each bat individual at the t-th optimization search to obtain the clusters at the t-th optimization search, where each bat individual corresponds to a cluster, the initial value of t is 1, and when t is 1, the position of any bat individual at the t-th optimization search is the initial position of any bat individual; Based on the clustering clusters at the t-th optimization search, the fitness of each bat individual at the t-th optimization search is calculated, wherein the greater the fitness of any bat individual, the higher the clustering accuracy of the cluster corresponding to any bat individual; Determine whether the fitness of each bat individual at the tth optimization search is greater than the historical optimal fitness of each bat individual; If not, the maximum fitness, search weight factor and pulse frequency of the t-th optimization search are determined, wherein the following formula (4) is used to calculate the corresponding pulse frequency of the t-th optimization search; （4） In the above formula (4), represents the pulse frequency of any bat individual during the t-th optimization, represent the maximum pulse frequency and the minimum pulse frequency respectively, and It represents a random number on [0,1]; According to the maximum fitness, search weight factor and pulse frequency, the speed of each bat individual at the t-th optimization search is updated to obtain the updated speed of each bat individual, and the position of the bat individual at the t-th optimization search is updated using the updated speed of each bat individual to obtain the updated position of each bat individual, wherein when t is 1, the speed of any bat individual at the t-th optimization search is the initial speed corresponding to the bat individual; For any individual bat, a random vector is generated, where the elements in the random vector have values ​​between [-1, 1] and have the same length as the position vector corresponding to the individual bat; then, the updated position corresponding to the individual bat is determined based on the updated velocity corresponding to the individual bat and the random vector; Use the following formula (6) to calculate the updated position of any bat individual （6） The above formula (6) represents the updated position corresponding to any bat individual, represents a random vector, represents the norm of the vector, represents the updated speed corresponding to any bat individual, represents the position of any bat individual at the tth optimization time; The pulse emission frequency of each bat individual during the t-th optimization is used to perturb the updated position of each bat individual to obtain the perturbed position of each bat individual. When t is 1, the pulse emission frequency of any bat individual during the t-th optimization is the initial pulse emission frequency of the bat individual. Based on the pulse loudness of each bat individual at the t-th optimization, determine whether to retain the disturbance position corresponding to each bat individual. When t is 1, the pulse loudness of any bat individual at the t-th optimization is the initial pulse loudness of any bat individual. If so, then increment t by 1, and use the disturbance position corresponding to each bat individual as the position of each bat individual at the t-th optimization search. Then, cluster the real-time transaction data stream based on the position of each bat individual at the t-th optimization search until the fitness of each bat individual at the t-th optimization search is greater than the historical optimal fitness of each bat individual. Then, multiple optimal initial clustering centers of the real-time transaction data stream are determined based on the position of each bat individual at the t-th optimization search. Determine the search weight factor for the tth optimization search, including: According to the following formula (1), the search weight factor for the tth optimization is determined; （1） In the above formula (1), Represents the search weight factor during the t-th optimization search, represents the maximum number of optimization attempts, denotes the maximum search weight and the minimum search weight, respectively, and represents the weight index; Correspondingly, the pulse emission frequency of each bat individual during the t-th optimization is used to perturb the updated position corresponding to each bat individual to obtain the perturbed position corresponding to each bat individual, which includes: For any individual bat, generate a first random number, and determine whether the first random number is greater than the pulse emission frequency of any individual bat during the t-th optimization search; If so, obtain the maximum disturbance factor, the minimum disturbance factor, and the total number of individuals in the bat population; According to the maximum disturbance factor, the minimum disturbance factor and the total number of individuals, the disturbance coefficient of the t-th optimization search is calculated; Using the perturbation coefficient during the t-th optimization search, the updated position corresponding to any bat individual is perturbed to obtain the perturbed position corresponding to any bat individual; According to the maximum disturbance factor, the minimum disturbance factor and the total number of individuals, the disturbance coefficient at the t-th optimization is calculated, including: According to the following formula (2), the disturbance coefficient of the t-th optimization search is calculated; （2） In the above formula (2), represents the perturbation coefficient during the t-th optimization search, represent the maximum disturbance factor and the minimum disturbance factor respectively, represents the total number of individuals described; Performing cluster correction processing on the plurality of initial data clusters to obtain the at least one data cluster includes: Based on each initial data cluster, incorrectly classified data is filtered out from the real-time transaction data stream; wherein, for any transaction data in the real-time transaction data stream, the distance between the any transaction data and the cluster center of each initial data cluster can be first calculated, and the calculated distances are sorted in ascending order to obtain a sorted sequence; then, the first two distances in the sorted sequence are screened out, and it is determined whether the absolute value of the difference between the first two distances is less than a preset threshold; wherein, if so, the any transaction data is treated as incorrectly classified data, and when all transaction data in the real-time transaction data stream are polled, all incorrectly classified data is filtered out from the real-time transaction data stream; All the incorrectly classified data that have been screened out are used to form a set to be classified; Remove all misclassified data from the real-time transaction data stream and use the remaining transaction data to form a correctly classified set; For any misclassified data in the set to be classified, determine the transaction data closest to the misclassified data from the correct classification set as the nearest data to the misclassified data; According to the nearest data of any of the erroneously classified data, the erroneously classified data is reclassified, and after all the erroneously classified data in the set to be classified are polled, the clustering correction processing of multiple initial data clusters is completed to obtain the at least one data cluster.

2. The method according to claim 1, characterized in that Performing data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center, including: Performing sampling compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain the sampling compressed data corresponding to each cluster center and the sampling compressed individual difference data corresponding to each cluster center; The sampled compressed data corresponding to each cluster center and the sampled compressed individual difference data corresponding to each cluster center are subjected to lossless compression processing, so as to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center after the lossless compression processing.

3. A real-time distribution device for quantitative trading data based on cloud transmission, characterized in that: The method for real-time distribution of quantitative trading data based on cloud transmission according to any one of claims 1 to 2 is used, wherein the device comprises: An acquisition unit, used to acquire real-time transaction data streams; A clustering unit, configured to perform clustering processing on each transaction data in the real-time transaction data stream to obtain at least one data cluster; A compression unit, configured to determine the cluster center of each data cluster, and to determine the individual difference data between each cluster center and each remaining transaction data in the corresponding data cluster, as well as the difference position corresponding to each individual difference data; A compression unit is used to perform data compression processing on each cluster center and the individual difference data corresponding to each cluster center to obtain each compressed cluster center and the compressed individual difference data corresponding to each cluster center; The compression unit is further configured to form a compressed data stream corresponding to the real-time transaction data stream using the compressed cluster centers, the compressed individual difference data corresponding to the cluster centers, and the difference positions corresponding to the individual difference data; The distribution unit is used to distribute the compressed data stream to each transaction device to complete the real-time distribution of the real-time transaction data stream.

4. A real-time distribution device for quantitative trading data based on cloud transmission, characterized in that: include: A memory, a processor, and a transceiver that are sequentially communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program, and execute the real-time distribution method for quantitative trading data based on cloud transmission as described in any one of claims 1 to 2.

Images