Financial data pushing method and device based on PYTHON and storage medium
Through memory mapping technology and multi-process architecture, data interaction between C++ and Python is realized, which solves the problem of insufficient efficiency in financial data push technology, improves the efficiency and speed of data push, and meets users' requirements for data timeliness and accuracy.
Patent Information
- Application Number
- CN202510560945.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2025-07-25
AI Technical Summary
The existing financial data push technology is insufficient in processing massive market data and cannot meet users' high requirements for data timeliness and accuracy, especially in data interaction in different programming language environments.
Memory mapping technology is used to realize data interaction between C++ and Python, and integrate, cache and standardize the data by obtaining real-time market data, and push it to the user side in parallel, and use a multi-process architecture to improve data push efficiency.
It realizes efficient interaction between market data between C++ and Python, improves the efficiency and speed of data push, improves the utilization rate of multi-core CPUs, and meets users' needs for data timeliness and accuracy.
Smart Images
Figure CN120378481A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of financial data management, and particularly relates to a financial data push method, device and storage medium based on PYTHON. Background Art
[0002] With the continuous development of the financial market and the increasing progress of information technology, the push efficiency and speed of financial data have become one of the key factors affecting the quality of financial services. There are many deficiencies in traditional financial data push services.
[0003] As the mainstream language for quantitative trading, PYTHON has a large user base. Its syntax is simple, the development efficiency is relatively high, and the quantitative trading ecosystem is perfect. However, as an interpreted language, its running efficiency is relatively low, and its processing efficiency is not sufficient to support the numerous and massive market data from the market service center. It needs to rely on the CPP statically typed language to speed up, and there are certain obstacles in data interaction in different programming language environments. As a result, most of the existing financial data push technologies use a single language for data processing and transmission, which cannot meet the high requirements of users for data timeliness and accuracy. Due to the ever-changing nature of the financial market, investors need to obtain accurate market data in a timely manner in order to make correct investment decisions.
[0004] Therefore, a financial data push service that can ensure efficiency is needed. Summary of the Invention
[0005] The purpose of the present invention is to provide a financial data push method, device and storage medium based on PYTHON. By using the memory mapping technology to read the cached market data, the interaction of market data between C++ and Python is realized, and thus the efficient data push between the market service center and the user terminal is realized.
[0006] To achieve the above object, the present invention is implemented by the following technical solutions:
[0007] In a first aspect, the present invention provides a financial data push method based on PYTHON, including: Obtaining real-time market data and integrating the real-time market data, wherein the real-time market data is C language data; Caching the integrated real-time market data according to user requests, wherein the user requests include user subscription requests and user query requests; Obtaining the cached user request data through a memory mapping method, wherein the user request data is C language data recognizable by Python; Performing standardization processing on the user request data; Push the standardized user request data in parallel.
[0008] Optionally, the obtaining real-time market data and integrating the real-time market data includes: Access the real-time market data of the exchange through the market service center; Integrate the real-time market data of different categories, access methods, and sources, and provide a unified data model and service method.
[0009] Optionally, the caching the integrated real-time market data according to the user request includes: Apply for two pieces of memory through the C++ module in the circular buffer, where the two pieces of memory are the index data cache and the service data cache respectively; According to the user subscription request and the user query request, write the service data in the real-time market data into the service data cache in a circular manner, and update the address in the index data cache.
[0010] Optionally, the service data cache is stored in binary data, and the binary data includes the identifier, length, version, and checksum of the message.
[0011] Optionally, the obtaining the cached user request data through the memory mapping method includes: Memory map the index data cache and the service data cache through the Python-SDK; Read the index data cache at a high frequency, and loop to read the service data cache data until the address matches; Receive the read service data cache and index data cache.
[0012] Optionally, the standardizing the user request data includes: using the Pandas library to convert the JSON format user request data into the DataFrame format, and cleaning the data after format conversion.
[0013] Optionally, the pushing the standardized user request data in parallel includes: through the Python's multiprocessing module, push the standardized user request data to the corresponding user terminals through multiple processes.
[0014] In a second aspect, the present invention provides a Python-based financial data pushing device, including: A market data acquisition module: used to acquire real-time market data and integrate the real-time market data, where the real-time market data is C language data; Market data caching module: used to cache the integrated real-time market data according to user requests, where the user requests include user subscription requests and user query requests; User request data acquisition module: used to acquire the cached user request data through memory mapping, where the user request data is C language data recognizable by Python; Standardization processing module: used to perform standardization processing on the user request data; Data push module: used to push the standardized user request data in parallel.
[0015] Optionally, the user request data acquisition module includes: Memory mapping module: used to read user request data through memory mapping; Data receiving module: used to receive the read user request data.
[0016] Thirdly, the present invention provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the PYTHON-based financial data push method described in any item of the first aspect is implemented.
[0017] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: By caching the C language market data after unified standardization, and then reading the cached market data through memory mapping technology, and receiving the market data recognizable by Python, the market data can be efficiently interacted between C++ and Python, making the processing process of the static basic information of the market data and the process of pushing the market data to the user side more efficient. And a multi-process architecture is adopted to perform parallel processing on multiple push tasks, and each process runs independently and is not restricted by the Python global interpreter lock GIL, further improving the efficiency of data push and the utilization rate of multi-core CPUs, thereby improving the overall performance. Description of the Drawings
[0018] Figure 1 The flowchart of the PYTHON-based financial data push method in an embodiment of the present invention is shown;
[0019] Figure 2 The flowchart of the push method implemented by the PYTHON-based financial data push system in an embodiment of the present invention is shown;
[0020] Figure 3 The schematic diagram of the memory mapping message structure of the business data cache in an embodiment of the present invention is shown. Detailed Embodiments
[0021] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0022] Embodiment 1
[0023] As Figure 1 shown, this embodiment provides a Python-based financial data push method, including: Obtain real-time market data and integrate the real-time market data, wherein the real-time market data is C language data; Cache the integrated real-time market data according to user requests, wherein the user requests include user subscription requests and user query requests; Obtain the cached user request data through memory mapping, wherein the user request data is C language data recognizable by Python; Perform standardization processing on the user request data; Push the standardized user request data in parallel.
[0024] By writing the C language market data obtained from user requirements into memory and then using memory mapping to read the user requirement data recognizable by the Python language, data interaction between C++ and Python is realized, enabling the market data to be efficiently pushed to the user side.
[0025] Embodiment 2
[0026] On the basis of Embodiment 1, the following design is also made in this embodiment.
[0027] As Figure 2 shown, a Python-based financial data push process is implemented through a financial data push system. The financial data push system includes: a data access module, a data subscription and query module, an asynchronous communication module, a data processing module, and a data push module. The specific execution steps of each module are as follows:
[0028] The data access module provides data access services: The data access module is the market data service center, which is used to access the real-time market data of the exchange, integrate market information of different categories, access methods, and sources, and provide a unified data model and service method for downstream users. The market data service center is directly connected to the exchange data center through a dedicated ground line to receive real-time market data of stocks, bonds, funds, and indexes. After receiving the data, it performs cleaning and formatting processing to ensure the accuracy and consistency of the data, supports the data playback function, and users can request Tick data for a specific time period for playback analysis. It provides a securities information query interface, and users can query the basic information and historical data of specific securities, with minute-level Tick data playback, microsecond-level processing delay, and multi-channel backup high availability.
[0029] The data subscription and query module is used to connect to the market data service center, subscribe to the high-frequency market data of the exchange, where the high-frequency market data includes stocks, indexes, bonds, and funds, and is used to provide historical K-line data of market stocks and query of basic securities information.
[0030] In this embodiment, the user subscription request is sent to the market data service center through the TCP and GRPC communication protocols. The market data service center, according to the subscription request, pushes relevant data to the data receiving module in real time; for the user query request (such as for the Shanghai and Shenzhen markets) for the historical K-line data and basic securities information of stocks, the query request is sent to the market data service center through the TCP and GRPC communication protocols, and the market data service center returns the corresponding historical data and securities information according to the query request.
[0031] The asynchronous communication module provides data interaction between C++ and Python: It is set as a circular buffer, including business data cache and index data cache. The user subscription module and the user query module respectively write the subscribed high-frequency market data and the queried securities-related information into the corresponding data caches in the circular buffer, and then the data receiving module reads and receives the cached data in the circular buffer by means of memory mapping. Among them, the C++ module in the asynchronous communication module accelerates data processing and generates a C function interface recognizable by Python through SWIG.
[0032] In this embodiment, the writing method of the asynchronous communication module includes:
[0033] When the C++ module is initialized, two pieces of memory are allocated, namely the index data cache and the business data cache;
[0034] The business data is written into the business data cache in a circular manner, and the address in the index data cache is updated.
[0035] In this embodiment, the reading method of the asynchronous communication module includes:
[0036] Python-SDK memory mapping index data cache and business data cache;
[0037] The Python-SDK reads the index data cache frequently and reads the business data cache data in a loop until the address matches.
[0038] In this embodiment, the business data cache of the asynchronous communication module is stored in the memory mapping message in binary data, and the identity, length, version, and checksum of the message are identified by special fields.
[0039] The message data structure is as Figure 3 shown. Except for the message, all other fields use network byte order. The specific meanings of each field are as follows:
[0040] Message identifier, which serves as the start of the message;
[0041] Synchronous / asynchronous identifier, which indicates the synchronous / asynchronous identifier of the message. Among them, 0 represents asynchronous and 1 represents synchronous;
[0042] Function identifier, which indicates the module meaning represented by this message. Among them, 1 represents initialization, 2 represents login, 3 represents subscription, 4 represents query, and 5 represents backtest;
[0043] Sub-function identifier, which is within the module content of the function identifier of this message. Among them, 1 represents request, 2 represents response, and 3 represents data;
[0044] Response status, which indicates the module content of this message. Among them, 1 represents success and 2 represents failure;
[0045] Time / ID. For synchronous messages, it represents the unique ID of the message, and for asynchronous messages, it represents the number of milliseconds elapsed since the operating system was started;
[0046] Data length, which records the length of the message body;
[0047] Data, which indicates the message body of the request, response, and data.
[0048] The data processing module provides data reception and standardization services: First, it receives subscription and query data from the asynchronous communication module in real time through the TCP and GRPC communication protocols; then it uses the Pandas library to convert the received JSON format data of the asynchronous communication into the DataFrame format for subsequent data analysis and processing; after format conversion, it cleans and standardizes the data.
[0049] The data push module performs data push services: pushing the data processed by the data processing module to the corresponding client. The system adopts a multi-process architecture and uses Python's multiprocessing module to manage multiple push processes. Each process runs independently and is not restricted by the GIL, improving the parallel efficiency of data processing and pushing. The processes communicate through message queues or shared memory to ensure data consistency and synchronization.
[0050] In this embodiment, the push system also includes an integrated Cython module. By writing Cython code that does not depend on the GIL, the Cython code is compiled into efficient C code to improve data processing performance. The Cython module is used in key data processing links, especially in scenarios of large-scale data calculation and real-time data push.
[0051] In this embodiment, the push system adopts an asynchronous programming model and uses Python's asyncio library to manage asynchronous tasks. The asynchronous tasks optimize I / O-intensive operations, including but not limited to network communication and file reading and writing. Through the asynchronous programming model, the responsiveness and throughput of the system are improved.
[0052] In this embodiment, the push system reduces CPU-intensive operations by optimizing algorithms and data structures, uses efficient data structures such as NumPy arrays to process large-scale data, and the optimized algorithms and data structures indirectly reduce the impact of the GIL and improve system performance. The optimized algorithms include but are not limited to quicksort, hash table lookup, and dynamic programming, which are used to improve the efficiency and speed of data processing.
[0053] Quicksort is a divide-and-conquer algorithm. It divides an array into two sub-arrays and then recursively sorts the sub-arrays. It selects a "pivot" element and divides the array into two sub-arrays, where all elements in one sub-array are less than the pivot and all elements in the other sub-array are greater than the pivot. Then the same process is recursively applied to these two sub-arrays.
[0054] Sorting algorithms are used to sort the elements in an array or list. The expression of the sorting algorithm is as follows: function quicksort(array, low, high) if low<high pi = partition(array, low, high) quicksort(array, low, pi - 1) quicksort(array, pi + 1, high) function partition(array, low, high) pivot = array[high] i = low - 1 for j in range(low, high) if array[j]<pivot i = i + 1 swap array[i] and array[j] swap array[i + 1] and array[high] return i + 1
[0055] Among them, array represents the array to be sorted, low represents the starting index of the array, high represents the ending index of the array, pi represents the partition index, that is, the final position of the pivot element. In this formula, the quicksort function is the main function of quicksort. It recursively divides the array into two sub-arrays and sorts them. The partition function is used to select a pivot element and rearrange the elements in the array according to the pivot element, so that the elements on the left of the pivot element are all smaller than the pivot, and the elements on the right are all larger than the pivot.
[0056] A hash table is a data structure that uses a hash function to map keys to positions in the table, so that data can be retrieved quickly. The hash function converts the key into a fixed-size index, and then directly accesses the data in the table through this index.
[0057] Data structure operations are used to find the value corresponding to a specific key in the hash table. The data structure operation expression is as follows: function hash_lookup(key, table) index = hash_function(key) if table[index] is not None and table[index].key == key return table[index].value else return None
[0058] Among them, key represents the key to be searched, table represents the hash table that stores key-value pairs, index represents the index calculated by the hash function, and value represents the value corresponding to the key if found; otherwise, it returns None. In this formula, the hash_lookup function calculates the index of the key using the hash function and then looks up the key-value pair at that index position in the hash table. If a matching key is found, the corresponding value is returned; otherwise, None is returned.
[0059] Dynamic programming is an optimization technique that solves complex problems by decomposing them into smaller overlapping subproblems. It is commonly used to solve problems with overlapping subproblems and optimal substructure properties. Dynamic programming stores the solutions to subproblems to avoid redundant calculations, thereby improving efficiency.
[0060] Optimization algorithms are used to solve problems with overlapping subproblems and optimal substructure properties, such as calculating the Fibonacci sequence. The expression of the optimization algorithm is as follows: function fibonacci(n) if n == 0 return 0 if n == 1 return 1 fib = [0, 1] for i in range(2, n+1) fib.append(fib[i-1] + fib[i-2]) return fib[n]
[0061] Among them, n represents the index of the Fibonacci sequence to be calculated, and fib represents the array storing the Fibonacci sequence. In this formula, the fibonacci function calculates the nth number of the Fibonacci sequence using the dynamic programming method. It stores the previously calculated Fibonacci numbers to avoid redundant calculations, thereby improving efficiency.
[0062] Example 3
[0063] This embodiment provides a financial data push device based on PYTHON, including: Market data acquisition module: used to acquire real-time market data and integrate the real-time market data, where the real-time market data is C language data; Market data caching module: used to cache the integrated real-time market data according to user requests, where the user requests include user subscription requests and user query requests; User Request Data Acquisition Module: Used to acquire cached user request data through memory mapping, where the user request data is C language data recognizable by Python; Standardization Processing Module: Used to perform standardization processing on the user request data; Data Push Module: Used to push the standardized user request data in parallel.
[0064] Embodiment 4
[0065] This embodiment provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the PYTHON-based financial data push method described in any step of Embodiment 2.
[0066] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0067] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0068] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 in one block or a plurality of blocks.
[0070] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can still make many forms without departing from the spirit and scope of the present invention as protected by the claims. All of these are within the protection scope of the present invention.
Claims
1. A Python-based financial data push method, characterized in that, It includes: Obtain real-time market data and integrate the real-time market data, where the real-time market data is C language data; Cache the integrated real-time market data according to user requests, where the user requests include user subscription requests and user query requests; Obtain the cached user request data through memory mapping, where the user request data is C language data recognizable by Python; Perform standardization processing on the user request data; Push the standardized user request data in parallel.
2. The Python-based financial data pushing method according to claim 1, wherein The obtaining real-time market data and integrating the real-time market data includes: Access the real-time market data of the exchange through the market service center; Integrate the real-time market data of different categories, access methods, and sources, and provide a unified data model and service method.
3. The financial data push method based on PYTHON according to claim 1, wherein The caching the integrated real-time market data according to user requests includes: Apply for two pieces of memory through the C++ module in the circular buffer, where the two pieces of memory are the index data cache and the service data cache respectively; According to the user subscription request and the user query request, write the service data in the real-time market data into the service data cache in a circular manner, and update the addresses in the index data cache.
4. The Python-based financial data push method according to claim 3, wherein The service data cache is stored in binary data, and the binary data includes the identifier, length, version, and checksum of the message.
5. The PYTHON-based financial data push method according to claim 3, characterized in that The obtaining the cached user request data through memory mapping includes: Memory map the index data cache and the service data cache through Python-SDK; Read the index data cache at a high frequency and loop to read the service data cache data until the addresses match; Receive the read service data cache and index data cache.
6. The Python-based financial data push method according to claim 1, wherein The performing standardization processing on the user request data includes: using the Pandas library to convert the user request data in JSON format to DataFrame format, and cleaning the data after format conversion.
7. The Python-based financial data push method according to claim 1, wherein The pushing the standardized user request data in parallel includes: through the multiprocessing module of Python, push the standardized user request data to the corresponding user terminals through multiple processes.
8. A financial data push device based on PYTHON, characterized in that, It includes: Market data acquisition module: used to obtain real-time market data and integrate the real-time market data, where the real-time market data is C language data; Market data caching module: used to cache the integrated real-time market data according to user requests, where the user requests include user subscription requests and user query requests; User request data acquisition module: used to obtain the cached user request data through memory mapping, where the user request data is C language data recognizable by Python; Standardization processing module: used to perform standardization processing on the user request data; Data pushing module: used to push the standardized user request data in parallel.
9. The Python-based financial data push device according to claim 8, wherein The user request data acquisition module includes: Memory mapping module, used to read user request data through memory mapping; A data receiving module for receiving the read user request data.
10. A computer storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, the PYTHON-based financial data pushing method according to any one of claims 1-7 is implemented.