Multi-source heterogeneous data dynamic integration and unified access method and system based on distributed concurrency control

Abstract

The invention provides a multi-source heterogeneous data dynamic integration and unified access method and system based on distributed concurrency control, and relates to the technical field of distributed computation.The method comprises the steps that protocol metadata, original binary data streams, network state parameters and user access requests of multi-exchange data sources are collected, and the protocol metadata, the original binary data streams, the network state parameters and the user access requests of the multi-exchange data sources are obtained; analyzing the original binary stream into a target message object with a uniform structure in real time according to the protocol metadata; on the basis of the network state parameters, target quality scores of all links are calculated through a link quality evaluation algorithm, an optimal physical network link is selected accordingly, and a message object transmission session is routed; the user request type is identified through the optimal link, and buffering and management are carried out by using a lock-free annular queue; and finally, in combination with a distributed concurrency control method, the requests in the processing queue are extracted by a lock-free synchronization mechanism, and dynamic integration and unified access of multi-source heterogeneous data are realized. According to the invention, the throughput and response time efficiency of high-frequency real-time market data processing are improved.

Description

Technical Field

[0001] This application relates to the field of distributed computing technology, and in particular to a method and system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control. Background Technology

[0002] With the deepening integration of global financial markets, the demand for high-frequency real-time data integration and access across multiple regions and exchanges is becoming increasingly prominent. In such application scenarios, technical solutions need to be able to effectively cope with protocol differences between heterogeneous data sources, network link quality fluctuations, and the pressure of high-concurrency access from massive users. The core objective is to achieve low-latency data transmission, high-reliability integration, and ultimately provide stable and efficient unified services.

[0003] Currently, existing solutions adopt a centralized data middleware architecture. This solution first standardizes heterogeneous data from different sources through a pre-built protocol conversion module, then completes data transmission using fixed primary and backup network links, and handles user access requests based on a locked thread pool mechanism. This solution aims to shield the differences in underlying data sources through its unified access layer, while using load balancing devices to share the access pressure brought by high concurrency.

[0004] However, when this existing solution is applied to high-frequency real-time scenarios, its link switching mechanism still has room for improvement in terms of actual response agility because it relies on a preset fixed threshold. At the same time, the lock-based concurrency control method often causes fluctuations in system throughput when encountering instantaneous high concurrency requests. Summary of the Invention

[0005] This application provides a method and system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control, in order to solve the problems of low throughput and poor response time in high-frequency real-time market data processing in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control, comprising:

[0007] Collect multi-source heterogeneous data from multiple data sources, including: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests;

[0008] Based on the protocol metadata, the original binary data stream is parsed in real time to generate a target message object;

[0009] Based on the real-time network status parameters, the target quality score of each physical network link is dynamically calculated using a link quality assessment algorithm. The optimal physical network link is determined from multiple physical network links based on the target quality score, and the transmission session corresponding to the target message object is routed to the optimal physical network link.

[0010] Based on the optimal physical network link, the multiple user access requests are identified to obtain multiple request types. Based on different request types, a lock-free circular queue is used for buffering and management.

[0011] By combining distributed concurrency control methods, user access requests are extracted and processed from the lock-free circular queue through a lock-free synchronization mechanism to achieve dynamic integration and unified access to multiple data sources.

[0012] Optionally, the step of identifying the types of the multiple user access requests based on the optimal physical network link to obtain multiple request types, and using a lock-free circular queue for buffering and management based on different request types, includes:

[0013] By using the optimal physical network link, the message content of each user access request is parsed to identify the request of each user access request;

[0014] Using a priority scheduling algorithm, a corresponding processing priority is assigned to each request type, and each user access request is bound to the processing priority to obtain the user access request with the assigned processing priority;

[0015] User access requests with assigned processing priorities are input into a lock-free circular queue for buffering. After buffering is complete, the positions of the write and read pointers are adjusted, and the user access requests in the lock-free circular queue are scheduled and managed using a flow shaping algorithm to obtain the managed user access requests.

[0016] Secondly, this application provides a system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control, comprising:

[0017] The acquisition module is used to collect multi-source heterogeneous data from multiple data sources. The multi-source heterogeneous data includes: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests.

[0018] The parsing module is used to parse the original binary data stream in real time based on the protocol metadata and generate a target message object;

[0019] The calculation module is used to dynamically calculate the target quality score of each physical network link based on the real-time network status parameters using a link quality assessment algorithm, determine the optimal physical network link from multiple physical network links according to the target quality score, and route the transmission session corresponding to the target message object to the optimal physical network link.

[0020] The identification module is used to identify the types of the multiple user access requests based on the optimal physical network link, obtain multiple request types, and use a lock-free circular queue for buffering and management based on different request types.

[0021] The extraction module is used to extract and process managed user access requests from the lock-free circular queue by combining distributed concurrency control methods and a lock-free synchronization mechanism, so as to complete the dynamic integration and unified access of multiple data sources.

[0022] Thirdly, this application provides an electronic device, comprising:

[0023] Memory, used to store computer programs;

[0024] A processor, configured to execute the computer program to implement the steps of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control as described in the first aspect above.

[0025] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control as described in the first aspect above.

[0026] The technical solution provided in this application has the following beneficial effects:

[0027] This application ensures that the data transmission path is always in an optimal state through a dynamic link selection mechanism, effectively reducing the risk of transmission delay and interruption caused by network quality fluctuations, and improving the stability and overall efficiency of the data transmission process. Furthermore, this application realizes the orderly acceptance and preliminary organization of massive concurrent access requests, effectively avoiding request congestion and preparing for subsequent priority processing. Finally, this application achieves efficient and stable integration and unified access services for multi-source heterogeneous data, can easily cope with high-concurrency scenarios, and ensures that the multi-source heterogeneous data dynamic integration and unified access system has comprehensive performance of high throughput and low response latency.

[0028] Furthermore, this application also identifies the type of user requests by parsing the optimal network link, then uses a priority scheduling algorithm to assign processing priorities to different types and bind them to the requests. Subsequently, these priority requests are sent to a lock-free circular queue for buffering. Finally, the requests are scheduled and managed by maintaining queue pointers and combining them with a traffic shaping algorithm. Therefore, this process, through request classification, priority sorting, and queue management, effectively sorts and schedules high-concurrency access traffic, and ensures that critical requests in the multi-source heterogeneous data dynamic integration and unified access system can receive priority and timely processing. This optimizes the resource utilization of the multi-source heterogeneous data dynamic integration and unified access system as a whole, and improves the response speed and orderliness of the service provided by the multi-source heterogeneous data dynamic integration and unified access system.

[0029] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control, provided in an embodiment of this application;

[0032] Figure 2 A schematic diagram illustrating a specific implementation of a method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control, provided in this application embodiment;

[0033] Figure 3 This is a schematic diagram of the structure of a multi-source heterogeneous data dynamic integration and unified access system based on distributed concurrent control, provided in an embodiment of this application. Detailed Implementation

[0034] To address the problems existing in the prior art, this application proposes a method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control. This method can adaptively select the optimal data transmission path, and then efficiently and orderly process high-concurrency user requests by adopting a lock-free data structure and a priority scheduling mechanism. By combining dynamic optimization of link selection with concurrent control of request processing, this method effectively improves the timeliness of data transmission and the overall stability of the multi-source heterogeneous data dynamic integration and unified access system in providing services to the outside world. Thus, it achieves low-latency, high-reliability integration and unified access of multi-source heterogeneous data in complex heterogeneous network environments.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The core of this application is to provide a method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrency control. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0037] Step 101: Collect multi-source heterogeneous data from multiple data sources, including: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests.

[0038] In step 101, the data source can be understood as the exchange market data source, and the protocol metadata can refer to the detailed specifications of the communication protocol used by each exchange market data source. These detailed specifications include: protocol version, message structure, meaning and length of each data field, communication port number, etc.

[0039] Raw binary data streams refer to market data packets that conform to specific protocols and are received from various data sources through physical network links. These market data packets are unprocessed data sequences consisting of 0s and 1s, which cannot be directly understood by humans and need to be parsed based on protocol metadata.

[0040] Physical network links refer to the actual physical communication channels connecting a distributed environment with multiple data sources. These actual physical communication channels include specific transmission media such as fiber optic leased lines and satellite links. These links provide the physical connection foundation for acquiring multi-source heterogeneous data.

[0041] Real-time network status parameters refer to network performance indicators that are continuously measured during data transmission and are used to reflect the current communication quality of physical network links. These network performance indicators mainly include the delay time from data packet transmission to reception, jitter, and packet loss rate. Jitter is the degree of fluctuation in delay time, and packet loss rate is the proportion of data packets lost during transmission.

[0042] A user access request refers to an instruction initiated by multiple user clients in a distributed environment to obtain or manipulate data. For example, such a user access request may include: a market data retrieval instruction, a transaction submission instruction, or a historical data query instruction.

[0043] Step 102: Based on the protocol metadata, perform real-time parsing of the original binary data stream to generate a target message object.

[0044] In step 102, the target message object refers to the data object obtained after parsing the original, heterogeneous binary data stream according to a unified internal standard. This object contains fields with clear meaning and consistent type, such as stock code, latest price, and trading volume.

[0045] In this embodiment, the corresponding parsing rules are dynamically configured using the protocol metadata collected in step 101. Then, the raw binary data stream received in step 101 is parsed bit by bit according to these rules to identify and extract meaningful data fields. Next, these extracted fields are converted into temporary data bodies in memory with uniform field names and data types according to predefined mapping relationships. Finally, after performing necessary checks on this temporary data body, it is encapsulated into a target message object with a uniform format and structured content to prepare for subsequent transmission.

[0046] For example, in this embodiment of the application, the protocol metadata collected in step 101 is used to parse the original binary data stream. For example, for a piece of binary data received from an exchange in location A, this embodiment of the application knows from the protocol metadata that the first 8 bytes are the message header, the next 6 bytes are the stock code field, and the following 8 bytes are the price field. Therefore, this embodiment of the application parses the data segment according to this rule, first extracting the binary value of the stock code field and converting it into the string "000001", and then extracting the binary value of the price field and converting it into the floating-point number 12.35.

[0047] Next, in this embodiment of the application, the string "000001" is mapped to the "symbol" field of the target object and the floating-point number 12.35 is mapped to the "price" field of the target object according to the internally defined unified data standard, thereby forming a structured target message object. The content of this object is {symbol: "000001", price: 12.35, ...}. For data received from the exchange in location B, a similar parsing process is performed according to its corresponding FIX protocol metadata, but with different rule details, and a unified target message object with the same standard structure is also generated.

[0048] Step 103: Based on the real-time network status parameters, dynamically calculate the target quality score of each physical network link using the link quality assessment algorithm, determine the optimal physical network link from multiple physical network links according to the target quality score, and route the transmission session corresponding to the target message object to the optimal physical network link.

[0049] In step 103, the target quality score can refer to a value that represents the current overall quality of the link, calculated by combining multiple real-time network status parameters such as latency, jitter, and packet loss rate, and taking into account the differences in the importance of these parameters to market data transmission.

[0050] A transmission session can refer to the state information maintained for transmitting one or a batch of target message objects. This state information includes the data source address, destination address, port number used, and sequence number, and the transmission session ensures that the data can be delivered accurately and in an orderly manner.

[0051] In this embodiment of the application, the real-time network status parameters of each physical network link monitored in step 101 are first obtained. Then, these parameters are input into a link quality assessment algorithm. The algorithm assigns a weight coefficient to each parameter and generates a target quality score for each link through weighted calculation and other methods.

[0052] Next, this embodiment compares the target quality scores of all links and determines the link with the highest score as the current optimal physical network link. Finally, this embodiment smoothly switches the target message object generated in step 102 and its related transmission session information to this optimal link for transmission.

[0053] For example, in this embodiment of the application, the network status parameters monitored in step 101 are first obtained, wherein the parameters of fiber optic leased line one are a delay of 15ms, jitter of 3ms, and a packet loss rate of 0.01%; and the parameters of fiber optic leased line two are a delay of 25ms, jitter of 5ms, and a packet loss rate of 0.05%.

[0054] Then, the target quality score of the fiber optic leased line is calculated using a preset link quality assessment algorithm. This algorithm can be calculated using the following formula: ,in Represents the target quality score. Represents delay, Represents shaking, Represents packet loss rate. , , These are the weight coefficients for the corresponding items. For example, they are set as follows: , , Therefore, the score of fiber optic lease line one was calculated as follows: The score for fiber optic leased line 2 is ;

[0055] After comparison, the fiber optic leased line 1 with the higher score is selected as the current optimal physical network link; then, the target message object about stock "000001" generated in step 12 is switched from its current transmission link to the selected fiber optic leased line 1 for subsequent data transmission.

[0056] Step 104: Based on the optimal physical network link, identify the types of the multiple user access requests to obtain multiple request types, and use a lock-free circular queue for buffering and management based on different request types.

[0057] In step 104, a lock-free circular queue refers to a data structure used to efficiently and safely buffer data in a multi-threaded environment, and to temporarily store and manage user access requests that are about to be processed. "Circular" means that the storage space is used cyclically, and "lock-free" means that thread synchronization is achieved through special atomic operations rather than traditional locking mechanisms, thereby avoiding performance degradation caused by threads waiting for locks.

[0058] In this embodiment of the application, the user access request mentioned in step 101 is first received through the optimal physical network link determined in step 103; then, the content of each request is parsed to identify the specific type of request; then, a priority scheduling algorithm is used to assign different processing priorities to different types of requests according to preset rules.

[0059] Next, each request is bound to its priority and placed into a lock-free circular queue for buffering in the order of arrival. The requests in the queue are managed by maintaining the positions of the write and read pointers of the queue and combining them with a flow shaping algorithm. Finally, a batch of sorted requests is output.

[0060] For example, in this embodiment of the application, requests from users are first received through the optimal link fiber optic leased line. Then, the content of these user requests is parsed. For example, request A is identified as a market data subscription request because its content contains "action": "subscribe", and request B is identified as a trading order request because its content contains "action": "order". Subsequently, according to the preset request processing rules, the priority of the trading order request is set to priority 3, and the priority of the market data subscription request is set to priority 2. Accordingly, priority 3 is assigned to request B, and priority 2 is assigned to request A. Priority 3 is higher than priority 2 and is the highest priority.

[0061] Next, these priority-assigned requests are written into a lock-free circular queue with a capacity of 1000 in the order of arrival for buffering. When the number of requests in the lock-free circular queue reaches 800 and approaches its capacity limit, the preset traffic shaping algorithm will be automatically activated and the elimination strategy will be determined according to the priority rules, thereby achieving effective buffering and orderly management of high-concurrency user requests.

[0062] Step 105: Combining distributed concurrency control methods, user access requests are extracted and processed from the lock-free circular queue through a lock-free synchronization mechanism to complete the dynamic integration and unified access of multiple data sources.

[0063] In step 105, the distributed concurrency control method refers to the core service responsible for scheduling and processing user access requests from the lock-free circular queue. This method can ensure that high-priority requests are processed first and quickly.

[0064] Lock-free synchronization is a technique for achieving coordination between threads. Instead of using traditional locks, it utilizes atomic operation instructions provided by the processor to ensure data consistency, thereby reducing thread blocking and improving concurrency performance.

[0065] In this embodiment, the state of the lock-free circular queue used in step 104 is periodically checked using a distributed concurrency control method. When a user request to be processed is found in the lock-free circular queue, the highest priority request is extracted from it using a lock-free synchronization mechanism. Then, according to the specific type of the extracted request, it is distributed to a pre-defined dedicated thread that is specifically responsible for handling that type of request. The dedicated thread performs the actual data operation required by the request and returns the final response result to the corresponding user, thereby completing the dynamic integration and unified access service of multi-source heterogeneous data.

[0066] For example, the distributed concurrency control method checks the state of the lock-free circular queue and finds that the read pointer is lagging behind the write pointer, indicating that there are pending requests in the queue. Then, it scans the queue using a lock-free synchronization mechanism, finds the delegate instruction request B with a priority of 3, and extracts request B through an atomic operation, while moving the read pointer forward. Next, based on the type of request B as a "delegation instruction", it distributes it to a dedicated thread responsible for processing delegate instructions. This dedicated thread executes the specific operation required by request B, such as sending a buy order to an exchange in location A, and finally feeding back the confirmation result returned by the exchange to the user, thus completing this call to the data integration and access service.

[0067] This method improves integration efficiency, access speed, and overall service stability and reliability when processing high-frequency real-time market data from multiple global exchanges by executing step 105.

[0068] To address the challenges of differentiated processing of user access requests and efficient utilization of system resources in high-concurrency scenarios, some embodiments include step 104: based on the optimal physical network link, the multiple user access requests are type-identified to obtain multiple request types. Based on different request types, a lock-free circular queue is used for buffering and management, such as... Figure 2 As shown, it includes:

[0069] Step 201: Parse the message content of each user access request through the optimal physical network link to identify the request type of each user access request.

[0070] In step 201, the message content refers to the complete data packet that constitutes the user access request, and the message content includes header information and body information; wherein, the header information usually includes the request target, request method, etc., and the body information includes specific request parameters, and the request type includes market data subscription request, order instruction request or historical query request.

[0071] In this embodiment, the raw data of user access requests are first received through the determined optimal physical network link; then, the message content of each received user access request is parsed, focusing on checking specific fields or keywords contained in its header and body; finally, based on the parsed information, this embodiment classifies each user access request into one of three predefined categories, namely market data subscription request, order instruction request, or historical query request, thereby completing the identification of the type of user access request.

[0072] Step 202: Using a priority scheduling algorithm, assign a corresponding processing priority to each request type, and bind each user access request to the processing priority to obtain the user access request with the assigned processing priority.

[0073] In step 202, processing priority refers to the level identifier assigned to the request, usually represented by a number, with the size of the number representing the order of processing; the user access request with assigned processing priority not only contains the original request content, but also includes a processing priority identifier assigned by the priority scheduling algorithm.

[0074] In this embodiment, the priority scheduling algorithm maintains a preset priority policy mapping table, which clearly defines the processing priority value corresponding to each request type. Therefore, the priority scheduling algorithm can be used to match the request type identified in step 201 with the priority policy mapping table to retrieve the corresponding processing priority value. Subsequently, this value is bound to the corresponding user access request, so that each user access request carries a clear priority identifier, thereby obtaining the user access request with the assigned processing priority.

[0075] Step 203: Input the user access requests with assigned processing priorities into the lock-free circular queue for buffering. After buffering is complete, adjust the positions of the write pointer and read pointer, and combine the flow shaping algorithm to schedule and manage the user access requests in the lock-free circular queue to obtain the managed user access requests.

[0076] In step 203, the write pointer and read pointer are logical pointers inside the lock-free circular queue used to track the data write and read positions. They are the core components that constitute the data structure and implement its lock-free buffering function.

[0077] Managed user access requests refer to requests that have been buffered by a lock-free circular queue and scheduled by a traffic shaping algorithm and are ready to be extracted by subsequent processing units. These requests have been initially sorted and managed in the queue according to their priority.

[0078] In this embodiment, firstly, the user access requests with assigned processing priorities obtained in step 202 are sequentially stored in a lock-free circular queue for buffering according to their arrival order. This embodiment uses atomic operations to update the position of the write pointer to safely complete the addition of requests. Subsequently, when the queue utilization rate approaches its capacity limit, the position relationship between the write pointer and the read pointer is adjusted collaboratively using a flow shaping algorithm, and a management strategy is executed according to the priority identifier carried by the user access request. That is, when a request needs to be discarded, low-priority requests are discarded first, or the dequeue rate of different priority requests is adjusted differently, thereby realizing the effective scheduling and management of user access requests in the queue, and finally outputting the user access requests after the above management.

[0079] Here is a specific example:

[0080] Following the aforementioned embodiments, this embodiment of the application, while transmitting the target message object through the optimal physical network link fiber optic leased line, also begins to process the 150 previously cached user access requests in parallel. This embodiment of the application first receives the raw data of these 150 user access requests through the fiber optic leased line, and then parses the message content of each user access request. By checking specific fields in the message body, the request type is identified. For example, it is found that the message body of 50 requests contains keywords representing subscription, thus identifying them as market data subscription requests; 60 requests contain keywords representing order placement, thus identifying them as order placement instructions; and 40 requests contain keywords representing historical query, thus identifying them as historical query requests. In this way, 150 classified request types are obtained.

[0081] Subsequently, given that the priority strategy mapping table specifies that the order instruction request corresponds to processing priority 3, the market data subscription request corresponds to processing priority 2, and the historical query request corresponds to processing priority 1, this embodiment of the application calls the priority scheduling algorithm, uses the internally preset matching function to assign processing priority 2 to 50 market data subscription requests, processing priority 3 to 60 order instruction requests, and processing priority 1 to 40 historical query requests, and binds each user access request with the corresponding processing priority value, thereby obtaining 150 user access requests with assigned processing priorities.

[0082] Subsequently, in this embodiment, the 150 user access requests with assigned processing priorities are sequentially input into a lock-free circular queue with a capacity of 1000 for buffering according to their receiving order; when the 800th request is written, the queue utilization rate approaches the capacity limit, and the buffering operation is completed. At this time, this embodiment adjusts the relative relationship between the write pointer pointing to position 800 and the read pointer pointing to position 50, and combines the traffic shaping algorithm to schedule and manage the requests in the queue.

[0083] The algorithm calculates the retention weight of different priority requests based on the current queue fullness (800 ÷ 1000 = 80%) and the proportion of requests of each priority. The retention weight of the entrustment instruction request is set to 1.0, the retention weight of the market data subscription request is set to 0.9, and the retention weight of the historical query request is set to 0.7. When it is necessary to make room for newly arrived requests, the system will prioritize eliminating requests with lower retention weights, thereby ensuring that high-priority requests can be retained in the queue to the greatest extent. Finally, the algorithm outputs the set of user access requests after the above management, which prepares for the subsequent request extraction and processing.

[0084] In this embodiment, the above steps effectively streamline and schedule the request flow under high concurrency access pressure, thereby improving the overall response speed, processing efficiency, and stability of the external services provided by this embodiment.

[0085] To further improve the accuracy and automation of request priority allocation, in some embodiments, step 202: using a priority scheduling algorithm to assign a corresponding processing priority to each request type, and binding each user access request with the processing priority to obtain the user access request with the assigned processing priority, includes:

[0086] Step 301: For each user access request to be assigned a processing priority, the request type is matched with the known request types in the preset priority policy mapping table using the matching function of the priority scheduling algorithm.

[0087] In step 301, the user access request to be assigned processing priority refers to the original user access request that has been identified in step 201 as a market data subscription request, order instruction request, or historical query request, and is waiting to be assigned processing priority.

[0088] The matching function represents the core component inside the priority scheduling algorithm used for pattern comparison. This function compares an input request type with a known list of request types to determine whether the two match.

[0089] The preset priority strategy mapping table represents a predefined mapping table, which stores the fixed mapping relationship between all known request types and their corresponding processing priority values ​​in this application embodiment.

[0090] In this embodiment of the application, for each user access request to be assigned a processing priority, the priority scheduling algorithm first calls its internal matching function and compares the request type identified in step 201 with the known request types registered in the priority policy mapping table preset in the algorithm one by one, and finally determines whether the request type exists in the priority policy mapping table.

[0091] Step 302: When a match is successful, retrieve the corresponding processing priority from the priority strategy mapping table.

[0092] In this embodiment of the application, when the matching function in step 301 confirms that the request type of the user access request is consistent with a known type in the priority policy mapping table, the priority scheduling algorithm uses this successfully matched request type as an index to search in the preset priority policy mapping table, thereby retrieving the processing priority value that strictly corresponds to the request type.

[0093] Step 303: Using the allocation function of the priority scheduling algorithm, the processing priority is assigned to the corresponding user access request to obtain the user access request with the assigned processing priority.

[0094] In step 303, the allocation function represents the functional component inside the priority scheduling algorithm responsible for assigning priority values. This function associates the retrieved processing priority value with the specific user access request.

[0095] In this embodiment of the application, the binding operation is performed by the allocation function of the priority scheduling algorithm. The function receives the processing priority value retrieved in step 302 and assigns it to the user access request to be assigned the processing priority in step 301, so that the request is closely associated with the processing priority value, thereby generating a user access request with an assigned processing priority. This binding process is performed one by one on each user access request in the queue until all user access requests have completed the processing priority allocation.

[0096] In this embodiment, the above steps lay a reliable foundation for subsequent priority-based request scheduling and management, thereby improving the intelligence level and overall efficiency of request processing in this embodiment.

[0097] As a specific implementation, the real-time network status parameters include latency measurement, jitter measurement, and packet loss rate measurement. To further improve the accuracy and adaptability of network link selection, in some embodiments, step 103: based on the real-time network status parameters, dynamically calculating the target quality score of each physical network link using a link quality assessment algorithm, determining the optimal physical network link from multiple physical network links according to the target quality score, and routing the transmission session corresponding to the target message object to the optimal physical network link, includes:

[0098] Step 401: Assign corresponding weight coefficients to the delay measurement value, the jitter measurement value, and the packet loss rate measurement value respectively.

[0099] In step 401, the real-time network status parameters refer to the latency measurement, jitter measurement, and packet loss rate measurement values ​​periodically collected from each physical network link;

[0100] The weighting coefficient is a value set for each of the three network state parameters: latency measurement, jitter measurement, and packet loss rate measurement. This value represents the proportion of that parameter in the final link quality score.

[0101] In this embodiment of the application, according to the preset evaluation strategy, a corresponding weight coefficient is assigned to each of the three parameters obtained in step 401: latency measurement value, jitter measurement value, and packet loss rate measurement value. These coefficients determine the magnitude of the impact of each parameter on the final link quality score.

[0102] Step 402: Based on the weighting coefficients and combined with the link quality assessment algorithm, the various measurement values ​​are weighted and calculated to obtain the initial quality score for each physical network link.

[0103] In step 402, the initial quality score refers to the preliminary quantitative assessment result of the physical network link quality obtained through weighted calculation.

[0104] In this embodiment of the application, a link quality assessment algorithm is used to multiply the latency measurement, jitter measurement, and packet loss rate measurement of each physical network link by their corresponding weight coefficients, and then the three product results are added together to calculate the initial quality score of each physical network link.

[0105] Step 403: Process the initial quality score using a fuzzy logic decision algorithm to generate a target quality score.

[0106] In this embodiment, a fuzzy logic decision algorithm is used to divide the initial quality score of each link into different preset quality level ranges, and then converts it into a standardized target quality score according to the membership function, thereby generating a clearer and easier-to-compare target quality score between different links.

[0107] Step 404: Compare the target quality scores of all physical network links and determine the physical network link with the highest target quality score as the optimal physical network link.

[0108] In this embodiment of the application, the target quality scores obtained by all physical network links in step 403 are compared, and the physical network link with the highest target quality score is determined as the current optimal physical network link by comparing the numerical values.

[0109] Step 405: Identify transmission session information from the target message object and establish the transmission connection channel of the optimal physical network link.

[0110] In step 405, the transmission session information refers to the information extracted from the target message object for establishing and maintaining the transmission connection. This information includes the data sender address, data receiver address, port number, etc.; the transmission connection channel refers to the logical path established on the optimal physical network link for transmitting data.

[0111] Step 406: Migrate the target message object and the transmission session information to the corresponding transmission connection channel.

[0112] In this embodiment, the target message object to be transmitted and its complete transmission session information are migrated together to the transmission connection channel established on the optimal physical network link in step 405, thereby completing the routing process and ensuring that subsequent data is transmitted through the optimal link.

[0113] In the embodiments of this application, the above-described steps effectively improve the timeliness and reliability of data transmission and ensure the stability of key market information transmission.

[0114] To further improve the accuracy and modularity of link quality scoring calculations, in some embodiments, step 402: based on the weighting coefficients and combined with the link quality assessment algorithm, weighted calculations are performed on the various measurement values ​​to obtain the initial quality score for each physical network link, including:

[0115] Step 501: Based on the link quality assessment algorithm, construct the first scoring function, the second scoring function, the third scoring function, and the summation function.

[0116] In step 501, the first scoring function, the second scoring function, and the third scoring function represent three independent computational units within the link quality assessment algorithm, each specifically responsible for performing basic scoring on a network state parameter; wherein, the first scoring function processes latency measurements, the second scoring function processes jitter measurements, and the third scoring function processes packet loss rate measurements.

[0117] The summation function represents the calculation unit within the link quality assessment algorithm used to combine multiple weighted scores into a single total score. Specifically, the summation function accumulates the three weighted scores, i.e., it sums the three input values: weighted delay score, weighted jitter score, and weighted packet loss score.

[0118] In this embodiment of the application, four functional functions required by the link quality assessment algorithm are constructed: a first scoring function specifically for handling latency, a second scoring function specifically for handling jitter, a third scoring function specifically for handling packet loss rate, and a summation function for summation.

[0119] Step 502: Input the latency measurement value, the jitter measurement value, and the packet loss rate measurement value into the corresponding scoring function to obtain the latency scoring component, jitter scoring component, and packet loss scoring component.

[0120] In step 502, the latency score component refers to the base score obtained after the latency measurement value is transformed by the first scoring function. This latency score component reflects the link quality in terms of latency alone. The jitter score component refers to the base score obtained after the jitter measurement value is transformed by the second scoring function. This jitter score component reflects the link quality in terms of jitter alone. The packet loss score component refers to the base score obtained after the packet loss rate measurement value is transformed by the third scoring function. This packet loss score component reflects the link quality in terms of packet loss rate alone.

[0121] Step 503: Multiply the delay score component, the jitter score component, and the packet loss score component by their respective weighting coefficients to obtain the weighted delay score, the weighted jitter score, and the weighted packet loss score.

[0122] In step 503, the weighted delay score is the contribution of the delay parameter to the overall score, the weighted jitter score is the contribution of the jitter parameter to the overall score, and the weighted packet loss score is the contribution of the packet loss rate parameter to the overall score.

[0123] Step 504: Using the summation function, sum the weighted delay score, the weighted jitter score, and the weighted packet loss score to obtain the initial quality score for each physical network link.

[0124] In this embodiment of the application, the weighted delay score, weighted jitter score and weighted packet loss score are summed by the summation function constructed in step 501 to obtain the initial quality score of the physical network link. This process is performed independently once for each link to be evaluated.

[0125] In the embodiments of this application, the above-mentioned steps decompose the comprehensive evaluation into independent scoring, weighting and summing stages, making the scoring process clearer, more modular, easier to adjust and maintain. At the same time, by using specialized scoring functions for different parameters, the actual impact of each parameter on link quality can be reflected more precisely, thereby improving the accuracy and rationality of the initial quality score.

[0126] To address the accuracy and efficiency issues in parsing multi-source heterogeneous data, in some embodiments, step 102: real-time parsing of the original binary data stream based on the protocol metadata to generate a target message object includes:

[0127] Step 601: Parse the protocol version information, message structure definition, and field mapping relationship in the protocol metadata.

[0128] In step 601, the protocol version information refers to the specific version number of the communication protocol used by the data source, which is used to determine the basic framework and compatibility requirements of the parsing rules;

[0129] The message structure definition refers to the information that describes the arrangement order, length, and boundary rules of the various components in a data packet, and the message structure definition specifies key structures such as message header length, message body format, and checksum location;

[0130] Field mapping relationship refers to the correspondence between specific fields in the original data packet and the unified field names within the system. In this application, the field mapping relationship is equivalent to a translation dictionary, indicating how to map multi-source heterogeneous data fields to standardized target fields.

[0131] In this embodiment of the application, the collected protocol metadata is parsed to identify and separate key protocol version information, detailed message structure definitions, and field mapping relationships, providing a basis for subsequent construction of parsing rules.

[0132] Step 602: Based on the protocol version information and the message structure definition, construct a protocol descriptor that describes the layout of binary data.

[0133] In step 602, the protocol descriptor represents a formatted description file generated based on protocol metadata to guide data parsing. In this application, the protocol descriptor clearly defines the layout rules of the raw binary data stream by integrating the protocol version and message structure.

[0134] In this embodiment of the application, the protocol version information and message structure definition parsed in step 601 are integrated and constructed into a complete protocol descriptor, wherein the descriptor describes in detail the organization of binary data.

[0135] Step 603: Dynamically assemble the parsing instruction sequence according to the protocol descriptor.

[0136] In step 603, the parsing instruction sequence represents a set of executable low-level operation commands dynamically generated according to the protocol descriptor. In this application, the parsing instruction sequence contains a series of specific instructions to guide how to process binary data streams.

[0137] In this embodiment of the application, a set of parsing instruction sequences is dynamically generated based on the protocol descriptor constructed in step 602, wherein these instruction sequences can transform the abstract rules in the protocol descriptor into specific, sequentially executable operation steps.

[0138] Step 604: Using the parsing instruction sequence, the original binary data stream is parsed bit by bit to extract the data fields.

[0139] In step 604, a data field refers to a segment of original data that has independent meaning and is parsed from the original binary data stream.

[0140] In this embodiment of the application, the parsing instruction sequence assembled in step 603 is used as an operation guide to perform bit-by-bit parsing operations on the received raw binary data stream, and extract each independent data field contained therein in turn.

[0141] Step 605: Convert the data field into a memory object according to the field mapping relationship.

[0142] In step 605, a memory object refers to a temporary data entity created in computer memory that contains a specific data type and value.

[0143] In this embodiment of the application, the data fields extracted in step 604 are converted and reorganized according to the field mapping relationship parsed in step 601 to create a memory object that conforms to the internal specifications of the system.

[0144] Step 606: Perform a semantic integrity check on the memory object, and encapsulate the memory object that passes the check into a target message object.

[0145] In this embodiment of the application, firstly, the memory object created in step 605 is subjected to semantic integrity check to verify the validity and integrity of its data. Then, the memory object that passes the check is finally encapsulated, and meta-information such as timestamp and data source is added to generate a unified target message object.

[0146] In this embodiment of the application, the above-described steps lay a reliable foundation for subsequent data processing and transmission through a modular parsing process.

[0147] To further improve the efficiency and orderliness of request processing in high-concurrency request scenarios, in some embodiments, step 105: the step of extracting and processing managed user access requests from the lock-free circular queue using a lock-free synchronization mechanism in conjunction with a distributed concurrency control method includes:

[0148] Step 701: Using a distributed concurrency control method, compare the positional relationship between the read pointer and the write pointer in the lock-free circular queue, and based on the comparison result, determine whether there are any pending requests in the lock-free circular queue.

[0149] In step 701, the positional relationship between the read pointer and the write pointer refers to the relative positions of the two key pointers in the lockless circular queue, where the read pointer points to the position of the next pending read request, and the write pointer points to the position of the next pending write request; pending requests refer to user access requests that have been stored in the lockless circular queue and are waiting to be retrieved and executed.

[0150] In this embodiment of the application, the distributed concurrency control method periodically checks the current positions of the read pointer and the write pointer in the lock-free circular queue, and then compares the positional relationship between the two pointers. If the write pointer is after the read pointer, it is determined that there is a request to be processed in the queue; or, if the positions of the two pointers coincide, it is determined that the queue is currently empty.

[0151] Step 702: When the judgment result is that there are pending requests, all managed user access requests are extracted from the lock-free circular queue using a lock-free synchronization mechanism. Based on the priority identifier of the managed user access requests, the pending request with the highest priority identifier is identified.

[0152] In step 702, the request with the highest priority is the one with the highest priority value among all pending requests in the lock-free circular queue.

[0153] In this embodiment of the application, when step 701 determines that there are pending requests in the lock-free circular queue, this embodiment of the application uses a lock-free synchronization mechanism to safely extract all managed user access requests in batches from the lock-free circular queue through atomic operations; then, the priority identifiers carried by these requests are scanned, and the request with the highest priority identifier is identified by comparing the numerical values ​​of the priority identifiers, and it is determined as the pending request to be processed.

[0154] Step 703: Based on the request type of the pending request, distribute the pending request to the corresponding target processing thread, and execute the data access operation corresponding to the pending request through the target processing thread.

[0155] In step 703, the target processing thread refers to the execution thread created or allocated within the distributed concurrency control method to handle specific types of requests, and each thread is specifically responsible for handling a certain type of request in order to achieve request classification and processing and optimized scheduling of system resources.

[0156] In this embodiment of the application, the request is distributed to the corresponding target processing thread according to the request type of the request to be processed identified in step 702. For example, a delegated instruction request will be distributed to the delegated processing thread; then the target processing thread that receives the request will execute the data access operation corresponding to the request to complete the request processing process.

[0157] In this embodiment of the application, the above steps combine intelligent request scheduling with dedicated line processing to ensure efficient and orderly processing of user requests in high-concurrency scenarios, thereby improving the overall processing capacity and response speed of this embodiment of the application.

[0158] Figure 3This application provides a schematic diagram of the structure of a multi-source heterogeneous data dynamic integration and unified access system based on distributed concurrent control, and the specific implementation details are as follows:

[0159] The acquisition module 31 is used to acquire multi-source heterogeneous data from multiple data sources, including: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests.

[0160] The parsing module 32 is used to parse the original binary data stream in real time based on the protocol metadata and generate a target message object.

[0161] The calculation module 33 is used to dynamically calculate the target quality score of each physical network link based on the real-time network status parameters using a link quality assessment algorithm, determine the optimal physical network link from multiple physical network links according to the target quality score, and route the transmission session corresponding to the target message object to the optimal physical network link.

[0162] The identification module 34 is used to identify the types of the multiple user access requests based on the optimal physical network link, obtain multiple request types, and use a lock-free circular queue for buffering and management based on different request types.

[0163] Extraction module 35 is used to extract and process managed user access requests from the lock-free circular queue by combining distributed concurrency control methods and lock-free synchronization mechanism, so as to complete the dynamic integration and unified access of multiple data sources.

[0164] The system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control in this application is used to implement the aforementioned method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control. Therefore, the specific implementation of the system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control can be found in the embodiment section of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0165] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-described method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control.

[0166] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control.

[0167] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0168] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control.

[0169] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] The foregoing has provided a detailed description of a method and system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control, characterized in that, include: Collect multi-source heterogeneous data from multiple data sources, including: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests; Based on the protocol metadata, the original binary data stream is parsed in real time to generate a target message object; Based on the real-time network status parameters, the target quality score of each physical network link is dynamically calculated using a link quality assessment algorithm. The optimal physical network link is determined from multiple physical network links based on the target quality score, and the transmission session corresponding to the target message object is routed to the optimal physical network link. Based on the optimal physical network link, the multiple user access requests are identified to obtain multiple request types. Based on different request types, a lock-free circular queue is used for buffering and management. By combining distributed concurrency control methods, user access requests are extracted and processed from the lock-free circular queue through a lock-free synchronization mechanism to achieve dynamic integration and unified access to multiple data sources.

2. The method according to claim 1, characterized in that, Based on the optimal physical network link, the process involves identifying the types of multiple user access requests to obtain multiple request types. Based on different request types, a lock-free circular queue is used for buffering and management, including: By using the optimal physical network link, the message content of each user access request is parsed to identify the request type of each user access request; Using a priority scheduling algorithm, a corresponding processing priority is assigned to each request type, and each user access request is bound to the processing priority to obtain the user access request with the assigned processing priority; User access requests with assigned processing priorities are input into a lock-free circular queue for buffering. After buffering is complete, the positions of the write and read pointers are adjusted, and the user access requests in the lock-free circular queue are scheduled and managed using a flow shaping algorithm to obtain the managed user access requests.

3. The method according to claim 2, characterized in that, The step of using a priority scheduling algorithm to assign a corresponding processing priority to each request type and binding each user access request with the processing priority to obtain user access requests with assigned processing priorities includes: For each user access request to be assigned a processing priority, the matching function of the priority scheduling algorithm is used to match the request type with the known request types in the preset priority policy mapping table; When a match is successful, the corresponding processing priority is retrieved from the priority strategy mapping table; The processing priority is assigned to the corresponding user access request by the allocation function of the priority scheduling algorithm, thus obtaining the user access request with the assigned processing priority.

4. The method according to claim 1, characterized in that, The real-time network status parameters include latency measurement, jitter measurement, and packet loss rate measurement. The step of dynamically calculating the target quality score of each physical network link based on the real-time network status parameters using a link quality assessment algorithm, determining the optimal physical network link from multiple physical network links based on the target quality score, and routing the transmission session corresponding to the target message object to the optimal physical network link includes: Assign corresponding weighting coefficients to the latency measurement value, the jitter measurement value, and the packet loss rate measurement value respectively; Based on the aforementioned weighting coefficients and combined with the link quality assessment algorithm, the various measurement values ​​are weighted and calculated to obtain the initial quality score for each physical network link. The initial quality score is processed using a fuzzy logic decision algorithm to generate a target quality score; Compare the target quality scores of all physical network links, and determine the physical network link with the highest target quality score as the optimal physical network link; Identify transmission session information from the target message object and establish a transmission connection channel for the optimal physical network link; The target message object and the transmission session information are migrated to the corresponding transmission connection channel.

5. The method according to claim 4, characterized in that, Based on the weighting coefficients and combined with the link quality assessment algorithm, the various measurement values ​​are weighted and calculated to obtain the initial quality score for each physical network link, including: Based on the link quality assessment algorithm, multiple scoring functions and a summation function are constructed. The latency measurement value, jitter measurement value, and packet loss rate measurement value are respectively input into the corresponding scoring function to obtain the latency scoring component, jitter scoring component, and packet loss scoring component; The delay score component, jitter score component, and packet loss score component are multiplied by their respective weighting coefficients to obtain the weighted delay score, weighted jitter score, and weighted packet loss score. The initial quality score for each physical network link is obtained by summing the weighted delay score, the weighted jitter score, and the weighted packet loss score using the summation function.

6. The method according to claim 1, characterized in that, The step of real-time parsing of the original binary data stream based on the protocol metadata to generate a target message object includes: Parse the protocol version information, message structure definition, and field mapping relationships in the protocol metadata; Based on the protocol version information and the message structure definition, a protocol descriptor describing the binary data layout is constructed; Dynamically assemble and parse instruction sequences based on the protocol descriptor; The original binary data stream is parsed bit by bit using the parsing instruction sequence to extract data fields; Based on the field mapping relationship, the data fields are converted into memory objects; Perform semantic integrity checks on the memory objects, and encapsulate the memory objects that pass the checks into target message objects.

7. The method according to claim 1, characterized in that, The method, which combines distributed concurrency control and uses a lock-free synchronization mechanism to extract and process managed user access requests from the lock-free circular queue, includes: By using a distributed concurrency control method, the positional relationship between the read pointer and the write pointer in the lock-free circular queue is compared, and based on the comparison result, it is determined whether there are any pending requests in the lock-free circular queue. When the judgment result is that there are pending requests, a lock-free synchronization mechanism is used to extract all managed user access requests from the lock-free circular queue, and the pending request with the highest priority is identified based on the priority identifier of the managed user access requests. Based on the request type of the pending request, the pending request is distributed to the corresponding target processing thread, and the data access operation corresponding to the pending request is executed by the target processing thread.

8. A system for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control, characterized in that, include: The acquisition module is used to collect multi-source heterogeneous data from multiple data sources. The multi-source heterogeneous data includes: protocol metadata, raw binary data streams, real-time network status parameters, and multiple user access requests. The parsing module is used to parse the original binary data stream in real time based on the protocol metadata and generate a target message object; The calculation module is used to dynamically calculate the target quality score of each physical network link based on the real-time network status parameters using a link quality assessment algorithm, determine the optimal physical network link from multiple physical network links according to the target quality score, and route the transmission session corresponding to the target message object to the optimal physical network link. The identification module is used to identify the types of the multiple user access requests based on the optimal physical network link, obtain multiple request types, and use a lock-free circular queue for buffering and management based on different request types. The extraction module is used to extract and process managed user access requests from the lock-free circular queue by combining distributed concurrency control methods and a lock-free synchronization mechanism, so as to complete the dynamic integration and unified access of multiple data sources.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program, implements the steps of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the method for dynamic integration and unified access of multi-source heterogeneous data based on distributed concurrent control as described in any one of claims 1 to 7.

Images