Data processing method and device based on shared memory, equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device based on shared memory, equipment and storage medium. The method comprises: obtaining queue identification information to be read from a first master queue; wherein the queue identification information is used to represent a first sub-queue, the first sub-queue is used to cache service data, and the first sub-queue is a shared memory queue; obtaining at least one service data from the first sub-queue corresponding to the queue identification information to be read; and performing data processing on the obtained service data. The sub-queue caches real data, and the master queue only retains sub-queue identification, thereby reducing scheduling overhead and improving data transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and storage medium based on shared memory. Background Technology

[0002] As enterprise-level businesses increasingly demand real-time and reliable data, especially in fields such as finance, telecommunications, and the Internet of Things, systems need to achieve efficient transmission and processing of large-scale data in environments spanning data centers and clusters.

[0003] Currently, it is difficult to achieve a balance between transmission performance and resource management, which leads to problems such as resource contention, latency jitter, and overload packet loss in high-load scenarios, affecting the efficiency and accuracy of data processing. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium based on shared memory to improve the efficiency and accuracy of data processing.

[0005] In a first aspect, embodiments of this application provide a data processing method based on shared memory, including:

[0006] Obtain the queue identifier information to be read from the first main queue; wherein, the queue identifier information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue;

[0007] Obtain at least one piece of business data from the first sub-queue corresponding to the queue identifier information to be read;

[0008] The acquired business data is processed.

[0009] Secondly, embodiments of this application provide a data processing apparatus based on shared memory, comprising:

[0010] The first acquisition unit is used to acquire queue identification information to be read from the first main queue; wherein, the queue identification information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue;

[0011] The second acquisition unit is used to acquire at least one business data from the first sub-queue corresponding to the queue identifier information to be read;

[0012] The data processing unit is used to process the acquired business data.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the processor to perform the implementation method described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the embodiments described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the implementation methods described in the first aspect above.

[0018] This application provides a data processing method, apparatus, device, and storage medium based on shared memory. Different business data are cached in different first sub-queues, achieving multi-dimensional resource isolation and avoiding data chaos. The queue identifier information of the first sub-queue is written into the first main queue, and the business data is written into the first sub-queue. That is, the first sub-queue records the actual business data, and the first main queue records the first sub-queue containing the business data. The queue identifier information is obtained from the first main queue, and then each business data is retrieved from the first sub-queue corresponding to the queue identifier information. Finally, the business data is processed. By using a sub-queue + main queue approach, the problems of resource contention and scheduling fairness are solved. The actual data is cached in the sub-queues, and the main queue only retains the sub-queue identifiers, reducing scheduling overhead and ultimately achieving high efficiency and reliability in data processing under high concurrency scenarios. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A flowchart illustrating a data processing method based on shared memory provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of data flow transmission during the data processing stage provided in an embodiment of this application;

[0022] Figure 3 A flowchart illustrating a data processing method based on shared memory provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of data flow transmission during the data enqueueing stage provided in an embodiment of this application;

[0024] Figure 5 A schematic diagram of a data processing device based on shared memory provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0028] First, let me explain the terms used in this application:

[0029] ShmQueue: A shared memory queue that supports efficient data transfer across processes;

[0030] DataPackage: The serialized data packet, containing business data and metadata;

[0031] queueInform: Notification that the queue is not empty, triggering the subsequent transmission process;

[0032] Thread pool: refers to a resource pool composed of multiple worker threads, used to process task data in parallel. For example, a thread pool contains 10 worker threads, each of which independently obtains sub-queue information from the main queue to obtain business data.

[0033] In distributed systems and cloud computing environments, data transmission and processing technologies are increasingly important as core infrastructure supporting critical applications such as financial transactions, log processing, and message queues. With the continuous expansion of business scale and the ever-increasing demands for real-time performance, the industry widely adopts message middleware, remote procedure call frameworks, and inter-process communication mechanisms to achieve efficient data flow. However, current data processing processes generally suffer from uneven resource allocation, low scheduling efficiency, and data source latency, posing challenges to overall system performance and real-time capabilities.

[0034] This application provides a data processing method, apparatus, device, and storage medium based on shared memory, which are applied in the field of data processing technology to improve the efficiency and accuracy of data processing.

[0035] It should be noted that the data in this embodiment is not specific to any particular user and does not reflect the personal information of any particular user. It should also be noted that the data in this embodiment comes from a publicly available dataset.

[0036] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] To enable readers to have a deeper understanding of the implementation principles of this application, the embodiments are further detailed.

[0038] Figure 1 This is a flowchart illustrating a shared memory-based data processing method according to an embodiment of the present disclosure. This method can be executed by a shared memory-based data processing device. Figure 1 As shown, the method includes the following steps:

[0039] S101. Obtain the queue identifier information to be read from the first main queue; wherein, the queue identifier information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue.

[0040] For example, the first main queue can be a lock-free circular buffer running in user space or a FIFO (First In First Out) queue with a lightweight spinlock. Its logical structure only stores string or integer queue identification information and does not carry actual business data. The queue identification information is a unique identifier for the first sub-queue. For example, the queue identification information can be represented as "shm_q_{src_client_name}_{dest_center_id}_{dest_pod_namespace}", where shm_q indicates that the queue type is a shared memory queue, {src_client_name} is the identifier of the source business process, {dest_center_id} is the number of the geographic center to which the business data needs to reach, and {dest_pod_namespace} is the number of the pod cluster or pod node. The first sub-queue is a cross-process queue built on shared memory, supporting both blocking push / pop and non-blocking try_push / try_pop modes.

[0041] The first subqueue is decoupled from the first main queue. The first main queue only serves as a "registration book of active first subqueues." When a first subqueue changes from empty to non-empty, its queue identification information is atomically written to the first main queue once. It is not re-registered during subsequent periods of non-emptiness, thus reducing the access frequency of the first main queue to O(1). The first main queue can also be deployed in a dedicated scheduling coroutine, polling the status of each first subqueue in an event-driven manner, replacing thread-based polling and reducing CPU idle overhead. The queue identification information can be extended into a structure, embedding timestamps, priority flags, or checksum fields to support future extended timing control or integrity verification functions.

[0042] When processing data, one or more queue identifiers are first retrieved from the first main queue. Each queue identifier corresponds to a first sub-queue, and different first sub-queues cache different business data.

[0043] In this embodiment, a thread pool is pre-set, which includes at least two worker threads; obtaining queue identifier information to be read from the first main queue includes: determining the target thread from each worker thread according to the current state of each worker thread in the thread pool; wherein the current state is an idle state or a running state; and obtaining the queue identifier information to be read from the first main queue through the target thread.

[0044] Specifically, a thread pool refers to a set of reusable worker threads dynamically created during system initialization or runtime. These threads are used for unified scheduling and execution of data processing tasks, avoiding the overhead of frequently creating and destroying threads. Thread pools can employ a hybrid strategy of bounded queues, preset core thread counts, and maximum thread counts, supporting dynamic scaling. When the task submission rate consistently exceeds the processing capacity of currently active threads, new threads are automatically created until the maximum thread count is reached. When idle time exceeds a threshold and the number of threads exceeds the core count, excess threads are reclaimed. Each worker thread in the thread pool can be an independent user-mode execution flow, possessing its own stack space, register context, and task queue pointer. Its lifecycle is maintained uniformly by the thread pool manager. A minimum of two worker threads are required to ensure multi-task concurrent scheduling capabilities.

[0045] The current state of each thread can be monitored in real time or periodically. The current state is a real-time health snapshot of the thread during runtime, which can be periodically collected and maintained in a shared memory area or atomic variables. The current state can include two mutually exclusive states: idle state and running state. The idle state means that the worker thread is not currently performing data reading or processing operations and is in a blocked / polling state waiting for the first main queue to distribute tasks. The running state means that the worker thread is performing any step such as retrieving queue identification information from the first main queue, retrieving business data from the first sub-queue, or performing data processing. Its state can be determined through the task context identifier in the thread's local storage or the global state bitmap.

[0046] The scheduler iterates through the current state of all worker threads in the thread pool, prioritizing idle worker threads as target threads. If no idle threads are available, the scheduler selects the worker thread with the shortest execution time or the fewest processed tasks as the target thread, achieving a load balancing effect similar to shortest job priority. This process does not rely on lock contention but updates thread state bits and task pointers through lock-free CAS (Compare-and-Swap) operations, ensuring low-latency response under high concurrency. In this embodiment, the current state can also be extended to three levels, such as idle, running, and blocked waiting for the sub-queue to be ready, and a weighted round-robin mechanism is introduced to assign differentiated weights to different priority business channels.

[0047] The elements in the first main queue are of type queue identifier information. The queue identifier information in the first main queue represents a non-empty first sub-queue. If the first sub-queue is empty, its queue identifier information is removed from the first main queue. The target thread can extract queue identifier information from the head of the first main queue as queue identifier information to be read. The target thread can extract one or more queue identifier information to be read at a time.

[0048] As a resource container, the thread pool provides a unified foundation for lifecycle management and state maintenance for its internal worker threads. The current state of a worker thread is the input for scheduling decisions, directly determining the selection of the target thread. Once the target thread is determined, the queue identifier information is obtained from the first main queue to ensure that the task distribution path is clear and unambiguous.

[0049] The advantages of this setup are that the thread pool contains at least two worker threads, and the scheduling process dynamically selects the best threads based on their real-time idle / running status, avoiding the problem of some threads being continuously busy while others are idle for long periods, which is common with traditional polling or random allocation. The atomic acquisition of thread states and the lock-free decision-making mechanism significantly reduce scheduling overhead, allowing CPU cycles to be used more for actual data processing rather than coordination and waiting. This improves the overall system throughput and real-time response, making it particularly suitable for high-frequency, low-latency, and highly stable cross-center data transmission scenarios in disaster recovery construction and multi-center architecture optimization projects.

[0050] S102. Obtain at least one business data from the first sub-queue corresponding to the queue identifier information to be read.

[0051] For example, the queue identifier information extracted from the first main queue is determined as the queue identifier information to be read. The first sub-queue corresponding to the queue identifier information to be read is determined, and at least one piece of business data is obtained from this first sub-queue. For example, a batch harvesting mechanism can be used to extract multiple pieces of business data. A preset batch size N can be used to continuously read N pieces of business data in a single memory access. If there are fewer than N pieces of data remaining in the first sub-queue, the actual number that can be obtained is returned to avoid blocking and waiting.

[0052] In this embodiment, a client pool is pre-set, which includes at least two communication clients. Each communication client corresponds to a confidence level, and the confidence level characterizes the communication capability of the communication client. Obtaining at least one piece of business data from the first sub-queue corresponding to the queue identifier information to be read includes: determining a target client from among the communication clients based on the confidence levels of each communication client in the client pool; determining a target quantity based on the confidence level of the target client; wherein the target quantity characterizes the quantity of business data to be obtained; and obtaining the target quantity of business data from the first sub-queue corresponding to the queue identifier information to be read through a target thread, and sending the target quantity of business data to the target client.

[0053] Specifically, the client pool and thread pool reside on the same computer device. The client pool can be represented as URPCClientManager, which stands for Unified RPC Client Manager, a system component that integrates and manages multiple RPC (Remote Procedure Call) clients. In this embodiment, the communication client is an RPC client.

[0054] Each communication client can have its own confidence level, which is a scalar value that quantifies the current communication capability of the client and can be dynamically updated. Its calculation is based on, but is not limited to, the number of business data packets successfully processed and acknowledged per unit time, average end-to-end round-trip latency, remaining connection buffer capacity, success rate of the most recent N calls, and confidence values ​​actively fed back by the server. In this embodiment, the method for determining the confidence level is not specifically limited. A higher confidence level indicates that the communication client currently possesses stronger throughput, lower latency risk, and higher reliability.

[0055] Based on the confidence level of each communication client in the client pool, one or more communication clients are selected as target clients. For example, communication clients can be sorted according to their confidence levels, and the communication client with the highest confidence level can be selected as the target client.

[0056] The target quantity is the number of business data entries read in batches from the first sub-queue and prepared for transmission. Its value is not fixed but rather positively correlated with the current confidence level of the target client. The higher the confidence level, the greater the target quantity. For example, when the confidence level ∈ [0, 50), the target quantity = 1; when the confidence level ∈ [50, 80), the target quantity = 4.

[0057] The target thread retrieves the queue identifier information to be read from the first main queue, retrieves the target number of business data from the first sub-queue corresponding to the queue identifier information to be read, and then sends the retrieved target number of business data to the target client.

[0058] The client pool provides a client view based on confidence level, providing a basis for decision-making in selecting target clients. Confidence level, as a core control parameter, both drives the selection of target clients and determines the quantifiable value of the target number, ensuring a strict match between data supply intensity and downstream processing capacity.

[0059] The benefits of this setup are that, in scenarios with significant differences in client processing capabilities, it avoids forcibly pushing large amounts of data to low-confidence clients, which could lead to buffer overflows, connection timeouts, or a surge in response latency. It ensures that high-confidence clients receive sufficient computing bandwidth and data supply while limiting the request load on low-confidence clients, thereby significantly improving overall system throughput stability, reducing tail latency, enhancing fault isolation capabilities, and providing a verifiable flow control governance foundation for multi-tenant, multi-center, and elastic scaling scenarios in cloud-native environments.

[0060] In this embodiment, the method further includes: obtaining the current capability information of each communication client in the client pool; wherein the current capability information represents the capacity of service data occupied by the communication client; and updating the confidence level of the communication client based on the current capability information of the communication client.

[0061] Specifically, each communication client corresponds to an independent network connection channel. Its underlying communication protocol supports gRPC, HTTP / 2, or a custom binary protocol, and it possesses capabilities for connection keep-alive, heartbeat detection, abnormal reconnection, and traffic statistics. Current capability information characterizes the capacity of business data held by the communication client; that is, it indicates how many data items the client has already received or how much more data it can receive. Current capability information can be obtained through periodic reporting from the client side or proactive probing by the server. For example, when a target client sends business data to the server for processing, the server has multiple queues, each corresponding to a communication client. The server can determine the current communication capability of the client based on the amount of business data in the queue and provide feedback. For instance, if there is a large amount of business data in the queue, the current capability of the communication client corresponding to that queue is considered poor, and the amount of business data sent to that client needs to be appropriately reduced.

[0062] Based on the current capability information of the communication client, the confidence level of the communication client can be dynamically adjusted. For example, the stronger the current capability information of the communication client, the higher the confidence level. In this embodiment, the algorithm for updating the confidence level is not specifically limited; for example, nonlinear decay or incremental correction can be performed on the confidence level.

[0063] The beneficial effect of this setup is that, under the two-level queue scheduling framework, the real-time occupied capacity of the local buffer of the communication client is used as an objective input to drive the dynamic, non-linear, and configurable update of the confidence value. This ensures that the client pool always maintains a view of available clients that reflects the actual processing capacity, effectively avoiding the coexistence of overload impact and resource idleness, and improving the end-to-end data flow's steady-state maintenance capability and adaptability to sudden traffic.

[0064] S103. Process the acquired business data.

[0065] For example, after retrieving business data from the first sub-queue, the business data is processed. In this embodiment, data processing can broadly refer to any computational logic performed on the business data, including but not limited to protocol parsing, field validation, routing forwarding, state updates, format conversion, and persistent writing. This processing can be completed in user space by a pre-defined data processing thread, without involving cross-process scheduling. In this embodiment, no specific limitations are placed on the specific data processing logic.

[0066] In this embodiment, the first main queue serves as the top-level task discovery mechanism, the first sub-queue serves as the intermediate data carrying layer, and batch acquisition and zero-copy transmission are key constraints at the execution layer, determining how to acquire data efficiently. Because the main queue only registers sub-queue identifiers rather than carrying data, the complexity of task scheduling is significantly reduced, eliminating the contention lock overhead and task backlog inherent in traditional global queues. Since the first sub-queue is implemented based on shared memory and supports batch harvesting, it bypasses the kernel protocol stack, reducing cross-process data transmission latency. The main queue registration mechanism is triggered only when a sub-queue transitions from empty to non-empty, ensuring that all sub-queues are polled equally, avoiding starvation of low-priority queues caused by continuous preemption of high-priority queues, and improving data processing efficiency.

[0067] In this embodiment, data processing of the acquired business data includes: determining the queue identifier information of the second sub-queue corresponding to the acquired business data based on the communication client corresponding to the acquired business data; writing the acquired business data into the second sub-queue corresponding to the acquired business data; if the second sub-queue corresponding to the acquired business data is not empty for the first time, then writing the queue identifier information of the second sub-queue corresponding to the acquired business data into the second main queue; using a preset data processing thread to obtain target identifier information from the second main queue; wherein, the target identifier information represents the queue identifier information existing in the second main queue; and obtaining business data from the second sub-queue corresponding to the target identifier information for processing.

[0068] Specifically, the second sub-queue refers to the shared memory queue ShmQueue used to receive business data flowing through communication clients. Its physical implementation is a named shared memory segment built based on POSIX or System V standards. The first and second sub-queues have the same form, using spinlocks or condition variables to achieve safe cross-process concurrent access. The second sub-queue is logically isolated from the first sub-queue and named independently. Each second sub-queue also has its own queue identifier information and is used to cache the actual business data. After receiving business data, the target client can send the data to a specific second sub-queue. Business data from different communication clients can also be sent to different second sub-queues, determining the queue identifier information of the second sub-queue where the business data resides.

[0069] The business data in the communication client is written to the corresponding second sub-queue via the communication client. A second main queue is pre-set, isolated from the first main queue, and is used to cache the queue identification information of the second sub-queue. If the second sub-queue was previously empty, i.e., when it becomes non-empty for the first time, its queue identification information is written to the second main queue. The second main queue is the global scheduling hub; essentially, it is a lock-free single-producer multi-consumer queue, storing only the queue identification information of the second sub-queue and not carrying actual business data, thus achieving separation of control flow and data flow.

[0070] A pre-defined data processing thread, a dedicated execution unit independent of the thread pool, can extract queue identifier information from the head of the second main queue in real time via CAS atomic operations, using it as the target identifier information. This process does not involve copying business data, only transmitting lightweight metadata, ensuring that the scheduling overhead remains constant at O(1). The obtained target identifier information is the queue identifier string of the second sub-queue.

[0071] The data processing thread locates the shared memory address of the corresponding second sub-queue based on the target identifier information, and calls the pop interface to read a batch of business data. Each data item is a serialized DataPackage structure, containing the original payload, context header, timestamp, and checksum. The data processing thread processes the business data, including but not limited to deserialization parsing, business rule engine matching, related data querying, result aggregation calculation, anomaly classification and labeling, and log tracking. In this embodiment, the processing logic is not specifically limited. The processing results can be directly output to downstream services or repackaged into new business data for subsequent processes. All processing is completed in user space, avoiding kernel-mode switching overhead.

[0072] Figure 2 This is a schematic diagram of data flow transmission during the data processing stage. Figure 2 In this diagram, A2, B2, and C2 are second sub-queues, and D2 is the second main queue. When the first piece of business data enters each second sub-queue, its queue identifier is written to the second main queue. The same sub-queue's identifier can be written to the second main queue multiple times, but duplicate identifiers will not appear simultaneously. For example, if the identifier for sub-queue C2 is written the first time it is not empty, and then rewritten the next time it is not empty after the current business data in C2 is cleared, the identifier will be rewritten. Multi-threaded data processing refers to a pre-defined data processing thread retrieving queue identifier information from the second main queue, then retrieving business data from the corresponding second sub-queue for processing, and finally outputting the processing result.

[0073] The beneficial effects of this setup are that it enables standardized modeling and efficient execution of multi-stage asynchronous data processing flows. By employing a two-tier scheduling structure consisting of a second sub-queue and a second main queue, it solves the problems of single-level data processing being unable to support complex chained processes and lacking the ability to reschedule processing results. By setting up two types of sub-queues and two types of main queues, it achieves resource decoupling and fault domain convergence between different processing stages, improving the efficiency and accuracy of data processing.

[0074] In this embodiment, determining the queue identifier information of the second sub-queue corresponding to the acquired service data based on the communication client corresponding to the acquired service data includes: determining the queue identifier information of the second sub-queue corresponding to the client identifier of the communication client corresponding to the acquired service data based on a preset first association relationship; wherein, the preset first association relationship represents the association relationship between the client identifier of the communication client and the queue identifier information of the second sub-queue.

[0075] Specifically, each communication client has its own unique identifier, which serves as the client identifier. Each communication client also has a corresponding second sub-queue, meaning that there is a pre-defined association between the client identifier and the queue identifier information of the second sub-queue; this association is the first association.

[0076] When writing business data into the second sub-queue, the communication client where the business data resides is determined. Based on the identification information of the communication client and the preset first association relationship, the second sub-queue corresponding to the communication client can be determined, that is, the queue identification information of the second sub-queue corresponding to the business data is obtained.

[0077] Since the second sub-queue corresponds to the communication client, the current communication capability of the communication client can be determined by the amount of business data in the second sub-queue, and the current communication capability can be fed back to the client pool for timely update of confidence.

[0078] The advantage of this setup is that by adopting an identifier mapping mechanism based on a preset first association relationship, the data stream can maintain semantic coherence between the writing end and the processing end, improving the accuracy and continuity of the data processing link, and increasing the efficiency and precision of data processing.

[0079] This application provides a data processing method based on shared memory. Different business data are cached in different first sub-queues, achieving multi-dimensional resource isolation and avoiding data chaos. The queue identifier information of the first sub-queue is written into the first main queue, and the business data is written into the first sub-queue. That is, the first sub-queue records the actual business data, and the first main queue records the first sub-queue containing the business data. The queue identifier information is obtained from the first main queue, and then each business data is retrieved from the first sub-queue corresponding to the queue identifier information. Finally, the business data is processed. By using a sub-queue + main queue approach, the problems of resource contention and scheduling fairness are solved. The actual data is cached in the sub-queues, and the main queue only retains the sub-queue identifiers, reducing scheduling overhead and ultimately achieving high efficiency and reliability in data processing under high concurrency scenarios.

[0080] Figure 3 This application provides a flowchart illustrating a data processing method based on shared memory, as shown in the embodiments below. Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed description of a data processing method based on shared memory, which includes:

[0081] S301. Obtain the service data to be transmitted based on the preset data interface, and determine the queue identifier information of the first sub-queue corresponding to the service data to be transmitted based on the attribute information of the service data to be transmitted.

[0082] For example, obtaining the business data to be transmitted based on a preset data interface refers to receiving the raw business data generated by the upstream business system through a standardized and configurable data access channel. This data interface supports multiple protocol formats, and this embodiment does not specifically limit the preset data interface.

[0083] The attribute information of business data can refer to metadata fields extracted from the business semantic layer that can structurally represent the data ownership relationship. These fields include at least one or more of the following: business type identifier (such as "liquidation log" or "account change"), data source process identifier, target server location location identifier, target Pod cluster namespace, and target Pod instance name. This attribute information can be automatically injected by the SDK (Software Development Kit) when the business process calls the DataTransformClient.writeAsync(data) interface, without relying on external configuration files or manual intervention. Here, DataTransformClient.writeAsync(data) is an asynchronous data write interface; DataTransformClient represents the client class, writeAsync represents the asynchronous write method, and data represents the data object.

[0084] In this embodiment, the attribute information may also include the data transmission execution mode, which can be either UNORDERED or ORDERED. UNORDERED mode is a multi-channel / multi-target concurrent model, where parameters such as shared_executor.thread_num and client_manager.connection_num can be configured independently. shared_executor.thread_num represents the number of threads in the shared thread pool, and client_manager.connection_num represents the number of connections in the client manager, i.e., the number of communicating clients in the client pool. ORDERED mode is a single-channel directed mode, where threads and clients are all assigned a single thread, with strict order, and the granularity of "process exclusive / process shared" can be selected.

[0085] The first sub-queue is a shared memory queue. Multiple first sub-queues can be pre-defined, each with a unique queue identifier. For example, based on the attribute information of business data, the queue identifier corresponding to that attribute information can be found from existing queue identifier information; this is the queue identifier for the first sub-queue corresponding to the business data. Alternatively, the queue identifier for the first sub-queue can be a unique resource name generated by string concatenating attribute information according to a preset template rule. That is, queue identifier information can be generated based on attribute information to obtain a new first sub-queue, and the queue identifier for this new first sub-queue is the generated queue identifier. The queue identifier can also serve as the naming key for shared memory segments and the naming prefix for mutexes / condition variables, ensuring that the same queue identifier corresponds to a unique IPC (Inter-Process Communication) resource entity.

[0086] In this embodiment, the service data to be transmitted refers to the service data that needs to be written into the first sub-queue, while in the above embodiment, the service data obtained refers to the service data extracted from the first sub-queue.

[0087] In this embodiment, determining the queue identifier information of the first sub-queue corresponding to the service data to be transmitted based on the attribute information of the service data to be transmitted includes: determining the context information of the service data to be transmitted based on the attribute information of the service data to be transmitted and a preset routing strategy; wherein the context information includes at least one of the center identifier information of the server center where the service data is to be received, the node identifier information of the node to be received, and the client identifier information of the process client from which the service data comes; and determining the queue identifier information of the first sub-queue corresponding to the service data to be transmitted based on the context information of the service data to be transmitted and a preset second association relationship; wherein the preset second association relationship represents the association relationship between the context information and the queue identifier information.

[0088] Specifically, the preset routing strategy is a lightweight strategy engine deployed on the data writing end. Its strategy rules are configured in JSON or Protobuf format and are hot-loaded, supporting various logics such as conditional expressions, regular expression matching, hash sharding, and weighted round-robin. The preset routing strategy is used to determine how business data reaches its destination. For example, when the business type encoding in the attribute information is "BINLOG_SYNC" and the timeliness level is "REALTIME", a "strong order routing branch" is triggered, prioritizing the matching of the combination of dest_pod_namespace and dest_pod_name; when the business type encoding is "CLEARING_LOG" and the data length is greater than 1MB, a "multi-channel sharding routing branch" is triggered, distributing the data to N target Pod clusters according to hash(data_id) % N.

[0089] Context information is a structured intermediate result output after the routing policy is executed. Its data structure can be a set of key-value pairs, containing at least one or more of the following fields: center identifier information (dest_center_id), used to characterize the geographical or logical data center to which the target server belongs; the value can be a string. Node identifier information (dest_pod_name, dest_pod_namespace, dest_pod_ip), used to characterize the granularity of the target server's running instance in the cloud-native environment, where dest_pod_name is the Kubernetes Pod name (e.g., "service-impl-7f8c9d4b5-xvq2t"), dest_pod_namespace is the namespace (e.g., "prod-clearing"), and dest_pod_ip is the internal network IP address assigned to the Pod (e.g., "10.244.3.15"). Client identifier information (src_client_name), used to characterize the data source process. In this embodiment, the client of the data source process is different from the communication client.

[0090] This context information does not directly participate in data transmission, but serves as an input credential for subsequent resource binding and queue addressing. Its generation process is completed entirely in user space, with no system call overhead and low average time consumption.

[0091] The pre-defined second association is a static / dynamic association structure residing in memory in the form of a mapping table or hash index. Its key is a normalized string representation of the context information, and its value is the queue identifier information of the first sub-queue. This association supports a template expansion construction mode, which uses a placeholder replacement mechanism to substitute the context information fields into a predefined named template to generate queue identifier information. For example, the template "shm_q_{src_client_name}_{dest_center_id}_{dest_pod_namespace}" expands to "shm_q_12345_001_SHANGHAI_DC_prod-clearing" when the context is {src_client_name:“12345_001”, dest_center_id:“SHANGHAI_DC”, dest_pod_namespace:“prod-clearing”}.

[0092] The default first association can support hot updates. When a new center is added or a new Pod cluster is launched, incremental mapping entries can be pushed through the configuration center without restarting the process. It can also support a fallback strategy. When the context information does not match any association entries, it will automatically fall back to the default template "shm_q_default_{src_client_name}".

[0093] The advantage of this setup is that, based on dynamic context generation driven by business data attribute information, combined with configurable and hot-update primary associations, abstract business routing intentions are accurately mapped to specific shared memory queue resource identifiers. This solves the problem that relying solely on static configuration cannot adapt to complex routing requirements in cloud-native multi-center, multi-tenant, and elastic scaling scenarios. It provides a scalable and governable underlying resource addressing foundation for upper-layer mechanisms such as two-layer queue scheduling, confidence-driven backpressure, and Pod-level proxy connection optimization.

[0094] S302. Write the service data to be transmitted into the first sub-queue corresponding to the service data to be transmitted. If the first sub-queue corresponding to the service data to be transmitted is not empty for the first time, write the queue identifier information of the first sub-queue corresponding to the service data to be transmitted into the first main queue.

[0095] For example, for each piece of service data to be transmitted, the queue identifier information of the first sub-queue corresponding to that service data is determined, that is, it is determined in which first sub-queue the service data needs to be cached. The first sub-queue can be a pre-generated shared memory queue. Other service data may have already been written to the first sub-queue, or no service data may have been written to it yet. It can be determined whether the first sub-queue is empty. An empty first sub-queue means that the sub-queue is not currently occupied by any service data, its internal counter is zero, and the queue identifier information of the sub-queue is not registered in the first main queue. The first main queue does not store specific service data, but stores the queue identifier information of the first sub-queues that have been written with any service data. The emptiness of the first sub-queue can be determined by atomically reading the size field of the queue header structure in the first sub-queue, avoiding the overhead of locking. If it is determined that the first sub-queue is empty, it means that the first sub-queue is not empty for the first time when service data is written to it. Alternatively, after writing the business data into the first main queue, it can be determined whether there is only this one piece of business data in the first sub-queue. If so, the first sub-queue is determined to be non-empty for the first time; otherwise, the first sub-queue is determined not to be non-empty for the first time.

[0096] The primary queue is a lock-free circular buffer residing in the address space of the data writing process. Its elements are fixed-length queue identifier strings, supporting concurrent access by multiple producers and a single consumer. When the first sub-queue is determined to be writing data for the first time, the SDK uses an atomic CAS operation to append the queue identifier information of the first sub-queue to the end of the primary queue. This registration is only performed once; subsequent writes to the first sub-queue will not trigger another registration action unless the first sub-queue is empty and then data is written again. This lazy registration mechanism ensures that the actual number of entries in the primary queue is strictly equal to the number of currently active sub-queues, rather than the theoretical maximum.

[0097] Regardless of whether the first sub-queue is empty, once the first sub-queue corresponding to the business data is determined, the business data must be written into that first sub-queue.

[0098] In this embodiment, attribute information provides the business dimension, queue identifiers enable resource addressing, and main queue registration triggers scheduling activation. The "registration only for the first non-empty instance" constraint directly compresses the main queue size. This achieves refined resource isolation across multiple centers, nodes, and clients. Furthermore, by registering entries in the main queue only when the sub-queue is first non-empty, the number of entries in the main queue strictly matches the number of active business flows, maintaining scheduling complexity at a stable O(1) level and eliminating polling overhead. Because the registration action and data writing are atomically coupled, it ensures that the reading end can read valid data from the corresponding sub-queue after obtaining the identifier from the main queue, preventing false triggering of empty queues. This solves problems such as main queue redundancy expansion, low polling efficiency, and rigid resource binding, laying the foundation for a fair scheduling model of two-layer queues.

[0099] In this embodiment, it further includes: if the first sub-queue corresponding to the service data to be transmitted is not empty for the first time, then an enqueue notification is issued according to the queue identifier information of the first sub-queue corresponding to the service data to be transmitted; wherein, the enqueue notification indicates that a new first sub-queue is written into the first main queue.

[0100] Specifically, the first sub-queue refers to a buffer queue uniquely named with queue identification information, deployed in a shared memory area. It can be used to cache business data from the same business source process with the same routing context. The physical implementation of the first sub-queue can be a memory-mapped circular buffer, using named mutexes and condition variables to achieve cross-process synchronization, supporting blocking / non-blocking write modes. "First Not Empty" refers to the transient event where the length of data within the first sub-queue changes from 0 to greater than 0. This can be detected in real time using atomic counters or flags to avoid race conditions. "First Not Empty" does not necessarily mean the first time data is written to the first sub-queue since its creation; the first sub-queue can have multiple "First Not Empty" events. For example, if the first sub-queue is initially empty, writing a piece of data constitutes "First Not Empty." When this data is removed, the first sub-queue becomes empty again. If another piece of data is written, it again constitutes "First Not Empty."

[0101] If the first subqueue corresponding to the business data to be transmitted is not empty for the first time, an enqueue notification can be issued based on the queue identifier information of the first subqueue. If it is not not empty for the first time, no enqueue notification is issued. If the first subqueue corresponding to the business data to be transmitted is not empty for the first time, the queue identifier information of the first subqueue needs to be written into the first main queue. That is, the enqueue notification can indicate that a new first subqueue has been written into the first main queue. The enqueue notification is a lightweight asynchronous signal mechanism. Specific implementations may include: sending a short message to the scheduling component SrcDataTransform (source agent) via UDS (Unix Domain Socket), with the message carrying the queue identifier information and timestamp; or triggering kernel event notification using an event file descriptor; or setting a preset notification bitmap in shared memory and waking up waiting threads. This notification does not carry the business data itself, but only serves as a scheduling trigger signal, ensuring low overhead and high timeliness.

[0102] SrcDataTransform internally maintains a thread pool and a client pool. Upon receiving an enqueue notification, it immediately wakes up the idle target thread in its thread pool, which then performs the subsequent main queue scanning and sub-queue harvesting operations. Specifically, based on the enqueue notification, it retrieves the queue identifier information from the first main queue. For example, it can retrieve the queue identifier information of the first queue in the first main queue according to the order of the queue identifier information. It then retrieves at least one piece of business data from the first sub-queue corresponding to the retrieved queue identifier information, and performs data processing on the retrieved business data.

[0103] When the business process calls DataTransformClient.writeAsync(data) to complete a data enqueue, the internal state detection module of ShmQueue captures the state transition from "empty to non-empty" in real time and immediately triggers the notification mechanism. This notification is sent to SrcDataTransform with low latency through the UDS channel. After SrcDataTransform parses the queue identification information, the idle worker thread obtains the queue identification information from the first main queue through polling or condition variable wake-up, and then locates the corresponding first sub-queue and harvests data in batches.

[0104] The advantage of this setup is that it uses an event-driven mechanism of "notification upon first non-empty" to replace the passive waiting method of periodically scanning all sub-queues, significantly reducing idle waiting time. The notification message only contains the queue identifier string and does not require copying business data. Combined with the zero-copy transmission feature of UDS, this reduces CPU consumption and improves data processing efficiency.

[0105] In this embodiment, writing the service data to be transmitted into the first sub-queue corresponding to the service data to be transmitted includes: packaging the service data to be transmitted to obtain a data packet; and writing the data packet into the first sub-queue corresponding to the service data to be transmitted.

[0106] Specifically, the business data to be transmitted refers to the raw data units input from the business process layer, including but not limited to settlement logs, transaction logs, monitoring metrics, or configuration synchronization messages. The format of this business data is heterogeneous, potentially including JSON, Protobuf, Avro, custom binary structures, or plain text streams. Its length is variable, and it lacks a unified boundary identifier. This data has not undergone standardized encapsulation before entering the shared memory queue. Direct writing to this data leads to complex parsing logic on the consumer side, weak fault tolerance, poor cross-language / cross-platform compatibility, and is prone to parsing failures or data truncation due to missing fields, version mismatches, or inconsistent byte order.

[0107] Before writing the business data to be transmitted into the first sub-queue, the data can be packaged. This packaging process yields a data packet containing the business data to be transmitted. For example, structured encapsulation can be performed on the original business data to generate a unified data packet with a fixed header and a variable payload. This can include serialization, header padding, checksum injection, and optional compression and encryption operations. The packaging process can be completed internally within the DataTransformClient (data conversion and transmission client), running in user space and independent of kernel system calls. The DataTransformClient can be a client component that handles data conversion and transmission, processing data conversion, encapsulation, and asynchronous transmission.

[0108] The data packet is the output entity after packaging and processing. DataTransformClient serializes the data into a unified data packet, which is then enqueued through the shared memory queue, i.e., written to the first sub-queue.

[0109] The beneficial effect of this setup is that, given the highly heterogeneous format of the original business data, by performing standardized packaging processing on the data and writing it into the first sub-queue of shared memory, it solves the problems of difficult parsing on the consumer side, low transmission efficiency, and poor cross-platform compatibility caused by inconsistent data structures, thereby improving the efficiency and accuracy of data processing.

[0110] Figure 4 This is a schematic diagram of data flow transmission during the data queuing phase. Figure 4 In this process, the business data to be transmitted is written to the first sub-queue after passing through its respective process. A1, B1, C1, E1, F1, and G1 are the first sub-queues, and D1 and H1 are the second sub-queues. Different data within the same process can be written to different first sub-queues, but a single piece of business data can only be written to one first sub-queue. When the first sub-queue is not empty for the first time, its queue identifier information is written to its parent first main queue, and a notification is sent to the thread pool. The thread pool retrieves the queue identifier information of the first sub-queue from the first main queue until the first sub-queue is empty. When the first sub-queue is not empty again for the first time, its queue identifier information is written back to its parent first main queue, and so on. Figure 4 As shown in the dashed box, in this embodiment, the transmission mechanism of business data between the first sub-queue and the second sub-queue is the same during the data queuing stage and the data processing stage.

[0111] This application provides a data transmission method that, when data transmission is required, determines the queue identifier information of the first sub-queue corresponding to the business data based on the attribute information of the business data. This determines which first sub-queue the business data should be cached in. This achieves multi-dimensional resource isolation and avoids data corruption. If the first sub-queue corresponding to the business data is empty, the queue identifier information of that first sub-queue is written into the first main queue, and the business data is written into that first sub-queue. In other words, the first sub-queue records the actual business data, while the first main queue records the first sub-queue containing the business data. The data writing end sends a read request to the data reading end. The data reading end retrieves the queue identifier information from the first main queue, then retrieves each piece of business data from the first sub-queue corresponding to that queue identifier information, and finally sends the business data to the server for data processing. By using a sub-queue + main queue approach, the resource contention and scheduling fairness issues are resolved. The sub-queues cache the actual data, while the main queue only retains the sub-queue identifiers, reducing scheduling overhead and ultimately achieving high efficiency and reliability of data transmission in high-concurrency scenarios.

[0112] Figure 5 This application provides a schematic diagram of the structure of a data processing device based on shared memory, as shown in the embodiments. Figure 5 As shown, the data processing device 50 based on shared memory provided in this embodiment includes:

[0113] The first acquisition unit 501 is used to acquire queue identification information to be read from the first main queue; wherein, the queue identification information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue;

[0114] The second acquisition unit 502 is used to acquire at least one business data from the first sub-queue corresponding to the queue identifier information to be read;

[0115] The data processing unit 503 is used to process the acquired business data.

[0116] In one possible implementation, a thread pool is pre-configured, comprising at least two worker threads; the first acquisition unit 501 includes:

[0117] The thread determination module is used to determine the target thread from among the worker threads in the thread pool based on the current state of each worker thread; where the current state is either idle or running.

[0118] The identifier acquisition module is used to obtain the queue identifier information to be read from the first main queue through the target thread.

[0119] In one possible implementation, a client pool is pre-set, comprising at least two communication clients, each corresponding to a confidence level, the confidence level representing the communication capability of the communication client; the second acquisition unit 502 includes:

[0120] The client determination module is used to determine the target client from among the communication clients based on the confidence level of each communication client in the client pool;

[0121] The quantity determination module is used to determine the target quantity based on the confidence level of the target client; whereby the target quantity represents the amount of business data to be acquired.

[0122] The data acquisition module is used to obtain a target amount of business data from the first sub-queue corresponding to the queue identifier information to be read through the target thread, and send the target amount of business data to the target client.

[0123] One possible implementation also includes:

[0124] The capability acquisition unit is used to acquire the current capability information of each communication client in the client pool; wherein, the current capability information represents the capacity of service data occupied by the communication client;

[0125] The confidence update unit is used to update the confidence of the communication client based on the current capability information of the communication client.

[0126] In one possible implementation, the data processing unit 503 includes:

[0127] The sub-queue determination module is used to determine the queue identifier information of the second sub-queue corresponding to the acquired business data based on the communication client corresponding to the acquired business data.

[0128] The data writing module is used to write the acquired business data into the second sub-queue corresponding to the acquired business data. If the second sub-queue corresponding to the acquired business data is not empty for the first time, the queue identifier information of the second sub-queue corresponding to the acquired business data is written into the second main queue.

[0129] The target determination module is used to obtain target identification information from the second main queue using a preset data processing thread; wherein, the target identification information represents the queue identification information existing in the second main queue;

[0130] The data processing module is used to retrieve business data from the second sub-queue corresponding to the target identification information and process it.

[0131] In one possible implementation, the sub-queue determination module is specifically used for:

[0132] Based on the preset first association relationship, the queue identifier information of the second sub-queue corresponding to the client identifier of the communication client corresponding to the acquired business data is determined; wherein, the preset first association relationship represents the association relationship between the client identifier of the communication client and the queue identifier information of the second sub-queue.

[0133] One possible implementation also includes:

[0134] The identifier determination unit is used to obtain the service data to be transmitted based on the preset data interface, and determine the queue identifier information of the first sub-queue corresponding to the service data to be transmitted according to the attribute information of the service data to be transmitted.

[0135] The identifier writing unit is used to write the service data to be transmitted into the first sub-queue corresponding to the service data to be transmitted. If the first sub-queue corresponding to the service data to be transmitted is not empty for the first time, the queue identifier information of the first sub-queue corresponding to the service data to be transmitted is written into the first main queue.

[0136] In one possible implementation, the identifier determining unit is specifically used for:

[0137] Based on the attribute information of the business data to be transmitted and a preset routing strategy, the context information of the business data to be transmitted is determined; wherein, the context information includes at least one of the following: the center identifier information of the server to which the business data is to be transmitted, the node identifier information of the node to which it is to be transmitted, and the client identifier information of the process client from which the business data is to be transmitted.

[0138] Based on the context information of the service data to be transmitted, and based on the preset second association relationship, the queue identifier information of the first sub-queue corresponding to the service data to be transmitted is determined; wherein, the preset second association relationship represents the association relationship between the context information and the queue identifier information.

[0139] One possible implementation also includes:

[0140] The notification unit is used to issue an enqueue notification based on the queue identifier information of the first subqueue corresponding to the service data to be transmitted if the first subqueue is not empty for the first time; wherein, the enqueue notification indicates that a new first subqueue is written into the first main queue.

[0141] In one possible implementation, the identifier writing unit is specifically used for:

[0142] The business data to be transmitted is packaged to obtain data packets;

[0143] Write the data packet into the first sub-queue corresponding to the business data to be transmitted.

[0144] This embodiment provides a data processing device based on shared memory, which can execute the methods provided in the above-described method embodiments. Its implementation principle and technical effects are similar, and will not be described in detail here.

[0145] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0146] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0147] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0148] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0149] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0150] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0152] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0153] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0154] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0155] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0156] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0158] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0160] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method based on shared memory, characterized in that, include: Obtain the queue identifier information to be read from the first main queue; wherein, the queue identifier information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue; Obtain at least one piece of business data from the first sub-queue corresponding to the queue identifier information to be read; The acquired business data is processed.

2. The method according to claim 1, characterized in that, A thread pool is pre-configured, comprising at least two worker threads; the queue identifier information to be read is obtained from the first main queue, including: Based on the current state of each worker thread in the thread pool, the target thread is determined from each worker thread; wherein, the current state is either an idle state or a running state; The target thread retrieves the queue identifier information to be read from the first main queue.

3. The method according to claim 2, characterized in that, A client pool is pre-configured, which includes at least two communication clients. Each communication client corresponds to a confidence level, and the confidence level characterizes the communication capability of the communication client. Obtain at least one piece of business data from the first sub-queue corresponding to the queue identifier information to be read, including: Based on the confidence level of each communication client in the client pool, the target client is determined from among the communication clients; The target quantity is determined based on the confidence level of the target client; wherein the target quantity represents the amount of business data to be acquired. The target thread retrieves the target amount of business data from the first sub-queue corresponding to the queue identifier information to be read, and sends the target amount of business data to the target client.

4. The method according to claim 3, characterized in that, Also includes: Obtain the current capability information of each communication client in the client pool; wherein, the current capability information represents the capacity of service data occupied by the communication client; The confidence level of the communication client is updated based on its current capability information.

5. The method according to claim 3, characterized in that, The acquired business data is processed, including: Based on the communication client corresponding to the acquired business data, determine the queue identifier information of the second sub-queue corresponding to the acquired business data; The acquired business data is written into the second sub-queue corresponding to the acquired business data. If the second sub-queue corresponding to the acquired business data is not empty for the first time, the queue identifier information of the second sub-queue corresponding to the acquired business data is written into the second main queue. A preset data processing thread is used to obtain target identification information from the second main queue; wherein, the target identification information represents queue identification information existing in the second main queue; Business data is retrieved from the second sub-queue corresponding to the target identification information and processed.

6. The method according to claim 5, characterized in that, Based on the communication client corresponding to the acquired service data, determine the queue identifier information of the second sub-queue corresponding to the acquired service data, including: Based on a preset first association relationship, the queue identifier information of the second sub-queue corresponding to the client identifier of the communication client corresponding to the acquired service data is determined; wherein, the preset first association relationship represents the association relationship between the client identifier of the communication client and the queue identifier information of the second sub-queue.

7. The method according to claim 1, characterized in that, Also includes: The service data to be transmitted is obtained based on the preset data interface, and the queue identifier information of the first sub-queue corresponding to the service data to be transmitted is determined according to the attribute information of the service data to be transmitted. The service data to be transmitted is written into the first sub-queue corresponding to the service data to be transmitted. If the first sub-queue corresponding to the service data to be transmitted is not empty for the first time, the queue identifier information of the first sub-queue corresponding to the service data to be transmitted is written into the first main queue.

8. The method according to claim 7, characterized in that, Based on the attribute information of the service data to be transmitted, determine the queue identifier information of the first sub-queue corresponding to the service data to be transmitted, including: Based on the attribute information of the service data to be transmitted, and based on a preset routing strategy, the context information of the service data to be transmitted is determined; wherein, the context information includes at least one of the following: the center identifier information of the server to which the service data is to be received, the node identifier information of the node to which it is to be received, and the client identifier information of the process client from which the service data originates; Based on the context information of the service data to be transmitted, and based on a preset second association relationship, the queue identifier information of the first sub-queue corresponding to the service data to be transmitted is determined; wherein, the preset second association relationship represents the association relationship between the context information and the queue identifier information.

9. The method according to claim 7, characterized in that, Also includes: If the first subqueue corresponding to the service data to be transmitted is not empty for the first time, then an enqueue notification is issued according to the queue identifier information of the first subqueue corresponding to the service data to be transmitted; wherein, the enqueue notification indicates that a new first subqueue is written into the first main queue.

10. The method according to claim 7, characterized in that, Writing the service data to be transmitted into the first sub-queue corresponding to the service data to be transmitted includes: The service data to be transmitted is packaged to obtain a data packet; Write the data packet into the first sub-queue corresponding to the service data to be transmitted.

11. A data processing device based on shared memory, characterized in that, include: The first acquisition unit is used to acquire queue identification information to be read from the first main queue; wherein, the queue identification information is used to identify the first sub-queue, the first sub-queue is used to cache business data, and the first sub-queue is a shared memory queue; The second acquisition unit is used to acquire at least one business data from the first sub-queue corresponding to the queue identifier information to be read; The data processing unit is used to process the acquired business data.

12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images