High-scalability data processing system and method based on multi-strategy collaboration, readable storage medium and electronic equipment

Abstract

The invention discloses a high-scalability data processing system and method based on multi-strategy collaboration, a readable storage medium and electronic equipment. The system comprises a strategy abstraction and implementation module; a concurrency control and synchronization module; the strategy collaboration and factory scheduling module comprises a data flow reactor which is used for coordinating event driving and state management of the whole data processing flow and receiving original data; the transmission strategy factory module is used for receiving the original data sent by the data flow reactor and selecting a data format and a transmission protocol from the strategy abstraction and implementation module according to the configuration parameters so as to dynamically create a specific implementation instance based on the original data; and the data model packer and the concurrency control and synchronization module carry out parallel batch data processing on the specific implementation examples generated by the transmission strategy factory module, and output a final processing result to a target system. The system has the advantages that the system can overcome the defects of high coupling, poor expansibility, insufficient strategy collaboration, low concurrency efficiency and the like in the prior art.

Description

Technical Field

[0001] This invention relates to the field of computer distributed computing and real-time data stream processing technology, and in particular to a highly scalable data processing system, method, readable storage medium, and electronic device based on multi-strategy collaborative processing. Background Technology

[0002] With the large-scale application of IoT (Internet of Things), IIoT (Industrial Internet of Things), and ISCS (Integrated System for Monitoring and Controlling Systems), data processing systems need to handle massive amounts of heterogeneous real-time data (such as analog, digital, and cumulative data) and reliably transmit it to downstream terminals such as big data platforms, real-time monitoring systems, and historical databases. For example, in the operation of urban rail transit, key subsystems such as signaling systems, integrated monitoring systems, and passenger information systems generate massive amounts of multi-source, heterogeneous real-time and historical data. This data includes, but is not limited to, train location, speed, equipment status, passenger flow density, power consumption, and various alarm information. As the construction of smart urban rail transit deepens, how to efficiently and reliably process and integrate this data to achieve advanced applications such as predictive maintenance of equipment status, intelligent passenger flow management, and refined energy management has become a core challenge facing the industry.

[0003] Currently, existing data processing technologies suffer from the following core pain points, making it difficult to meet the demands of highly dynamic and highly reliable business operations: Excessive coupling: Data processing logic, transmission protocol (e.g., Kafka client), and data format (e.g., JSON parsing) are hard-coded together. Switching transmission protocols (e.g., from Kafka to MQTT) or adding new data formats (e.g., from JSON to Protobuf) requires modifying the core business code, resulting in extremely high iteration costs.

[0004] Insufficient scalability: When adding new data sources (such as industrial sensors) or output targets (such as time series databases), it is necessary to carry out invasive modifications to the existing architecture, or even start from scratch, which makes it impossible to respond quickly to business changes and violates the "open / closed principle".

[0005] Lack of strategic coordination: There is a lack of a unified mechanism to coordinate the strategies of each stage of data processing (such as packaging format and transmission method). During runtime, the strategy cannot be adjusted according to the amount of data and network status, resulting in rigid system behavior and a tendency for performance bottlenecks or data loss.

[0006] Low concurrency efficiency: Multi-threaded task scheduling lacks a unified synchronization mechanism (such as task status tracking and cross-thread notification), which can easily lead to resource contention or task blocking, making it impossible to fully utilize multi-core CPU resources, and limiting the throughput of processing large amounts of data.

[0007] High complexity of operation and maintenance: There is no unified control plane to manage the entire process of data "collection-packaging-transmission", each module has an independent status, and fault diagnosis requires locating each one individually, resulting in low operation and maintenance efficiency.

[0008] Therefore, existing data processing technologies need to be improved.

[0009] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art. Summary of the Invention

[0010] Based on the aforementioned technical problems, the purpose of this invention is to solve the problems of high coupling, poor scalability, insufficient strategy coordination and low concurrency efficiency in the prior art. This invention provides a highly scalable data processing system, method, readable storage medium and electronic device based on multi-strategy coordination. The system is a decoupled, pluggable, strategy-coordinated, highly concurrent, safe and reliable highly scalable data processing architecture.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solution: A highly scalable data processing system based on multi-strategy collaborative processing includes: The strategy abstraction and implementation module contains multiple strategy abstraction interfaces. Each strategy abstraction interface is used to provide at least multiple data format processing strategies and transmission protocol strategies. The data format processing strategies and transmission protocol strategies of the strategy abstraction and implementation module are extensible. The concurrency control and synchronization module is used to enable parallel processing of multiple tasks; A strategy coordination and factory scheduling module, which is connected to both the external data source and the strategy abstraction and implementation module, includes: A data flow reactor is used to coordinate event-driven and state management of the entire data processing flow, and the data flow reactor is also used to receive raw data from external data sources; A transmission strategy factory module is connected to the data stream reactor. The transmission strategy factory module receives raw data sent by the data stream reactor and selects data format and transmission protocol from the strategy abstraction and implementation module according to configuration parameters to dynamically create and manage specific implementation instances based on the raw data. A data model packer is connected to the transmission strategy factory module. The data model packer and the concurrency control and synchronization module perform parallel batch data processing on the specific implementation instances generated by the transmission strategy factory module through a concurrency control mechanism, and output the final processing results to the target system. The parallel batch data processing includes sharding, mapping and aggregation processing.

[0012] Optionally, the strategy abstraction and implementation module includes: The transport strategy abstraction interface is used to define a unified operation interface for transport protocols. The specific strategies of the transport strategy abstraction interface are used to adapt to at least two transport protocols among Kafka, MceBus, and MQTT. The transport protocol strategies can be extended through the transport strategy abstraction interface. The data packaging format strategy abstract interface is used to define the serialization or deserialization specifications of data formats. The specific strategies of the data packaging format strategy abstract interface are used to adapt to various data types.

[0013] Optionally, when a new transmission protocol strategy needs to be added, a corresponding transmission strategy producer class is added through the transmission strategy abstract interface; when a new data format processing strategy needs to be added, a corresponding data packaging format strategy packaging class is added through the data packaging format strategy abstract interface.

[0014] Optionally, the strategy abstraction and implementation module further includes: The Kafka producer class inherits the transport strategy abstract interface. The Kafka producer class is specifically instantiated through the Kafka client pattern to implement the message sending function based on Kafka. The MceBus producer class inherits the transport strategy abstract interface. The MceBus producer class internally implements the message bus client through MceBus to realize the message sending function based on MceBus. The stream feature encapsulation base class is used to encapsulate the basic features and logic of each message stream for reuse by specific producers.

[0015] Optionally, the transmission strategy factory module inherits the transmission strategy abstract interface. When the transmission strategy factory module is initialized, it creates specific Kafka producer class or MceBus producer class instance objects as member variables based on the transmission mode options in the configuration parameters of the configuration file.

[0016] Optionally, the strategy abstraction and implementation module further includes: A JSON packaging class, which inherits the data packaging format strategy abstract interface, internally formats various types of data using JSON, and can be used to implement JSON text format data serialization or deserialization; The Protobuf packing class inherits the data packing format strategy abstract interface. The Protobuf packing class internally formats various types of data using Protobuf. The Protobuf packing class can be used to implement data serialization or deserialization in Protobuf binary format.

[0017] Optionally, the concurrency control and synchronization module includes a thread pool, which is used to manage thread resources for the parallel execution of multiple tasks. It supports dynamic adjustment of the number of threads, and the thread pool can dynamically adjust the number of threads according to the task queue length or system resource utilization.

[0018] Optionally, the data model packager includes: The Map phase processing component is used to split the processing task of the specific implementation instance into data, and submit each subtask corresponding to each split data piece to the thread pool for parallel processing. The Map phase processing component is also used to receive each data after processing by the thread pool. The Reduce phase processing component receives the data sent by the Map phase processing component after parallel processing by a thread pool, and performs mapping and aggregation processing on the data to obtain the final processing result.

[0019] Optionally, the concurrency control and synchronization module further includes: A multi-threaded notification queue uses a blocking or waking mechanism to distribute tasks and synchronize states between threads.

[0020] Optionally, the thread pool includes two methods: creation and destruction. When created, a predetermined number of worker threads are initialized through its constructor. These worker threads are used to process and package data. When destroyed, the worker threads are recycled through its destructor. The Map phase processing component writes split and sharded data to the multi-threaded notification queue. The worker thread objects in the thread pool extract the sharded data from the multi-threaded notification queue, execute the processing function to process it, and package it according to the predetermined data format.

[0021] Optionally, when the Map phase processing component completes the data segmentation of the current batch of data and writes each data segment into the multi-threaded notification queue, the task status of the current batch of data is in a blocked critical resource state. When all data segments are processed, the multi-threaded notification queue wakes up the next batch of data that is currently in a blocked critical resource state.

[0022] Optionally, the concurrency control and synchronization module further includes: An atomic counter is used for thread-safe task counting and status identification.

[0023] Optionally, when the Map phase processing component writes a shard of data to the multi-threaded notification queue, the atomic counter is incremented by 1; when the worker thread finishes processing a shard of data, the atomic counter is decremented by 1. After delivering all sharded data to the multi-threaded notification queue, the Map phase processing component waits for the atomic counter to return to zero. When the last worker thread completes its task and the atomic counter returns to zero, it wakes up the waiting Reduce phase processing component to begin the aggregation work.

[0024] Optional, also includes: The resource and configuration management module uses smart pointers to automatically allocate and release memory resources. It also provides a unified configuration access interface based on configuration files or service registry centers and supports hot reloading of configuration items.

[0025] Optionally, when the data flow reactor starts, it calls the device data service module. The registration manager of the device data service module adds the event source to the processor list to complete the registration. When the device data service module collects raw data, the data flow reactor receives the raw data and coordinates the workflow to process the raw data.

[0026] Optionally, one of the aforementioned working methods for a multi-strategy collaborative highly scalable data processing system includes: The data flow reactor receives raw data from external data sources and sends it to the transmission strategy factory module; The transmission strategy factory module selects the data format and transmission protocol from the strategy abstraction and implementation module according to the configuration parameters, so as to dynamically create and manage specific implementation instances based on the original data; The data model packer performs parallel batch data processing on the specific implementation instances generated by the transmission strategy factory module through a concurrency control mechanism, and outputs the final processing results to the target system. The parallel batch data processing includes sharding, mapping and aggregation processing.

[0027] Optionally, the strategy abstraction and implementation module includes a transmission strategy abstraction interface and a data packaging format strategy abstraction interface. When a new transmission protocol strategy needs to be added, a corresponding transmission strategy producer class is added through the transmission strategy abstraction interface. When a new data format processing strategy needs to be added, a corresponding data packaging format strategy packaging class is added through the data packaging format strategy abstraction interface.

[0028] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned working method of a multi-strategy cooperative highly scalable data processing system.

[0029] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned working method of a multi-strategy collaborative highly scalable data processing system.

[0030] Compared with the prior art, the present invention has the following advantages: The present invention discloses a highly scalable data processing system, method, readable storage medium, and electronic device based on multi-strategy collaboration. The system combines a strategy abstraction and implementation module, a strategy collaboration and factory scheduling module, and a concurrency control and synchronization module. It can achieve flexible switching of transmission protocols and data formats without modifying the core code, achieve unified collaboration of strategies in all aspects of data processing to adapt to different business scenarios, support parallel processing under large data volumes to improve throughput, avoid memory safety issues through automated resource management, and reduce operation and maintenance complexity by relying on unified configuration and control.

[0031] Furthermore, in the field of distributed computing and real-time data stream processing technology, this invention solves the problems of high coupling, poor scalability, insufficient strategy coordination, and low concurrency efficiency in existing technologies. In other words, this invention solves the problems of adaptability, real-time performance, and stability in multi-source heterogeneous data processing, and provides strong technical support for the effective processing of multi-source heterogeneous data. Attached Figure Description

[0032] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the drawings in the following description are one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a schematic diagram illustrating the architecture inheritance and implementation of a highly scalable data processing system based on multi-strategy collaboration according to the present invention; Figure 2 This is a flowchart illustrating the collaborative process between the transmission strategy factory module and the strategy abstraction and implementation module of the present invention. Figure 3 This is a schematic diagram illustrating a data format inheritance relationship according to the present invention; Figure 4 This is a timing diagram of a data model packer according to the present invention; Figure 5 This is a schematic diagram of the various stages of a data stream reactor according to the present invention. Detailed Implementation

[0033] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more detailed account of a multi-strategy collaborative, highly scalable data processing system, method, readable storage medium, and electronic device proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clarify the illustration of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0034] As mentioned above, existing data processing technologies have significant limitations. For example, in the rail transit field, the data formats and communication protocols of various subsystems differ (for instance, some systems use Kafka message queues, while traditional equipment may use industrial protocols such as Modbus / TCP and CAN), leading to difficulties in data access and integration, and the formation of "data silos." Secondly, data processing logic is usually hard-coded in applications, lacking flexibility. When a new data format needs to be added (for example, introducing Protobuf to replace JSON for new line construction) or a transmission protocol needs to be changed, the core code must be modified and the system redeployed. This not only results in long development cycles and high costs, but also makes it difficult to quickly adapt to customized projects for different lines or different owners.

[0035] Furthermore, rail transit data exhibits a significant burst-like characteristic, with massive amounts of sensor data being reported simultaneously during morning and evening rush hours. Traditional serial processing architectures struggle to meet real-time requirements, easily leading to data backlog. Existing technologies lack a unified data governance framework capable of dynamically configuring processing strategies and supporting parallel batch processing, resulting in low system throughput and poor scalability, failing to meet the urgent needs of smart urban rail transit for real-time data analysis and decision-making.

[0036] Based on the above, this invention provides a highly scalable data processing system based on multi-strategy collaboration. This system includes a strategy abstraction and implementation module, a strategy collaboration and factory scheduling module, and a concurrency control and synchronization module (see [link to documentation]). Figure 1 and Figure 2The strategy abstraction and implementation module includes multiple strategy abstraction interfaces, each providing at least several data format processing strategies and transmission protocol strategies. These strategies are extensible. The concurrency control and synchronization module enables parallel processing of multiple tasks. The strategy coordination and factory scheduling module connects to both the external data source and the strategy abstraction and implementation module. This module includes a DataReactor, a ProducerFactory, and a MapReduce. The DataReactor is used to coordinate the event-driven and state management of the entire data processing flow. It also receives raw data from external data sources. The ProducerFactory (or simply the strategy factory) is connected to the DataReactor. It receives the raw data from the DataReactor and selects the data format and transmission protocol from the strategy abstraction and implementation module based on configuration parameters to dynamically create and manage concrete implementation instances based on the raw data. The MapReduce data model packager is connected to the ProducerFactory. The MapReduce and the concurrency control and synchronization module perform parallel batch data processing on the concrete implementation instances generated by the ProducerFactory through a concurrency control mechanism, and output the final processing results to the target system. This parallel batch data processing includes sharding, mapping, and aggregation processing.

[0037] In practical applications, the DataReactor is used to coordinate multi-module workflows. It receives and parses data streams through an event-driven mechanism, and then distributes the parsed data to the corresponding processing units based on the specific implementation instance (i.e., the strategy instance). Specifically, for example... Figure 1 and Figure 2As shown, after receiving raw data from external data sources (such as signaling systems or passenger information systems), the DataReactor drives the ProducerFactory module to select a strategy type through the strategy abstraction and implementation module. Based on the strategy type, it dynamically creates and returns a specific implementation instance of the corresponding transmission or data format. Then, the DataReactor drives the MapReduce data model packager, combined with the concurrency control and synchronization module, to perform sharding, mapping, and aggregation processing on the specific implementation instance based on the MapReduce computing model to obtain the final processing result, which is then sent to the downstream terminal (such as a big data platform or real-time monitoring system).

[0038] As described above, the multi-strategy collaborative high-scalability data processing system of the present invention adopts a layered decoupling and modular collaborative design. It combines a strategy abstraction and implementation module, a data flow reactor (DataReactor), a transmission strategy factory module (ProducerFactory), a data model packager (MapReduce), and a concurrency control and synchronization module to achieve a highly scalable data processing architecture. Specifically, the high scalability achieved through the strategy abstraction and implementation module relies on abstract interfaces and a strategy registration mechanism. Adding a new transmission protocol / data format only requires adding the corresponding strategy implementation class (without modifying the core code, conforming to the open / closed principle). Replacing the message middleware only requires adjusting the configuration, without affecting the business logic. Furthermore, the strong strategy collaboration achieved through the strategy collaboration and factory scheduling module specifically utilizes the transmission strategy factory module (ProducerFactory) and the data flow reactor (DataReactor) to achieve end-to-end collaboration of "collection-packaging-transmission," dynamically adjusting strategies according to data volume and network status to adapt to different scenarios. Furthermore, the MapReduce data model packager significantly improves the throughput of large data volumes by reusing the ThreadPool in the MapReduce computing model's parallel packaging and concurrency control and synchronization modules, effectively enhancing the efficiency of parallel computing on large datasets. Each module in this invention has a single responsibility, improves code reusability through interface interaction, and features clear modularity that facilitates development, testing, and iteration, contributing to low data coupling and high maintainability.

[0039] Therefore, the multi-strategy collaborative highly scalable data processing system of this invention abstracts data format processing and transmission protocols into pluggable strategies through a strategy abstraction and implementation module. New strategies only need to implement a unified interface to be connected, enabling the system to quickly adapt to diverse data sources and protocol changes, exhibiting high scalability and flexibility. Simultaneously, this invention utilizes the MapReduce data model packager and the concurrency control module to work collaboratively, achieving automatic sharding, parallel mapping, and aggregation processing of massive amounts of rail transit data, significantly improving data processing throughput. Its efficient parallel processing capability meets real-time requirements. This invention also employs the DataReactor data flow reactor as the core coordinator, which uses an event-driven mechanism to dynamically schedule the entire data processing process, achieving coupled collaboration between modules and intelligent process collaboration, ensuring stable and efficient system operation even when facing data fluctuations. In short, this invention solves the challenges of adaptability, real-time performance, and stability in multi-source heterogeneous data processing, providing strong technical support for the effective processing of multi-source heterogeneous data.

[0040] In this invention, the strategy abstraction and implementation module is the core decoupling layer. This module hides the specific implementation details by abstracting the strategy interface and at least two specific strategy implementations to support flexible switching between multiple strategies. For example... Figure 1 and Figure 3 As shown, the abstract strategy interface includes a transport strategy abstract interface (AbstractProducer) and a data packaging format strategy abstract interface (PackagingStrategy). The transport strategy abstract interface (AbstractProducer) defines a unified operation interface for transport protocols, and its specific strategies are used to adapt to at least two transport protocols among Kafka, MceBus, and MQTT. The transport protocol strategies can be extended through the transport strategy abstract interface to adapt to other protocols. The data packaging format strategy abstract interface (PackagingStrategy), also known as the data format strategy abstract interface, defines the serialization / deserialization specifications for data formats (various data types). The specific strategies of the data packaging format strategy abstract interface (PackagingStrategy) are used to adapt to various data types. The various specific strategy implementations of the strategy abstraction and implementation module are developed based on the above interfaces, such as KafkaProducer (Kafka), McebusProducer (MceBus), JsonPackaging (JSON), and ProtobufPackaging (Protobuf). They are dynamically loaded into the system through the strategy registration mechanism to adapt to different transmission protocols (such as TCP and HTTP) and different data formats (such as JSON and Protobuf), thereby achieving decoupling of multiple transmission protocols and multiple data formats.

[0041] In practical applications, the AbstractProducer abstract base class interface defines a unified behavioral specification for all producers. It can include pure virtual function interface methods such as create (create connection), publish (send data / message), and destroy (destroy connection / resource). It is the contract for all specific producers, and its specific strategy implementation is adapted to transmission protocols such as Kafka, MceBus, and MQTT.

[0042] like Figure 1 As shown, the strategy abstraction and implementation module also includes a transport strategy producer class and a stream feature encapsulation base class. The transport strategy producer class, i.e., the concrete producer implementation class, is the component that actually handles message production, and it includes a Kafka producer class and a MceBus producer class. The Kafka producer class, KafkaProducer, inherits from the transport strategy abstraction interface AbstractProducer. Internally, KafkaProducer is specifically instantiated using the Kafka client pattern to implement Kafka-based message sending functionality. KafkaProducer is specifically responsible for producing messages through the Kafka cluster and handling producer logic for interacting with Kafka (e.g., connecting to Kafka, sending messages to a specified topic, etc.). The MceBus producer class, MceBusProducer, inherits from the transport strategy abstraction interface AbstractProducer. Internally, MceBusProducer is specifically instantiated using the MceBus message bus client to implement MceBus-based message sending functionality. MceBusProducer is specifically responsible for producing messages through the MceBus protocol and handling message sending logic based on the MceBus protocol / channel.

[0043] The stream feature encapsulation base class is a stream-type-specific base class used to encapsulate the basic features and logic of their respective message streams for reuse by specific producers. The stream feature encapsulation base class includes KafkaStream and MceBusStream, which respectively encapsulate the basic features of Kafka streams and MceBus streams (such as connection parameters and data formats). They also encapsulate the basic logic of their respective streams (such as connection establishment and format conversion). Specific producers can directly reuse these functionalities through inheritance, avoiding code duplication.

[0044] In practical applications, the data packaging format strategy abstract interface PackagingStrategy defines a package method to adapt to data formats such as JSON and Protobuf. For example, it defines pure virtual function interface methods packageAnalog, packageDigital, and packageAccumulator to adapt to various data types such as analog, digital, and cumulative quantities, respectively.

[0045] like Figure 3 As shown, the data packaging format strategy packaging class of the strategy abstraction and implementation module includes a JSON packaging class and a Protobuf packaging class. The JSON packaging class, JsonPackaging, inherits from the data packaging format strategy abstraction interface PackagingStrategy. Internally, JsonPackaging formats corresponding analog, numerical, or cumulative data types using JSON. The JSON packaging class can be used to implement JSON text format data serialization or deserialization. The Protobuf packaging class, ProtobufPackaging, inherits from the data packaging format strategy abstraction interface PackagingStrategy. Internally, ProtobufPackaging formats corresponding analog, numerical, or cumulative data types using Protobuf. The Protobuf packaging class can be used to implement Protobuf binary format data serialization or deserialization.

[0046] In practical applications, the architecture design of the strategy abstraction and implementation module reserves clear extension points, and both its transmission strategy and data packaging format strategy are extensible. When a new transmission protocol strategy is needed, a corresponding transmission strategy producer class is added through the transmission strategy abstraction interface AbstractProducer; when a new data format processing strategy is needed, a corresponding data packaging format strategy packaging class is added through the data packaging format strategy abstraction interface PackagingStrategy. The specific strategy protocols of the newly added transmission strategy producer class and data packaging format strategy packaging class are derived from the above. For example, when a new messaging system MQTT needs to be integrated, only an MqttStream class needs to be added to encapsulate MQTT-related stream characteristics; an MqttProducer class needs to be added, inheriting the transmission strategy abstraction interface AbstractProducer and MqttStream, to implement the specific production logic; a line of judgment logic needs to be added to the transmission strategy factory module ProducerFactory, and an MqttProducer instance can be created through the transmission strategy factory module ProducerFactory. The entire process does not require modification of the existing class code, conforms to the open / closed principle, and allows the system to iterate smoothly with business needs.

[0047] As described above, the core function of the strategy abstraction and implementation module of this invention is to achieve data standardization and reduce the complexity of downstream packaging and transmission. It can be regarded as a unified data model (semantic consistency layer). This module can define standardized data objects (such as AnalogData, DigitalData, AccumulatorData) circulating within the system, including core fields such as data identifier, timestamp, value, and quality label. All modules operate on the unified data model, achieving seamless integration and semantic consistency in each stage of "collection-packaging-transmission" and avoiding data format chaos.

[0048] The strategy abstraction and implementation module can define standardized data objects containing data metadata (such as type identifiers and timestamps), content bodies, and standard operation interfaces (such as retrieval, modification, and verification) to achieve consistency in data semantics and seamless interaction between modules.

[0049] The strategy abstraction and implementation module may include data identifier, timestamp, numerical value, and quality label fields to ensure semantic consistency in the "collection-packaging-transmission" process.

[0050] In this invention, the strategy coordination and factory scheduling module is the core coordination layer. As the core of the architecture strategy coordination, this module is responsible for dynamic strategy selection, module coordination and object creation.

[0051] The ProducerFactory class of the strategy coordination and factory scheduling module is responsible for creating concrete implementation instances, i.e., concrete producer instances. The ProducerFactory inherits from the abstract interface AbstractProducer. During initialization, the ProducerFactory can create concrete Kafka producer class or MceBus producer class instances as member variables based on the transmission mode options in the preset configuration file.

[0052] In practical applications, the ProducerFactory module selects a policy object from the policy abstraction and implementation module based on configuration parameters (e.g., creating a KafkaProducer instance when configuring the Kafka mode). (See also: [link to relevant documentation]). Figure 2 This allows for the dynamic instantiation and transmission of concrete implementation instances. Furthermore, based on this strategy object, the concrete implementation instances generated by the ProducerFactory module hide the creation details of the specific producers. Clients do not need to know the construction process of KafkaProducer or McebusProducer; they only need to call the factory method to obtain the instance, facilitating subsequent expansion. In practical applications, the ProducerFactory module can also select data format strategy objects (e.g., creating a ProtobufPackaging instance when configuring the Protobuf format), isolating object creation from business logic to support seamless replacement of message middleware. Through this factory pattern, clients do not need to directly depend on the construction logic of specific producers; they only need to obtain the concrete implementation instance through the factory, thereby reducing the coupling between system modules.

[0053] like Figure 2 The diagram illustrates the collaborative process between the ProducerFactory module and the policy abstraction and implementation module. It describes the logic of configuring parameters → policy factory → instantiating specific policies → execution processing, demonstrating the dynamic collaborative capability of policies. As shown in the diagram, the ProducerFactory module implements the creation logic for KafkaProducer and McebusProducer, determining which producer to create based on conditional judgments (such as configuration parameters). AbstractProducerPtr ProducerFactory::createProducer(const std::string& type) { if (type == "kafka") return new KafkaProducer(); if (type == "mcebus") return new McebusProducer(); return nullptr; } The data packaging format strategy abstract interfaces PackagingStrategy, JsonPackaging, and ProtobufPackaging are linked through the Strategy Pattern. The core purpose is to decouple the data packaging logic from the business usage logic, so as to support flexible switching of different data serialization formats (such as JSON and Protobuf).

[0054] like Figure 3 As shown, the abstract base class (interface) of the data packaging strategy abstraction interface, PackagingStrategy, defines a unified interface that all data packaging strategies must implement, serving as the abstract strategy role in the strategy pattern. It doesn't concern itself with the specific packaging details, only specifying which types of data need to be packaged, and typically includes the following abstract methods: class PackagingStrategy{ public: / / Package analog data (such as continuous sensor values) virtual std::string packageAnalog(const std::vector <analogdata>&data)=0; / / Pack digital data (such as switch status) virtual std::string packageDigital(const std::vector <digitaldata>&data) = 0; / / Pack accumulator data (such as cumulative traffic) virtual std::string packageAccumulator(const std::vector <accumulatordata>&data) = 0; } The concrete implementation classes JsonPackaging and ProtobufPackaging inherit from the abstract interface PackagingStrategy, representing the concrete strategy role in the strategy pattern. They implement all methods in the abstract class, but employ different serialization logic (JSON text format and Protobuf binary format).

[0055] Specifically, JsonPackaging serializes business data into JSON text format (key-value pair structure), suitable for scenarios requiring high readability and lightweight cross-platform interaction. ProtobufPackaging serializes business data into Protobuf binary format, suitable for high-performance, large-volume, and cross-language transmission scenarios.

[0056] Upper-level modules (such as KafkaProducer) only need to select the data packaging format strategy abstract interface PackagingStrategy through configuration parameters and call the packaging method, without needing to know whether it is JSON or Protobuf. If a format change is required, simply replace the strategy instance; no modification to the calling logic is needed.

[0057] The architecture design allows the system to flexibly choose data formats based on business needs (such as transmission efficiency, readability, and cross-language requirements), while reducing code coupling and facilitating maintenance and expansion. If XML format packaging is added, simply add an `XmlPackaging` class that inherits from `PackagingStrategy` and implements the interface; the existing code remains unchanged, adhering to the open / closed principle.

[0058] On the other hand, in this invention, the concurrency control and synchronization module is a performance optimization layer that ensures the stability and efficiency of the architecture in high-concurrency scenarios. The concurrency control and synchronization module includes a thread pool (ThreadPool), which manages thread resources for parallel execution of multiple tasks and supports dynamic adjustment of the number of threads. In practical applications, the thread pool can dynamically adjust the number of threads based on the task queue length or system resource utilization.

[0059] In this invention, the MapReduce data model packer addresses the performance bottleneck of large-scale data processing through a divide-and-conquer logic of "data partitioning → parallel packing → result merging." Specifically, it employs a divide-and-conquer approach to process data (see [link to relevant documentation]). Figure 4 The process involves splitting large-scale data into multiple smaller tasks (data shards) using the map function, allowing the ThreadPool to process them in parallel; and combining the intermediate processing results from the map function and the ThreadPool using the reduce function to obtain the final result.

[0060] Specifically, the data model packager MapReduce (PackageMapReduce object) bases the computation process on two main functions: the map function and the reduce function, which correspond to the Map phase processing component and the Reduce phase processing component, respectively. These are applied in the Map phase and the Reduce phase, respectively. The Map phase processing component, i.e., the map function, is used to split the processing task of the specific implementation instance into data, and then submits the subtasks corresponding to each split data piece to a ThreadPool for parallel processing. The Map phase processing component also receives the data processed by the ThreadPool. The Reduce phase processing component, i.e., the reduce function, receives the data sent by the Map phase processing component after parallel processing by the ThreadPool, and performs mapping and aggregation processing on the data to obtain the final processing result.

[0061] Furthermore, the concurrency control and synchronization module also includes a multi-threaded notification queue (ThreadQueue), which uses a blocking or waking mechanism to distribute tasks and synchronize states between threads, ensuring the orderly processing of multiple tasks in parallel. This multi-threaded synchronization mechanism can effectively reduce response latency. The multi-threaded notification queue (ThreadQueue) can transmit task state change information in real time, avoiding thread blocking.

[0062] In practical applications, the ThreadPool can include two methods: creation (constructor) and destruction (destructor). During creation, a predetermined number of worker threads (PackageWorker) are initialized through the constructor (e.g., 32). These worker threads (PackageWorker) are used to process and package batch data. During destruction, the destructor reclaims the worker threads (PackageWorker). The `map` function, i.e., the Map phase processing component, writes split and fragmented data to the multi-threaded notification queue (ThreadQueue). The worker thread (PackageWorker) objects in the ThreadPool extract the fragmented data from the ThreadQueue, execute the `work` function to process it, and package it according to a predetermined data format (e.g., JsonPackaging or ProtobufPackaging).

[0063] As described above, the MapReduce data model packer uses a divide-and-conquer approach to split large datasets by type or time slice, and then merges the results after multi-threaded parallel packing, thus solving the single-threaded performance bottleneck. Meanwhile, this invention employs a DataReactor as the overall process controller, monitoring changes in the data source (such as changes in state values) and triggering the entire "collection-packaging-transmission" process according to configured rules, coordinating the orderly operation of the strategy module and the factory module.

[0064] Furthermore, the concurrency control and synchronization module also includes an atomic counter, AtomicCounter, which is used for thread-safe task counting and status identification. In practical applications, the atomic counter AtomicCounter is used to track the number of tasks completed in the Map phase, ensuring that the Reduce phase is triggered after all subtasks are completed.

[0065] During the operation, when the Map phase processing component pushes / writes a data shard to the multi-threaded notification queue ThreadQueue, the atomic counter AtomicCounter is incremented by 1; when the worker thread PackageWorker completes processing a data shard, the atomic counter AtomicCounter is decremented by 1. During the operation, after delivering all data shards to the multi-threaded notification queue, the Map phase processing component waits for the atomic counter to reach zero; when the last worker thread completes its task and resets the atomic counter to zero, it wakes up the waiting Reduce phase processing component to begin the aggregation operation.

[0066] In practical applications, the Map phase is the process of splitting data into blocks. During this phase, the processing component completes the splitting of all data in the current batch and writes each data shard into the multi-threaded notification queue (ThreadQueue). At this point, the task status of the current batch is in a blocked, critical resource state. The Reduce phase is the result merging process. When all data shards have been processed, i.e., when the atomic counter (AtomicCounter) reaches 0, the multi-threaded notification queue (ThreadQueue) wakes up the next batch of data / tasks currently in a blocked, critical resource state, allowing them to continue executing the next operation step.

[0067] like Figure 4 The diagram shows the timing of MapReduce, a data model packer according to the present invention. It illustrates the time sequence and module interactions between the Map phase (data partitioning → ThreadPool task delivery → parallel packing) and the Reduce phase (task counting → result merging → transmission). The MapReduce computing model follows the classic two-phase pattern of Map (mapping) → Reduce (reduction), breaking down complex data processing tasks into two parallel steps. It acts as the system's data processing factory, transforming messy raw data into standardized, directly usable structured data through the Map phase (transformation and partitioning) and the Reduce phase (aggregation and merging). In practical applications, the Map phase processing component can partition data into blocks of 5000 records per group (default configuration). The Reduce phase processing component waits for all data blocks to be processed and formatted before sending the data to the target using a specified producer strategy object. For example, data can be sent through the ProducerFactory module; if sending fails, it will retry three times and log the relevant information. Alternatively, data can be sent through the DataReactor.

[0068] The MapReduce data model packer supports efficient batch processing and is suitable for high-concurrency big data scenarios. It also decouples data processing from business logic, improving system scalability. Furthermore, the MapReduce data model packer can completely separate data processing logic (cleansing, transformation, aggregation) from upper-layer business application logic (packaging, sending).

[0069] If you need to add new data processing rules (such as adding data filtering conditions or modifying the aggregation method), you only need to modify the working method of the Map stage processing component or the Reduce stage processing component. There is no need to change the upstream data receiving (such as AbstractObSimpleServer) or the downstream packaging and sending (such as PackagingStrategy, KafkaProducer).

[0070] As described above, in this invention, the data model packager MapReduce performs sharding, mapping, and aggregation processing on the data model based on the MapReduce computing model to optimize the efficiency of parallel computing on large datasets. Simultaneously, it maintains reusable thread resources through a ThreadPool to avoid the overhead of frequently creating / destroying threads and receives asynchronous tasks submitted by the Map phase processing components. In the multi-threaded synchronization mechanism, the atomic counter AtomicCounter tracks the number of subtasks split by the Map phase processing components that have been completed by the ThreadPool, ensuring that all subtasks are completed before entering the Reduce phase. The multi-threaded notification queue ThreadQueue notifies relevant modules in real time when the task status changes (such as packaging completion or transmission failure), avoiding thread blocking.

[0071] On the other hand, in this invention, the DataReactor is used to coordinate multi-module workflows. It is a data processing component based on an event-driven model, primarily responsible for data reception, parsing, distribution, and response. It efficiently processes real-time data streams and utilizes an asynchronous communication framework. Its core is to decouple data processing logic from underlying I / O operations through an event-triggered mechanism (such as changes in measured data values), thereby improving the system's concurrent processing capabilities. In practical applications, the DataReactor can trigger a "collection-packaging-transmission" process based on data volume thresholds, time intervals, and master / slave failover events.

[0072] Furthermore, such as Figure 5 As shown, when the DataReactor starts, it calls the Device Data Service module, specifically the Device Data Service method `register`. The registration manager of the Device Data Service module adds the event source to the processor list to complete the registration. When the Device Data Service module collects raw data from the sensor, it generates a raw data event and calls the `broadcast(event)` method of the registration manager. The registration manager iterates through the list and calls `onNotify(event)`. The DataReactor receives the raw data event and coordinates the workflow to process the raw data.

[0073] In practical applications, the DataReactor acts as an intermediary layer connecting data sources (such as sensors or external systems) and message producers (such as KafkaProducer and McebusProducer). Essentially, it's an event-driven coordinator. It receives raw data events, coordinates the PackagingStrategy and ProducerFactory to complete data serialization and transmission, and manages the start, stop, and exception handling of the entire process. The DataReactor only coordinates the process; it doesn't directly handle data packaging or transmission, but delegates this to the strategy and producer classes, adhering to the Single Responsibility Principle and facilitating maintenance.

[0074] In some embodiments, AbstractObSimpleServer acts as an abstract base class for observer services, defining a pure virtual method called notify. This method serves as the communication interface between the data source and the DataReactor, pushing external data to the DataReactor via the notify method. The DataReactor typically implements the observer interface of AbstractObSimpleServer (or registers a listener pattern) to become the data receiver, thus establishing a closed loop of "data generation → notification → processing".

[0075] The DataReactor needs to initialize downstream components and build a complete data processing chain upon startup: [Dependency on producer factories] The ProducerFactory module creates concrete producer instances (such as KafkaProducer) and holds references (or pointers) to them for the final data delivery.

[0076] [Binding and Packaging Strategy] Choose JsonPackaging or ProtobufPackaging as the current packaging strategy based on the configuration (e.g., configuration file), and ensure that the correct serialization logic is invoked during data processing by associating strategy instances.

[0077] [Related data source] Establish a registration mechanism with data input sources (such as observer services like AbstractObSimpleServer) and receive raw data through event callbacks (such as the notify method).

[0078] DataReactor operates on an event-driven model. Its core mechanism involves receiving raw data and triggering processing flows via the `notify` method of the observer service. [Data Reception] When an external system (such as a sensor or upstream service) generates data, it calls the notify method of the observer service. The Data Reactor acts as a listener to receive data sources of type AbstractSimpleDelegate (which encapsulates raw data such as analog, digital, and cumulative quantities).

[0079] [Data Transformation] Obtain the instance of the PackagingStrategy abstract interface (e.g., ProtobufPackaging) that is bound to the data packaging format strategy, and serialize the raw data into the target format (JSON text or Protobuf binary) using methods such as packageAnalog / packageDigital / packageAccumulator.

[0080] [Data transmission] The serialized data is sent to a message queue (e.g., a Kafka Producer) or a target system (e.g., a message bus implemented by Mcebus) through a producer (e.g., a Kafka cluster) created by the ProducerFactory module of the transmission strategy factory.

[0081] On the other hand, in practical applications, data processing systems may also encounter resource management risks. Existing technologies that manually manage the lifecycle of objects (such as message queue producers and strategy instances) using raw pointers pose security risks such as memory leaks and dangling pointers. Furthermore, this approach lacks unified configuration management, requiring manual maintenance of multi-source configurations during operation and maintenance, which increases the probability of errors.

[0082] Based on this, the multi-strategy collaborative highly scalable data processing system of the present invention also includes a resource and configuration management module, which serves as the security layer of the present invention. This module can solve the resource leakage and configuration chaos problems of the prior art and avoid memory risks.

[0083] Specifically, the resource and configuration management module uses smart pointers to automatically allocate and release memory resources, avoiding resource leaks. At the same time, the resource and configuration management module also provides a unified configuration access interface based on configuration files or service registry centers, and supports hot reloading of configuration items (smart pointers), which can be used to realize automated resource management and dynamic configuration updates.

[0084] In practical applications, the resource and configuration management module uses smart pointer management strategies in high-level programming languages, the lifecycle of message queue producers / consumers, and automatically releases resources when tasks end or terminate abnormally to avoid memory leaks and dangling pointers. At the same time, it provides a unified configuration access interface, supports loading configurations from INI files, environment variables, and configuration centers, and enables hot reloading of configurations that take effect without restarting the system, reducing operation and maintenance costs.

[0085] Based on the same inventive concept, this invention also provides a working method for a multi-strategy collaborative highly scalable data processing system. This method includes: a Data Reactor receiving raw data from an external data source and sending it to a ProducerFactory module; the ProducerFactory selecting a data format and transmission protocol from a strategy abstraction and implementation module according to configuration parameters to dynamically create and manage concrete implementation instances based on the raw data; and a MapReduce data model packager performing parallel batch data processing on the concrete implementation instances generated by the ProducerFactory through a concurrency control mechanism, and outputting the final processing results to the target system. The parallel batch data processing includes sharding, mapping, and aggregation processing.

[0086] Furthermore, in this method, the strategy abstraction and implementation module includes a transmission strategy abstraction interface AbstractProducer and a data packaging format strategy abstraction interface PackagingStrategy. When a new transmission protocol strategy needs to be added, a corresponding transmission strategy producer class is added through the transmission strategy abstraction interface AbstractProducer. When a new data format processing strategy needs to be added, a corresponding data packaging format strategy packaging class is added through the data packaging format strategy abstraction interface PackagingStrategy.

[0087] Based on the same inventive concept, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned working method of a multi-strategy cooperative highly scalable data processing system.

[0088] Based on the same inventive concept, the present invention also provides an electronic device, the electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned working method of a multi-strategy collaborative highly scalable data processing system.

[0089] In summary, the present invention discloses a highly scalable data processing system, method, readable storage medium, and electronic device based on multi-strategy collaborative processing. This system decouples multi-protocol / multi-format data through strategy abstraction and implementation modules to ensure data semantic consistency across modules. The ProducerFactory module enables dynamic strategy selection and large-scale data optimization. Concurrency control and synchronization modules guarantee high concurrency performance. The MapReduce data model packager, working in conjunction with the concurrency control module, achieves automatic sharding, parallel mapping, and aggregation processing of massive amounts of rail transit data, significantly improving data processing throughput. Its efficient parallel processing capability meets real-time requirements. Furthermore, the present invention employs a DataReactor as the core coordinator, which uses an event-driven mechanism to dynamically schedule the entire data processing flow, achieving coupled collaboration and intelligent process coordination between modules, ensuring stable and efficient system operation even in the face of data fluctuations. This system is particularly suitable for scenarios such as the Internet of Things (IoT), Industrial Internet of Things (IIoT), and Integrated Monitoring Systems (ISCS). It needs to support multiple transmission protocols (such as Kafka, MceBus, MQTT), multiple data formats (such as JSON, Protobuf), and has the ability to dynamically coordinate policies, efficiently handle concurrent processing, and automatically manage resources. It is suitable for real-time synchronization of multi-source heterogeneous data, and has the advantages of high scalability, strong collaboration, high performance, and high reliability. It is highly practical.

[0090] Furthermore, the core idea of ​​this invention is to construct a producer system of "general interface + concrete implementation + factory scheduling" by defining specifications through abstract interfaces, decoupling through the factory pattern, dividing responsibilities through layered inheritance, and ensuring extensibility through the open / closed principle. This design ensures both the simplicity of upper-layer calls and the flexibility of lower-layer implementations, making it suitable for business scenarios with multiple message systems coexisting and requiring frequent iterative expansion. In this invention, the modules interact through interfaces, with clear boundaries and strong cohesion. Simultaneously, this invention supports multiple data types (analog quantities, digital quantities, cumulative quantities, etc.) through generic methods (such as map<>). When adding new data types, only template extensions are needed, without refactoring the overall logic.

[0091] Furthermore, the resource and configuration management module of this invention can automatically manage resources and resolve memory leaks through smart pointers. Its built-in error handling and retry mechanisms ensure data consistency, thereby guaranteeing the high reliability of the entire system. Moreover, the resource and configuration management module can reduce manual intervention through unified configuration and hot reload, and its full-process status monitoring improves troubleshooting efficiency and reduces maintenance costs.

[0092] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] In the description of this invention, it should be understood that the terms "center," "height," "thickness," "upper," "lower," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0094] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "joining," and "fixation" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0095] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0096] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above. Therefore, the scope of protection of the present invention should be defined by the appended claims.< / accumulatordata> < / digitaldata> < / analogdata>

Claims

1. A multi-strategy based cooperative high-scalable data processing system, characterized in that, Comprise: a strategy abstraction and implementation module comprising a plurality of strategy abstraction interfaces, each strategy abstraction interface being used to provide at least a plurality of data format processing strategies and transmission protocol strategies, the data format processing strategies and transmission protocol strategies of the strategy abstraction and implementation module being extensible; a concurrency control and synchronization module used to implement parallel processing of a plurality of tasks; a strategy coordination and factory scheduling module connected with an external data source and the strategy abstraction and implementation module respectively, the strategy coordination and factory scheduling module comprising: a data stream reactor used to coordinate event driving and state management of the entire data processing flow, the data stream reactor also being used to receive raw data of the external data source; a transmission strategy factory module connected with the data stream reactor, the transmission strategy factory module receiving the raw data sent by the data stream reactor and selecting data formats and transmission protocols from the strategy abstraction and implementation module according to configuration parameters to dynamically create and manage specific implementation instances based on the raw data; a data model packager connected with the transmission strategy factory module, the data model packager and the concurrency control and synchronization module performing parallel batch data processing on the specific implementation instances generated by the transmission strategy factory module through a concurrency control mechanism, and outputting final processing results to a target system, wherein the parallel batch data processing comprises sharding, mapping and aggregation processing.

2. The multi-strategy based synergic high-scalable data processing system according to claim 1, wherein, The strategy abstraction and implementation module comprises: a transmission strategy abstraction interface used to define a unified operation interface of transmission protocols, specific strategies of the transmission strategy abstraction interface being used to adapt to at least two of Kafka, MceBus and MQTT transmission protocols, the transmission protocol strategies being extensible through the transmission strategy abstraction interface; a data packing format strategy abstraction interface used to define serialization or deserialization specifications of data formats, specific strategies of the data packing format strategy abstraction interface being used to adapt to various data types.

3. The multi-strategy coordination high-extensible data processing system based on the multi-strategy coordination high-extensible data processing system of claim 2, wherein: when a transmission protocol strategy needs to be added, a corresponding transmission strategy producer class is added through the transmission strategy abstraction interface, and when a data format processing strategy needs to be added, a corresponding data packing format strategy packing class is added through the data packing format strategy abstraction interface.

4. The multi-strategy based synergistic high-scalable data processing system according to claim 2, wherein, The strategy abstraction and implementation module further comprises: a Kafka producer class inheriting the transmission strategy abstraction interface, the Kafka producer class being internally instantiated through a Kafka client mode to implement Kafka-based message sending functions; a MceBus producer class inheriting the transmission strategy abstraction interface, the MceBus producer class being internally instantiated through a MceBus message bus client to implement MceBus-based message sending functions; a stream characteristic encapsulation base class used to encapsulate basic characteristics and basic logic of respective message streams for reuse by specific producers.

5. The multi-strategy coordination high-extensible data processing system based on the multi-strategy coordination high-extensible data processing system of claim 4, wherein: The transmission strategy factory module inherits the transmission strategy abstract interface. When the transmission strategy factory module is initialized, it creates specific Kafka producer class or MceBus producer class instance objects as member variables based on the transmission mode options in the configuration parameters of the configuration file.

6. The multi-strategy based synergistic high-scalable data processing system as claimed in claim 2, wherein, The strategy abstraction and implementation module also includes: A JSON packaging class, which inherits the data packaging format strategy abstract interface, internally formats various types of data using JSON, and can be used to implement JSON text format data serialization or deserialization; The Protobuf packing class inherits the data packing format strategy abstract interface. The Protobuf packing class internally formats various types of data using Protobuf. The Protobuf packing class can be used to implement data serialization or deserialization in Protobuf binary format.

7. The highly scalable data processing system based on multi-strategy collaborative processing as described in claim 1, characterized in that, The concurrency control and synchronization module includes a thread pool, which is used to manage thread resources for the parallel execution of multiple tasks. It supports dynamic adjustment of the number of threads, and the thread pool can dynamically adjust the number of threads according to the task queue length or system resource utilization.

8. The multi-strategy based synergistic high-scalable data processing system according to claim 7, wherein, The data model packager includes: The Map phase processing component is used to split the processing task of the specific implementation instance into data, and submit each subtask corresponding to each split data piece to the thread pool for parallel processing. The Map phase processing component is also used to receive each data after processing by the thread pool. The Reduce phase processing component receives the data sent by the Map phase processing component after parallel processing by a thread pool, and performs mapping and aggregation processing on the data to obtain the final processing result.

9. The multi-strategy based synergistic high-scalable data processing system as claimed in claim 8, wherein, The concurrency control and synchronization module also includes: A multi-threaded notification queue uses a blocking or waking mechanism to distribute tasks and synchronize states between threads.

10. The highly scalable data processing system based on multi-strategy collaborative processing as described in claim 9, characterized in that, The thread pool includes two methods: creation and destruction. When created, a predetermined number of worker threads are initialized through its constructor. These worker threads are used to process and package data. When destroyed, the worker threads are recycled through its destructor. The Map phase processing component writes split and sharded data to the multi-threaded notification queue. The worker thread objects in the thread pool extract the sharded data from the multi-threaded notification queue, execute the processing function to process it, and package it according to the predetermined data format.

11. The highly scalable data processing system based on multi-strategy collaborative processing as described in claim 10, characterized in that, When the Map phase processing component completes the data segmentation of the current batch of data and writes each data segment into the multi-threaded notification queue, the task status of the current batch of data is in a blocked critical resource state. When all data segments are processed, the multi-threaded notification queue wakes up the next batch of data that is currently in a blocked critical resource state.

12. The multi-strategy based synergistic high-scalable data processing system as claimed in claim 10, wherein, The concurrency control and synchronization module further comprises: An atomic counter for thread-safe task counting and state identification.

13. The multi-policy based cooperative high-scalable data processing system according to claim 12, wherein, When the Map phase processing component writes a piece of data into the multi-thread notification queue, the atomic counter is incremented by 1; and when the worker thread finishes processing a piece of data, the atomic counter is decremented by 1. After the Map phase processing component delivers all pieces of data to the multi-thread notification queue, it waits for the atomic counter to be zero; and when the last worker thread finishes the task and makes the atomic counter zero, the waiting Reduce phase processing component is woken up to start the aggregation work.

14. The multi-strategy based synergistic high-scalable data processing system as described in claim 1, wherein, Further comprising: A resource and configuration management module which uses an intelligent pointer to automatically apply and release memory resources, and which also provides a unified configuration access interface based on a configuration file or a service registry center and supports hot reloading of configuration items.

15. The multi-policy based cooperative high-scalable data processing system according to claim 1, wherein, When the data stream reactor is started, it calls the device data service module, the registration manager of which adds an event source to a processor list to complete registration; and when the device data service module collects raw data, the data stream reactor receives the raw data and coordinates the workflow to process the raw data.

16. A method of operating a multi-strategy based cooperative high- scalable data processing system according to any one of claims 1-15, characterized by, Comprising: The data stream reactor receives raw data from an external data source and sends it to the transmission strategy factory module; The transmission strategy factory module selects a data format and a transmission protocol from the strategy abstraction and implementation module according to configuration parameters, to dynamically create and manage specific implementation instances based on the raw data; The data model packager performs parallel batch data processing on the specific implementation instances generated by the transmission strategy factory module through a concurrency control mechanism, and outputs the final processing result to a target system, wherein the parallel batch data processing comprises sharding, mapping and aggregation processing.

17. The working method of the multi-policy based cooperative high-scalable data processing system according to claim 16, wherein, The strategy abstraction and implementation module comprises a transmission strategy abstraction interface and a data packing format strategy abstraction interface; When a transmission protocol strategy needs to be added, a corresponding transmission strategy producer class is added through the transmission strategy abstraction interface; and when a data format processing strategy needs to be added, a corresponding data packing format strategy packing class is added through the data packing format strategy abstraction interface.

18. A readable storage medium, characterized by, The computer program is stored on the readable storage medium and is executed by the processor to implement the steps of the working method of the multi-policy based cooperative high-scalable data processing system according to any one of claims 16-17.

19. An electronic device, comprising: The electronic device comprises a memory and a processor, wherein the computer program is stored on the memory and is executed by the processor to implement the steps of the working method of the multi-policy based cooperative high-scalable data processing system according to any one of claims 16-17.

Images