Distributed data processing system, distributed data processing method

Abstract

The embodiment of the present disclosure discloses a distributed data processing system and a distributed data processing method, comprising a publisher, an in-memory middleware and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system; wherein the publisher is used to collect initial sleep data and transmit the initial sleep data to the in-memory middleware; the in-memory middleware is used to store the initial sleep data; each subscriber is used to obtain corresponding initial sleep data and process the corresponding initial sleep data through a corresponding sub-distributed data processing system to obtain a corresponding processing result. Each subscriber in the distributed data processing system provided by the scheme can also comprise at least one sub-distributed data processing system, solving the problems of limited processing capacity of traditional single machines, low data processing efficiency, poor real-time performance and the like.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of data processing, and particularly relate to a distributed data processing system and a distributed data processing method. BACKGROUND

[0002] Sleep monitoring data is usually a large amount of high-density time series data, which requires a large amount of storage and processing capacity for storage and analysis. If the storage and processing capacity is insufficient, it may cause data loss or system crash. SUMMARY

[0003] Embodiments of the present disclosure provide a distributed data processing system and a distributed data processing method, which improve the performance of data processing.

[0004] In a first aspect, a distributed data processing system is provided, comprising a publisher, an in-memory middleware and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system, each sub-distributed data processing system comprising a sub-publisher, a sub-in-memory middleware and at least one sub-subscriber; wherein,

[0005] The publisher is configured to collect initial sleep data and transmit the initial sleep data to the in-memory middleware;

[0006] The in-memory middleware is configured to store the initial sleep data;

[0007] Each subscriber is configured to obtain corresponding initial sleep data and process the corresponding initial sleep data through the corresponding sub-distributed data processing system to obtain corresponding processing results.

[0008] In a second aspect, a distributed data processing method is provided, applied to the distributed data processing system provided in the first aspect, the distributed data processing system comprising a publisher, an in-memory middleware and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system, each sub-distributed data processing system comprising a sub-publisher, a sub-in-memory middleware and at least one sub-subscriber;

[0009] The method comprises:

[0010] The publisher collects initial sleep data and transmits the initial sleep data to the in-memory middleware;

[0011] The in-memory middleware stores the initial sleep data;

[0012] Each subscriber obtains corresponding initial sleep data according to the data type and processes the corresponding initial sleep data through the corresponding sub-distributed data processing system to obtain corresponding processing results.

[0013] The embodiment of the present disclosure discloses a distributed data processing system and a distributed data processing method. The system comprises a publisher, an in-memory middleware and at least one subscriber. Each subscriber comprises at least one sub-distributed data processing system. Each sub-distributed data processing system comprises a sub-publisher, a sub-in-memory middleware and at least one sub-subscriber. The publisher is configured to collect initial sleep data and transmit the initial sleep data to the in-memory middleware. The in-memory middleware is configured to store the initial sleep data. Each subscriber is configured to acquire corresponding initial sleep data through a corresponding sub-distributed data processing system and process the corresponding initial sleep data to obtain a corresponding processing result. The distributed data processing system provided by the technical solution comprises a publisher, an in-memory middleware and at least one subscriber. Each subscriber comprises at least one sub-distributed data processing system, thereby realizing a high-efficiency and reliable multi-process and multi-thread system and solving the problems of limited processing capacity, low data processing efficiency and poor real-time performance of a traditional single machine, thereby providing an efficient data processing solution for the sleep medical field.

[0014] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the embodiments of the present disclosure. Other features of the embodiments of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0016] Figure 1 is a structural block diagram of a distributed data processing system provided by the first embodiment of the present disclosure;

[0017] Figure 2 is a structural block diagram of a publish-subscribe mode provided by the first embodiment of the present disclosure;

[0018] Figure 3 is a flowchart of a distributed data processing method provided by the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the person skilled in the art better understand the scheme of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present disclosure.

[0020] It should be noted that the terms "first", "second" and the like in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] Embodiment one

[0022] Figure 1 A structural block diagram of a distributed data processing system 10 provided by the first embodiment of the present disclosure is provided, and the present embodiment can be applied to the case of distributed processing of data. As shown in the figure, the distributed data processing system 10 includes a publisher 101, an in-memory middleware 102 and at least one subscriber 103; each subscriber 103 includes at least one sub-distributed data processing system 104, and each sub-distributed data processing system 104 includes a sub-publisher 1041, a sub-in-memory middleware 1042 and at least one sub-subscriber 1043; wherein, Figure 1

[0023] The publisher 101 is configured to collect initial sleep data and transmit the initial sleep data to the in-memory middleware 102;

[0024] The in-memory middleware 102 is configured to store the initial sleep data;

[0025] Each subscriber 103 is configured to obtain corresponding initial sleep data through the corresponding sub-distributed data processing system 104 and process the corresponding initial sleep data to obtain corresponding processing results.

[0026] ​The embodiment provides a distributed data processing system 10, which comprises a publisher 101, in-memory middleware 102 and at least one subscriber 103; the publisher 101 is used for collecting initial sleep data and pushing the initial sleep data to the in-memory middleware 102, the in-memory middleware 102 stores the received initial sleep data; the subscriber 103 can obtain the initial sleep data from the in-memory middleware 102 and process the initial sleep data by using a sub-distributed data processing system 104.

[0027] The publisher 101 is used for collecting initial sleep data and transmitting the initial sleep data to the in-memory middleware 102.

[0028] In the embodiment, the publisher 101 can be used for collecting initial sleep data, and the publisher 101 can comprise a plurality of data collection devices, the data collection device can be a polysomnograph, the polysomnograph can be a medical device, and can be used for polysomnography (PSG) in a sleep monitoring room, which is an examination technique capable of continuously and synchronously collecting, recording and analyzing a plurality of sleep physiological parameters and pathological events; the data type of the initial sleep data collected by the data collection device can at least comprise electroencephalogram (EEG) data, electrocardiogram (ECG) data, electromyogram (EMG) data and / or electrooculogram (EOG) data. The EEG data can be used for recording the electrical activity of the brain, the ECG data can be used for recording the electrical activity of the heart, the EMG data can be used for recording the electrical signal generated by the muscle when the muscle is active, and the EOG data can be used for recording the electrical signal generated by the eye movement.

[0029] Specifically, after collecting the initial sleep data, the publisher 101 can transmit the collected initial sleep data to the in-memory middleware 102.

[0030] The in-memory middleware 102 is used for storing the initial sleep data.

[0031] In this embodiment, the memory middleware 102 can be implemented based on a data management and processing architecture. The data management and processing architecture can be an architecture for managing and / or processing acquired data. In this embodiment, the data management and processing architecture can include an Internet of Things message server EMQ, a distributed streaming platform Kafka, and / or a relational database management system MySQL, etc. It should be noted that the data management and data processing architecture in this embodiment can also include a time series database model, a column storage structure, a cloud database structure, a graph database model, an object-relational database model, and / or a non-relational database model, etc. having a storage function. This embodiment does not limit this.

[0032] The EMQ is an open-source Internet of Things message server, and its main function is to process communication between devices. The EMQ is designed for high concurrency and low latency, can support tens of millions of device connections, and can provide message publishing and subscription services. Kafka is a distributed streaming platform, mainly used for building real-time data pipelines and streaming applications. MySQL is a relational database management system, and the SQL language used by MySQL is the most commonly used standardized language for accessing databases. MySQL has the characteristics of small size, high speed, and open source code.

[0033] Specifically, after receiving the initial sleep data transmitted by the publisher 101, the memory middleware 102 can store the initial sleep data. In this embodiment, the memory middleware 102 can also be used to store the initial sleep data in the form of a queue when the data volume of the initial sleep data exceeds a first preset threshold, so as to perform peak clipping on the initial sleep data. The first preset threshold can be a threshold that is preset to limit the storage quantity of the memory middleware 102.

[0034] Based on the above description, peak clipping can be to temporarily store a large amount of data in the data structure (queue) of the memory middleware 102 to reduce the instantaneous pressure of the downstream system of data transmission. This processing method can smooth the peak of data flow and prevent system overload. The peak clipping strategy of the memory middleware 102 can include: setting the maximum capacity of the memory middleware 102 to avoid memory overflow, dynamically adjusting the queue capacity of the memory middleware 102 according to the current system load, controlling the production rate of data by the publisher 101 or pausing the production of new data when the queue capacity of the memory middleware 102 approaches the upper limit, increasing the processing speed of the subscriber 103, increasing the number of subscribers 103, setting priorities according to the importance of tasks, and / or preferentially processing high-priority tasks, etc.

[0035] Each of the subscribers 103 is configured to acquire and process the corresponding initial sleep data through the corresponding sub-distributed data processing system 104 to obtain a corresponding processing result.

[0036] It can be understood that each distributed data processing system 10 can include at least one subscriber 103, and each subscriber 103 can include at least one sub-distributed data processing system 104. Specifically, the subscriber 103 can acquire the corresponding initial sleep data from the memory middleware 102 according to the data type through the sub-distributed data processing system 104. For example, if the distributed data processing system 10 can include four subscribers 103, and the memory middleware 102 stores four data types of EEG data, ECG data, EMG data and EOG data, then the four subscribers 103 can process one type of data respectively.

[0037] It should be noted that the subscriber 103 can process the initial sleep data after acquiring the initial sleep data to obtain a corresponding processing result, wherein the processing of the initial sleep data can be a processing step of data format conversion, data mining, data enhancement, data coding, data fusion and / or data visualization, and the present embodiment does not limit this.

[0038] According to the above description, the initial sleep data can include different types of sleep data, and after obtaining the processing results of the multiple types of sleep data in the initial sleep data, the processing results of each type of sleep data can be summarized by summation or weighted summation, and the summarized result can be used as the processing result of the initial sleep data.

[0039] The present embodiment provides a distributed data processing system, which includes a publisher, a memory middleware and at least one subscriber, each of the subscribers includes at least one sub-distributed data processing system, each of the sub-distributed data processing systems includes a sub-publisher, a sub-memory middleware and at least one sub-subscriber; wherein the publisher is configured to acquire initial sleep data and transmit the initial sleep data to the memory middleware; the memory middleware is configured to store the initial sleep data; each of the subscribers is configured to acquire and process the corresponding initial sleep data through the corresponding sub-distributed data processing system to obtain a corresponding processing result. The technical scheme provided by the present embodiment solves the problems of limited processing capacity, low data processing efficiency and poor real-time performance of the traditional single machine, and provides an efficient data processing solution for the sleep medical field.

[0040] As an optional implementation, each of the subscribers 103 is further configured to:

[0041] a1) acquiring, by the sub-publisher 1041 in the corresponding sub-distributed data processing system 104, data of the same data type in the initial sleep data, and transmitting the data of the same data type in the initial sleep data to the sub-memory middleware 1042;

[0042] Specifically, each subscriber 103 can include at least one sub-distributed data processing system 104, and the sub-distributed data processing system 104 can include a sub-publisher 1041, a sub-memory middleware 1042, and at least one sub-subscriber 1043. The subscriber 103 can acquire data of the same type in the initial sleep data from the memory middleware 102 through the sub-publisher 1041 in the corresponding sub-distributed data processing system 104, and transmit the acquired data of the same type in the initial sleep data to the sub-memory middleware 1042.

[0043] b1) receiving, by the sub-memory middleware 1042 in the corresponding sub-distributed data processing system 104, data of the same data type in the initial sleep data, and storing the data of the same data type in the initial sleep data;

[0044] In this embodiment, the sub-memory middleware 1042 is implemented based on a data management and processing architecture. The data management and processing architecture can be an architecture for managing and / or processing acquired data. The data management and processing architecture can include an Internet of Things message server EMQ, a distributed streaming media platform Kafka, and / or a relational database management system MySQL. It should be noted that the data management and data processing architecture in this embodiment can also include a time series database model, a column storage structure, a cloud database structure, a graph database model, an object-relational database model, and / or a non-relational database model, etc. having a storage function. This embodiment does not limit this.

[0045] Specifically, after receiving the data of the same data type in the initial sleep data transmitted by the sub-publisher 1041, the sub-memory middleware 1042 can store the data of the same data type in the initial sleep data.

[0046] It should be explained that the sub-memory middleware 1042 can also be used to store the data of the same data type in the initial sleep data in the form of a queue when the data amount of the data of the same data type in the initial sleep data exceeds a second preset threshold, so as to perform secondary peak clipping on the data of the same data type in the initial sleep data. The second preset threshold can be a threshold for limiting the storage amount of the sub-memory middleware 1042.

[0047] In light of the foregoing, the peak clipping can be to temporarily store a large amount of data by using a data structure (queue) in the sub-memory middleware 1042, so as to alleviate the instantaneous pressure of the downstream system of the data transmission. This processing manner can smooth the peak of the data flow, and prevent the system from being overloaded.

[0048] c1) The sub-subscriber 1043 in the corresponding sub-distributed data processing system 104 randomly collects data of the same data type in the initial sleep data from the sub-memory middleware 1042, and processes the data of the same data type in the initial sleep data to obtain a processing result.

[0049] It can be known that the sub-subscriber 1043 in the sub-distributed data processing system 104 can randomly collect data of the same data type in the initial sleep data from the sub-memory middleware 1042, and process the data of the same data type in the initial sleep data to obtain a processing result. The processing of the data of the same data type in the initial sleep data can be a processing step of data format conversion, data mining, data enhancement, data coding, data fusion and / or data visualization, and the embodiment is not limited thereto. For different types of sleep data in the initial sleep data, processing results of multiple types of sleep data can be obtained, and the processing results of the multiple types of sleep data can be summarized by using summation or weighted summation, and the summarized result is taken as the processing result of the initial sleep data.

[0050] The distributed data processing system provided in the embodiment can be implemented based on a publish-subscribe mode, Figure 2 A structure block diagram of the publish-subscribe mode provided in the embodiment is shown in Figure 2 The publish-subscribe mode provided in the embodiment includes a publisher, a memory middleware and at least one subscriber, and Figure 2 On the basis of the publish-subscribe mode provided in the embodiment, each subscriber can further include at least one sub-distributed data processing system, and each sub-distributed data processing system includes a sub-publisher, a sub-memory middleware and at least one sub-subscriber. The publish-subscriber mode is used to collect and process the initial sleep data: a memory middleware (memory queue) is additionally added, and the coupling is reduced. Each subscriber can be a publish-subscriber mode, that is, each subscriber can include at least one sub-distributed data processing system, each sub-distributed data processing system includes a sub-publisher, a sub-memory middleware and at least one sub-subscriber, each subscriber is a process, and each sub-distributed data processing system can be regarded as a thread. The multiple threads are processed in parallel in each process, so as to accelerate the data processing speed and improve the processing efficiency.

[0051] Embodiment two

[0052] Figure 3A flowchart of a distributed data processing method provided for Embodiment Two of the present disclosure is applied to a distributed data processing system, the distributed data processing system comprising a publisher, an in-memory middleware, and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system, each sub-distributed data processing system comprising a sub-publisher, a sub-in-memory middleware, and at least one sub-subscriber. As shown in Figure 3 The method comprises the following steps:

[0053] S210, the publisher collects initial sleep data and transmits the initial sleep data to the in-memory middleware.

[0054] Specifically, the publisher can be used to collect initial sleep data, and the publisher can comprise a plurality of collection devices for collecting different types of initial sleep data. The data types of the initial sleep data can at least include Electroencephalogram (EEG) data, Electrocardiogram (ECG) data, Electromyogram (EMG) data, and / or Electrooculogram (EOG) data. The EEG data can be used to record the electrical activity of the brain, the ECG data can be used to record the electrical activity of the heart, the EMG data can be used to record the electrical signals generated by the muscle when it is active, and the EOG data can be used to record the electrical signals generated by the eye movement.

[0055] According to the above description, after the publisher collects the initial sleep data, the publisher can transmit the collected initial sleep data to the in-memory middleware.

[0056] S220, the in-memory middleware stores the initial sleep data.

[0057] In this embodiment, the in-memory middleware is implemented based on a data management and processing architecture. The data management and processing architecture can be an architecture for managing and / or processing acquired data. The data management and processing architecture comprises an Internet of Things message server EMQ, a distributed streaming media platform Kafka, and / or a relational database management system MySQL. It should be noted that the data management and processing architecture in this embodiment can also comprise a time series database model, a column storage structure, a cloud database structure, a graph database model, an object-relational database model, and / or a non-relational database model, etc. having a storage function. This embodiment does not limit the data management and processing architecture.

[0058] Specifically, the memory middleware can be configured to receive initial sleep data and store the initial sleep data. It should be noted that the memory middleware can also be configured to store the initial sleep data in the form of a queue when the data volume of the initial sleep data exceeds a first preset threshold, so as to perform peak clipping on the initial sleep data. The first preset threshold can be a threshold that is preset to limit the storage quantity of the memory middleware.

[0059] According to the above description, the peak clipping can be to temporarily store a large amount of data in the data structure (queue) in the memory middleware, so as to reduce the instantaneous pressure of the downstream system of data transmission. This processing manner can smooth the peak of data flow and prevent system overload. The strategy of peak clipping can include: setting the maximum capacity of the memory middleware to avoid memory overflow, dynamically adjusting the queue capacity of the memory middleware according to the current system load, controlling the production rate of data of the publisher or pausing the production of new data when the queue capacity of the memory middleware approaches the upper limit, increasing the processing speed of the subscriber or increasing the number of subscribers when the queue length of the memory middleware increases, and / or setting priorities according to the importance of tasks, and processing high-priority tasks first.

[0060] S230, each subscriber obtains corresponding initial sleep data through a corresponding sub-distributed data processing system and processes the corresponding initial sleep data to obtain a corresponding processing result.

[0061] Specifically, the distributed processing system can include a plurality of subscribers, and each subscriber can include at least one sub-distributed data processing system. The sub-distributed data processing system can obtain initial sleep data from the memory middleware according to the data type, and process the obtained initial sleep data by using the sub-distributed data processing system to obtain a processing result. In this embodiment, the processing of the initial sleep data can be a processing step of data format conversion, data mining, data enhancement, data coding, data fusion, and / or data visualization, and this embodiment does not limit the same.

[0062] The embodiment provides a distributed data processing method, which includes: the publisher collects initial sleep data and transmits the initial sleep data to the memory middleware; the publisher collects initial sleep data and transmits the initial sleep data to the memory middleware; the memory middleware stores the initial sleep data; each subscriber obtains corresponding initial sleep data through a corresponding sub-distributed data processing system and processes the corresponding initial sleep data to obtain a corresponding processing result, thereby improving the speed of data processing.

[0063] As an optional implementation, each of the subscribers obtains corresponding initial sleep data and processes the corresponding initial sleep data through a corresponding sub-distributed data processing system to obtain a corresponding processing result, including:

[0064] a2) each of the subscribers obtains data of the same data type in the initial sleep data through a sub-publisher in the corresponding sub-distributed data processing system, and transmits the data of the same data type in the initial sleep data to the sub-memory middleware;

[0065] Specifically, each of the subscribers can include a sub-distributed data processing system, and the subscriber can process data of the same data type in the initial sleep data through the sub-distributed data processing system. Specifically, the sub-distributed data processing system can include a sub-publisher, a sub-memory middleware, and a sub-subscriber, and the sub-publisher can obtain data of the same data type in the initial sleep data from the memory middleware and transmit the obtained data of the same data type in the initial sleep data to the sub-memory middleware.

[0066] b2) receiving data of the same data type in the initial sleep data through the sub-memory middleware in the corresponding sub-distributed data processing system, and storing the data of the same data type in the initial sleep data;

[0067] In this embodiment, the sub-memory middleware can be implemented based on a data management and processing architecture, which includes an Internet of Things message server EMQ, a distributed streaming media platform Kafka, and / or a relational database management system MySQL. It should be noted that the data management and data processing architecture in this embodiment can also include a time series database model, a column storage structure, a cloud database structure, a graph database model, an object-relational database model, and / or a non-relational database model, etc. having a storage function, which is not limited in this embodiment.

[0068] Specifically, the sub-memory middleware can be used to store data of the same data type in the initial sleep data. It should be noted that the sub-memory middleware can also be used to store data of the same data type in the initial sleep data in the form of a queue when the amount of data of the same data type in the initial sleep data exceeds a second preset threshold, so as to perform secondary peak clipping on the data of the same data type in the initial sleep data. The second preset threshold can be a threshold for limiting the storage quantity of the sub-memory middleware.

[0069] c2) The sub-subscriber in the sub-distributed data processing system randomly collects data of the same data type in the initial sleep data from the sub-memory middleware and processes the data of the same data type in the initial sleep data to obtain a processing result.

[0070] It can be known that the sub-subscriber in the sub-distributed data processing system can randomly collect data of the same data type in the initial sleep data from the sub-memory middleware and process the data of the same data type in the initial sleep data, wherein the processing of the data of the same data type in the initial sleep data can include data format conversion, data mining, data enhancement, data coding, data fusion and / or data visualization, etc. The present embodiment does not limit this. For different types of sleep data in the initial sleep data, processing results of multiple types of sleep data can be obtained, and the processing results of multiple types of sleep data can be summarized by summation or weighted summation, etc. The result after the summary is taken as the processing result of the initial sleep data.

[0071] The distributed data processing method provided in the embodiment includes the following steps: a publisher collects initial sleep data and transmits the initial sleep data to the memory middleware; the memory middleware stores the initial sleep data; each subscriber acquires data of the same data type in the corresponding initial sleep data according to the data type by using a corresponding sub-distributed data processing system, and stores the data of the same data type in the initial sleep data by using a sub-memory middleware; and a sub-subscriber randomly collects data of the same data type in the initial sleep data from the sub-memory middleware and processes the data of the same data type in the initial sleep data to obtain a processing result. The method aims to solve the problems of limited processing capacity, low data processing efficiency and poor real-time performance of the traditional single machine, and provides an efficient data processing solution for the sleep medical field.

Claims

1. A distributed data processing system, characterized by, Comprise: a publisher, an in-memory middleware and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system, each sub-distributed data processing system comprising a sub-publisher, a sub-in-memory middleware and at least one sub-subscriber; wherein, the publisher is configured to collect initial sleep data and transmit the initial sleep data to the in-memory middleware; the in-memory middleware is configured to store the initial sleep data; each subscriber is configured to obtain and process corresponding initial sleep data through a corresponding sub-distributed data processing system to obtain a corresponding processing result; the processing result of the initial sleep data is generated by aggregating a plurality of results obtained by processing different types of sleep data respectively; each subscriber is configured to: obtain data of the same data type in the initial sleep data through a sub-publisher in the corresponding sub-distributed data processing system, and transmit the data of the same data type in the initial sleep data to the sub-in-memory middleware; receive the data of the same data type in the initial sleep data through the sub-in-memory middleware in the corresponding sub-distributed data processing system, and store the data of the same data type in the initial sleep data; randomly collect the data of the same data type in the initial sleep data from the sub-in-memory middleware through a sub-subscriber in the corresponding sub-distributed data processing system, and process the data of the same data type in the initial sleep data to obtain the processing result.

2. The system of claim 1, wherein, The data types of the initial sleep data at least include electroencephalogram data, electrocardiogram data, electromyogram data and / or electrooculogram data.

3. The system of claim 1, wherein, The in-memory middleware and the sub-in-memory middleware are implemented based on a data management and processing architecture, which comprises an Internet of Things message server EMQ, a distributed streaming media platform Kafka and / or a relational database management system MySQL.

4. The system of claim 1, wherein, The in-memory middleware is configured to store the initial sleep data in the form of a queue to perform peak clipping on the initial sleep data when the data volume of the initial sleep data exceeds a first preset threshold; The sub-in-memory middleware is configured to store the data of the same data type in the initial sleep data in the form of a queue to perform secondary peak clipping on the data of the same data type in the initial sleep data when the data volume of the data of the same data type in the initial sleep data exceeds a second preset threshold.

5. A distributed data processing method, characterized by, The method is applied to a distributed data processing system, which comprises a publisher, an in-memory middleware and at least one subscriber, each subscriber comprising at least one sub-distributed data processing system, each sub-distributed data processing system comprising a sub-publisher, a sub-in-memory middleware and at least one sub-subscriber; The method comprises: the publisher collects initial sleep data and transmits the initial sleep data to the in-memory middleware; the in-memory middleware stores the initial sleep data; Each of the subscribers obtains corresponding initial sleep data and processes the corresponding initial sleep data to obtain a corresponding processing result through a corresponding sub-distributed data processing system; the processing result of the initial sleep data is generated by aggregating a plurality of results obtained by processing different types of sleep data respectively; Each of the subscribers obtains corresponding initial sleep data and processes the corresponding initial sleep data to obtain a corresponding processing result through a corresponding sub-distributed data processing system, including: Each of the subscribers obtains data of the same data type in the initial sleep data through a sub-publisher in the corresponding sub-distributed data processing system, and transmits the data of the same data type in the initial sleep data to the sub-memory middleware; The data of the same data type in the initial sleep data is received through the sub-memory middleware in the corresponding sub-distributed data processing system, and the data of the same data type in the initial sleep data is stored; The data of the same data type in the initial sleep data is randomly collected from the sub-memory middleware through a sub-subscriber in the corresponding sub-distributed data processing system, and the data of the same data type in the initial sleep data is processed to obtain the processing result.

6. The method of claim 5, wherein, The data type of the initial sleep data at least includes electroencephalogram data, electrocardiogram data, electromyogram data and / or electrooculogram data.

7. The method of claim 5, wherein, The memory middleware and the sub-memory middleware are implemented based on a data management and processing architecture, and the data management and processing architecture includes an Internet of Things message server EMQ, a distributed streaming media platform Kafka and / or a relational database management system MySQL.

8. The method of claim 6, wherein, The memory middleware is used to store the initial sleep data in the form of a queue when the data volume of the initial sleep data exceeds a first preset threshold, so as to perform peak clipping on the initial sleep data; The sub-memory middleware is used to store the data of the same data type in the initial sleep data in the form of a queue when the data volume of the data of the same data type in the initial sleep data exceeds a second preset threshold, so as to perform secondary peak clipping on the data of the same data type in the initial sleep data.

Images