Edge cloud real-time data processing system based on multi-source sensor

Abstract

The invention relates to an edge cloud real-time data processing system based on a multi-source sensor. The problems that cloud computing is low in real-time performance, low in bandwidth and the like are solved by enabling computing and storage resources to be close to a data source. The system comprises a multi-source sensor access end, a data exchange end and an edge cloud real-time data processing end, the multi-source sensor access end transmits and forwards collected access data of photoelectric and radar sensors and the like to different nodes of the edge cloud real-time data processing end through the data exchange end, and various types of storage or calculation tasks are executed. According to the invention, access of different types of sensors can be coped, different computing units can be combined, computing power can be flexibly distributed, computing resources are effectively utilized, and the designed data processing unit has the characteristics of high bandwidth and strong real-time performance.

Description

Technical Field

[0001] The present invention belongs to the field of Ethernet network communication and edge cloud computing, and in particular to the design of an edge cloud real-time data processing architecture based on multi-source sensors. Background Art

[0002] With the widespread application of emerging technologies such as big data, big models, artificial intelligence, and the Internet of Things, the scale of big data, the number of microservices, and the number of intelligent applications have shown explosive growth. The increasing number of sensors and edge devices has driven the exponential growth of data volume, and the growth in the number of intelligent applications has driven the demand for computing power to increase continuously, bringing unprecedented challenges to the network, communication, and core computing power of the computing platform.

[0003] More importantly, in future airborne missions, optoelectronic perception, game confrontation, radar processing, human-computer interaction, and intelligent air combat will have many requirements for computing cloud platforms, which makes it difficult for the current CPU-based customized computing to adapt to tasks with complex scenarios and huge amounts of data. The development of future airborne computing platforms will focus on data-centric computing cloud platform computing architectures. Cloud platforms can better support future mission requirements, cope with the challenges of surging data volumes and computing pressure, and provide strong support and technical foundations for applications in various fields.

[0004] At present, the traditional cloud computing platform with CPU as the core performs data analysis through a centralized processing mode. In the cloud computing mode, the terminal device needs to package and upload the data to the cloud server for calculation, and then transmit the processed results back to the terminal device. Although the traditional cloud computing model can solve the current problem of large-scale data processing, it still has the following disadvantages:

[0005] 1. Low bandwidth: Traditional centralized cloud computing platforms cannot cope with the real-time massive data generated by large-scale edge devices. The bandwidth resources for data transmission have reached limits and cannot meet high performance requirements.

[0006] 2. High latency: The data generated by edge devices and sensor data are continuously transmitted and interacted with the cloud platform, resulting in high network latency and unable to meet real-time requirements.

[0007] 3. Low reliability: Transmitting data to a remote cloud computing platform will bring about a series of problems such as data security and privacy leakage, resulting in low reliability. Summary of the invention

[0008] In order to solve the above-mentioned technical problems existing in the background technology, the present invention has developed an edge cloud real-time data processing system based on multi-source sensors, which greatly improves the real-time performance by placing computing and storage resources close to the data source. The present invention designs a flexible multi-source sensor access terminal, a data exchange terminal, and a scalable and expandable edge cloud real-time data processing terminal. The present invention can cope with different types of sensor access, can be combined with different computing units, flexibly allocate computing power, effectively utilize computing resources, and the designed data processing unit has the characteristics of high bandwidth and strong real-time.

[0009] The technical solution of the present invention is: the present invention is an edge cloud real-time data processing system based on multi-source sensors, and its special feature is: the edge cloud real-time data processing system based on multi-source sensors includes a multi-source sensor access terminal, a data exchange terminal and an edge cloud real-time data processing terminal. The multi-source sensor access terminal transmits and forwards the collected access data of photoelectric and radar sensors to different nodes of the edge cloud real-time data processing terminal through the data exchange terminal to perform various types of storage or computing tasks.

[0010] Furthermore, the multi-source sensor access end includes a big data sensor access end and a microservice network aggregation access end. The big data sensor access end is used for access scenarios with high data dimensions and large data scales, and the microservice network aggregation access end is used for scenarios with small data dimensions but large volumes. The big data sensor access end and the microservice network aggregation access end are respectively connected to the data exchange end.

[0011] Furthermore, the big data sensor access end includes a first data processing unit. For access scenarios with high data dimensions and large data scale, the first data processing unit is used to quickly process and transmit the data. A customized data transmission protocol is used to achieve CPU zero copy through kernel bypass to achieve low latency and high performance transmission. The microservice network aggregation access end includes a second data processing unit and a microprocessor. For scenarios with small data dimensions but large data volumes, the data will be accessed and processed through the microprocessor, and then processed and transmitted through the second data processing unit, and finally reach the edge cloud real-time data processing end.

[0012] Furthermore, the first data processing unit and the second data processing unit support high-precision clock synchronization.

[0013] Furthermore, the data exchange end uses a switching unit to form a redundant backup, providing equipment protection and traffic load sharing.

[0014] Furthermore, the data exchange end is a switch.

[0015] Furthermore, the edge cloud real-time data processing end includes a third data processing unit, a control unit, a computing unit and a storage unit. The edge cloud real-time data processing end takes the data processing unit as the core and is interconnected with the control unit, the computing unit and the storage unit through different forms and different transmission protocols.

[0016] Furthermore, the third data processing unit supports DSA direct, Multihost, RDMA, container hot start and traffic scheduling technology.

[0017] The present invention provides an edge cloud real-time data processing system based on multi-source sensors. By bringing computing and storage resources close to data sources, data transmission delay is significantly reduced, and real-time performance is improved, aiming to solve many problems of traditional cloud computing platforms and future airborne missions. The system designed by the present invention is mainly composed of three parts: a multi-source sensor access terminal, a data exchange terminal, and an edge cloud real-time data processing terminal. The present invention pushes the edge cloud real-time data processing terminal close to terminal devices and data acquisition devices, and designs nodes such as data exchange and data control processing in the system to improve the platform processing concurrent services. The present invention is mainly based on data as the center and combines the edge cloud computing model. Through the combination of data processing units and other computing units, the system can achieve high performance, strong real-time performance, low latency, low CPU occupancy and other requirements. Secondly, the present invention supports flexible access to sensors, and the system computing power is scalable and expandable. The edge cloud real-time data system is mainly formed with the data processing unit as the core, so that data interaction does not need to be copied multiple times, effectively solving the problem of large-scale data processing; the system integrates computing power units such as control units, computing units, and storage units, so that the edge cloud real-time data processing system can effectively adapt to intelligent computing tasks.

[0018] The present invention is data-centric and combines edge cloud computing technology, uses data processing units as data access and data transmission processing, can adapt to large-scale data transmission, and distributes to specific nodes through switching units to complete general computing tasks, intelligent computing tasks, storage tasks, etc. Therefore, the beneficial effects of the present invention are as follows:

[0019] 1) The present invention uses edge computing and data localization processing technology to significantly reduce data transmission delays and improve real-time performance by bringing computing and storage resources closer to data sources.

[0020] 2) The present invention uses a data processing unit based on high-speed Ethernet and edge cloud technology to solve the problems of low bandwidth, high latency and poor reliability of traditional cloud computing.

[0021] 3) The present invention can support flexible sensor access, high-precision clock synchronization, and scalable and expandable system computing power, which can meet the needs and tasks in various scenarios and make full use of computing resources.

[0022] 4) The present invention uses RDMA technology to achieve the characteristics of high performance, strong real-time performance, low latency, low CPU occupancy and low power consumption.

[0023] 5) The present invention can realize the training and reasoning of intelligent tasks, and distributed storage, such as game confrontation, intelligent task interaction, data collection, etc.

[0024] 6) The switching module designed by the present invention can provide equipment protection and traffic load sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural block diagram of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0027] See also Figure 1 The structure of the specific embodiment of the edge cloud real-time data processing system based on multi-source sensors designed by the present invention is mainly composed of a multi-source sensor access terminal, a data exchange terminal and an edge cloud real-time data processing terminal. The multi-source sensor access terminal quickly transmits and forwards the collected photoelectric and radar sensor data to different nodes of the edge cloud real-time data processing terminal through the data exchange terminal to perform various types of storage or computing tasks. The main invention contents are described in detail as follows:

[0028] 1) Multi-source sensor access terminal design

[0029] The multi-source sensor access terminal supports flexible sensor access, including big data sensor access terminal, microservice network aggregation access terminal, etc. Among them, the data processing unit in the access terminal supports high-precision clock synchronization, the main purpose is to quickly synchronize and transmit large-scale data. The access terminal design covers applications in most current scenarios. The following is a detailed introduction.

[0030] For access scenarios with high data dimensions and large data scale, the big data sensor access end quickly processes and transmits data through the first data processing unit, and implements CPU zero copy through kernel bypass through a customized data transmission protocol, achieving low latency and high-performance transmission. For scenarios with small data dimensions but large data volumes, the microservice network aggregation access end accesses and processes data through a microprocessor, and then processes and transmits it through the second data processing unit, and finally reaches the data processing end.

[0031] 2) Data exchange end design

[0032] The data exchange end uses a switching unit to form a redundant backup, providing equipment protection and traffic load sharing. The data exchange end can support link aggregation of multiple devices, or support a switching device to enhance point-to-point time certainty. The data exchange end can use a switch.

[0033] 3) Edge cloud real-time data processing design

[0034] The edge cloud real-time data processing end is based on the third data processing unit and is interconnected with other control units, computing units, and storage units through different forms and different transmission protocols. The third data processing unit supports technologies such as DSAdirect, Multihost, RDMA, container hot start, and traffic scheduling, with lower network communication overhead, interaction delay, and bandwidth cost.

[0035] Taking the data collection process of a high-speed camera as an example, data transmission and processing are performed through the edge cloud real-time data processing system based on multi-source sensors of the present invention, and the high-speed transmission camera can collect images with a 4k resolution.

[0036] The specific processing flow is as follows:

[0037] 1) The high-speed camera collects image data with a resolution of 4k in real time.

[0038] 2) Rapidly transmit high-resolution data through multi-source sensor access terminals, and then encapsulate and process the data through the data processing unit.

[0039] 3) Data is forwarded and backed up through the data exchange terminal and sent to a specific node for processing.

[0040] 4) Send the data from the data exchange end to the edge cloud real-time data processing end for calculation and storage.

[0041] 5) The inference results can be fed back in real time to other nodes or control units at the edge cloud real-time data processing end.

[0042] The above are only specific embodiments disclosed in the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

[0043] The content of the present invention and the technical content not specifically described in the above embodiments are the same as the prior art.

[0044] The present invention is not limited to the above embodiments, and all the contents of the present invention can be implemented and have the above good effects.

Claims

1. An edge cloud real-time data processing system based on multi-source sensors, characterized by: The edge cloud real-time data processing system based on multi-source sensors includes a multi-source sensor access terminal, a data exchange terminal and an edge cloud real-time data processing terminal. The multi-source sensor access terminal transmits and forwards the collected photoelectric and radar sensor data to different nodes of the edge cloud real-time data processing terminal through the data exchange terminal to perform various types of storage or computing tasks.

2. The edge cloud real-time data processing system based on multi-source sensors according to claim 1 is characterized in that: The multi-source sensor access end includes a big data sensor access end and a microservice network aggregation access end. The big data sensor access end is used for access scenarios with high data dimension and large data scale, and the microservice network aggregation access end is used for scenarios with small data dimension but large volume. The big data sensor access end and the microservice network aggregation access end are respectively connected to the data exchange end.

3. The edge cloud real-time data processing system based on multi-source sensors according to claim 2 is characterized in that: The big data sensor access end includes a first data processing unit. For access scenarios with high data dimensions and large data scale, the first data processing unit is used to quickly process and transmit the data. A customized data transmission protocol is used to implement CPU zero copy through kernel bypass to achieve low latency and high-performance transmission. The microservice network aggregation access end includes a second data processing unit and a microprocessor. For scenarios with small data dimensions but large data volumes, the data will be accessed and processed through the microprocessor, and then processed and transmitted through the second data processing unit, and finally reach the edge cloud real-time data processing end.

4. The edge cloud real-time data processing system based on multi-source sensors according to claim 3 is characterized in that: The first data processing unit and the second data processing unit support high-precision clock synchronization.

5. The edge cloud real-time data processing system based on multi-source sensors according to any one of claims 2 to 4, characterized in that: The data exchange end uses a switching unit to form a redundant backup, providing equipment protection and traffic load sharing.

6. The edge cloud real-time data processing system based on multi-source sensors according to claim 5 is characterized in that: The data exchange end is a switch.

7. The edge cloud real-time data processing system based on multi-source sensors according to claim 6 is characterized in that: The edge cloud real-time data processing end includes a third data processing unit, a control unit, a computing unit and a storage unit. The edge cloud real-time data processing end takes the data processing unit as the core and is interconnected with the control unit, the computing unit and the storage unit through different forms and different transmission protocols.

8. The edge cloud real-time data processing system based on multi-source sensors according to claim 7 is characterized in that: The third data processing unit supports DSA direct, Multihost, RDMA, container hot start and traffic scheduling technology.

Images