Multi-source sensor data fusion control system and method

Abstract

The invention relates to the technical field of data processing, and provides a multi-source sensor data fusion control system and method, and the system comprises an operation resource layer which is used for providing a system operation environment; the data access layer is used for being in butt joint with the at least two sensors through a hardware and software development kit so as to obtain original data transmitted by the at least two sensors; the infrastructure layer is used for constructing a communication link in the system, and the communication link comprises an instruction transmission channel and a data transmission channel which are mutually independent; the service application layer is used for issuing a control instruction to the data access layer through an instruction transmission channel to control sensor hardware and receiving original data through a data transmission channel; the service application layer is also used for performing space-time synchronization and fusion processing on the original data to obtain target data; and the presentation layer is used for receiving the target data through a network protocol and performing visual display.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multi-source sensor data fusion control system and method. Background Technology

[0002] With the rapid development of technologies such as intelligent sensing, unmanned systems, autonomous driving, robotics, and industrial automation, multi-source fusion sensing systems have become a key means of acquiring environmental and equipment status information. Existing systems typically employ various types of sensors, including cameras, lidar, inertial navigation units (IMUs), and synchronous triggers, to collect environmental data from different dimensions. However, due to significant differences in the working principles, communication protocols, sampling frequencies, and data formats of different sensors, the data fusion processing between multi-source sensors faces considerable technical challenges. Summary of the Invention

[0003] In view of this, embodiments of this application provide a multi-source sensor data fusion control system and method to solve the problem that multi-source sensor data cannot be fused in the prior art.

[0004] A first aspect of this application provides a multi-source sensor data fusion control system, comprising: a runtime resource layer for providing a system operating environment; a data access layer for interfacing with at least two types of sensors via a hardware-software development kit (HDK), wherein the data access layer is configured with a sensor abstract interface, which is used to adapt to the private protocols corresponding to the at least two types of sensors through a unified base class function to obtain raw data transmitted by the at least two types of sensors; an infrastructure layer for constructing communication links within the system, wherein the communication links include mutually independent command transmission channels and data transmission channels; wherein the command transmission channel adopts a request-response communication protocol, and the data transmission channel adopts a publish-subscribe communication protocol; a service application layer for issuing control commands to the data access layer through the command transmission channel to control the sensor hardware, and receiving raw data through the data transmission channel; the service application layer is also used to perform spatiotemporal synchronization and fusion processing on the raw data to obtain target data; and a presentation layer for receiving the target data through a network protocol and performing visualization display. A second aspect of this application provides a multi-source sensor data fusion control method, which includes: interfacing with at least two sensors using a hardware software development kit and establishing a unified data acquisition channel for the at least two sensors using a sensor abstraction interface; issuing control commands to the at least two sensors through an instruction transmission channel to control the at least two sensors to start acquiring data, and receiving raw data uploaded by the at least two sensors through a data transmission channel; performing spatiotemporal synchronization processing and fusion processing on the raw data received through the data transmission channel to obtain target data, and visualizing the target data.

[0005] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0006] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0007] The beneficial effects of this application embodiment compared with the prior art are as follows: The system in this application embodiment includes: a runtime resource layer for providing a system operating environment; a data access layer for interfacing with at least two types of sensors through a hardware software development kit (HDK), wherein the data access layer is configured with a sensor abstract interface, which is used to adapt to the private protocols corresponding to at least two types of sensors through a unified base class function to obtain the raw data transmitted by at least two types of sensors; an infrastructure layer for constructing the internal communication link of the system, wherein the communication link includes mutually independent command transmission channels and data transmission channels; wherein the command transmission channel adopts a request-response communication protocol, and the data transmission channel adopts a publish-subscribe communication protocol; a service application layer for issuing control commands to the data access layer through the command transmission channel to control the sensor hardware, and receiving raw data through the data transmission channel; the service application layer is also used for spatiotemporal synchronization and fusion processing of the raw data to obtain target data; and a presentation layer for receiving the target data through a network protocol and performing visualization. It has the following beneficial effects: 1. The system achieves decoupling and unified management of the underlying hardware, significantly improving its scalability. By configuring a sensor abstraction interface at the data access layer, a unified base class function shields the differences in private SDKs between different manufacturers and sensor models (such as panoramic cameras and LiDAR). This eliminates the dependency of upper-layer business logic on specific hardware interfaces. When changing sensor hardware or adapting to domestically produced equipment, there is no need to refactor the upper-layer code; only the derived classes need to be extended at the lower level. This greatly reduces system coupling and maintenance costs, and solves the problem of system integration difficulties caused by hardware heterogeneity in existing technologies.

[0008] 2. Balancing the reliability of control commands with the efficiency of massive data transmission, the system's robustness is enhanced. An innovative dual-channel independent communication mechanism is constructed. Utilizing a command transmission channel employing a request-response model (such as ZeroMQ REQ-REP), the system ensures the delivery and confirmation of critical control signals (such as start / stop and configuration), preventing command loss due to network jitter. Simultaneously, a data transmission channel employing a publish-subscribe model (such as ROS Pub-Sub) leverages its high-concurrency distribution capabilities, enabling real-time transfer of large amounts of data such as laser point clouds and video streams. This design effectively resolves the contradiction between the limitations of traditional single communication protocols in simultaneously meeting the demands of "high-reliability control" and "high-throughput transmission."

[0009] 3. High-precision spatiotemporal synchronization and fusion of multi-source heterogeneous data has been achieved, improving the accuracy of environmental perception. The service application layer is not just a simple data forwarding layer, but also incorporates spatiotemporal synchronization and fusion processing logic. By using a unified timestamp (such as GNSS time) as a benchmark, the problems of inconsistent sampling frequencies (such as 1Hz and 10Hz) and time asynchrony between different sensors are solved. Combined with the fusion algorithm, high-quality target data containing position, attitude, and confidence level can be output, avoiding the technical difficulties in related technologies caused by spatiotemporal misalignment of multi-source sensor data, which leads to fusion failure or "ghosting".

[0010] 4. Clear hierarchical architecture facilitates modular development and fault localization. The system employs a layered architecture design (runtime resource layer, data access layer, infrastructure layer, service application layer, and presentation layer), with each layer having a single responsibility and clear boundaries. This architecture not only facilitates parallel development by the team but also enables rapid location of specific layers and modules in the event of system failures (such as network outages or sensor disconnections), improving system maintainability and industrial deployment capabilities.

[0011] The multi-source sensor data fusion control system provided in this application avoids the problem of multi-source sensor data not being able to be fused in related technologies. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a multi-source sensor data fusion control system provided in an embodiment of this application; Figure 2This is a flowchart illustrating another multi-source sensor data fusion control method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] A multi-source sensor data fusion control system and method according to embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0016] Figure 1 This is a basic schematic diagram of a multi-source sensor data fusion control system provided in an embodiment of this application, such as... Figure 1 As shown, the system includes: The runtime resource layer is used to provide the system's operating environment; The data access layer is used to interface with at least two types of sensors through the hardware SDK. The data access layer is configured with a sensor abstraction interface, which is used to adapt to the private protocols corresponding to at least two types of sensors through a unified base class function in order to obtain the raw data transmitted by at least two types of sensors. The infrastructure layer is used to build the internal communication links of the system. The communication links include independent command transmission channels and data transmission channels. The command transmission channel adopts a request-response communication protocol, and the data transmission channel adopts a publish-subscribe communication protocol. The service application layer is used to send control commands to the data access layer through the command transmission channel to control the sensor hardware, and to receive raw data through the data transmission channel; the service application layer is also used to perform spatiotemporal synchronization and fusion processing on the raw data to obtain the target data; The presentation layer is used to receive target data via network protocols and to visualize it.

[0017] Specifically, such as Figure 1 As shown, the runtime resource layer is located at the bottom layer of the system and is used to provide the system runtime environment required for software operation. Depending on the actual deployment requirements, this runtime resource layer is compatible with multiple operating system platforms, including common Windows systems, Linux systems (such as Ubuntu 20.04 LTS), and Kylin / Tongxin UOS, providing cross-platform underlying support for upper-layer services.

[0018] The data access layer is primarily responsible for shielding the differences in the underlying hardware, enabling unified access to heterogeneous sensors. For example... Figure 1 As shown, this layer interacts with the physical hardware by calling the hardware software development kits (SDKs) provided by various sensor manufacturers—such as camera SDKs, laser SDKs, inertial navigation SDKs, and synchronizer SDKs. To address the issue of inconsistent interface definitions across different manufacturers' SDKs, this embodiment constructs a sensor abstract interface (corresponding to the various SDK calling modules in the diagram) at the data access layer. The sensor abstract interface is used to adapt the private protocols corresponding to at least two types of sensors through a unified base class function, in order to obtain the raw data transmitted by at least two types of sensors.

[0019] Specifically, in a preferred embodiment of this application, the sensor abstract interface is implemented based on the polymorphic mechanism of object-oriented programming (OOP). The system defines a unified abstract base class (e.g., named SensorBase), which declares a series of pure virtual functions. These functions constitute the common operating standards that all sensors must follow. The unified base class functions include, but are not limited to: a device self-check function for detecting the online status of the device (corresponding to selfCheck() in the code), a device initialization function for establishing device connections (corresponding to initDevice() in the code), a data acquisition start function for starting data stream transmission (corresponding to startGrab() in the code), a data acquisition stop function for ending transmission (corresponding to stopGrab() in the code), and a parameter configuration function for setting the resolution or frame rate (corresponding to setConfig() in the code).

[0020] To adapt to different types or manufacturers of sensors, the system derives specific subclasses from an abstract base class. For example, for a specific panoramic camera model, the system defines a derived class (e.g., named YTH_lishi_Pano). Within this derived class, developers override the base class functions, internally calling the private SDK interfaces provided by the panoramic camera manufacturer (such as private initialization instructions or stream extraction instructions from company A or company B). Through this design, the device management module in the upper-layer application layer only needs to hold a pointer to a SensorBase type to uniformly control various sensors, eliminating the need to write hardware-specific if-else conditional logic in the business logic, thus significantly reducing system coupling.

[0021] In terms of communication link construction, such as Figure 1As shown in the "Network Communication" module, this embodiment creatively constructs a system communication link based on a multi-channel independent communication mechanism. This communication link includes mutually isolated command transmission channels and data transmission channels, each performing its own function, effectively resolving the contradiction between control stability and data throughput.

[0022] The command transmission channel is primarily used for transmitting low-frequency but highly reliable signaling such as equipment control (e.g., start / stop), parameter configuration (e.g., resolution modification), and status queries. In a preferred embodiment of this application, the system employs the ZeroMQ (ZMQ) message queue framework, constructing the channel based on a request-response (REQ-REP) pattern. The advantage of using the REQ-REP pattern is that it ensures that every control command (e.g., "start acquisition") issued by the client receives an explicit acknowledgment (ACK) from the server, preventing command loss during network fluctuations and thus guaranteeing the absolute stability of system control. It should be noted that this application does not strictly limit the specific implementation protocol of the command transmission channel. In addition to the preferred ZeroMQ framework described above, in other embodiments, the command transmission channel can also be constructed based on at least one of the standard TCP / IP socket or gRPC (Google Remote Procedure Call) frameworks, as long as it can satisfy the reliable interaction logic of "request-acknowledgment".

[0023] This data transmission channel is primarily used to transmit high-frequency, high-throughput raw sensor data such as laser point clouds, panoramic video, and inertial navigation pose data. In a preferred embodiment of this application, the system employs the communication mechanism of the Robot Operating System (ROS), constructing the channel based on a publish-subscriber pattern. Leveraging ROS's efficient message distribution capabilities and node decoupling characteristics, real-time flow and concurrent processing of massive amounts of data between different algorithm nodes (such as preprocessing nodes and fusion nodes) can be achieved, avoiding the complexity of point-to-point connections. Similarly, this application does not strictly limit the specific implementation protocol of the data transmission channel. Besides the preferred ROS communication protocol described above, in other embodiments, the data transmission channel can also be constructed based on at least one of the LCM (Lightweight Communications and Marshalling) communication protocol and the publish-subscribe pattern of the ZeroMQ framework, as long as it can meet the real-time distribution requirements of high-concurrency data.

[0024] like Figure 1 As shown, this infrastructure layer also encapsulates common tools such as thread pools, memory pools, AES encryption algorithms, Base64 encoding and decoding, JSON parsing libraries, and Lzma compression algorithms, providing underlying support for upper-layer business applications.

[0025] The multi-source sensor data fusion control system also includes a service application layer (containing data processing logic). This service application layer is the "brain" of the system, executing specific business logic based on the interfaces provided by the infrastructure layer. The service application layer is primarily responsible for the system's control logic scheduling and state management, playing a pivotal role in connecting the upper and lower layers. For example... Figure 1 As shown, this layer contains core modules such as device management, message management, and simulation services.

[0026] The message management module is primarily responsible for connecting to the "command transmission channel" of the infrastructure layer. When ZMQ receives a JSON-formatted control command from the front end, the message interpretation unit parses the command, and the message distribution unit dispatches the parsed task to the corresponding processing unit.

[0027] Device Management Module: Responds to instructions distributed by the message management module and is responsible for the full lifecycle management of sensor hardware. Specifically, this includes device loading (loading SO libraries), device operation (calling the SensorBase interface of the data access layer to perform initialization, start and stop), device parameters (configuring resolution and frame rate), and device policies (setting acquisition modes).

[0028] Simulation service module: When no real hardware is connected, local data can be replayed through units such as Camera simulation and PTZ simulation for development and debugging.

[0029] The service application layer also includes a data processing layer, which contains modules such as spatiotemporal calibration, data cleaning, data fusion, and AI-powered intelligent recognition. Data cleaning: The system subscribes to the raw data stream uploaded by the data access layer via ROS. First, it enters the data cleaning module to perform distortion correction on the panoramic image and denoise and distortion removal on the laser point cloud.

[0030] Spatiotemporal calibration: Since each sensor operates independently, this module uses the high-precision timestamp of the Global Navigation Satellite System (GNSS) as a unified benchmark. Utilizing time synchronization algorithms (such as sliding window matching), and based on the high-precision timestamp of GNSS, a time synchronizer (such as ROS's message_filters) is used to align image data and point cloud data of different frequencies along the time axis, transforming them to the same coordinate system.

[0031] Data fusion: This is the core function of this layer. This module uses a strategy that combines weighted fusion and Kalman filtering to perform state estimation calculations on time-aligned image data and point cloud data, and calculate high-precision positioning information and environmental perception results (i.e., the aforementioned "target data").

[0032] AI-powered intelligent recognition and localization matching: Based on the fusion of data, further target detection algorithms (such as vehicle and pedestrian recognition) and SLAM localization matching are executed to output structured perception results.

[0033] The service application layer also supports parallel processing of data distribution paths: the service application layer directly stores the raw data to the local file system through the first path; the service application layer sends the parsed raw data to the external client via the HTTP protocol through the second path. Specifically, for the first path, the service application layer mainly utilizes the "data buffer pool" and "Lzma compression algorithm" components (such as...) provided by the infrastructure layer. Figure 1 The system is implemented as shown in the basic components. When massive amounts of raw data (such as high-resolution panoramic video streams or high-frequency laser point clouds) are input, the system first writes them into a circular data buffer in memory to avoid frequent disk I / O operations blocking the main thread. Subsequently, the background storage thread reads the data from the buffer, calls the "Lzma compression algorithm" to encode the data with a high compression ratio, and finally persists the compressed data as a file to the local file system (such as a Linux file directory). This path is mainly used for subsequent data playback, offline algorithm debugging, and accident tracing.

[0034] For the second path, the service application layer mainly utilizes the "interface service" module provided by the infrastructure layer (such as...). Figure 1 The system implements the RESTful interface shown. First, it performs protocol parsing on the raw data (e.g., parsing a private binary stream into general structured data), then encapsulates it into standard HTTP messages (such as JSON or binary stream format). The application layer calls the "RESTful" interface service, pushing the parsed data in real-time to a configured external client (such as a third-party cloud platform or remote monitoring center) via HTTP POST / GET requests. This path is primarily used to meet the real-time monitoring and business integration needs of external systems for on-site data.

[0035] The multi-source sensor data fusion control system also includes a presentation layer, which is responsible for user interaction and data visualization. For example... Figure 1 As shown, this presentation layer is the direct window for user interaction with the system, responsible for data visualization and human-computer interaction control. For example... Figure 1 As shown at the top layer, this layer specifically includes functional modules such as data monitoring and display, human-machine control and interaction, data analysis and alarm, target measurement, and real-time mapping.

[0036] In a typical embodiment of this application, the presentation layer adopts an architecture design of "Web frontend + backend service middleware," with the two working together to implement the issuance of instructions and the reporting of data. Specifically, the Web frontend (corresponding to...) Figure 1The "Human-Computer Interaction" module responds to user actions (such as clicking the "Start Collection" button) and triggers control commands. Format conversion: The backend service middleware (e.g., written in Python or Node.js and running on the server) receives control commands from the frontend. To adapt to the communication specifications of the infrastructure layer, the middleware encapsulates the commands into standard JSON format data packets (e.g., containing fields like { "cmd": "start_grab", "device_id": "cam_01"}).

[0037] Command transmission: The middleware, acting as a client, sends the encapsulated JSON data packet to the service application layer through the aforementioned "command transmission channel" (i.e., ZeroMQ's REQ-REP channel).

[0038] Command parsing and execution: The service application layer is internally configured with a command parsing module (corresponding to...). Figure 1 The "Message Interpretation" unit in the service application layer receives a JSON data packet, parses its fields, extracts the operation type and target device, and calls the corresponding sensor abstraction interface in the data access layer (such as calling the startGrab() function) based on the parsing results, thereby completing the control of the physical hardware.

[0039] The "target data" processed by the server (including the fused high-precision pose, the distortion-free point cloud map, the real-time mapping results, and the target measurement data) is transmitted back to the presentation layer in real time via an efficient network protocol (preferably WebSocket full-duplex communication or HTTP stream).

[0040] The web front-end uses a graphics rendering engine (such as WebGL or Canvas) to render the above data in real time on the browser interface. At the same time, the "Data Analysis and Alarm" module monitors the data status in real time, and when the sensor data is abnormal (such as frame loss or timestamp jump), an alarm prompt will pop up on the interface.

[0041] In summary, through the aforementioned layered architecture and dual-channel communication design, this system effectively resolves the contradiction between "unreliable control commands" and "massive data congestion" in multi-source sensor integration, achieving efficient and stable system operation.

[0042] The system in this embodiment includes: a runtime resource layer for providing the system's operating environment; a data access layer for interfacing with at least two types of sensors via a hardware / software development kit (HDK), the data access layer being configured with a sensor abstraction interface (SIA), which adapts to the private protocols corresponding to the at least two types of sensors through a unified base class function to obtain raw data transmitted by the at least two types of sensors; an infrastructure layer for constructing internal communication links, the communication links including independent command transmission channels and data transmission channels; wherein the command transmission channel adopts a request-response communication protocol, and the data transmission channel adopts a publish-subscribe communication protocol; a service application layer for issuing control commands to the data access layer via the command transmission channel to control the sensor hardware, and receiving raw data via the data transmission channel; the service application layer is also used for spatiotemporal synchronization and fusion processing of the raw data to obtain target data; and a presentation layer for receiving the target data via a network protocol and performing visualization. It has the following beneficial effects: 1. The system achieves decoupling and unified management of the underlying hardware, significantly improving its scalability. By configuring a sensor abstraction interface at the data access layer, a unified base class function shields the differences in private SDKs between different manufacturers and sensor models (such as panoramic cameras and LiDAR). This eliminates the dependency of upper-layer business logic on specific hardware interfaces. When changing sensor hardware or adapting to domestically produced equipment, there is no need to refactor the upper-layer code; only the derived classes need to be extended at the lower level. This greatly reduces system coupling and maintenance costs, and solves the problem of system integration difficulties caused by hardware heterogeneity in existing technologies.

[0043] 2. Balancing the reliability of control commands with the efficiency of massive data transmission, the system's robustness is enhanced. An innovative dual-channel independent communication mechanism is constructed. Utilizing a command transmission channel employing a request-response model (such as ZeroMQ REQ-REP), the system ensures the delivery and confirmation of critical control signals (such as start / stop and configuration), preventing command loss due to network jitter. Simultaneously, a data transmission channel employing a publish-subscribe model (such as ROS Pub-Sub) leverages its high-concurrency distribution capabilities, enabling real-time transfer of large amounts of data such as laser point clouds and video streams. This design effectively resolves the contradiction between the limitations of traditional single communication protocols in simultaneously meeting the demands of "high-reliability control" and "high-throughput transmission."

[0044] 3. High-precision spatiotemporal synchronization and fusion of multi-source heterogeneous data has been achieved, improving the accuracy of environmental perception. The service application layer is not just a simple data forwarding layer, but also incorporates spatiotemporal synchronization and fusion processing logic. By using a unified timestamp (such as GNSS time) as a benchmark, the problems of inconsistent sampling frequencies (such as 1Hz and 10Hz) and time asynchrony between different sensors are solved. Combined with the fusion algorithm, high-quality target data containing position, attitude, and confidence level can be output, avoiding the technical difficulties in related technologies caused by spatiotemporal misalignment of multi-source sensor data, which leads to fusion failure or "ghosting".

[0045] 4. Clear hierarchical architecture facilitates modular development and fault localization. The system employs a layered architecture design (runtime resource layer, data access layer, infrastructure layer, service application layer, and presentation layer), with each layer having a single responsibility and clear boundaries. This architecture not only facilitates parallel development by the team but also enables rapid location of specific layers and modules in the event of system failures (such as network outages or sensor disconnections), improving system maintainability and industrial deployment capabilities.

[0046] The multi-source sensor data fusion control system provided in this application avoids the problem of multi-source sensor data not being able to be fused in related technologies.

[0047] Based on the same concept, this application also provides a multi-source sensor data fusion control method. The multi-source sensor data fusion control method provided in this embodiment is applied to the multi-source sensor data fusion control system described in the above embodiment, such as... Figure 2 As shown, the method includes: S201. Connect to at least two types of sensors using a hardware and software development kit, and establish a unified data acquisition channel for at least two types of sensors using a sensor abstraction interface; The system interfaces with at least two types of sensors (such as panoramic cameras and LiDAR) using a hardware software development kit (SDK). To shield the underlying differences, a pre-configured sensor abstraction interface (such as the SensorBase class) is used, and a unified base class function adapts to the private protocols corresponding to each sensor. In this step, the system completes the mapping from the "private SDK" to the "unified access mechanism" and establishes a standardized data acquisition channel.

[0048] S202. Send control commands to at least two types of sensors through the command transmission channel to control at least two types of sensors to start collecting data, and receive raw data uploaded by at least two types of sensors through the data transmission channel; Control commands (such as initialization and start data acquisition) are sent to the sensor via a command transmission channel (based on a request-response mode). The sensor responds to the command and begins operation, uploading raw data via a data transmission channel (based on a publish-subscribe mode). This process achieves parallel transmission of both "commands without packet loss" and "high data throughput."

[0049] S203. Perform spatiotemporal synchronization and fusion processing on the raw data received through the data transmission channel to obtain the target data, and then visualize the target data.

[0050] The received raw data undergoes spatiotemporal synchronization processing (based on GNSS time) and fusion processing (such as Kalman filtering) to obtain target data containing pose and confidence, which is then visualized in the presentation layer.

[0051] To enable those skilled in the art to more clearly understand the technical solution of this application, the above process will be described in detail below with reference to a specific preferred embodiment: 1. Preferred Operating Environment This embodiment is built on an embedded system. The preferred hardware platform is a computing unit based on an ARM Cortex-A9 processor (or a higher-performance domestically produced chip), with a memory configuration of no less than 8GB DDR3. The software environment uses the Ubuntu 20.04 LTS operating system, with ROS Noetic as the middleware, and ZeroMQ (ZMQ) version 4.3.2 as the message queue component.

[0052] 2. Specific implementation of the core modules.

[0053] Sensor control library (data access layer): Written in C++, its core lies in the definition of the abstract base class SensorBase. This base class contains pure virtual functions such as selfCheck() (self-check), initDevice() (initialization), startGrab() (start acquisition), stopGrab() (stop acquisition), and setConfig() (configuration). For specific hardware, such as a "historical panoramic camera," the system derives the YTH_lishi_Pano class, overrides the above base class functions, and calls the vendor's SDK's private communication protocol to achieve polymorphic control.

[0054] Transport protocol encapsulation (infrastructure layer): Command library: It adopts ZMQ's REQ-REP (request-response) pattern and encapsulates the sendCommand() and recvResponse() interfaces to ensure that every control command (such as initialization command) receives a clear response, thus guaranteeing reliability.

[0055] Database: The system adopts the ROS Publisher-Subscriber pattern. A conversion adapter has been written to convert raw data from different manufacturers (such as proprietary streams from Hikvision cameras) into ROS-compatible message types, such as sensor_msgs::Image or geometry_msgs::Pose, to facilitate unified processing by subsequent nodes.

[0056] Data Fusion Module (Service Application Layer): This module is developed based on a custom ROS node, sensor_fusion_node, and executes a strict three-level process of "preprocessing-synchronization-fusion". Preprocessing: Subscribe to multi-source topics under the / sensor_data / prefix, perform distortion correction on LiDAR point clouds, perform distortion correction on panoramic images, and perform jump filtering on GNSS data.

[0057] Time synchronization: Using the message_filters::TimeSynchronizer class in ROS, the GNSS timestamp (1Hz) is used as the "master clock" to interpolate and align the high-frequency panoramic image (10Hz) and the LiDAR point cloud (10Hz) to solve the time deviation caused by the difference in sampling frequency of multiple sensors.

[0058] Fusion Algorithm: For environmental perception scenarios, a combined strategy of "weighted fusion + Kalman filtering" is adopted. The output is encapsulated into a custom ROS message type fusion_msgs::FusedPose (containing timestamp, x / y / z coordinates, heading angle, and fusion confidence), and published to the topic / fusion_data / fused_pose for upper-layer calls.

[0059] Client-side interaction flow (presentation layer): Taking panoramic camera initialization as an example: The web frontend sends an initialization command via JavaScript -> the Python backend receives it and converts it into JSON format -> it is sent to the C++ server via ZMQ -> the server parses the command and calls initDevice() -> the execution result is returned along the same path and displayed in a pop-up window on the frontend.

[0060] 3. Data flow closed loop: As shown in the figure, the system realizes a complete closed loop from data acquisition to multi-target distribution: After the data acquired by the sensors (CCD, laser, inertial navigation) enters the system, it is divided into two parallel paths: Path 1 (Local Archive): The raw data is compressed and saved to the local file system for offline analysis.

[0061] Path 2 (Real-time Distribution): After parsing, the raw data is sent to ROS / LCM shared memory. The parsed data is further distributed: Internal distribution: The data is sent back to the front-end interface for real-time preview (such as video streaming) via a backend service and HTTP protocol.

[0062] External distribution: Directly send structured data (such as fused location coordinates) to the cloud platform of external customers via the HTTP protocol.

[0063] 4. Adaptation to Domestic Scenarios (Industrial Applications) It is worth mentioning that the architecture proposed in this application has strong portability and can seamlessly adapt to domestic embedded hardware and software environments (such as Kylin operating system and Phytium / Loongson processors), solving the core pain point of multi-sensor integration in domestic embedded systems. Specific application scenarios include: Domestically produced surveying and mapping equipment: compatible with domestic lidar and BeiDou combined inertial navigation, enabling high-precision autonomous acquisition of geographic information.

[0064] Domestically produced intelligent robots: Support domestically produced vision sensors and ultrasonic sensors to meet the autonomous environmental perception needs of industrial robots.

[0065] Domestic autonomous driving: compatible with domestic millimeter-wave radar and CCD camera to achieve real-time monitoring of the vehicle's surrounding environment.

[0066] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0067] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0068] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above.

[0069] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0070] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0071] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0073] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0074] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-source sensor data fusion control system, characterized by, The system comprises: a running resource layer for providing a system running environment; a data access layer for interfacing at least two sensors through a hardware software development kit, the data access layer being configured with a sensor abstract interface for adapting private protocols respectively corresponding to the at least two sensors through a unified base class function to obtain raw data transmitted by the at least two sensors; an infrastructure layer for constructing a communication link within the system, the communication link comprising a mutually independent instruction transmission channel and a data transmission channel; wherein the instruction transmission channel adopts a communication protocol in a request-response mode, and the data transmission channel adopts a communication protocol in a publish-subscribe mode; a service application layer for issuing control instructions to the data access layer through the instruction transmission channel to control sensor hardware, and receiving the raw data through the data transmission channel; the service application layer is further configured to perform spatio-temporal synchronization and fusion processing on the raw data to obtain target data; a presentation layer for receiving the target data through a network protocol and performing visual display.

2. The system of claim 1, wherein, The instruction transmission channel is constructed based on at least one of a TCP / IP socket, a gRPC framework, and a ZeroMQ framework in a request-response mode. The data transmission channel is constructed based on at least one of an LCM communication protocol, a ROS communication protocol, and a ZeroMQ framework in a publish-subscribe mode.

3. The system of claim 1, wherein, The unified base class function at least includes a device self-check function, a device initialization function, a data acquisition start function, a data acquisition stop function, and a parameter configuration function.

4. The system of claim 1, wherein, The spatio-temporal synchronization and fusion processing on the raw data to obtain target data comprises: performing distortion correction on image data in the raw data, and performing de-distortion processing on point cloud data in the raw data; taking a global navigation satellite system timestamp as a reference, using a time synchronizer to time-align the image data and the point cloud data of different frequencies, and generating the target data based on the time-aligned image data and point cloud data.

5. The system of claim 4, wherein, Generating the target data based on the time-aligned image data and point cloud data comprises: using a strategy combining weighted fusion and Kalman filtering to perform state estimation calculation on the time-aligned image data and point cloud data; and obtaining the target data.

6. The system of claim 1, wherein, The service application layer is further configured to support parallel processing of data distribution paths: the service application layer directly stores the raw data to a local file system through a first path; and the service application layer sends the parsed raw data to an external client through an HTTP protocol through a second path.

7. The system of claim 1, wherein, The presentation layer comprises a web front end and a background service middleware; the web front end is used to send control instructions in response to user operations; the background service middleware is used to receive the control instructions, convert the control instructions into data packets in JSON format, and send the data packets to the service application layer through an instruction transmission channel; the service application layer is configured with an instruction analysis module, which is used to analyze the data packets in JSON format and call the sensor abstract interface according to the analysis result.

8. A multi-source sensor data fusion control method, characterized by, The method comprises: At least two sensors are connected through a hardware software development kit, and a unified data acquisition channel is established for the at least two sensors by using a sensor abstract interface; Control instructions are sent to the at least two sensors through an instruction transmission channel to control the at least two sensors to start collecting data, and raw data uploaded by the at least two sensors is received through a data transmission channel; The raw data received through the data transmission channel is subjected to spatio-temporal synchronization processing and fusion processing to obtain target data, and the target data is subjected to visual display.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of claim 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the method of claim 8.

Images