Multi-modal sensor abstract description and packaging method and system based on software definition

Abstract

The invention discloses a multi-modal sensor abstract description and packaging method and system based on software definition. Firstly, a unified sensor abstract description model based on software definition is introduced, so that hardware independence is realized, and the differences of a bottom-layer multi-mode sensor in the aspects of interfaces, protocols and data formats can be effectively shielded; secondly, abstract packaging of data is achieved through an autonomous information processing module, and high-bandwidth and high-redundancy original data is converted into low-bandwidth and high-value structured information; in addition, by means of automatic registration, discovery, scheduling and control mechanisms of the sensor management control module on sensor resources, the system realizes a real plug-and-play function. According to the invention, by deploying the autonomous information processing modules at various sensor nodes, the original sensing data is preprocessed, and standardized data packaging with semantic consistency is output, so that the operation load of the central controller is reduced, and the information processing efficiency and the system response capability are also improved.

Description

Technical Field

[0001] This invention relates to the fields of sensor technology, Internet of Things and artificial intelligence, and in particular to a software-defined method and system for abstract description and encapsulation of multimodal sensors. Background Technology

[0002] With the deepening development of intelligent systems (such as autonomous driving, intelligent inspection, and service robots), their requirements for environmental perception capabilities have become unprecedentedly complex. This typically necessitates the integration of multiple sensor modalities, including LiDAR, vision cameras, infrared thermal imagers, sonar, and gas sensors. These sensors originate from different manufacturers, exhibiting significant differences in hardware interfaces (such as USB, RS485, Ethernet, and CAN), communication protocols, and data formats. This results in high complexity during system integration, long development cycles, and severely limited scalability.

[0003] Currently, the mainstream sensor integration methods can be summarized into two categories:

[0004] One approach is the tightly coupled integration method based on traditional hardware drivers. This method requires the central controller to develop dedicated drivers and data parsing libraries for each specific sensor model. For example, when integrating a new USB interface industrial camera, developers must use the manufacturer's dedicated SDK (Software Development Kit) for device initialization, parameter configuration, and image data acquisition; while integrating a gas sensor using an RS485 interface requires writing code for splicing and parsing command frames according to its proprietary communication protocol, and handling low-level details such as CRC checks. This approach results in a system architecture filled with numerous repetitive and heterogeneous code modules, with each module deeply bound to specific hardware, leading to a cumbersome system with low maintainability. When a sensor needs to be upgraded or replaced, even if the old and new sensors have identical functions, it often requires re-adapting the driver, modifying the central processing logic, and even adjusting the human-machine interface, resulting in high system upgrade costs and lengthy cycles.

[0005] The second approach is integration based on certain common industry standard protocols (such as OPC UA and MQTT). While this approach achieves a certain degree of interface unification at the data transmission level, it does not fundamentally solve the semantic consistency and plug-and-play issues of sensor functional descriptions. Taking OPC UA as an example, although it provides an information modeling framework, system integrators still need to manually create information models for different sensors, and it cannot automatically understand the functional equivalence between "LiDAR point cloud data" and "depth camera 3D information." Furthermore, such solutions generally lack the ability to uniformly manage the entire sensor lifecycle (such as registration, discovery, hibernation, and wake-up), and fail to address the crucial aspects of intelligent preprocessing and standardized encapsulation at the data source. Their "plug-and-play" experience is mostly limited to network connection discovery, rather than automatic identification and integration at the functional semantic level.

[0006] Therefore, existing technologies fail to provide an end-to-end solution that can uniformly abstractly describe sensors at the functional semantic level and automatically complete data preprocessing and encapsulation. Currently, traditional sensor integration methods mostly employ one-to-one customized driver development, lacking a unified abstract description model. Controllers need to write specific data parsing and processing code for each specific sensor, resulting in high system coupling, poor robustness, and difficulty in maintenance and upgrades. When sensors need to be replaced or added, most of the software code often needs to be redeveloped, failing to achieve true plug-and-play functionality.

[0007] In summary, there is an urgent need in this field for a systematic solution that can fundamentally shield the differences in underlying sensor hardware, provide unified and standardized data interfaces and services for upper-layer application algorithms, and support plug-and-play functionality at the semantic level. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a software-defined multimodal sensor abstract description and encapsulation method and system. By constructing a unified sensor abstract description model, designing an autonomous information processing module, realizing multi-source data fusion, and implementing centralized sensor resource management, this invention completely decouples upper-layer application algorithms from lower-layer sensor hardware, significantly improving the flexibility, scalability, and reliability of intelligent sensing systems.

[0009] The software-defined multimodal sensor abstraction description and encapsulation method of this invention includes the following steps:

[0010] Step a) Based on the Resource Description Framework (RDF), generate a unified sensor abstract description file for each sensor. The description file is in XML format and includes an ID for globally uniquely identifying the sensor, an OperationID for characterizing its function rather than its hardware type, a State for characterizing its real-time operating state, and a Config for carrying its static configurable parameters and metadata.

[0011] Step b) Deploy a corresponding autonomous information processing module for each sensor or each type of sensor. The autonomous information processing module receives raw data through the sensor's own hardware interface, and sequentially completes protocol parsing, data cleaning and unit standardization, semantic information extraction, and data encapsulation in a unified standard format. The encapsulated data is then uploaded through a unified communication interface.

[0012] Step c) Set up a data fusion and decision-making module in the intelligent control layer to receive and fuse standardized data from their respective main information processing modules to generate a joint estimate of the environmental state;

[0013] Step d) Set up a sensor management and control module in the intelligent control layer. By reading the sensor abstract description file, the module can automatically complete the registration, discovery, scheduling and control of the sensor, and achieve plug-and-play functionality at the semantic level.

[0014] The software-defined multimodal sensor abstraction description and encapsulation system of this invention includes:

[0015] The sensor abstract description unit is used to store and manage sensor description files based on the Resource Description Framework (RDF) and carried by XML. The description file includes at least the sensor's globally unique ID, function OperationID, real-time state State, and static configuration Config.

[0016] The autonomous information processing module corresponds one-to-one with the sensor or is shared by category. It is used to receive raw data through the hardware interface of the sensor, and sequentially complete protocol parsing, data cleaning and unit standardization, semantic information extraction and unified standard format encapsulation, and output standardized data packets through a unified communication interface.

[0017] The data fusion and decision-making module, located in the intelligent control layer, is used to receive and fuse standardized data packets from their respective main information processing modules to generate a joint estimate of the environmental state.

[0018] The sensor management and control module, located in the intelligent control layer, is used to read the description file of a sensor when it is connected to automatically complete registration, discovery, scheduling and control, achieving plug-and-play functionality at the semantic level.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] First, this invention fundamentally achieves hardware independence by introducing a unified, software-defined sensor abstraction description model. This effectively shields the differences in interfaces, protocols, and data formats among underlying multimodal sensors, significantly reducing the complexity of system integration and adaptation. Second, through an autonomous information processing module, data abstraction and encapsulation are achieved, transforming high-bandwidth, highly redundant raw data into low-bandwidth, high-value structured information, improving data consistency and processability. Furthermore, with the sensor management and control module's automatic registration, discovery, scheduling, and control mechanism for sensor resources, the system achieves true plug-and-play functionality, significantly improving the overall architecture's scalability and maintainability, and enabling flexible responses to the dynamic addition or removal of sensor types.

[0021] Furthermore, by deploying autonomous information processing modules on various sensor nodes, the raw sensor data is preprocessed and output as semantically consistent standardized data encapsulation. This not only reduces the computational load on the central controller but also improves information processing efficiency and system responsiveness. Based on this, the system generates a more comprehensive and accurate environmental state estimate through deep fusion processing of multi-source heterogeneous sensor data, thereby significantly enhancing the robustness and reliability of the intelligent decision-making module.

[0022] Ultimately, the method proposed in this invention demonstrates excellent versatility and industry adaptability, and can be widely applied in intelligent robots, security monitoring, industrial IoT, and other technical fields that rely on complex multimodal perception, exhibiting outstanding practical value and promising prospects for promotion. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application example of the method described in this invention in an intelligent safety inspection robot system.

[0025] Figure 2 This is a block diagram of the overall architecture of the system hierarchy described in this invention.

[0026] Figure 3 This is a schematic diagram of the unified sensor abstract description model (RDF quadruple) of the method described in this invention.

[0027] Figure 4This is a data flow diagram of how the autonomous information processing module of the present invention processes sonar sensor signals. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This application provides a software-defined method for abstracting and encapsulating multimodal sensors, specifically including the following steps:

[0030] Step 1: Create a unified sensor abstract description model for various sensors in the physical layer based on the Resource Description Framework (RDF).

[0031] Furthermore, step 1 specifically includes the following steps:

[0032] Step 1.1, Model Framework Selection: The Resource Description Framework (RDF) is adopted as the modeling foundation, and Extensible Markup Language (XML) is selected as the specific syntax implementation carrier. At the same time, the RDF schema (RDFS) specification is strictly followed to predefine the classes, properties, their hierarchical relationships and constraints used in the model.

[0033] Step 1.2, Constructing the Abstract Description Tuple: The core of the unified sensor abstract description model is a four-tuple structure, defined as: RDF Machine={ID,OperationID,State,Config}. This four-tuple is constructed based on the RDF triple (subject-predicate-object), where each element is an RDF resource or literal, and its specific meaning and implementation are as follows:

[0034] ID (Device Unique Identifier): This element serves as a globally unique identifier for a sensor instance, and is essentially an RDF resource defined in the Uniform Resource Identifier (URI) format.

[0035] OperationID (Function Identifier): This element is key to achieving "hardware independence." It describes the function or service provided by the sensor, rather than its specific hardware type.

[0036] State (Device Status): This element is used to dynamically characterize the real-time operating status of the sensor. Its value is an enumerated literal constrained by RDFS, such as Online, Offline, Busy, etc.

[0037] Config (Device Configuration Attribute): This element is a container that describes all static configurable attributes and metadata of the sensor.

[0038] Step 1.3, Sensor Description File Generation and Usage: For each type and even each physical sensor, a corresponding XML description file is generated according to its actual capabilities and parameters, following the RDF four-tuple model described above. This file is stored in the system's sensor abstract description unit. When a sensor connects to the network, the sensor management and control module reads and parses this file to obtain the sensor's identity (ID), function (OperationID), state, and configuration attributes (Config), thereby automatically completing resource integration and scheduling without needing to write dedicated driver code for specific sensors.

[0039] Step 2: For each type of sensor, design an autonomous information processing module to abstract and encapsulate the raw sensor data. This module is the core connecting the physical sensor and the upper-layer application algorithm, responsible for transforming the raw, heterogeneous, high-bandwidth, and highly redundant low-level data into a lightweight, semantically clear, and standard-format information encapsulation.

[0040] This step specifically includes the following sub-steps:

[0041] Step 2.1, Raw Data Access and Protocol Parsing: The autonomous information processing module first receives the raw data stream through the sensor's specific hardware interface. The module has built-in parsers for different communication protocols, which extract the valid sensor data payload from the data stream according to a predefined frame structure.

[0042] Step 2.2, Data Preprocessing and Standardization: The parsed raw data is cleaned and initially screened to remove outliers or invalid data packets caused by transmission errors. Subsequently, unit conversions for physical quantities are performed (e.g., converting Fahrenheit to Celsius and feet to meters) and data type standardization is performed (e.g., converting 16-bit signed integers to 32-bit floating-point numbers).

[0043] Step 2.3, Perceptual Analysis and Semantic Information Extraction: In this core step, the module runs a built-in specific algorithm model to extract high-level semantic information from the preprocessed data that can be directly used for decision-making.

[0044] Step 2.4, Standardized Data Encapsulation: The semantic information extracted in Step 2.3, along with necessary metadata (such as timestamps and data source sensor IDs), is encapsulated according to a predefined, hardware-independent standard data format.

[0045] Step 2.5, Communication Interface Adaptation and Upload: The encapsulated standard data packet is uploaded to the data fusion and decision-making module of the intelligent control layer through a unified communication interface (such as MQTT topic publishing, HTTP RESTful API push, gRPC call, etc.).

[0046] Step 3: In the intelligent control layer, a data fusion and decision-making module is designed to receive standard data from various autonomous information processing modules. Data fusion algorithms, such as weighted average, Kalman filtering, multi-Bayes estimation, neural networks, or rough set theory, are used to deeply fuse the multi-source heterogeneous standard data, generating a more accurate and comprehensive joint estimate of the environmental state to support decision-making in upper-layer application algorithms.

[0047] Step 4: In the intelligent control layer, design a sensor management and control module. This module provides sensor registration, discovery, scheduling, and control functions; when a new sensor is connected to the system, it automatically completes registration by reading its sensor description file, and the system can automatically identify and schedule the sensor resource, thereby realizing plug-and-play functionality for sensors.

[0048] This application also provides a software-defined multimodal sensor abstraction description and encapsulation system, including:

[0049] The system comprises the following modules: a sensor abstraction description unit (SDR unit) for storing and managing RDF-based sensor description files; an autonomous information processing module (Autonomous Information Processing Module) corresponding one-to-one with physical sensors or one module per type, for preprocessing and analyzing raw sensor data and outputting data encapsulation in a standard format; a data fusion and decision-making module located in the intelligent control layer, for fusing standard data from multiple autonomous signal processing units; and a sensor management and control module located in the intelligent control layer, for registering, scheduling, and uniformly controlling all sensors. The focus of this invention lies in the above method steps themselves, which can be implemented in software, hardware, or a combination of both.

[0050] In summary, this application, based on the concept of software-defined networking, constructs a unified sensor abstract description model, deploys an autonomous information processing module at the data source to achieve data abstraction and encapsulation, and finally completes fusion and scheduling at the intelligent control layer, thereby realizing plug-and-play functionality for multimodal sensors.

[0051] The following section uses an intelligent safety inspection robot system as a typical application scenario to elaborate on the embodiments of the present invention.

[0052] like Figure 1 and Figure 2 As shown in this embodiment, the intelligent safety inspection robot system is equipped with multiple sensors such as 3D LiDAR, visible light vision sensor, infrared thermal imager, gas sensor and sonar sensor. It can transmit the data of the sensors back to the background or edge computing node. At the same time, the robot can be applied to factory environments with multiple pipelines, multiple shelves and multiple aisles.

[0053] First, such as Figure 3 As shown, a unified sensor abstraction description model is created for various sensors at the physical layer. This step aims to transform heterogeneous physical sensor resources into standardized descriptions that are machine-readable and semantically clear, which is the foundation for achieving plug-and-play sensors.

[0054] In this specific implementation, the Resource Description Framework (RDF) is used as the theoretical basis for modeling, and Extensible Markup Language (XML) is selected as the specific syntax implementation carrier. Simultaneously, the RDF Schema (RDFS) specification is strictly followed to predefine the classes, properties, their hierarchical relationships, and constraints used in the model. In this embodiment, the RDFS specification plays a role similar to a "constitution," predefining a standard vocabulary (e.g., all valid OperationID function identifiers and Config configuration parameters) and specifying the hierarchical relationships between classes and properties (e.g., "Visible Light Camera" is a subclass of "Imaging Sensor"), thereby ensuring that all sensor description files generated according to this specification have consistency and interoperability.

[0055] The core of the unified sensor abstraction description model is a four-tuple structure, defined as: RDF Machine = {ID, OperationID, State, Config}. This four-tuple is constructed based on the RDF triple (subject-predicate-object), where each element is an RDF resource or literal, and its specific meaning and implementation are as follows:

[0056] As a globally unique identifier for a device, the ID functions similarly to the MAC address of a network device. It is usually fixed during production and used to uniquely identify physical sensor devices in the system. For example, the ID of a LiDAR sensor can be represented as: urn:device:laser_scanner:001.

[0057] OperationID describes the function or service provided by a sensor, rather than its specific hardware type, and is key to achieving "hardware independence". It uses a function vocabulary uniformly defined by RDFS, allowing different hardware such as LiDAR and depth cameras to share the same OperationID (e.g., urn:function:3d-ranging) when providing the same "3D ranging" function. This enables upper-layer application algorithms to schedule sensor resources based on functional requirements rather than hardware model.

[0058] The State element is used to characterize the real-time status of the sensor. Its values ​​are based on RDFS predefined values, including common sensor states such as Online, Offline, Busy, Freezing, SensorFailure, ConfigurationError, Overheated, Obstructed, and LowBattery, providing a status basis for system operation monitoring and task scheduling.

[0059] The `Config` element, as a container element, describes all configurable properties and metadata of the sensor. Its content is organized in key-value pairs, and the data type and value range of each property are strictly constrained by RDFS to ensure the validity and security of configuration operations. Examples include setting the range (…). <range unit="meter"> 0.1-100< / range> ), accuracy ( <accuracy> 0.02< / accuracy> ), Output data format ( <outputdataformat> PointCloud< / outputdataformat> ), interface type (<Interface Type> Ethernet< / Interface Type> ), communication protocol ( <protocol> TCP / IP< / protocol> ) and sampling frequency ( <scan frequency unit="Hz">10) etc.

[0060] The RDFS specification not only predefines a standard vocabulary (such as all valid OperationID and Config parameters), but also specifies the hierarchical relationship between classes and attributes (e.g., "Visible Light Camera" is a subclass of "Imaging Sensor"). The description file for any specific sensor must conform to this specification, ensuring that all device descriptions use a unified semantics, achieving unambiguous machine readability and laying the foundation for automatic sensor discovery, plug-and-play functionality, and integration.

[0061] For each type and even each physical sensor, a corresponding XML description file is generated according to its actual capabilities and parameters, following the RDF quadruple model described above. This file is stored in the sensor abstract description unit of the system.

[0062] Taking a gas sensor with an RS485 interface as an example, its description file can set the ID to "urn:device:Gas_Sensor_001", the OperationID to "Detect_CH4_Concentration", the State to dynamically reflect states such as "Online" or "Offline", and the Config to include key parameters such as range, accuracy, sampling frequency, RS485 address, and baud rate.

[0063] When a sensor connects to the network, the sensor management and control module can understand the sensor's identity (ID), function (OperationID), real-time status (State), and configuration (Config) by reading and parsing this XML file. This allows the module to automatically complete resource registration, integration, scheduling, and control without the need to write dedicated driver code for specific sensors, laying a solid foundation for automatic sensor discovery and plug-and-play functionality.

[0064] Each sensor is connected to an independent (or shared) autonomous information processing module. This module is the core of the connection between the physical sensors and the upper-layer application algorithms, and is responsible for transforming the raw, heterogeneous, high-bandwidth, and highly redundant low-level data into lightweight, semantically clear, and standard-format information encapsulations.

[0065] The autonomous information processing module first receives the raw data stream through the sensor's specific hardware interface (such as USB, Ethernet, CAN, RS485, etc.). The module has built-in parsers for different communication protocols (such as vendor-specific protocols, Modbus, TCP / IP custom frames, etc.). Based on a predefined frame structure, the parser extracts the valid sensor data payload from the data stream. For example, for a gas sensor connected via an RS485 interface, the module will parse specific register data containing the gas concentration value according to its communication protocol.

[0066] Secondly, this module cleans and initially filters the parsed raw data, removing outliers or invalid data packets caused by transmission errors. Subsequently, it performs unified unit conversions for physical quantities (e.g., converting Fahrenheit to Celsius and feet to meters) and data type standardization (e.g., converting 16-bit signed integers to 32-bit floating-point numbers) to ensure that the data is consistent in both numerical value and dimension.

[0067] The module then performs perception analysis and semantic information extraction. In this core step, the module runs a built-in specific algorithm model to extract high-level semantic information that can be directly used for decision-making from the preprocessed data. The algorithm is specifically selected according to the sensor type and functional requirements.

[0068] For visual image sensors: Real-time target detection and recognition are performed by calling built-in lightweight deep learning models (such as YOLOv5s, MobileNetV3-SSD). The output results include target category (such as "person" or "vehicle"), confidence score, and bounding box coordinates (x, y, width, height) in the image.

[0069] For LiDAR point cloud sensors: run point cloud clustering algorithms (such as Euclidean clustering) or segmentation algorithms to identify independent obstacles in the scene and output their three-dimensional dimensions, center point coordinates, number of points, and other features.

[0070] For sonar / microphone array sensors: The audio signal is subjected to Fast Fourier Transform (FFT) to obtain the spectrum, and then specific events (such as abnormal sounds, decibel values ​​exceeding limits) are detected by threshold judgment or simple classifier, and the event type, main frequency, sound pressure level, etc. are output.

[0071] For the inertial measurement unit (IMU): complementary filtering or Kalman filtering algorithms are used to fuse gyroscope and accelerometer data to calculate the attitude angles (roll angle, pitch angle, yaw angle) of the carrier in three-dimensional space.

[0072] This module encapsulates the extracted semantic information along with necessary metadata (such as timestamps and data source sensor IDs) in a predefined, hardware-independent standard data format.

[0073] Preferably, a relatively lightweight data exchange format, such as JSON, is used. The encapsulated data no longer contains massive amounts of raw data (such as entire images or complete point clouds), but rather structured key information.

[0074] Example JSON format (object detection): {"sensor_id":"urn:device:camera:front_001","timestamp":"2023-10-27T08:00:00.000Z","data_type":"object_detection_result","objects":[{"class":"person","confidence":0.92,"bbox":{"x":320,"y":240,"w":80,"h":160}},{"class":"car","confidence":0.88,"bbox":{"x":600,"y":300,"w":120,"h":90}]}

[0075] Finally, this module encapsulates the standard data packets and uploads them to the data fusion and decision-making module of the intelligent control layer through a unified communication interface (such as MQTT topic publishing, HTTP RESTful API push, gRPC calls, etc.). The ultimate effect of the entire process is to achieve abstract encapsulation of data, meaning that the upper-layer application algorithms no longer receive raw bitstreams that are difficult to process directly, but rather standardized information objects with clear business semantics.

[0076] For example, the autonomous information processing module corresponding to the sonar sensor (such as...) Figure 4 (As shown) It acquires the raw audio signal in real time, performs analog-to-digital conversion and noise reduction, then uses the Fast Fourier Transform (FFT) method for time-frequency conversion, and runs an event detection algorithm (such as detecting abnormal sounds at specific frequencies). Finally, it outputs a structured JSON data packet to the upper-layer application algorithm instead of a large raw audio stream, for example: {"timestamp":"2023-10-27T08:00:00Z","db_level":65.2,"main_freq":12000,"anomaly_detected":"true","anomaly_type":"bearing_overspeed"}.

[0077] The specific implementation of the autonomous information processing module can be flexibly deployed according to computing power requirements. For micro sensors with limited computing power, this module can be simplified into a lightweight protocol converter and data formatter. For smart sensors with certain edge computing capabilities (such as smart cameras), this module is a powerful embedded AI processing unit.

[0078] Taking a visible light vision sensor as an example, the operation flow of its autonomous information processing module is as follows:

[0079] ① Capture raw Bayer format images using the vendor's SDK. Raw Bayer format images are massive in size and contain a large number of redundant pixels. For example, a 1920×1080 Bayer image (each pixel contains only one color channel of RAW data) occupies approximately 3MB of storage space, and the uncompressed raw data stream can reach hundreds of MB per second (assuming a 30fps frame rate). This data not only consumes a lot of bandwidth but also contains a large amount of environmental noise, repetitive textures, and other redundant information; direct transmission would place a huge burden on the network and CPU.

[0080] ② Perform ISP (Image Signal Processing) processing, including de-mosaicing, white balance, color correction, noise reduction, etc., and output an RGB image;

[0081] ③ Call the built-in lightweight neural network model (such as YOLO, MobileNet-SSD) for real-time target detection and recognition;

[0082] ④ The recognition results (such as target category, confidence level, and bounding box coordinates) along with timestamps, sensor IDs, and other information are encapsulated into a standard data packet in a predefined JSON format and output. The following is a JSON example demonstrating lightweight structured information transformed from highly redundant raw image data. This JSON data packet occupies only a few hundred bytes to a few KB, far less than the original image data.

[0083] {"sensor_id":"urn:device:camera:front_001","timestamp":"2023-10-27T08:00:00.000Z","data_type":"object_detection_result","image_resolution":{"width":1920,"height":1080},"objec ts":[{"class":"person","confidence":0.92,"bbox":{"x":320,"y":240,"width":80,"height":16 0}},{"class":"car","confidence":0.88,"bbox":{"x":600,"y":300,"width":120,"height":90}}]}

[0084] This process transforms high-bandwidth, highly redundant raw data into low-bandwidth, high-value structured information, greatly reducing the burden on network transmission and the central processing unit.

[0085] All these standardized data packets are sent to the data fusion and decision-making module, which employs a hierarchical fusion strategy.

[0086] The first level is low-level data fusion, for example, fusing homogeneous sensor data (such as inertial data from two IMUs) using Kalman filtering to obtain more accurate and reliable state estimates.

[0087] The second level is feature-level fusion, such as extracting texture features from visible light images and temperature features from infrared thermal imagers, and performing correlation analysis.

[0088] The third level is advanced decision-making fusion, which integrates all available information and applies more complex models (such as DS evidence theory and Bayesian networks) to perform joint reasoning and decision-making.

[0089] Taking the inspection robot's assessment of equipment status as an example: LiDAR provides high-precision point clouds for positioning and obstacle avoidance; visible light cameras identify instrument panel readings; infrared thermal imagers detect surface temperature distribution; and gas sensors monitor for any flammable gas leaks. A fusion algorithm processes these heterogeneous data sources in parallel. When both "abnormally high temperature" and "excessive methane concentration" events are simultaneously identified, its decision confidence is far higher than that of a single sensor, thus triggering the highest level alarm and allowing for preliminary location of faulty equipment.

[0090] The sensor management and control module in this embodiment implements a resource virtualization mechanism, which abstracts all physical sensor resources into a unified "sensor resource pool".

[0091] This module maintains a dynamic resource mapping table that records the ID of each physical sensor, its corresponding OperationID (function), current state, load status, and the access endpoint (such as IP address and port) of the autonomous information processing module to which it belongs.

[0092] When the upper-layer application algorithm requests a "temperature monitoring" function through the service interface, the sensor management and control module does not specify a particular sensor. Instead, it queries the resource pool for all sensors with the OperationID urn:function:measure-temperature and an Online status, and selects the optimal instance based on a load balancing strategy, returning its access endpoint to the upper-layer application algorithm. The application then communicates directly with the autonomous information processing module corresponding to that sensor. The entire process is transparent to the application, achieving on-demand allocation and elastic scheduling of sensor resources.

[0093] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.< / scan>

Claims

1. A software-defined based multi-modal sensor abstraction description and packaging method, characterized in that: Comprising the following steps: Step a) generating a unified sensor abstract description file for each sensor based on Resource Description Framework (RDF), the description file being in XML carrier, containing an ID for globally uniquely identifying the sensor, an Operation ID for characterizing its function rather than hardware type, a State for characterizing its real-time running status, and a Config for carrying its static configurable parameters and metadata; Step b) deploying a corresponding autonomous information processing module for each sensor or each type of sensor, the autonomous information processing module receiving raw data through the hardware interface of the sensor itself, sequentially completing protocol analysis, data cleaning and unit standardization, semantic information extraction, and data encapsulation in a unified standard format, and uploading the encapsulated data through a unified communication interface; Step c) setting a data fusion and decision module at the intelligent control layer to receive and fuse standardized data from each autonomous information processing module to generate a joint estimation of the environment state; Step d) setting a sensor management and control module at the intelligent control layer to automatically complete sensor registration, discovery, scheduling and control by reading the sensor abstract description file, realizing plug-and-play at the functional semantic level.

2. The method for software-defined multi-modal sensor abstraction description and packaging according to claim 1, characterized in that: The sensor abstract description file in step a) complies with the RDF Schema (RDFS) specification, which predefines sensor classes, properties, and their hierarchical relationships and value constraints to ensure consistency and interoperability between different sensor description files.

3. The method of software-defined multi-modal sensor abstraction description and packaging according to claim 2, wherein: The Operation ID in step a) references the function vocabulary defined uniformly by RDFS, so that different hardware but functionally identical sensors share the same Operation ID, realizing resource scheduling based on functional requirements rather than hardware models at the upper layer application.

4. The method of software-defined multi-modal sensor abstraction description and packaging according to claim 1, wherein: The semantic information extraction in step b) is completed by a lightweight algorithm model running in the autonomous information processing module, which is selected from target detection, point cloud clustering, spectrum analysis or pose solving according to the sensor type, to output high-level information that can be directly used for decision-making.

5. The method for software-defined multi-modal sensor abstraction description and packaging according to claim 1 or 4, characterized in that: The unified standard format in step b) is JSON, and the encapsulated content at least includes timestamp, sensor ID, data type and key fields after semantic extraction, and the data packet size is reduced by at least two orders of magnitude relative to the original data stream.

6. The method of software defined multi-modal sensor abstraction description and packaging according to claim 1, wherein: The data fusion and decision module in step c) adopts a hierarchical fusion strategy, including low-level data fusion, feature-level fusion and high-level decision-level fusion, to improve the accuracy and robustness of the environment state estimation.

7. The method of software defined multi-modal sensor abstraction description and packaging according to claim 6, wherein: The sensor management and control module in step d) maintains a dynamic resource mapping table, records the ID, Operation ID, State, load and access endpoint of each sensor, and dynamically allocates the optimal sensor instance for the upper layer application based on the load balancing strategy, realizing elastic scheduling of the sensor resource pool.

8. A software-defined multi-modal sensor abstraction description and packaging system, characterized in that, Comprising: a sensor abstract description unit for storing and managing sensor description files based on Resource Description Framework (RDF) and in XML carrier, the description file at least containing a globally unique ID of the sensor, a function Operation ID, a real-time State, and a static Config; An autonomous information processing module, corresponding to each sensor or shared by sensors of the same type, is configured to receive raw data from the hardware interface of the sensor, sequentially complete protocol analysis, data cleaning and unit standardization, semantic information extraction and unified standard format packaging, and output a standardized data packet through a unified communication interface; A data fusion and decision module, located in the intelligent control layer, is configured to receive and fuse the standardized data packets from the autonomous information processing modules to generate a joint estimation of the environment state; A sensor management and control module, located in the intelligent control layer, is configured to read the description file of the sensor when the sensor is connected to automatically complete registration, discovery, scheduling and control, and realize plug-and-play at the functional semantic level.

9. The software-defined based multi-modal sensor abstraction description and packaging system of claim 8, wherein, The sensor abstract description unit follows the RDF mode RDFS specification, which predefines the sensor class, attribute and hierarchical relationship and value constraint to ensure consistency and interoperability between different sensor description files.

10. The software-defined based multi-modal sensor abstraction description and packaging system of claim 8, wherein, The autonomous information processing module is internally provided with a lightweight algorithm model, which is selected from target detection, point cloud clustering, spectrum analysis or attitude solution according to the sensor type, to output high-level semantic information that can be directly used for decision-making.

Images