Mobile robot-oriented multi-mode database storage and query method and system

Abstract

The invention belongs to the technical field of mobile robots and multi-mode database storage, and discloses a multi-mode database storage and query method and system for a mobile robot. The storage method comprises the following steps of: distributing a corresponding table name for each type of sensing data, defining specific structure information of each table, creating the table in a database by utilizing an SQL (Structured Query Language) statement, and distributing a corresponding storage space; all sensor nodes are controlled to be started, and sensing data of all modes are collected; and performing corresponding data conversion on the acquired modal sensing data, and then storing the converted result into a pre-allocated storage space in a database in the form of a relation model. Unified storage of multi-mode data is achieved at the mobile robot end, the multi-mode database of the mobile robot end can be accessed through the mobile device under the offline condition, the multi-mode database is introduced to be used for data storage and management, the memory capacity needed by storage is reduced, and the data query efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of mobile robots and multi-modal database storage, and particularly relates to a multi-modal database storage, query method and system for mobile robots. Background Art

[0002] In existing mobile robot scenarios, such as robot cars, drones, etc., a large amount of data needs to be obtained from the environment to perform tasks such as path planning, speech recognition or similarity search. Usually, the data required to perform these tasks comes from different sensors and has different data formats. In order to effectively organize these multi-modal data on the mobile robot side, the current mainstream strategy is to deploy different databases for management according to different data models. However, deploying multiple databases on a mobile robot will undoubtedly cause a large storage overhead. In addition, when using multiple databases for multi-modal data management, it is also easier to generate time delays caused by insufficient coordination.

[0003] Moreover, when the robot performs tasks such as path planning, speech recognition and similarity search, it usually involves a large amount of multi-modal data processing and operations. However, the databases currently used on the mobile robot side, such as the SQLite database, do not have complex and efficient data analysis and processing capabilities, such as query operations (k-nearest neighbor query), etc.

[0004] In the mobile robot scenario, the database system needs to meet the requirements of high accuracy and high execution efficiency at the same time. Among them, accuracy is reflected in the accuracy of data storage, the reliability of query and processing, and the correctness of multi-modal data fusion, so as to ensure the spatio-temporal consistency of sensor data during storage, indexing and calculation, and support tasks such as high-precision positioning and path planning. The execution efficiency is reflected in aspects such as data access speed, query response time, and calculation resource optimization, so as to ensure that the robot can quickly obtain and process multi-modal data in real-time tasks, and thus make accurate decisions.

[0005] Therefore, it can be seen that the mobile robot database system needs to have a powerful multi-modal data management ability, that is, it can efficiently store, organize, index and query multi-modal data. Therefore, it is particularly important to propose a database that can support multi-modal data storage on the mobile robot side and has a high multi-modal data operation efficiency in the mobile robot scenario. Summary of the Invention

[0006] The object of the present invention is to propose a multi-modal database storage method for mobile robots. By collecting data acquired by various sensors on the robot trolley and uniformly storing them in a multi-modal database system deployed on the robot trolley, it is possible to achieve unified storage and management of multi-modal data, and at the same time, it is beneficial to reduce the storage overhead caused by the deployment of multiple databases.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A multi-modal database storage method for mobile robots, comprising:

[0009] A multi-modal data table creation step for assigning corresponding table names to each type of sensing data, defining the specific structure information of each table, and at the same time using SQL statements to create tables in the same database and allocate corresponding storage spaces;

[0010] A sensing data acquisition control step for controlling the activation of all sensor nodes and acquiring sensing data of each modality;

[0011] And a multi-modal data conversion step for respectively performing corresponding data conversions on the acquired sensing data of each modality, and then storing the converted results in the storage spaces pre-allocated in the database in a relational model.

[0012] In addition, based on the above multi-modal database storage method for mobile robots, the present invention also proposes a multi-modal database storage system for mobile robots, which adopts the following technical solutions:

[0013] A multi-modal database storage system for mobile robots, located at the mobile robot terminal. The multi-modal database storage system for mobile robots includes the following modules:

[0014] Database;

[0015] A multi-modal data table creation module for assigning corresponding table names to each type of sensing data, defining the specific structure information of each table, and at the same time using SQL statements to create tables in the same database and allocate corresponding storage spaces;

[0016] A sensing data acquisition control module for controlling the activation of all sensor nodes and acquiring sensing data of each modality;

[0017] And a multi-modal data conversion module for respectively performing corresponding data conversions on the acquired sensing data of each modality, and then storing the converted results in the storage spaces pre-allocated in the database in a relational model.

[0018] Preferably, the multi-modal database storage system for mobile robots further includes:

[0019] A syntax parsing module for receiving an input of an SQL query statement, converting the query according to the results of lexical analysis and syntax analysis, and generating a syntax tree for the corresponding modal data;

[0020] And a query processing module for generating a query execution plan corresponding to the SQL query statement according to the syntax tree, and performing an actual physical query operation at the physical level.

[0021] Preferably, a Bluetooth module is provided on the mobile robot terminal for establishing a wireless communication connection between the mobile robot and the mobile terminal to implement a query of the database in the multi-modal database storage system for the mobile robot.

[0022] In addition, based on the above multi-modal database storage system for mobile robots, the present invention also proposes a multi-modal database query method for mobile robots, which includes the following steps:

[0023] A multi-modal client input step on the mobile terminal side for receiving an SQL query statement for queries of different data types; the SQL query statement includes the table name of the corresponding data and the storage id for different data types respectively;

[0024] A data query request sending step for establishing Bluetooth communication between the mobile robot terminal and the mobile terminal, and transmitting the key query information of the SQL query statement parsed by the mobile terminal to the mobile robot via Bluetooth;

[0025] Wherein, the key query information of the SQL query statement includes the table name of the data table and the id of the corresponding data;

[0026] A syntax parsing step on the mobile robot side for receiving an input of an SQL query statement, converting the query according to the results of lexical analysis and syntax analysis, and generating a syntax tree for the corresponding data model;

[0027] And a query processing step on the mobile robot side for generating a query execution plan according to the syntax tree, and driving the corresponding data processing engine to perform an actual query operation.

[0028] The present invention has the following advantages:

[0029] As described above, the present invention relates to a multi-modal database storage, query method and system for mobile robots. Among them, the present invention configures a multi-modal database on the mobile robot side. Through this multi-modal database, five types of data, namely documents, point clouds, images, trajectories, and environmental maps, input by various sensors carried on the mobile robot can be uniformly stored and managed, that is, the unified storage and management of multi-modal data is realized. This multi-modal database supports multi-modal data storage on the mobile robot side and has a high multi-modal data operation efficiency, which well meets the requirements for high accuracy and high execution efficiency of the database system in the mobile robot scenario. In addition, the present invention can also access the multi-modal database on the mobile robot side through a mobile device under offline conditions, that is, without relying on the cloud or external servers, but independently completing data storage, query and processing functions on the robot trolley. This enables the robot trolley to complete data management relying on the local database even when the network is unstable or completely disconnected. By introducing a multi-modal database in the mobile robot side for storing and managing various modal data, the present invention reduces the memory capacity required for storage and improves the data query efficiency at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of the multi-modal database storage method for mobile robots in the embodiment of the present invention;

[0031] Figure 2 It is a schematic diagram of data model conversion and storage;

[0032] Figure 3 It is a schematic diagram of image data model conversion;

[0033] Figure 4 It is a schematic diagram of environmental map data model conversion;

[0034] Figure 5 It is a schematic diagram of document data model conversion;

[0035] Figure 6 It is a schematic diagram of point cloud data model conversion;

[0036] Figure 7 It is a schematic diagram of trajectory data model conversion;

[0037] Figure 8 It is an overall framework diagram composed of a robot trolley and a mobile terminal in the embodiment of the present invention;

[0038] Figure 9 It is a query flowchart of the multi-modal database query method for mobile robots in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0040] Embodiment 1

[0041] This embodiment describes a multi-modal database storage method for mobile robots. By collecting data collected by various sensors on the robot trolley and uniformly storing them in a multi-modal database system deployed on the robot trolley, unified storage and management of multi-modal data can be achieved.

[0042] As Figure 1 shown, the multi-modal database storage method for mobile robots includes:

[0043] A multi-modal data table creation step for assigning corresponding table names to each type of sensing data, defining the specific structure information of each table, and using SQL statements to create tables in the same database and allocate corresponding storage spaces;

[0044] A sensing data acquisition control step for controlling the activation of all sensor nodes and collecting sensing data of each modality;

[0045] And a multi-modal data conversion step for respectively performing corresponding data conversions on the collected sensing data of each modality, and then storing the converted results in the storage space previously allocated in the database in a relational model.

[0046] Among them, the multi-modal data table creation step is specifically as follows:

[0047] First, according to the organizational structure of the data, the data input by various sensors carried on the mobile robot is divided into five modalities: document, point cloud, image, trajectory, and environmental map. Then, according to the structure information of each modality data input by the sensor, the corresponding table storage structure is defined, including attribute names, attribute data types, and constraint conditions such as primary keys (such as unique identifiers), foreign keys (unique identifiers of other tables), uniqueness constraints, non-null constraints, default values, and indexes.

[0048] Attribute names refer to the field names in the database table, and each field corresponds to a data attribute. For example, in the table storing point cloud data, x, y, and z are attribute names used to describe the point cloud coordinates, and timestamp is the attribute name used to record the timestamp. Attribute data types refer to the data types of each field, which determine the data formats that can be stored in this field. For example, the integer type (INTEGER) is used to store numerical IDs, counts, etc.; the floating-point types (FLOAT, DOUBLE) are used to store sensor measurement values; the string type (TEXT) is used to store text information; the time type (TIMESTAMPE) is used to store time information; the JSON type is used to store semi-structured data.

[0049] The constraints are to ensure data consistency, integrity, and query efficiency.

[0050] Common constraints include:

[0051] Primary key - used to uniquely identify each row of data in a table. For example, ID is used as a unique identifier to ensure that the data of each sensor is unique;

[0052] Foreign key - used to establish associations between different tables. For example, sensor_id (in the relational model, each sensor has its own sensor_id) is used as a foreign key to reference the sensor_id field in the sensors table, indicating that the data belongs to a specific sensor;

[0053] Unique constraint - ensures that the values of a certain field are not repeated;

[0054] Not-null constraint - ensures that the field cannot be null;

[0055] Default value - provides a default value for the field;

[0056] Check constraint - used to define the value range of a field.

[0057] Next, create tables in the database through SQL statements and allocate corresponding storage space for each table.

[0058] Finally, verify whether the table storage structure of each modal data can meet the storage requirements of the data collected by the corresponding modal sensors to ensure that different modal data can be stored in the predetermined format. The process is as follows:

[0059] First, perform pattern matching checks to ensure that the field definitions of the database tables match the data formats collected by the sensors. For example, verify whether the field names, data types, and constraints are consistent with the sensor data requirements.

[0060] Second, perform data insertion and constraint verification. By inserting simulated or actual sensor data, check for errors such as data type mismatches, violations of primary key / foreign key constraints, or exceeding field length limits.

[0061] Finally, execute typical query operations and perform performance analysis to verify whether different modal data can be correctly retrieved according to the query conditions, and ensure that while efficiently executing queries, the data structure can meet the predetermined storage requirements and query performance requirements.

[0062] In the sensing data acquisition control step, different sensor nodes are mounted on the mobile robot, which are respectively used to collect different sensing data, including document data, point cloud data, image data, trajectory data, and environmental map data.

[0063] As Figures 2 to 7 shown, the multi-modal data conversion step further includes:

[0064] A document data conversion step of storing the entire JSON type document data as a variable, assigning a unique identifier id and representing it using the built-in JSON variable format of the database, and storing the converted vector data using a relational model;

[0065] A point cloud data conversion step of converting each point cloud data point into a tuple in a relational model, assigning a unique identifier id, splitting the coordinates of the point cloud data point into several floating-point attributes, each floating-point attribute representing a coordinate, and converting the additional attributes (such as timestamp, label, etc.) of the point cloud data point into attributes in a relational table respectively;

[0066] The label here is used to identify the category or type to which the point cloud data belongs, for example, to distinguish different objects or regions, such as "road", "building" or "pedestrian", etc.;

[0067] An image data conversion step of converting the entire image into BLOB type data using a key-value pair storage method, assigning a unique identifier id for storage, and storing the converted image model data using a relational model;

[0068] A trajectory data conversion step of converting the trajectory data (i.e., the position and time of the mobile robot) into a relational table, where each row in the relational table represents a trajectory point of the robot car (i.e., the position of the mobile robot at a certain moment), including position coordinates and timestamp, and generating a unique identifier id for each trajectory point for storage;

[0069] And an environmental map data conversion step of converting each environmental map into a tuple in a relational model. During the conversion process, converting each environmental map data into a binary object BLOB, assigning a unique identifier id for storage, and converting the width, height, and coordinates of the map origin of the environmental map into attributes in a relational table respectively.

[0070] Through the above image data conversion step, when the user initiates an image query, the system will query and return the BLOB data of the image through the database, and then decode it on the mobile terminal to restore it to a displayable image.

[0071] Through the above trajectory data conversion method, the trajectory data is split into multiple records for storage and can support querying in chronological order, facilitating subsequent trajectory analysis and playback.

[0072] Through the above environmental map data conversion steps, it is ensured that queries and updates can be effectively carried out. When querying, the system extracts the BLOB data of the map and transmits it to the mobile terminal. The mobile terminal decodes and renders the map for display in the user interface.

[0073] In the mobile robot terminal, the present invention deploys a multi-modal database, which can realize the unified storage and management of multi-modal data on the mobile robot. This multi-modal database supports multi-modal data storage on the mobile robot terminal and has a high operation efficiency for multi-modal data, well meeting the requirements for high accuracy and high execution efficiency of the database system in the mobile robot scenario.

[0074] Embodiment 2

[0075] This Embodiment 2 describes a multi-modal database storage system for a mobile robot. This system is based on the same inventive concept as the multi-modal database storage method for a mobile robot in the above Embodiment 1.

[0076] As Figure 8 shown, the multi-modal database storage system for a mobile robot in this embodiment is deployed on the mobile robot terminal. This multi-modal database storage system for a mobile robot includes the following modules:

[0077] Database;

[0078] A multi-modal data table creation module, which is used to assign corresponding table names to each type of sensing data, define the specific structure information of each table, and at the same time use SQL statements to create tables in the same database and allocate corresponding storage spaces;

[0079] A sensing data acquisition control module, which is used to control the activation of all sensor nodes and collect sensing data of each modality;

[0080] And a multi-modal data conversion module, which is used to perform corresponding data conversions on the collected sensing data of each modality respectively, and then store the converted results in the storage space pre-allocated in the database in a relational model.

[0081] The database here is the multi-modal database, which stores data of multiple modalities (that is, uses one database to store data of multiple modalities at the same time). The stored data includes documents, point clouds, images, trajectories, environmental map data, and so on.

[0082] The mobile robot in this embodiment is, for example, a robot car.

[0083] Among them, in the sensing data acquisition control module, there is a node that can activate all the sensors (radar, RGB, depth camera, etc.) and the mobile base plate of the robot trolley, a node that extracts the valid data of all sensors, and a node that can subscribe to all the processed sensor data.

[0084] Just running this module can start all the sensors and collect all the required sensor data.

[0085] In the sensing data acquisition control module, the node mentioned refers to a configuration file in the ROS system, namely the.launch file. This file is used to start and manage multiple sensor nodes. By defining the startup sequence and configuration parameters of each sensor, it ensures that all sensors can be activated at the correct time and in the correct order.

[0086] By executing the.launch file, the system can automatically start all the relevant sensors (such as radar, RGB camera, depth camera, etc.) on the robot trolley and assign corresponding configuration and processing tasks to each sensor.

[0087] After startup, various sensors mounted on the robot trolley will start collecting data and transmit the valid sensor data to the subsequent processing node (i.e., the node that extracts the valid data of all sensors), ensuring that the data can be uniformly collected and processed. Therefore, users only need to run this module once to easily start all the sensors and begin the data acquisition process.

[0088] Here, the node that extracts the valid data of all sensors discards the unnecessary fields when collecting sensor data (the data collected by various sensors is relatively messy. In this invention, only the desired data is taken, and other data is directly discarded. The process is relatively conventional), that is, only the required valid fields are stored in the database, greatly reducing the memory required to store data.

[0089] Taking point cloud data as an example, first run the radar activation node to operate the radar sensor. The radar emits electromagnetic waves and receives the reflected signals to generate the original point cloud data. Then, preprocessing such as filtering and denoising is performed to improve the data quality. Finally, the processed data is submitted to the multi-modal data conversion module, and the converted data is stored in the multi-modal database.

[0090] In the ROS system, data types are implemented by defining message types, and message types are usually defined in ".msg" files. Each ".msg" file describes the structure of the data and its fields.

[0091] After the definition is completed, these message types will be registered through a compilation tool and can be referenced in the code.

[0092] ROS provides multiple standard message packages, which contain common message types, such as "sensor_msgs / PointCloud" for point cloud data and "sensor_msgs / Image" for image data.

[0093] In the ROS system, data is published and subscribed through topics. Each topic has a specific message type to ensure data consistency. When a node publishes or subscribes to data, the message type determines the structure and processing method of the data.

[0094] Corresponding to the multi-modal data conversion steps in the above Embodiment 1, the multi-modal data conversion module includes a document conversion module, a point cloud data conversion module, an image data conversion module, a trajectory data conversion module, and an environmental map data conversion module.

[0095] The document conversion module will collect the document model data transmitted by each sensor, including the point cloud information from the radar, the trajectory information of the robotic vehicle on the mobile baseplate, and the pose information of the robotic vehicle.

[0096] All this information is converted into JSON format and assigned a unique identifier id. The converted JSON format data and the unique identifier id form a data adaptation relationship model similar to a key-value pair format and are stored in a relational table.

[0097] The point cloud data conversion module stores each three-dimensional coordinate point and the reflection intensity of this coordinate point in the point cloud data collected from the radar sensor as a separate row, and assigns a unique identifier id to this separate row and stores it in the relational table.

[0098] The image data conversion module converts the images captured from RGB and depth camera sensors into binary BLOB data, and assigns a unique identifier id to the converted BLOB data. The converted BLOB data and the unique identifier id form a data adaptation relationship model similar to a key-value pair format and are stored in the relational table.

[0099] When the user queries through the client, the BLOB data is transmitted to the client through the Bluetooth module and restored to image data through an algorithm and displayed on the client page.

[0100] The trajectory data conversion module collects the trajectory data from the sensor data of the moving baseplate of the robotic vehicle, stores a position point of the robotic vehicle as a row in the relational table, and assigns a unique identifier id to this row.

[0101] The environmental map data conversion module will run the mapping instruction on the ROS main control board Jetson Nano of the robotic vehicle. The mapping instruction will turn on the camera, radar, and moving baseboard on the robotic vehicle to jointly collect map information and convert it into the BLOB format, and assign a unique identifier id to the converted data.

[0102] The converted map data in BLOB format, along with the width and height of the map and the coordinates of the map origin, is stored as a single row, and a unique identifier id is assigned to this single row and stored in the relational table.

[0103] In addition, for the convenience of data query, the multi-modal database storage system for mobile robots in this embodiment further includes:

[0104] A syntax parsing module that is used to receive the input of an SQL query statement, convert the query according to the results of lexical analysis and syntactic analysis, and generate a syntax tree for the corresponding modal data;

[0105] And a query processing module that generates a query execution plan corresponding to the SQL query statement according to the syntax tree and executes the actual physical query operation at the physical level.

[0106] Among them, the specific processing process of the syntax parsing module is as follows:

[0107] First of all, after receiving the SQL query statement transmitted via Bluetooth, the syntax parsing module will decompose the SQL query statement into basic symbols or words through lexical analysis, including keywords, table names, field names, and operators.

[0108] Next, through syntactic analysis, according to the SQL syntax rules, it checks whether the query statement meets the SQL syntax requirements and converts it into a syntax tree; the syntax tree is a structured representation of the query statement. The generated syntax tree not only helps the system understand the specific intention of the query but also provides the operation sequence and logical structure for subsequent query execution. Each node in the syntax tree represents an operation or condition, and the leaf nodes usually represent fields or constant values.

[0109] Finally, the syntax parsing module will optimize and convert the syntax tree generated by the SQL statement according to the characteristics of different modal data (such as point cloud, image data, etc.) to ensure the efficient execution of the query.

[0110] Specifically, the optimization process includes parsing data patterns (identifying structured, semi-structured, time-series, or spatial data), adjusting query plans (selecting optimal indexes, join methods, or partitioning strategies), rewriting query statements (adapting SQL logic for different storage engines, e.g., converting JSON queries to key-value index queries), and operator optimization (optimizing computational operations for different data types, such as spatial index optimization for point cloud data). Through these optimizations, the system can execute efficient queries based on different data structures, thereby improving the query performance and response speed of the database.

[0111] Among them, the specific process of the query processing module is as follows:

[0112] The query processing module first parses the query target, filtering conditions, and association relationships in the SQL statement according to the query logic and data structure defined in the syntax tree, and identifies the data modalities involved in the query. Subsequently, in combination with different data storage methods, it selects the optimal data access strategy. For example, it uses indexes to accelerate queries for relational data and time partitioning to optimize retrieval for time-series data. Finally, the query processing module ensures the efficient execution of the query by optimizing the query execution plan, including adjusting the join order, optimizing the execution order of filtering conditions, and adopting parallel computing strategies, thereby improving data retrieval and computing performance.

[0113] The query execution plan is an optimization scheme for a specific data model, taking into account factors such as data storage methods, access paths, and index usage. The data storage method, i.e., how data is stored in the database (such as row storage, column storage, spatial data storage, etc.), directly affects query performance. The access path is the execution path of the query, referring to the path from the data storage location to the actual result obtained. The query optimizer selects the optimal access path according to the data storage method and query requirements. For complex queries, parallel processing or partitioning strategies are adopted to accelerate data access. Index usage is a crucial part of query optimization. Indexes can significantly improve data access speed, especially for queries with specific conditions (such as filtering point cloud data by time or coordinate range or querying image data by certain metadata). The query optimizer selects the appropriate index to accelerate data lookup according to the query conditions. For example, for point cloud data, the query plan needs to traverse multiple point cloud data points stored in the database and filter or aggregate them according to specific conditions; while for image data, the query plan involves extracting binary image data (BLOB) from the database and then performing decoding processing.

[0114] The generated query execution plan will specify the specific execution steps and order; the data processing engine then executes the query operation according to these steps using different algorithms and strategies, and finally returns the query result.

[0115] This process ensures efficient query execution for different types of data models, improving the system's response speed and processing capacity.

[0116] As Figure 8 shown, in order to achieve query access to the database in the multi-modal database storage system for mobile robots on a mobile terminal, a Bluetooth module and an SMT32 control chip processing module are provided on the robot trolley.

[0117] Specifically, a Jetson Nano small computer is included on the robot trolley, and the multi-modal database storage system for mobile robots is deployed on this Jetson Nano small computer.

[0118] The Bluetooth module on the robot trolley can receive information transmitted from an Android system mobile device to the robot trolley and transmit the query results retrieved from the multi-modal database of the mobile robot to the Android system mobile device.

[0119] In addition, a Bluetooth module is also installed on the mobile terminal, and data is transmitted between the two via Bluetooth, enabling query access to the multi-modal database on the robot trolley and control of the mobile base plate of the robot trolley on the mobile device.

[0120] Furthermore, on the mobile terminal, in order to achieve query access to the database on the robot trolley side, a multi-modal client input module is also provided, which is a software deployed on an Android system mobile device.

[0121] The input of the multi-modal client input module is a classic SQL query statement for querying the data content in the multi-modal database. The SQL query statement contains the table name of the corresponding data and the storage id for different data types respectively.

[0122] Specifically, the SQL statement input by the user will be parsed by the system to extract the key information in the statement and generate a syntax tree. The system first identifies the type of query and the data models involved (such as point cloud, image, trajectory, etc.) by extracting the key information in the SQL statement input by the user, and then different query rules will be executed corresponding to different data models.

[0123] For example, if the user wants to query the image information in the database, including the table name storing the image data and the ID of the corresponding image in the standard SQL statement input in the user interface can query the image with the corresponding ID and display it in the user interface, where the id is the unique identifier of the corresponding image, and image_data is the relational table used to store images in the multi-modal database.

[0124] The standard SQL statement input in the user interface usually contains multiple key parts. For example, it includes:

[0125] Query target, which is used to specify the fields or data to be queried, usually an ID;

[0126] Data source, which defines the table name of the data table involved in the query;

[0127] Filter condition, which is used to limit the query scope and filter data that meets specific conditions;

[0128] Data association, which supports cross-table queries to retrieve related data;

[0129] Sorting, which is used to define the sorting method of the query results.

[0130] These parts such as the query target, data source, filter condition, data association, and sorting together constitute the SQL query statement, enabling it to flexibly and efficiently perform operations such as data retrieval, filtering, calculation, and sorting.

[0131] Since users can execute queries for different data models through standard SQL query statements without the need to use other query languages again, unified query processing of the multi-modal data query language for users is achieved.

[0132] Taking the query of image data as an example, the mobile device first extracts key query information (such as the table name of the data table and the ID of the corresponding data) from this SQL query statement and transmits the query information to the robot car via Bluetooth.

[0133] The car then executes the corresponding query task, retrieves the BLOB data of the corresponding image and transmits it to the Android system mobile device, and the Android system mobile device then decodes the BLOB data into an image and displays it in the user interface.

[0134] In addition, in this embodiment, the multi-modal client input module also adds display statement statements for point cloud data model, image data model, trajectory data model, environmental map data model, and document data model other than the relational data model, which are used for the system to identify and execute syntax operations for different data models.

[0135] The display statement lists the storage formats of various modal data, the name of each field in the data table, and the data format. The display statement only serves as a reminder, and the input is the standard SQL statement. For simple queries, only the table name + ID need to be passed in.

[0136] In the query page of the user interface, the names of the relational tables storing various data (such as point cloud, image, trajectory, environmental map, and document data) are prompted for the user's reference, as shown in Table 1 below:

[0137] Table 1

[0138]

[0139] The multi - mode client input module receives the query input from the user and performs preliminary lexical and syntactic analysis to ensure that the input query statement conforms to the system's query rules.

[0140] In addition, the mobile terminal also includes a module for controlling the movement of the robotic cart. This module contains a user - facing interactive panel, and the user can control the movement and steering of the robotic cart by clicking the up, down, left, and right keys on the panel.

[0141] The control panel contains four buttons: up, down, left, and right, which represent the actions of the robotic cart moving forward, backward, and turning left and right. When the user clicks a button, a corresponding command is issued, and the mobile device collects the commands for movement and turning.

[0142] The mobile device transmits the control commands to the robotic cart via Bluetooth transmission. After the SMT32 control chip processing module of the robotic cart receives the movement and steering control commands of the robotic cart, it converts them into control instructions and transmits them to the servo motors and motors.

[0143] The moving base plate of the robotic cart contains two 65 - mm drive wheels and two 60 - mm omnidirectional driven wheels. After the servo motors and motors receive the control instructions, they control the driven wheels and drive wheels to steer and move the robotic cart.

[0144] It should be noted that in the module capable of controlling the movement of the robotic cart, it is also possible to control the movement of the robotic cart by running the cart control program through the terminal in the Jetson Nano control board mounted on the robotic cart.

[0145] Of course, in this embodiment, the mobile robot is not limited to the robotic cart. For example, it can also be a drone, etc., which will not be elaborated here.

[0146] This embodiment proposes a multi - mode database system deployed on a robotic cart, which is used to collect data from various sensors on the robotic cart and uniformly store it in the multi - mode database to reduce the storage overhead of data and improve the operating efficiency of the robotic cart. At the same time, the multi - mode database in this embodiment realizes offline multi - mode data management on the device. Here, offline means that it does not rely on the cloud or external servers, but independently completes data storage and query, etc. on the robotic cart. Even when the network of the robotic cart is unstable or completely disconnected, it can rely on the local database to complete data management.

[0147] Embodiment 3

[0148] Embodiment 3 describes a multi-modal database query method for mobile robots, which performs database queries based on the multi-modal database storage system for mobile robots in Embodiment 2 above.

[0149] As Figure 9 shown, the multi-modal database query method for mobile robots in this embodiment includes:

[0150] A multi-modal client input step on the mobile terminal side for receiving SQL query statements for queries of different data types; the SQL query statements respectively include the table names of the corresponding data and the storage ids for different data types.

[0151] A data query request sending step for establishing Bluetooth communication between the mobile robot terminal and the mobile terminal, and transmitting the key query information of the SQL query statement parsed by the mobile terminal to the mobile robot via Bluetooth.

[0152] Among them, the key query information of the SQL query statement includes the table name of the data table and the id of the corresponding data.

[0153] A syntax parsing step on the mobile robot side for receiving the input of the SQL query statement through a syntax parsing module, and converting the query according to the results of lexical analysis and syntax analysis to generate a syntax tree for the corresponding data model.

[0154] And a query processing step on the mobile robot side for generating a query execution plan according to the syntax tree through a query processing module, and driving the corresponding data processing engine to execute the actual query operation.

[0155] The advantages of the multi-modal database query method in this embodiment are mainly reflected in several aspects:

[0156] First of all, it realizes the unified storage and management of different sensor data, centralizes multi-modal data such as point clouds, images, timestamps, labels, etc. in the same database, simplifies the complexity of data storage and access, and improves data management efficiency.

[0157] Secondly, by optimizing the query execution plan, according to the differences in storage, access paths, and index usage for different data models (such as point cloud data, image data, etc.), the most suitable query execution strategy is selected, greatly improving query performance. For example, spatial indexes can be used to accelerate queries for point cloud data, and the processing efficiency can be improved by optimizing the extraction and decoding processes for image data.

[0158] In addition, the constraints in the database (such as primary keys, foreign keys, non-null, etc.) ensure data consistency and integrity, avoiding data redundancy and inconsistencies. The multi-modal database also supports flexible data processing and analysis. Users can perform complex queries based on conditions such as timestamps and spatial coordinates, providing a basis for data mining and analysis.

[0159] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to listing the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any person skilled in the art under the teaching of this specification fall within the substantial scope of this specification and should be protected by the present invention.

Claims

1. A multi-mode database storage method for mobile robots, characterized in that Including: A multi-modal data table creation step for assigning corresponding table names to each type of sensing data, defining the specific structure information of each table, and using SQL statements to create tables in the same database and allocate corresponding storage spaces; A sensing data acquisition control step for controlling the activation of all sensor nodes and collecting sensing data of each modality; And a multi-modal data conversion step for respectively performing corresponding data conversions on the collected sensing data of each modality, and then storing the converted results in the storage space pre-allocated in the database in a relational model.

2. The multi-modal database storage method for a mobile robot according to claim 1, characterized in that The multi-modal data table creation step is specifically as follows: First, according to the organizational structure of the data, the data input by various sensors mounted on the mobile robot is divided into five modalities: document, point cloud, image, trajectory, and environmental map; Then, define the corresponding table storage structure according to the structure information of each modality data input by the sensor, including attribute names, attribute data types, and primary keys, foreign keys, uniqueness constraints, non-null constraints, default values, and index constraint conditions; Immediately afterwards, create tables in the database through SQL statements and allocate corresponding storage spaces for each table; Finally, verify whether the table storage structure of each modality data can meet the storage requirements of the data collected by the corresponding modality sensor, so as to ensure that data of different modalities can be stored in a predetermined format.

3. The multi-modal database storage method for a mobile robot according to claim 1, characterized in that In the sensing data acquisition control step, different sensors are mounted on the mobile robot, which are respectively used to collect different sensing data, specifically including document, point cloud, image, trajectory, and environmental map data.

4. The multi-modal database storage method for a mobile robot according to claim 3, characterized in that The multi-modal data conversion step further includes: A document data conversion step of storing the JSON type document data as a whole as a variable, assigning a unique identifier id and representing it in the built-in JSON variable format of the database, and storing the converted vector data in a relational model; A point cloud data conversion step of converting each point cloud data point into a tuple in a relational model, assigning a unique identifier id, splitting the coordinates of the point cloud data point into several floating-point attributes, and respectively converting the additional attributes of the point cloud data point into attributes in the relational table; An image data conversion step of using a key-value pair storage method to convert the image as a whole into BLOB type data, assigning a unique identifier id for storage, and storing the converted image model data in a relational model; A trajectory data conversion step of converting the trajectory data into a relational table, where each row in the relational table represents a trajectory point of the robot car, including position coordinates and timestamps, and generating a unique identifier id for each trajectory point for storage. And converting each environmental map into a tuple in the relational model. During the conversion process, each environmental map data is converted into a binary object BLOB and stored with a unique identifier id. The width and height of the environmental map, as well as the coordinates of the map origin, are respectively converted into attributes in the relational table, which is the environmental map data conversion step.

5. A multi-mode database storage system for a mobile robot, located at the mobile robot terminal, characterized in that, The multi-modal database storage system for mobile robots includes: A database; A multi-modal data table creation module, which is used to assign corresponding table names to each type of sensing data, define the specific structure information of each table, and at the same time use SQL statements to create tables in the same database and allocate corresponding storage spaces; A sensing data acquisition control module, which is used to control the activation of all sensor nodes and collect sensing data of each modality; And a multi-modal data conversion module, which is used to perform corresponding data conversions on the collected sensing data of each modality respectively, and then store the converted results in the storage space allocated in advance in the database in a relational model.

6. The multi-modal database storage system for mobile robots according to claim 5, wherein The multi-modal database storage system for mobile robots further includes: A syntax parsing module, which is used to receive the input of SQL query statements, convert the query according to the results of lexical analysis and syntax analysis, and generate a syntax tree for the corresponding modality data; And a query processing module, which generates a query execution plan corresponding to the SQL query statement according to the syntax tree and performs the actual operation of physical query execution at the physical level.

7. The multi-modal database storage system for mobile robots according to claim 6, wherein The specific processing process of the syntax parsing module is as follows: First, the syntax parsing module decomposes the SQL query statement into basic symbols or words through lexical analysis, including keywords, table names, field names, and operators; then through syntax analysis, according to the SQL syntax rules, it checks whether the SQL query statement meets the SQL syntax requirements and converts it into a syntax tree; finally, according to the characteristics of different modality data, the syntax tree generated by the SQL statement is optimized and converted to ensure that the query can be executed efficiently.

8. The multi-modal database storage system for mobile robots according to claim 6, wherein The specific process of the query processing module is as follows: The query processing module first parses the query target, filtering conditions, and association relationships in the SQL statement according to the query logic and data structure defined in the syntax tree, and identifies the data modalities involved in the query; Subsequently, combined with different data storage methods, it selects the optimal data access strategy; Finally, by optimizing the query execution plan, including adjusting the join order, optimizing the execution order of filtering conditions, and adopting a parallel computing strategy, it ensures that the query can be executed efficiently and improves data retrieval and computing performance.

9. The multi-modal database storage system for mobile robots according to claim 6, wherein A Bluetooth module is provided on the mobile robot terminal for establishing a wireless communication connection between the mobile robot and the mobile terminal, so as to realize the query of the database in the multi-mode database storage system for the mobile robot.

10. A multi-modal database query method for a mobile robot, based on the multi-modal database storage system for a mobile robot described in claim 9, characterized in that, The multi-mode database query method includes: A multi-mode client input step on the mobile terminal side for receiving SQL query statements for queries of different data types; wherein, the SQL query statements respectively include the table names of the corresponding data and the storage ids for different data types. A data query request sending step for establishing Bluetooth communication between the mobile robot terminal and the mobile terminal, and transmitting the key query information of the SQL query statement parsed by the mobile terminal to the mobile robot through Bluetooth. Wherein, the key query information of the SQL query statement includes the table name of the data table and the id of the corresponding data. A syntax parsing step on the mobile robot side for receiving the input of the SQL query statement through the syntax parsing module and converting the query according to the results of lexical analysis and syntax analysis to generate a syntax tree for the corresponding data model. And a query processing step on the mobile robot side for generating a query execution plan according to the syntax tree through the query processing module and driving the corresponding data processing engine to execute the actual query operation.

Images