Robot positioning system and method

Abstract

The invention discloses a robot positioning system and method, and the system comprises a data preprocessing module which is used for carrying out the preprocessing operation of received sensor data and parameters; the tracking module is used for creating a Frame object for each frame of image data, and packaging related information obtained by processing the current frame of image data, I MU data pre-integration of the time corresponding to the current frame of image data and speedometer data pre-integration of the time corresponding to the current frame of image data into the Frame object corresponding to the current frame of image data; by using associated map data which is extracted from a map management module and is fused with semantic features as a spatial constraint, pre-integrating related information, an I MU pre-integration result and odometer data, taking the pre-integration result as an initial value of current frame image data to participate in optimization calculation, and jointly calculating robot pose data; and the map management module is used for managing a visual semantic map, storing associated map data required by current positioning, and correcting the associated map data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a robot system and method. BACKGROUND

[0002] With the development of AI technology, the intelligent level of robots is continuously improved, and service robots are rapidly developed and applied. For example, a companion robot can provide emotional support, life assistance, etc. for different family members.

[0003] In contrast to a relatively simple working environment, service robots need to face more complex and variable unstructured scenes. Here, the environment is complex and variable, and especially in dynamic scenes, various unexpected situations may occur at any time, which undoubtedly puts very high requirements on the positioning accuracy of the robot.

[0004] The positioning accuracy of a service robot directly affects its safety, functionality and user experience. For example, in a home environment, there are mobile targets such as children and pets, and low-precision positioning can cause the robot to be unable to determine the position of obstacles in real time, greatly increasing the risk of collision; fall detection, fire warning and other security functions rely on centimeter-level positioning, and if the positioning error is too large, it may cause alarm delay or false alarm, threatening the safety of the elderly or children; in a complex house type, low-precision positioning can cause path planning errors, and the cleaning coverage rate can be reduced by more than 30%; positioning drift can easily cause the robot to be stuck in a table leg or power line, and the average number of daily stuck times can reach several times, increasing the user's maintenance burden.

[0005] Therefore, it is currently an urgent problem to provide a service robot scheme that effectively improves the positioning accuracy of a household service robot, thereby improving the reliability of a household service robot in a complex environment and realizing the intelligent landing of a household service robot. SUMMARY

[0006] The main purpose of the present application is to disclose a robot positioning system and method to at least solve the problem that in the related art, due to the need for service robots to face more complex and variable unstructured scenes, low-precision positioning can cause the service robot to be unable to determine the position of obstacles in real time, thereby greatly increasing the risk of collision.

[0007] According to one aspect of the present application, a robot positioning system is provided.

[0008] The robot positioning system according to the application comprises: a data preprocessing module, which is used for performing a preprocessing operation on received sensor data and parameters, wherein the sensor data comprises image data collected by a multi-view vision module, IMU data and odometry data; a tracking module, which is connected with the data preprocessing module, creates a Frame object for each frame of image data from the data preprocessing module, encapsulates relevant information obtained by processing current frame image data, IMU data pre-integration at a time corresponding to the current frame image data and odometry data pre-integration at the time corresponding to the current frame image data in the Frame object corresponding to the current frame image data, uses associated map data with fused semantic features extracted from a map management module as spatial constraints, and uses the relevant information, the IMU pre-integration result and the odometry data pre-integration as initial values of the current frame image data to participate in optimization calculation, so as to jointly calculate robot pose data.

[0009] According to another aspect of the application, a robot positioning method is provided.

[0010] The robot positioning method according to the application comprises: performing a preprocessing operation on received sensor data and parameters, wherein the sensor data comprises image data collected by a multi-view vision module, IMU data and odometry data; creating a Frame object for each frame of image data, encapsulating relevant information obtained by processing current frame image data, IMU data pre-integration at a time corresponding to the current frame image data and odometry data pre-integration at the time corresponding to the current frame image data in the Frame object corresponding to the current frame image data; using associated map data as spatial constraints, and using the relevant information, the IMU pre-integration result and the odometry data pre-integration as initial values of the current frame image data to participate in optimization calculation, so as to jointly calculate robot pose data.

[0011] According to the application, a robot positioning system and method are provided, which are based on a multi-sensor fusion mode of a binocular vision module, an inertial measurement unit (IMU) and an odometer, and realize high-precision and high-robustness Vi-SLAM by combining semantic features, effectively improve the positioning accuracy of a household service robot, improve the reliability of the household service robot in a complex environment, and realize intelligent landing of the service robot. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a structural block diagram of a robot positioning system according to an embodiment of the application;

[0013] Figure 2 is an interaction diagram of a system including a robot positioning system, an SDK and a host computer according to a preferred embodiment of the present application;

[0014] Figure 3 is a service execution flowchart of a tracking module according to a preferred embodiment of the present application;

[0015] Figure 4 is a flowchart of a robot positioning method according to an embodiment of the present application;

[0016] Figure 5 is a flowchart of a robot positioning method according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0017] The specific implementation of the present application will be described in detail below in conjunction with the accompanying drawings of the specification.

[0018] According to an embodiment of the present application, a robot positioning system is provided.

[0019] Figure 1 is a structural block diagram of a robot positioning system according to an embodiment of the present application. As shown in Figure 1 , the robot positioning system includes:

[0020] a data preprocessing module 10, configured to perform preprocessing operations on received sensor data and parameters, wherein the sensor data includes image data collected by a multi-view vision module, IMU data and odometry data;

[0021] a tracking module 12, connected to the data preprocessing module 10, configured to create a Frame object for each frame of image data from the data preprocessing module, encapsulate relevant information processed according to the current frame of image data, IMU data pre-integration at a time corresponding to the current frame of image data and odometry data pre-integration at a time corresponding to the current frame of image data in the Frame object corresponding to the current frame of image data, use associated map data extracted from a map management module as spatial constraints, and use the relevant information, the IMU pre-integration result and the odometry data pre-integration as initial values of the current frame of image data to participate in optimization calculation, to jointly calculate robot pose data;

[0022] a map management module 14, connected to the tracking module 12, configured to manage a visual semantic map, store associated map data required for current positioning, and correct the associated map data.

[0023] According to an embodiment of the present application, a robot positioning method is provided. Figure 1The robot positioning system shown is based on a binocular vision module, an inertial measurement unit (IMU), an odometer, and other multi-sensor fusion methods, and combines semantic features to achieve high-precision and high-robustness Vi-SLAM, effectively improving the positioning accuracy of service robots, improving the reliability of service robots in complex environments, and realizing the intelligent landing of service robots.

[0024] As shown in Figure 2 The data preprocessing module 10 is an interface for the robot positioning system 1 to interface with the outside, used for preprocessing the received data (such as sensor data (image data collected by the multi-view vision module, IMU data, odometer data), various parameters, etc.), for example, performing data format conversion, data type, format and data itself correctness detection, and data time alignment processing.

[0025] The data preprocessing module 10 mainly involves the conversion of time difference, data format and unit of various data; time strong alignment of data needed in the robot positioning system is done to ensure that the binocular image, IMU data and odometer sampling time are aligned, with a time error <1ms; and format order transformation is performed, and the processed results are put into the corresponding structure of the corresponding object, so that the next part of the positioning system can be normally used.

[0026] Specifically, as shown in Figure 2 The data preprocessing module 10 will obtain data required for internal SLAM calculation from a software development kit (SDK) 2, including IMU data (including angular velocity and acceleration measurement values), image data collected by the multi-view vision module (such as binocular vision image data), and odometer data (such as wheeled odometer data). The data preprocessing module is also used to feed back abnormal data in the robot positioning system to the SDK.

[0027] The SDK encapsulates the underlying communication protocol and control instructions, providing a high-level API interface for the host computer. The host computer communicates with the robot controller through the SDK and processes data from the robot positioning system. The host computer is responsible for integrating the path generated by the global planner and the real-time control command generated by the local planner. At the same time, the host computer software also provides a visual interface to display the position, map and running trajectory of the robot in real time.

[0028] Preferably, the tracking module 12 can further comprise: a data receiving unit 120 configured to receive the image data, the IMU data comprising angular velocity and acceleration measurements, and the odometry data configured to provide motion estimates from the data pre-processing module; a frame creation and encapsulation unit 122 configured to create a frame object for each frame of image data, encapsulate relevant information obtained from processing the current frame of image data, pre-integrated IMU data corresponding to the time of the current frame of image data, and pre-integrated odometry data corresponding to the time of the current frame of image data in the frame object corresponding to the current frame of image data, wherein the relevant information comprises image feature points extracted from the image data and computed descriptors, multi-view image matching information, and 3D point information computed through geometric relationships; a map data extraction unit 124 configured to extract relevant map data required for current positioning from the map management module, wherein the relevant map data mainly comprises local map points and co-visible key frames; and a pose solving unit 126 configured to use the local map points and co-visible key frames extracted from the map management module as spatial constraints, and use the relevant information, the IMU pre-integrated results, and the odometry data pre-integration as initial values of the current frame of image data to participate in optimization solving, to jointly solve optimal robot pose data.

[0029] Preferably, the tracking module 12 can further comprise:

[0030] a data receiving unit 120 configured to receive the image data, the IMU data comprising angular velocity and acceleration measurements, and the odometry data configured to provide motion estimates from the data pre-processing module;

[0031] a frame creation and encapsulation unit 122 configured to create a frame object for each frame of image data, encapsulate relevant information obtained from processing the current frame of image data, pre-integrated IMU data corresponding to the time of the current frame of image data, and pre-integrated odometry data corresponding to the time of the current frame of image data in the frame object corresponding to the current frame of image data, wherein the relevant information comprises image feature points extracted from the image data and computed descriptors, multi-view image matching information, and 3D point information computed through geometric relationships;

[0032] a map data extraction unit 124 configured to extract relevant map data required for current positioning from the map management module, wherein the relevant map data mainly comprises local map points and co-visible key frames;

[0033] The pose solving unit 126 is configured to utilize the local map points extracted from the map management module and the co-visibility key frames as spatial constraints, and pre-integrate the relevant information, the IMU pre-integration result and the odometer data pre-integration as initial values of the current frame image data to participate in the optimization solving, and jointly solve the optimal robot pose data.

[0034] Preferably, the tracking module 12 further comprises:

[0035] The key frame judgment unit 128 is configured to judge whether the current frame is a key frame according to the robot pose data.

[0036] The processing unit 130 is configured to, when the key frame judgment unit outputs yes, create a key frame KeyFrame object and add the created KeyFrame object to a specified thread, and when the key frame judgment unit outputs no, continue to return to wait for the next frame data from the data preprocessing module.

[0037] Preferably, the tracking module 12 further comprises:

[0038] The semantic feature recognition unit 132 is configured to recognize semantic features using a semantic segmentation network for the image data, capture the context information of the image, and generate a semantic segmentation result.

[0039] The map generation unit 134 is configured to construct a 3D environment map containing semantic labels based on the semantic segmentation result and the depth information obtained from the image data collected by the multi-view vision module and the visual solving, project the 2D pixel-level semantic information to the 3D space, and form a point cloud map with semantic labels.

[0040] The feature extraction unit 136 is configured to evaluate the dynamic characteristics of each region in the current scene according to the semantic labels, and extract the image feature points according to the dynamic characteristics of each region.

[0041] Therefore, after receiving the data processed by the data preprocessing module, the tracking module fuses the sensor data of the binocular vision module, the IMU, the wheeled odometer and the like, specifically including: semantic feature extraction and matching, dynamic weight fusion of multi-sensor recursive results, solving of the current positioning information (robot pose data), and judgment of whether the current data is to be added as key data to the next thread for further processing.

[0042] The above preferred embodiments of the tracking module will be further described below. Figure 3

[0043] Figure 3 ​is a service execution flow chart of a tracking module according to a preferred embodiment of the present application. As shown in Figure 3 The main object used by the Tracking module is a Frame object, in which the main information processed for the frame image is stored, including IMU data pre-integration at the time corresponding to the current frame image data, odometer data pre-integration at the time corresponding to the current frame image data, which are used as initial values of the frame image data in the optimization solution. Other sensor data corresponding to the frame data will be processed as follows:

[0044] Step S301: The Tracking module receives sensor data processed by the data preprocessing module; wherein the sensor data includes image data collected by the binocular vision module, odometer data (providing motion estimation), and IMU data (including angular velocity and acceleration measurement values).

[0045] Step S302: The Tracking module creates a Frame object for each frame image data, which encapsulates the relevant information processed for the current frame image data, IMU data pre-integration at the time corresponding to the current frame image data, and odometer data pre-integration at the time corresponding to the current frame image data. The Frame part is mainly designed to preprocess the received image data and store data to be used in subsequent calculations. The preprocessing part mainly includes feature point extraction, descriptor calculation, feature point matching, and 3D point calculation.

[0046] That is, when a new frame of image data arrives, the Tracking module creates a Frame object, which is used to store: relevant information processed for the current frame image data, including matching information of multi-view (for example, binocular) images, 3D point information calculated through geometric relationships, and visual feature data (descriptors calculated by extracting feature points from image data); IMU pre-integration results at the time corresponding to the frame image data (relative pose changes (including increments of rotation, velocity, and displacement) calculated by pre-integrating IMU measurement data from the previous frame to the current frame), and wheeled odometer data (providing motion estimation) at the time corresponding to the frame image data. The feature points are used to match with the previous frame or key frame to estimate the relative motion between frames, and the purpose of IMU pre-integration is to integrate high-frequency IMU measurement data (acceleration and angular velocity) to obtain relative motion information between two frames, which is decoupled from the absolute pose.

[0047] In a specific implementation process, based on the image data collected by the multi-vision module (for example, a binocular vision module) of the service robot in the case of introducing semantic information, the semantic segmentation network (such as LR-ASPP) can be used to identify semantic features such as furniture and door frames, capture the context information of the image, and generate a high-precision semantic segmentation result. Based on the semantic segmentation result, as well as based on the image collected by the multi-vision module and the depth information obtained by visual calculation, the system constructs a 3D environment map containing semantic labels. The 2D pixel-level semantic information is projected into the 3D space to form a point cloud map with semantic labels. The above-mentioned semantic segmentation network can also be used to identify structured elements (such as wall surfaces and floor markings) in complex scenes as geometric constraints to optimize the pose estimation result of the IMU and the odometer. When the binocular vision module detects a low-texture area (such as a pure white wall), the binocular vision module is prone to failure due to the lack of feature points, at which time it can be switched to an IMU+odometer dominant mode, and by introducing semantic features (such as "corridor centerline"), the positioning accuracy can be improved.

[0048] According to the dynamic characteristics of different regions in the current scene according to the semantic labels, appropriate weights are assigned to different semantic regions according to the dynamic evaluation results. For example, "sofa area" and "bed area" are identified as low-dynamic weight areas, and areas such as "wall" are considered as high-static weight areas. High-static weight areas (such as walls) are assigned high weights (for example, 0.9), low-dynamic weight areas (such as sofas and beds) are assigned low weights (for example, 0.1), and unlabeled areas are dynamically adjusted by motion consistency detection.

[0049] Therefore, when extracting image feature points, for semantic regions with a weight value lower than a predetermined threshold (for example, 0.4), i.e. low-dynamic weight areas, only stable features such as corner points can be retained, edge features can be filtered, and stronger motion smoothing constraints (such as increasing the IMU integral error weight) can be applied to the low-dynamic area. For semantic regions with a weight value higher than or equal to a predetermined threshold (for example, 0.4), both corner features and edge features can be retained.

[0050] Step S303: Extracting associated map data required for current positioning from a map management (Map) module, wherein the associated map data mainly includes: local map points (environment feature points within the current field of view), and co-view key frames (other key frames observing the same map points as the current frame).

[0051] Step S304: fuse the observation information of vision, IMU and odometry together, use the local map points and co-visible keyframes extracted from the map management module as spatial constraints, use the relevant information processed according to the image data, and the pre-integration of the IMU data and the pre-integration of the odometry data corresponding to the time of the frame image data as the initial value of the frame image data to participate in the optimization calculation, and jointly calculate the robot pose data.

[0052] Therefore, by fusing multi-sensor information such as vision, IMU and odometry, and using the spatial constraints provided by the prior map, the accuracy of positioning and mapping can be further improved.

[0053] In the specific implementation process, in the case of introducing semantic information, the system constructs a multi-modal factor graph containing image feature points, IMU data pre-integration and odometry data pre-integration. Among them, the image feature points are used to constrain the pose and map points through feature matching, the IMU data pre-integration is used to describe the IMU motion constraint between adjacent keyframes, and the odometry data pre-integration data is used to describe the relative motion constraint. The above semantic weight value is introduced into the optimization objective function as a prior constraint to distinguish static background and dynamic objects. The optimal robot pose data (position and direction) and map state information are solved by a nonlinear least squares method. In the above graph optimization process, the semantic prior helps the system to better distinguish static background and dynamic objects, thereby improving the accuracy of pose estimation.

[0054] By introducing semantic information, the problem of decreased accuracy of robot pose estimation in complex environments is effectively solved, and the positioning accuracy of home service robots is effectively improved, providing more reliable environmental perception ability for service robot applications.

[0055] Step S305: determine whether the current frame is a key frame KeyFrame according to the pose information; if yes, execute step S306; if no, i.e. the current frame is not a key frame, wait for the next frame data to come. Return to execute step S301.

[0056] The Tracking thread performs specific data type conversion on the key data, and after creating the object of this type, some subsequent data in the Frame object is operated, and this data object is also the main data object stored in the local map.

[0057] Specifically, whether the current frame is a key frame can be determined by checking the pose difference between the current frame and the reference key frame. If the motion is large (the translation or rotation exceeds the set threshold), a new key frame is needed to capture the environmental changes. In addition, whether the current frame is a key frame can also be determined according to the result of pose optimization. For example, if the re-projection error after pose optimization is large, it may indicate that the current pose is unreliable, and a new key frame is needed to provide a more stable reference.

[0058] Step S306: If the current frame is a key frame, a KeyFrame object is created and added to the designated thread. If the current frame is a key frame, a KeyFrame object needs to be created. The KeyFrame object should contain the pose, timestamp, extracted feature points and their corresponding map points, camera intrinsic parameters and other information of the current frame. After creation, it needs to be added to the designated thread, usually added to the Local Mapping thread, so as to perform local map construction and optimization, such as inserting key frames, generating new map points, local bundle adjustment, etc.

[0059] In the specific implementation process, after obtaining new key frame data, the state of the current local bundle adjustment (LBA) thread will be judged. If the LBA thread can run normally, the current thread state will be modified so that the management thread can obtain the current thread running state. The image data, IMU data, predicted score results, and odometry data predicted score results are added to the local optimization window, an optimizer is constructed, optimization is performed, and finally the optimized results are updated.

[0060] If the current frame is not a key frame, the system does not need to perform key frame related processing, but continues to wait for the next frame data to arrive and continues to perform tracking and pose estimation. In this way, the system can only process frames that are important for map construction and positioning, reducing the amount of calculation and improving real-time performance.

[0061] Among them, the above-mentioned map management (Map) module is mainly responsible for the management of visual semantic map, stores the associated map data needed for current positioning, and corrects the associated map data according to the data processing results of each part.

[0062] The map management (Map) module will be called in the form of a thread and created in the form of a class. It mainly operates on the created and updated key frames KeyFrame, maintains a global map inside the positioning system, and places some unused data in the database when the memory exceeds the defined storage capacity.

[0063] Specifically, the map management module is responsible for the management of visual semantic map, including map data storage, associated data extraction, dynamic update and memory optimization, supporting environment perception and pose solution in real-time positioning process. Its core functions can be summarized as:

[0064] Semantic map storage and organization: store the global map containing geometric features and semantic information in a structured way;

[0065] Dynamic extraction of relevant data: provide local map points and co-visibility keyframes within the current field of view for positioning system;

[0066] Map correction and update: optimize map accuracy based on sensor data processing results;

[0067] Memory and database collaborative management: when the memory exceeds the defined storage capacity, some data that is not used for a short time will be stored in the database to ensure efficient operation of the system.

[0068] Preferably, as shown in Figure 2 The robot positioning system described above can also include a local mapping module 16 connected to the tracking module and the map management module, respectively, for monitoring whether there are keyframes from the tracking module, inserting the keyframes into the local map when keyframes from the tracking module are detected, establishing co-visibility relationships between keyframes, eliminating map points that do not meet predetermined conditions, creating new map points, fusing redundant map points through feature matching, and sending the processed local map data to the map management module.

[0069] In the specific implementation process, the local mapping (Local Mapping) module 16 combines local map data for joint optimization and feeds back and corrects the processing results to the local map. Through the co-visibility relationship of visual images and continuous IMU, odometer and other sensor data, multi-sensor tight coupling optimization is solved, which updates the local map data. The Local Mapping module further optimizes the data to obtain corrected data and transmits the data to the map management module for processing.

[0070] The Local Mapping module continuously monitors the arrival of new keyframes during system operation and activates the processing flow immediately when new keyframes are detected. First, the keyframe insertion operation is performed to add the keyframes transmitted by the tracking (Tracking) module to the local map and establish co-visibility relationships between the keyframes. Then, the map points are managed in detail, including eliminating map points that do not meet the quality requirements, such as dynamically identifying moving objects (such as pedestrians, etc.), which can actively eliminate these interference points to improve positioning robustness. New map points are created, and redundant map points are fused through feature matching. The data processed by the Local Mapping module is transmitted to the map management module for further processing. This modular design enables the system to effectively control error accumulation, providing an important guarantee for the real-time and accuracy of the SLAM system. Through the continuous feedback correction mechanism, the system can maintain stable operation for a long time, laying a foundation for higher-level global optimization.

[0071] Preferably, as shown inFigure 2 As shown in the above robot positioning system can also include: loop closure module 18, respectively, with local mapping module 16 and the above map management module 14 is connected for based on deep learning method to extract the depth semantic features of the current key frame, the current key frame and the history key frame are matched to find the loop similar area, when detecting the loop, based on visual calculation of the relative pose information of the current key frame and the candidate key frame, and using posegraph optimization error.

[0072] Loop closure and loop correction and posegraph correction loop error. LoopClosuring module will detect the current data, judge whether the current positioning is similar to the historical data (that is, whether the robot has been to the positioning position).

[0073] In the implementation process, the following methods can be included:

[0074] Loop closure: visual semantic feature loop, maximum probability loop similar area.

[0075] (1) Feature extraction: based on deep learning method to extract the depth semantic features of the image. Compared with traditional manual features (such as ORB), these features have stronger robustness to light changes, view changes and seasonal changes, and can more accurately describe the essence of complex change scenes.

[0076] (2) Generate candidate frames: the system matches the current key frame with the historical key frame retrieved from the bag-of-words database. The bag-of-words model quantizes image features into visual words and builds a database based on visual words, so that at least one candidate key frame most similar to the current frame can be quickly screened out, realizing fast loop candidate search.

[0077] (3) Similarity score: by calculating the similarity score of the current frame and the candidate key frame in the bag-of-words model, and sorting according to the score, the most probable similar area is selected as the loop hypothesis.

[0078] Loop solving: based on visual calculation of the relative pose of the key frame and the candidate key frame.

[0079] (1) Similarity transformation calculation: when detecting the loop, the similarity transformation (including rotation, translation and scale factor) between the current key frame and the loop candidate key frame needs to be calculated. This step usually uses RANSAC and other robust estimation algorithms to solve by matching the map points between the two frames, in order to eliminate the influence of the inherent scale drift in monocular SLAM.

[0080] (2) Consistency verification: after similarity transformation calculation, the similarity transformation is propagated to the key frames connected with the current frame for consistency check to ensure that the loop is valid within the local range.

[0081] (3) Map point matching and fusion: According to the computed similarity transformation, match and fuse the map points of the loop-closing frame and its connected keyframes with the map points of the current keyframe, thus updating the map and enhancing its consistency.

[0082] Posegraph corrects loop closing error: According to the loop, use the posegraph to correct the error.

[0083] (1) Constructing posegraph: Posegraph is a graph model in which nodes represent robot poses and edges represent constraints between poses. In loop closing correction, in addition to the conventional constraints between adjacent frames, new loop constraint edges will be added between the keyframes that form the loop.

[0084] (2) Optimization solution: The goal of posegraph optimization is to find a set of optimal poses that minimize the overall error of all constraints, including the newly added loop constraints.

[0085] (3) Global consistency: Through optimization, the strong constraints introduced by the loop can effectively "pull back" the accumulated error to the correct position, resulting in a topologically consistent and accurate trajectory and map. In systems such as OpenVSLAM, after posegraph optimization, global Bundle Adjustment may also be initiated for higher-precision global optimization.

[0086] In summary, the LoopClosing module effectively solves the accumulated error problem in SLAM systems through visual semantic feature search, loop calculation, and posegraph optimization correction, ensuring the accuracy and robustness of the system in long-term operation.

[0087] Preferably, as shown in Figure 2 the above robot positioning system further comprises at least one of the following:

[0088] Thread management module 20 for real-time monitoring of each thread of the above robot positioning system, performing repair operations when abnormal states are found, uploading current state information to the SDK, notifying the SDK to restart the positioning system when the exception cannot be repaired, and outputting exception information to the log when the above robot positioning system is detected to have a problem;

[0089] Data management module 22 for performing memory management and data management, wherein the variables inside the thread and the object data occupying more than the predetermined memory space are managed by pre-allocating fixed-size memory blocks, and the map data not used within the predetermined time is stored in the database.

[0090] As shown in Figure 2 Figure 2 ​Thread management module 20, connected with data preprocessing module 10, tracking module 12, map management module 14, local mapping module 16 and loop closure module 18 respectively, is responsible for management of threads, state bits, abnormal states and the like, and interacts with SDK for various states.

[0091] In the manner of thread management class, the thread management module is responsible for creation and destruction of threads, and related operations such as invocation and suspension. All threads will be monitored by the thread management module throughout the process, and when there is a problem, the thread management module will immediately trigger a special event handling mechanism. The thread management module defines corresponding start method, run method, stop method, wait method, reset method and destroy method for each thread, and each thread can call the corresponding method when needed, facilitating unified management.

[0092] To ensure the availability and reliability of the entire system, the thread management module will monitor the entire system in real time. When an abnormal state is detected, the system will first perform self-repair and upload the current state to the SDK. When the abnormality cannot be repaired, the SDK will be notified to restart the positioning system and perform corresponding work. At the same time, a log system is defined, and when a problem occurs in the positioning system, the abnormal information will be output to the log for subsequent problem positioning.

[0093] As shown in FIG. 1, Figure 2 Data management module 22, connected with map management module 14, local mapping module 16, tracking module 12 and thread management module 20, is responsible for memory management and data management. Among them, the data management module stores map data that will not be used for a short time into the database.

[0094] In terms of memory management, pre-allocated fixed memory is used for fixed variables. By pre-allocating a fixed-size memory block to manage internal variables and large memory objects (such as image data) in the thread, the overhead of dynamic memory allocation and memory fragmentation problems can be effectively reduced. In addition, smart pointers are used for automatic memory recovery. All feature points, ordinary frames, key frames and 3D points are managed by std smart pointers to prevent wild pointers and memory leaks. At the same time, smart pointers are used for data synchronization between all threads. When the memory usage exceeds the predetermined storage after a long time of running, the data management module will operate the map data that will not be used for a short time in the algorithm and place it in the database.

[0095] According to the embodiment of the present application, a robot positioning method is provided.

[0096] Figure 4is a flowchart of a robot positioning method according to an embodiment of the present application. As shown in Figure 4 the robot positioning method comprises the following processes:

[0097] Step S401: performing a preprocessing operation on the received sensor data and parameters, wherein the sensor data includes image data collected by a multi-vision module, IMU data, and odometry data;

[0098] Step S402: creating a Frame object for each frame of image data, and encapsulating relevant information obtained by processing the current frame of image data, pre-integrated IMU data corresponding to the time of the current frame of image data, and pre-integrated odometry data corresponding to the time of the current frame of image data in the Frame object corresponding to the current frame of image data;

[0099] Step S403: using the associated map data with fused semantic features as a spatial constraint, and using the relevant information, the IMU pre-integration result, and the odometry data pre-integration as initial values of the current frame of image data to participate in optimization calculation, to jointly calculate robot pose data.

[0100] The robot positioning method shown in Figure 4 based on a multi-sensor fusion mode of a binocular vision module, an inertial measurement unit (IMU), an odometer, and the like, and combines semantic features to achieve high-precision and high-robustness Vi-SLAM, effectively improves the positioning accuracy of a service robot, improves the reliability of the service robot in a complex environment, and realizes intelligent landing of the service robot.

[0101] Preferably, as shown in Figure 5 the robot positioning method can further comprise the following processes:

[0102] Step S404: determining whether the current frame is a key frame according to the robot pose data;

[0103] Step S405: in the case where the current frame is a key frame, creating a KeyFrame object, and adding the created KeyFrame object to a specified thread; in the case where the current frame is not a key frame, continuing to return to wait for the next frame of data;

[0104] Step S406: when a key frame is detected, inserting the key frame into a local map, establishing a co-view relationship between key frames, eliminating map points that do not meet predetermined conditions, creating new map points, fusing redundant map points through feature matching, and sending the processed local map data to a map management module;

[0105] Step S407: Extract deep semantic features of the current keyframe based on deep learning methods, match the current keyframe with historical keyframes to find closed loop similar regions, and when a closed loop is detected, calculate the relative pose information of the current keyframe and candidate keyframes based on vision, and use the pose graph to optimize the error.

[0106] It should be noted that the specific details of the above robot localization method can be found in the relevant references. Figures 1 to 3 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here. The robot positioning system and method provided have a wide range of applications, applicable not only to service robots but also to other types of robots, such as cleaning robots and delivery robots.

[0107] In summary, using the embodiments provided by this invention, data from multi-view vision modules, IMUs, wheeled odometers, and other sensors are fused. Semantic recognition is performed on the multi-view vision image data, and the dynamic characteristics of each region in the current scene are evaluated based on semantic labels. Image feature points are extracted based on the dynamic evaluation results. For example, in low-dynamic weight regions, only stable features such as corner points can be retained, edge features are filtered out, and stronger motion smoothness constraints are applied to low-dynamic regions (such as increasing the IMU integration error weight). For semantic regions with weight values ​​higher than or equal to a predetermined threshold, both corner and edge features can be retained. With the introduction of semantic information, the system constructs a multimodal factor graph containing image feature points, IMU data pre-integration, and odometer data pre-integration. During graph optimization, semantic weight values ​​are introduced as prior constraints into the optimization objective function, helping the system better distinguish between static backgrounds and dynamic objects, thereby improving the accuracy of pose estimation. Therefore, Vi-SLAM, which combines semantic features to achieve high precision and robustness, effectively solves the problem of decreased robot pose estimation accuracy in complex environments, thereby improving the positioning accuracy of service robots, providing more reliable environmental perception capabilities for service robot applications, and realizing the intelligent implementation of service robots.

[0108] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A robot positioning system, characterized in that, include: The data preprocessing module is used to perform preprocessing operations on the received sensor data and parameters, wherein the sensor data includes: image data acquired by the multi-view vision module, IMU data, and odometer data; The tracking module, connected to the data preprocessing module, creates a Frame object for each frame of image data from the data preprocessing module. It encapsulates the relevant information obtained from processing the current frame of image data, the IMU data pre-integration corresponding to the time of the current frame of image data, and the odometry data pre-integration corresponding to the time of the current frame of image data in the Frame object corresponding to the current frame of image data. It uses the associated map data with fused semantic features extracted from the map management module as spatial constraints, and uses the relevant information, the IMU pre-integration result, and the odometry data pre-integration as the initial values ​​of the current frame of image data to participate in the optimization solution, and jointly solves the robot pose data. The map management module, connected to the tracking module, is used to manage the visual semantic map, store the associated map data required for current positioning, and correct the associated map data.

2. The robot positioning system according to claim 1, characterized in that, The tracking module includes: A data receiving unit is configured to receive the image data from the data preprocessing module, the IMU data including angular velocity and acceleration measurements, and the odometry data for providing motion estimation; An encapsulation unit is created to create a Frame object for each frame of image data. The relevant information obtained from processing the current frame of image data, the pre-integration of IMU data at the time corresponding to the current frame of image data, and the pre-integration of odometer data at the time corresponding to the current frame of image data are encapsulated in the Frame object corresponding to the current frame of image data. The relevant information includes: image feature points extracted from the image data and the calculated descriptors, multi-view image matching information, and 3D point information calculated through geometric relationships. The map data extraction unit is used to extract the associated map data required for current positioning from the map management module. The associated map data mainly includes: local map points and common keyframes. The pose calculation unit is used to utilize local map points extracted from the map management module and common-view keyframes as spatial constraints, and to use the relevant information, the IMU pre-integration results, and the odometry data pre-integration as the initial values ​​of the current frame image data in the optimization calculation to jointly calculate the optimal robot pose data.

3. The robot positioning system according to claim 1 or 2, characterized in that, The tracking module includes: The keyframe determination unit is used to determine whether the current frame is a keyframe based on the robot pose data. The processing unit is configured to create a keyframe object and add the created keyframe object to a specified thread if the keyframe judgment unit outputs yes, and to continue waiting for the next frame of data from the data preprocessing module if the keyframe judgment unit outputs no.

4. The robot positioning system according to claim 1 or 2, characterized in that, The tracking module includes: The semantic feature recognition unit is used to recognize semantic features using a semantic segmentation network for the image data, capture the contextual information of the image, and generate semantic segmentation results. The map generation unit is used to construct a 3D environment map containing semantic labels based on semantic segmentation results, image data collected by multi-view vision modules, and depth information obtained by visual calculation. It projects 2D pixel-level semantic information into 3D space to form a point cloud map with semantic labels. The feature extraction unit is used to evaluate the dynamic characteristics of each region in the current scene based on the semantic labels, and extract the image feature points based on the dynamic evaluation results.

5. The robot positioning system according to claim 1, characterized in that, The robot positioning system also includes: The local mapping module is connected to both the tracking module and the map management module. It monitors whether there are keyframes from the tracking module. When a keyframe from the tracking module is detected, the keyframe is inserted into the local map, and a common-view relationship is established between keyframes. Map points that do not meet the predetermined conditions are removed, new map points are created, redundant map points are fused through feature matching, and the processed local map data is sent to the map management module.

6. The robot positioning system according to claim 1, characterized in that, The robot positioning system also includes: The loop closure module is connected to the local mapping module and the map management module respectively. It is used to extract the deep semantic features of the current keyframe based on deep learning methods, match the current keyframe with historical keyframes to find similar regions for loop closure, and when a loop closure is detected, calculate the relative pose information of the current keyframe and candidate keyframes based on vision and use the pose graph to optimize the error.

7. The robot positioning system according to claim 6, characterized in that, The loop closure module further includes: The closed-loop search unit is used to extract the deep semantic features of the current keyframe based on deep learning methods, match the current keyframe with historical keyframes, filter out at least one candidate keyframe, calculate the similarity score between the current keyframe and the candidate keyframe, sort them according to the similarity score, and determine the closed-loop similar region. The closed-loop solution unit is used to calculate the similarity transformation between the current keyframe and the closed-loop keyframe when a closed loop is detected. After the similarity transformation is calculated, the similarity transformation is propagated to the keyframes connected to the current keyframe for consistency check. Based on the calculated similarity transformation, the map points of the closed-loop keyframe and the keyframes connected to the closed-loop keyframe are matched and fused with the map points of the current keyframe. The closed-loop correction unit is used to construct and optimize the pose graph, determine a set of optimal pose data, and minimize the overall error of all constraints. The constraints include: inter-frame constraints and closed-loop constraint edges added between keyframes that form a closed loop.

8. The robot positioning system according to claim 1, characterized in that, The robot positioning system also includes at least one of the following: The thread management module is used to monitor each thread of the robot positioning system in real time, perform repair operations when an abnormal state is detected, and upload the current status information to the SDK. When the abnormality cannot be repaired, the SDK is notified to restart the positioning system. When a problem is detected in the robot positioning system, the abnormal information is output to the log. The data management module is used to perform memory management and data management. It manages internal variables and object data that occupy more than the predetermined memory space by pre-allocating memory blocks of a fixed size, and stores map data that is not used within a predetermined time into the database.

9. A robot localization method, characterized in that, include: Preprocessing operations are performed on the received sensor data and parameters, wherein the sensor data includes: image data acquired by the multi-view vision module, IMU data, and odometer data; For each frame of image data, a Frame object is created, and the relevant information obtained from processing the current frame of image data, the IMU data pre-integration at the time corresponding to the current frame of image data, and the odometry data pre-integration at the time corresponding to the current frame of image data are encapsulated in the Frame object corresponding to the current frame of image data. Using associated map data as spatial constraints, the relevant information, the IMU pre-integration result, and the odometry data pre-integration are used as initial values ​​of the current frame image data in the optimization solution to jointly calculate the robot pose data.

10. The robot localization method according to claim 9, characterized in that, Also includes: Based on the robot pose data, determine whether the current frame is a keyframe; If the current frame is a keyframe, create a keyframe object and add the created keyframe object to the specified thread. If the current frame is not a keyframe, continue to return and wait for the next frame data. When a keyframe is detected, it is inserted into the local map, and a common-view relationship is established between keyframes. Map points that do not meet the predetermined conditions are removed, new map points are created, redundant map points are fused through feature matching, and the processed local map data is sent to the map management module. Deep learning methods are used to extract deep semantic features of the current keyframe. The current keyframe is matched with historical keyframes to find closed loop similar regions. When a closed loop is detected, the relative pose information of the current keyframe and candidate keyframes is calculated based on vision, and the pose graph is used to optimize the error.

Images