A positioning method and system based on a semantic size chain of a semantic object

Abstract

This invention provides a localization method and system based on semantic object size chains. The method includes: acquiring a dataset; constructing a semantic object size chain to establish a two-dimensional semantic grid map; initializing a particle set; updating the particle set, calculating robot motion, performing coordinate transformation, predicting the position of each particle in the particle set to obtain a predicted particle set; calculating the weight value of each particle, extracting environmental semantics, identifying objects, performing foreground / background segmentation and coordinate transformation mapping, determining the corresponding position of the object in the semantic object size chain, using the information of each object in the current frame to inversely solve the robot's position, updating the weight value of the obtained estimated position value set; resampling the particles, continuously updating in a loop until the particles converge, and completing the localization. Compared with mainstream relocalization algorithms, this invention has higher stability, provides richer and more reliable features for robot localization and navigation, and thus can complete more advanced and complex tasks.

Description

Technical Field

[0001] This invention belongs to the field of robot platform mapping, localization and navigation, and more specifically, relates to a localization method and system based on semantic dimension chains of semantic objects. Background Technology

[0002] While the AMCL algorithm generally enables successful particle set convergence for robot localization, it can also fail to converge due to inherent limitations, indicating some areas for improvement. The AMCL algorithm addresses the potential for rapid particle updates during localization, which could lead to convergence errors. Furthermore, to increase data diversity during particle observation, it limits the update speed of particle filtering to prevent missing or rapidly degenerating particles. Consequently, due to insufficient prior information, the robot often requires considerable movement to accurately locate its initial global position. This process necessitates continuously issuing motion commands to the robot to maintain its movement until the particle set converges.

[0003] The current mainstream relocation algorithm AMCL suffers from a lack of semantic information about the environment during localization, resulting in slow convergence and a tendency to make localization errors in similar or dynamic environments. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a localization method and system based on semantic dimension chains of semantic objects. This invention improves the AMCL relocalization algorithm. The algorithm acquires the category attributes and distance relationships of objects in front of it through a camera, and determines the corresponding position of the object in the dimension chain based on the category attributes and distance relationships of objects in the current frame. Then, it uses the information of each object in the current frame to inversely solve the robot's pose, and optimizes the particle weight update method by calculating the matching probability of robot pose estimation and particle state distribution, thereby improving the relocalization success rate and speed.

[0005] According to a first aspect of the present invention, a method for locating semantic dimensions based on semantic objects is provided, comprising the following steps:

[0006] Step 1: Obtain the dataset, construct a semantic object size chain based on the dataset, and build a two-dimensional semantic raster map; the dataset includes semantic information of each object;

[0007] Step 2. Initialize the set of particles randomly scattered in the two-dimensional semantic raster map;

[0008] Step 3. Update the particle set. Calculate the robot's motion, perform coordinate transformation on the robot's motion, and predict the pose of each particle in the particle set to obtain the predicted particle set;

[0009] Step 4: Calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, and update the weight value of the obtained estimated pose value set a second time.

[0010] Step 5: Resample the particles and continuously update them until the particles converge, thus completing the localization.

[0011] Based on the above technical solution, the present invention can also be improved as follows.

[0012] Optionally, in step 1, constructing the semantic object size chain based on the dataset includes:

[0013] Synchronize localization information, depth camera color images, and depth image data. Input the color image into object detection to obtain object detection boxes and categories in the image. Calculate the corresponding positions of objects in the color image in the depth image. Convert the obtained depth data into point cloud information and project it into the global map coordinate system. Incrementally construct an object semantic graph and fully explore the relationships between semantic objects to construct a semantic object size chain.

[0014] Optionally, in step 1, establishing the two-dimensional semantic raster map includes:

[0015] The Bayesian algorithm is used to reduce the impact of false detections and duplicate detections on the object semantic map. The object semantic map and the grid map are aligned by coinciding the origin and positioning to construct a two-dimensional semantic grid map. At the same time, the established two-dimensional grid map and the optimized object semantic map are transformed by coordinate and merged to obtain a merged semantic map, which can basically reflect the semantic information of objects in the indoor environment.

[0016] Optionally, in step 2, the initialization of the randomly scattered particle set in the two-dimensional semantic raster map includes:

[0017] M particles are randomly scattered across the map, each with the same weight, and the sum of all particle weights is 1. The particle set is initialized in the laser map with a weight of 1 / M. During the particle initialization phase, the particle set is:

[0018]

[0019] In the formula, the size of the particle set is χ. t The number of particles is M, where M is greater than 1. When t = 0, it represents the initial particle set at time 0.

[0020] Optionally, in step 3, the prediction of the pose of each particle in the particle set is performed by predicting the particle set based on the odometry model, expressed by the following formula:

[0021]

[0022] In the formula, Let be the position of the robot in the world coordinate system at time t. Let V be the robot's position in the world coordinate system at time t-1, and let V represent the robot's velocity. t Indicates time difference.

[0023] Optionally, in step 4, the identification of objects in the environmental semantics and the background segmentation and coordinate transformation mapping include:

[0024] Once the color image and depth image of the object in the environmental semantics are obtained, the pre-trained YOLO model is used to detect and recognize the color image to obtain the detection result. At this time, the detection box contains the foreground and background. The foreground and background are removed using the segmentation algorithm. The detection box obtained by the deep learning model is used to replace the target box drawn by the user to obtain the target segmentation map of the color image in the viewpoint. The pixel coordinates of the target in the image coordinate system are transformed to the camera coordinate system for semantic mapping.

[0025] Optionally, in step 4, determining the corresponding position of the object in the semantic object size chain includes:

[0026] Sequentially identify the semantic information of objects in the environment, sort the identified objects from left to right, iterate through the constructed semantic object size chain, calculate the distance information between consecutive objects based on the category attributes and distance relationships of the objects identified in the current frame, and determine whether there is a corresponding position in the size chain, until the retrieval is successful.

[0027] Optionally, in step 4, the estimated pose value set obtained by inversely solving the robot's pose is represented as:

[0028] X c ={(x1,y1,θ2),(x2,y2,θ2),...};

[0029] In the formula, X c Estimate the set of position values ​​for the robot, x 1,2... Let x and y coordinates of the robot in the world coordinate system be... 1,2... Let θ be the robot's y-coordinate in the world coordinate system. 1,2... This is the robot's heading angle.

[0030] Optionally, in step 5, the resampling of particles and the continuous cyclical updating until particle convergence to complete the localization includes:

[0031] The resampler replicates high-weight particles and deletes low-weight particles, causing the particles to converge towards locations in the 2D semantic grid map where there are detected objects and similar structural environments. After multiple resampling operations, the particles finally converge to the robot's actual location.

[0032] According to a second aspect of the present invention, a positioning system based on semantic size chains of semantic objects is provided, comprising:

[0033] The raster map construction module is used to acquire the dataset, construct semantic object size chains based on the dataset, and build a two-dimensional semantic raster map; the dataset includes semantic information of each object.

[0034] The particle set preprocessing module is used to initialize the randomly scattered particle set in the two-dimensional semantic raster map;

[0035] The predictive particle set acquisition module is used to update the particle set, calculate the robot motion, perform coordinate transformation on the robot motion, predict the pose of each particle in the particle set, and obtain the predictive particle set.

[0036] The estimated pose value set calculation module is used to calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, and update the weight value of the obtained estimated pose value set a second time.

[0037] The particle convergence module is used to resample particles and continuously update them until the particles converge, thus completing the localization.

[0038] The technical effects and advantages of this invention are as follows:

[0039] This invention addresses the issue of AMCL localization algorithm's tendency to fail in certain scenarios by proposing a method that combines AMCL with a semantic map containing semantic object size chains. During localization, the YOLO deep learning model is used for object recognition and detection. The identified objects are sorted from left to right, and the constructed semantic object size chain is iterated. Based on the category attributes of the objects identified in the current frame and the calculated distances between consecutive objects, it is determined whether a corresponding position exists in the size chain. Visual pre-localization is achieved by constructing a semantic map with semantic object size chains, enabling the robot to achieve initial localization more quickly with limited prior information and motion. By combining AMCL algorithm with environmental map matching and improving the particle weight update method, fine localization is performed, effectively improving the localization efficiency and accuracy of the AMCL algorithm.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0041] Figure 1 A flowchart of a semantic dimension chain localization method based on semantic objects provided in an embodiment of the present invention;

[0042] Figure 2 Environment 1 and Environment 2 and their corresponding laser maps are provided for embodiments of the present invention;

[0043] Figure 3 This is a schematic diagram of the semantic raster probability value update process provided in an embodiment of the present invention;

[0044] Figure 4 The object semantic maps and merged semantic maps of Environment 1 and Environment 2 provided in this embodiment of the invention;

[0045] Figure 5 This refers to the semantic object size chains corresponding to Environment 1 and Environment 2 provided in this embodiment of the invention;

[0046] Figure 6 A flowchart of the dimension chain retrieval process provided for embodiments of the present invention;

[0047] Figure 7 This is a schematic diagram of robot pose, camera view, and pose calculation provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that the positioning method and system based on semantic dimension chains of semantic objects described in this embodiment of the invention are applied to a robot platform; wherein, the robot platform includes: a robot platform chassis, a main controller, a LiDAR sensor and a depth camera; the main controller, the LiDAR sensor and the depth camera are mounted on the robot platform chassis; the main controller is sequentially connected to the robot platform chassis, the LiDAR sensor and the depth camera via wired connections.

[0050] Understandably, given the deficiencies in the background technology, according to a first aspect of the present invention, an embodiment of the present invention proposes a localization method based on semantic size chains of semantic objects, the localization method comprising the following steps:

[0051] Step 1: Obtain the dataset, construct a semantic object size chain based on the dataset, and build a two-dimensional semantic raster map; the dataset includes semantic information of each object;

[0052] In step 1: the creation of a two-dimensional semantic grid map includes: using LiDAR and a robot experimental platform, the robot's real-time localization and grid map creation are completed in ROS based on the Gmapping algorithm.

[0053] The specific steps of constructing a semantic object size chain based on the dataset and establishing a two-dimensional semantic raster map include:

[0054] During the mapping process, localization information from the Gmapping algorithm, color images from the depth camera, and depth image data are synchronized. The color images are input into object detection to obtain object detection boxes and categories. Registration data is used to calculate the corresponding positions of objects in the color image within the depth image. The acquired depth data is converted into point cloud information and projected onto the global map coordinate system. An incremental object semantic map is constructed, fully exploring the relationships between semantic objects and building semantic object size chains. Simultaneously, a Bayesian method is employed to reduce the impact of false and duplicate detections on the object semantic map. The object semantic map and the raster map are aligned at their origins and poses to construct a two-dimensional semantic raster map.

[0055] Furthermore, the establishment of the two-dimensional raster semantic map involves constructing semantic maps in environment 1 and environment 2 based on the Gmapping algorithm, respectively. The corresponding raster maps are shown below. Figure 2 (a) and (b), Figure 2 middle, Figure 2 (a) represents the raster map corresponding to the real environment 1. Figure 2 (a) represents the raster map corresponding to the real environment 2. The experimental environment was constructed using foam boards, with a length and width of approximately 12.50m × 6.08m.

[0056] In actual operation, while the mobile robot traverses the indoor environment to build a grid map, the depth camera, driven by the robot's operating system, publishes RGB format image topics and corresponding depth information topics. Real-time object detection is achieved using a pre-trained model of five common objects in the indoor environment. After recognizing an object, the robot acquires the object detection box and category in the image, converts the acquired depth data into point cloud information, projects it onto the global map coordinate system, and incrementally constructs an object semantic map. Considering the real-time performance of the algorithm, after recognizing an object, the robot uses a nearest neighbor sampling algorithm to reduce the camera depth map and color image to 1 / 4 of the original image, significantly reducing the computational load while still meeting the requirement of a grid map resolution of 0.05m. Furthermore, for ease of observation, cabinets, doors, trash cans, chairs, and stools are visually represented using blue, cyan, red, purple, and green, respectively, when constructing the semantic grid map.

[0057] In this embodiment, a Bayesian method is used to reduce the impact of false detections on semantic map construction. During the robot's mapping process, the camera performs real-time target detection. If a door in the camera's field of view is mistakenly identified as a cabinet, the corresponding semantic grid is successfully updated to the color corresponding to the door as the number of correct identifications increases. The update process is described in [details omitted]. Figure 3 , Figure 3 (a) represents the probability that a grid cell in the corresponding range is occupied by a cabinet when a misidentification occurs. In subsequent frames, the cabinet is not identified in the corresponding grid cell range, therefore the probability that the corresponding grid cell is occupied by a cabinet decreases. (See...) Figure 3 (ad). The robot subsequently identified doors from multiple perspectives and in multiple frames. Therefore, the probability that a corresponding grid cell was occupied by a door increased, resulting in the final grid value. Figure 3 (e).

[0058] Finally, the object semantic raster maps constructed in Environment 1 and Environment 2 are as follows: Figure 4 As shown in (a) and (c). In the figure, Figure 4 (a) represents the semantic map of objects in environment 1. Figure 4 (a) Represents the object semantic map for Environment 2. The merged semantic map was obtained by performing coordinate transformation and merging the 2D grid map established by the LiDAR and the optimized object semantic map, as shown in [reference needed]. Figure 4 (b) and (d), in the figure, Figure 4 (b) is represented as the merged semantic map of environment 2. Figure 4 (d) represents the merged semantic map of Environment 2. In the figure, the obstacles in the corresponding areas of the object map and the grid map are basically aligned, which can basically reflect the semantic information of objects in the indoor environment. This facilitates subsequent localization based on the semantic map. Step 1 starts from the door and constructs a semantic object size chain along the wall in a clockwise direction. The semantic object size chains of Environment 1 and Environment 2 are as follows: Figure 5 As shown in (a) and 5(b). In the figure, Figure 5 (a) represents the semantic object size chain for environment 1. Figure 5 (b) represents the semantic object size chain for environment 2. The corresponding semantic object size chain lookup tables are shown in Table 1 and Table 2, respectively.

[0059] Table 1. Semantic object size chain retrieval table in Environment 1

[0060]

[0061] Table 2 Semantic Object Size Chain Retrieval Table in Environment 2

[0062]

[0063] Each object in the dimension chain has upstream and downstream objects. After the robot identifies the corresponding semantic object, it analyzes the category information of the upstream and downstream objects and retrieves the corresponding semantic object dimension chain to accurately obtain the position of each semantic object in the dimension chain from the current perspective.

[0064] Step 2. Initialize the particle set; wherein, the particle set includes: M particles randomly scattered in the two-dimensional semantic grid map;

[0065] In this embodiment, the particle set is initialized on the main control computer. When the robot has no prior information, its initial pose in the map is unknown. M particles are randomly scattered in the map, and the weight of each particle is set to the same value. The sum of the weights of all particles is 1. The particle set χ0 is initialized in the laser map with a weight of 1 / M.

[0066] In step 2, during the particle initialization phase, the particle set is:

[0067]

[0068] In the formula, the particle set size is χt, the number of particles is M, M is greater than 1, and when t = 0, it represents the initial particle set at time 0.

[0069] Step 3. Update the particle set, calculate the robot's motion, perform coordinate transformation on the robot's motion, predict the pose of each particle in the particle set, and obtain the predicted particle set χ. t ;

[0070] It should be noted that the main controller uses sensor data (such as odometry) of relevant motion quantities acquired by the robot itself, combined with the robot's motion model, to update the particle set. The robot's motion quantities calculated from the obtained sensor data are then transformed into coordinates from the robot's coordinate system to the world coordinate system. Using the particle state estimate from the previous moment, the robot's motion quantities, and the robot's motion model, the pose of each particle in the particle set is predicted, resulting in the predicted particle set χ.t ;

[0071] Specifically, in particle pose prediction, the particle set χ is predicted based on the odometry model. t The formula is:

[0072]

[0073] In the formula, Let be the position of the robot in the world coordinate system at time t. Let V be the robot's position in the world coordinate system at time t-1, and let V represent the robot's velocity. t Indicates time difference.

[0074] Step 4: Calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, obtain the estimated pose value set and update the weight value a second time.

[0075] In this embodiment, the main control combines a likelihood domain observation model and a beam model for laser matching, calculating the weight values ​​of each particle in the particle set. Then, based on the semantic object size chain visual pre-positioning method, the robot equipped with a depth camera extracts environmental semantics. The YOLO object detection model is used to identify objects in the environment, performing foreground-background segmentation and coordinate transformation mapping. Based on the object's category attributes and distance relationships in the current frame, a fast and effective size chain retrieval method is proposed to determine the object's corresponding position in the size chain. Then, the robot's pose is deduced using the object information in the current frame, and the resulting estimated pose value set is used for secondary weight value updates.

[0076] The background segmentation and coordinate transformation mapping of the object are as follows:

[0077] Once the camera acquires the color image and depth map from the current viewpoint, a pre-trained YOLO model is used to detect and identify the color image, yielding the detection results. It can be seen that the detection box contains both foreground and background; the GrabCut segmentation algorithm is first used to remove the background. The detection box obtained from the deep learning model replaces the user-drawn target box. After obtaining the target segmentation map from the color image at the viewpoint, the pixel coordinates of the target in the image coordinate system are transformed to the camera coordinate system by combining the corresponding depth map information and the depth camera calibration parameters. Then, semantic mapping is performed. To reduce the computational load of point cloud processing, only the central half of the detection box is processed. After point cloud filtering and coordinate transformation, the cluster of point clouds closest to the robot is mapped onto a two-dimensional plane.

[0078] Judging the corresponding position of an object in the semantic object size chain according to the category attributes and distance relationships of the objects in the current frame for identifying objects in the environmental semantics includes:

[0079] After the semantic object size chain is constructed, use the camera and YOLO network model to identify the object semantic information in the environment, and sort the identified objects in order from left to right (the object size chain is constructed in clockwise order, so sorting the objects identified in the current frame of the camera from left to right here is convenient for subsequent retrieval of the size chain). Loop through the constructed semantic object size chain, and judge whether there is a corresponding position in the size chain according to the category attributes of the objects identified in the current frame and the calculated distance information between each consecutive object until the retrieval is successful. See the corresponding size chain retrieval flowchart in Figure 6 , the robot identifies the object information in the environment, obtains the coordinate values of each object in the mapping graph, and calculates the distance d between each consecutive object. If d and the corresponding standard distance D in the retrieval table satisfy (1-α)D < d < (1+α)D at this time, the retrieval is successful; otherwise, the retrieval fails. Considering the influence of the hardware detection recognition accuracy, inaccurate object recognition, or point cloud missing on the detection result, α takes an empirical value of 0.2. Finally, retrieve the size chain to judge the position of each object in the size chain.

[0080] After determining the corresponding position of the object in the size chain, calculate the pose of the robot in the world coordinate system according to the object coordinates in the robot coordinate system and the position coordinates of the corresponding object in the grid coordinate system. First, the camera identifies the object to obtain the object point cloud in the camera coordinate system, and perform coordinate mapping on this point cloud to obtain the coordinates (X r , Y r ) of the object center in the robot coordinate system. Obtain the coordinates (X g , Y g ) of this object in the grid coordinate system under the known two-dimensional semantic grid map. Through the coordinate transformation matrix:

[0081]

[0082]

[0083]

[0084] In the formula, (x t , y t , θ t ) is the pose of the robot in the world coordinate system at time t, (X r , Y r ) is the coordinate of the object center in the robot coordinate system, (X w , Y w ) is the coordinate of the object center in the world coordinate system, (Xg ,Y g T represents the coordinates of the object's center in the grid coordinate system. w,r This is the transformation matrix from the robot coordinate system to the world coordinate system.

[0085] Substituting the positions of each object into the above equations, the robot's position and pose can be calculated. A diagram illustrating the robot pose calculation principle can be found below. Figure 7 .

[0086] The robot's pose is deduced by using the information of each object in the current frame, and the resulting set of estimated pose values ​​is expressed by the formula:

[0087] X c ={(x1,y1,θ2),(x2,y2,θ2),...};

[0088] In the formula, X c Estimate the set of position values ​​for the robot, x 1,2... Let x and y coordinates of the robot in the world coordinate system be... 1,2... Let θ be the robot's y-coordinate in the world coordinate system. 1,2... This is the robot's heading angle.

[0089] The formula for the secondary update of the weight value is:

[0090] w t =ηw t

[0091] In the formula, η is the similarity coefficient between laser scanning matching and visual pre-positioning results, and w t For particle weights, the weight value increases for particles concentrated near the estimated pose (η≥1), and decreases for particles farther away (0<η<1). The specific implementation method is as follows:

[0092] a) If Therefore, the particle weights are not updated, where η = 1, w t Keep the current value unchanged;

[0093] b) If For each particle in the particle set (x) t ,y t ,θ t The discrete value M can be calculated using the following formula. d Determine whether the particle is in X. c The weight update coefficient η is determined based on a certain position in the set.

[0094]

[0095] When M d ≤2 and The horizontal field of view of the camera is 70 degrees. The depth camera used in the invention has a horizontal field of view of 70 degrees. It is assumed that the particle is located near the pose obtained by the visual preposition. The probability value of it being near the real position is high, and the particle weight value should be increased. The coefficient η is calculated by the following formula.

[0096]

[0097] When M d ≤2 and It is assumed that although the particle is located near the pose obtained by visual prepositioning, the yaw angle error is large, so the weight value should be reduced, and the coefficient η is calculated according to the following formula;

[0098]

[0099] When M d If the value is greater than 2, the particle moves away from the pose obtained by the visual preposition, the weight value decreases, and the coefficient η is calculated according to the following formula:

[0100]

[0101] With ηw t As the new weight value for each particle, after traversing the particle set χ t After processing all particles, the particle set is weighted and normalized.

[0102] Step 5: Resample the particles and continuously update them until the particles converge, thus completing the localization.

[0103] In this embodiment, after the main controller performs robot pose estimation, it updates the particle weights by combining the particle state distribution with the matching probability of robot pose estimation and the matching probability of lidar observation model with grid map. Then, the particles are resampled and continuously updated until the particles converge.

[0104] When a robot initially finds itself in a similar region of the environment, localization failures are likely to occur. This study examines the effectiveness and superiority of the AMCL (Advanced AMCL Algorithm for Improving Localization) method in similar environments. The robot is operated in place and rotates within a similar environment to perform a self-localization experiment based on a semantic map. Initially, the convergence trend is similar to the original AMCL algorithm, with two particle clusters appearing, located in two similar regions. At this point, the laser data and grid map matching is basically accurate, but the number of particles with "false poses" remains relatively high. During self-localization using the method described in this invention, semantic information in front of the robot is acquired through a depth camera. This is combined with a visual pre-localization method to update particle weights. After multiple observations, the resampler replicates high-weight particles while gradually deleting low-weight particles. Overall, the particles converge towards locations in the map where there are detected objects and similar structural environments. After multiple resampling operations, the final convergence is to the robot's true pose.

[0105] It is understood that the semantic dimension chain-based localization method for semantic objects described in this invention first integrates robot odometry information, laser information, and visual information to establish a two-dimensional semantic grid map. This semantic map fully explores the relationships between semantic objects, constructs semantic object dimension chains, and proposes a fast and effective dimension chain retrieval method. Furthermore, it uses a particle filtering method to simultaneously match the grid map structure and environmental semantic information, enabling the mobile robot to achieve initial localization more quickly with limited prior information and motion in the visual pre-localization method. The particle weight update method is further improved, and the AMCL algorithm is simultaneously combined with environmental map matching for fine localization. Using a constructed robot experimental platform, comparative experiments under different environmental conditions verify the method's improvement in robot particle convergence rate, and its enhanced localization accuracy and robustness, especially when the environment has a certain similarity or undergoes certain changes.

[0106] According to a second aspect of the present invention, embodiments of the present invention provide a positioning system based on semantic size chains of semantic objects, comprising, wherein:

[0107] The raster map construction module is used to acquire the dataset, construct semantic object size chains based on the dataset, and build a two-dimensional semantic raster map; the dataset includes semantic information of each object.

[0108] The particle set preprocessing module is used to initialize the randomly scattered particle set in the two-dimensional semantic raster map;

[0109] The predictive particle set acquisition module is used to update the particle set, calculate the robot motion, perform coordinate transformation on the robot motion, predict the pose of each particle in the particle set, and obtain the predictive particle set.

[0110] The estimated pose value set calculation module is used to calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, obtain the pose estimation value set and perform secondary update of the weight value.

[0111] The particle convergence module is used to resample particles and continuously update them until the particles converge, thus completing the localization.

[0112] It is understood that the positioning system based on semantic dimension chain of semantic objects provided in the embodiments of the present invention corresponds to the positioning method based on semantic dimension chain of semantic objects provided in the foregoing embodiments. The relevant technical features of the positioning system based on semantic dimension chain of semantic objects can be referred to the relevant technical features of the positioning method based on semantic dimension chain of semantic objects, and will not be repeated here.

[0113] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a positioning method based on a semantic dimension chain of semantic objects.

[0114] According to a fourth aspect of the present invention, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a positioning method based on a semantic dimension chain of semantic objects.

[0115] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for locating semantic objects based on semantic dimension chains, characterized in that, The method includes the following steps: Step 1: Obtain the dataset, which includes semantic information of each object; construct a semantic object size chain based on the dataset, and build a two-dimensional semantic grid map, including: synchronous positioning information, depth camera color images and depth image data; input the color images into the object detection, obtain the object detection boxes and categories in the images, calculate the corresponding positions of the objects in the color images in the depth map, convert the obtained depth data into point cloud information and project it to the global map coordinate system, incrementally construct the object semantic map, and fully explore the relationship between each semantic object to construct the semantic object size chain; The Bayesian algorithm is used to reduce the impact of false detections and duplicate detections on the object semantic map. The origin of the object semantic map and the grid map are coincident and the positions are aligned to construct a two-dimensional semantic grid map. At the same time, the coordinate transformation of the constructed two-dimensional grid map and the optimized object semantic map are performed and merged to obtain a merged semantic map. Step 2. Initialize the set of particles randomly scattered in the two-dimensional semantic raster map; Step 3. Update the particle set, calculate the robot motion, perform coordinate transformation on the robot motion, predict the pose of each particle in the particle set, and obtain the predicted particle set. Step 4: Calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, obtain the estimated pose value set and update the weight value a second time. Step 5: Resample the particles and continuously update them until the particles converge, thus completing the localization.

2. The localization method based on semantic dimension chains of semantic objects according to claim 1, characterized in that, The set of particles randomly scattered in the initial two-dimensional semantic raster map includes: M particles are randomly scattered in a 2D semantic raster map, each with the same weight, and the sum of all particle weights is 1. The particle set is initialized in the laser map with a weight of 1 / M. During the particle initialization phase, the particle set is set as follows: In the formula, the size of the particle set is The number of particles is M, where M is greater than 1. When t=0, it represents the initial particle set at time 0.

3. The method for locating semantic objects based on semantic dimension chains according to claim 1, characterized in that, The prediction of the pose of each particle in the particle set is performed by predicting the particle set based on the odometry model, expressed by the following formula: In the formula, Let be the position of the robot in the world coordinate system at time t. Let be the robot's position in the world coordinate system at time t-1. Indicates the robot's speed. Indicates time difference, , , These represent the x and y coordinates and attitude angle of a particle at time t in the map coordinate system. , , These are represented by the x and y coordinates and attitude angle at time t-1, respectively. θ represents the difference in attitude angle from time t-1 to time t.

4. The method for locating semantic objects based on semantic dimension chains according to claim 1, characterized in that, The process of identifying objects in the environmental semantics, performing background segmentation, and coordinate transformation mapping includes: Once the color image and depth image of the object in the environmental semantics are obtained, the pre-trained YOLO model is used to detect and recognize the color image to obtain the detection result. At this time, the detection box contains the foreground and background. The foreground and background are removed using the segmentation algorithm. The detection box obtained by the deep learning model is used to replace the target box drawn by the user to obtain the target segmentation map of the color image in the viewpoint. The pixel coordinates of the target in the image coordinate system are transformed to the camera coordinate system for semantic mapping.

5. The method for locating semantic objects based on semantic dimension chains according to claim 4, characterized in that, The determination of the corresponding position of the object in the semantic object size chain includes: Sequentially identify the semantic information of objects in the environment, sort the identified objects from left to right, iterate through the constructed semantic object size chain, calculate the distance information between consecutive objects based on the category attributes and distance relationships of the objects identified in the current frame, and determine whether there is a corresponding position in the size chain, until the retrieval is successful.

6. The localization method based on semantic dimension chains of semantic objects according to claim 5, characterized in that, The formula for the estimated pose set obtained by inversely solving the robot's pose is expressed as follows: ； In the formula, Estimate the set of position values ​​for the robot. Let x be the robot's x-coordinate in the world coordinate system. Let y be the robot's y-coordinate in the world coordinate system. This is the robot's heading angle.

7. The localization method based on semantic dimension chains of semantic objects according to claim 1, characterized in that, The step of resampling the particles and continuously updating them until the particles converge to complete the localization includes: During the resampling process, high-weight particles are copied and low-weight particles are deleted, so that the particles converge to locations in the two-dimensional semantic grid map where there are detected objects and similar structural environments. After multiple resamplings, they finally converge to the robot's actual location.

8. A positioning system based on semantic dimension chains of semantic objects, characterized in that, include: The raster map construction module is used to acquire a dataset, which includes semantic information of each object. Based on the dataset, a semantic object size chain is constructed, and a two-dimensional semantic raster map is built. The dataset includes: synchronous positioning information, depth camera color images and depth image data. The color images are input into the object detection to obtain the object detection boxes and categories in the images. The corresponding positions of the objects in the color images in the depth images are calculated. The acquired depth data is converted into point cloud information and projected to the global map coordinate system. The object semantic map is incrementally constructed, and the relationships between each semantic object are fully explored to construct the semantic object size chain. The Bayesian algorithm is used to reduce the impact of false detections and duplicate detections on the object semantic map. The origin of the object semantic map and the grid map are coincident and the positions are aligned to construct a two-dimensional semantic grid map. At the same time, the coordinate transformation of the constructed two-dimensional grid map and the optimized object semantic map are performed and merged to obtain a merged semantic map. The particle set preprocessing module is used to initialize the randomly scattered particle set in the two-dimensional semantic raster map; The predictive particle set acquisition module is used to update the particle set, calculate the robot motion, perform coordinate transformation on the robot motion, predict the pose of each particle in the particle set, and obtain the predictive particle set. The estimated pose value set calculation module is used to calculate the weight value of each particle in the predicted particle set, extract the environmental semantics in the semantic object size chain, identify the objects in the environmental semantics and perform background segmentation and coordinate transformation mapping, determine the corresponding position of the object in the semantic object size chain, solve the robot's pose in reverse, and update the weight value of the obtained estimated pose value set a second time. The particle convergence module is used to resample particles and continuously update them until the particles converge, thus completing the localization.

Images