Robot simultaneous semantic localization and mapping method and system based on human-like thinking

Abstract

The application discloses a robot synchronous semantic positioning and mapping method and system based on a human-simulated thought, which comprises the following steps: acquiring environment pictures and spatial structure point cloud data of each frame; recognizing the environment pictures to obtain observed straight lines of positioning road marks; performing cluster division on all the observed straight lines, and dividing the observed straight lines belonging to the same positioning road mark into a cluster; analyzing the observed straight lines in each cluster by means of a least square method to determine the spatial coordinates of each positioning road mark; optimizing the spatial coordinates of the positioning road marks to obtain optimized coordinates of the positioning road marks; obtaining a positioning road mark map according to the optimized coordinates of all the positioning road marks; projecting each frame of point cloud data on a local path to a horizontal direction to generate a line segment map; associating all the line segment maps to obtain an environment structure map; and combining the positioning road mark map and the environment structure map to form a semantic map. The robot synchronous semantic positioning and mapping are realized.

Description

Technical Field

[0001] This invention relates to the field of map building technology, and in particular to a robot synchronous semantic localization and mapping method and system based on human-like concepts. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Current SLAM (Simultaneous Localization and Mapping) schemes for robots focus on low-level environmental features, generating large amounts of point cloud data for robot localization and mapping. Different mapping schemes employ different environmental perception devices, corresponding to different low-level features. Using devices capable of perceiving spatial structure, such as laser sensors and depth cameras, point clouds representing spatial structure can be acquired, and the organizational relationships between point clouds form low-level features. Cameras can capture environmental images, from which various pixel-level low-level features such as ORB and SIFT can be extracted. Based on these low-level features, appropriate mapping algorithms are used, along with backend optimization and loop closure detection methods, to achieve simultaneous localization and mapping for the robot. Because each point in the point cloud formed by low-level features contains relatively little information, a large number of point clouds are needed to represent environmental information. The denser the point cloud, the more precisely it can depict the environmental structure, but excessive pursuit of modeling precision will bring huge computational and storage costs. Localization-oriented mapping does not consider map usage and does not model environmental structure; in this mode, the number of point clouds is relatively small. However, these robot localization and mapping methods differ from human self-localization and map description modes. When humans locate themselves and describe space, they focus on prominent objects or the organizational structure between objects in the environment, using them as semantic symbols for their own location. At the same time, they focus on locations and the accessibility between locations while downplaying complex environmental structures, thus forming abstract semantic maps.

[0004] Early SLAM techniques struggled to quickly and accurately identify specific objects in the environment, making it difficult to obtain scene semantic information. With the development and application of deep learning, extracting semantic information from the environment has become much easier, leading to the proposal of increasingly sophisticated visual semantic SLAM methods. Semantic information is typically acquired using object detection or semantic segmentation. Object detection can obtain the spatial location of semantic landmarks of interest, represented using ellipsoids, cuboids, or known models. Semantic segmentation is generally combined with dense point clouds, adding semantic information to create labeled semantic point clouds. These methods largely follow the design principles of traditional SLAM, adding semantic information to low-level features to improve mapping accuracy, enhance loop closure detection performance, and expand the functionality of the original map. However, they fail to leverage the unique advantages of semantic information, resulting in inaccurate mapping. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method and system for synchronous semantic localization and mapping of robots based on human-like principles, thereby achieving synchronous semantic localization and spatial mapping of robots.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Firstly, a human-like approach to robot synchronous semantic localization and mapping is proposed, including:

[0008] Acquire environmental images and spatial structure point cloud data for each frame;

[0009] The environmental images in each frame are identified to obtain the observation lines of the corresponding positioning landmarks;

[0010] All observed lines are clustered, and observed lines belonging to the same location landmark are grouped into one cluster;

[0011] By using the least squares method, the observed lines in each cluster are analyzed to determine the intersection points of the observed lines in each cluster, which are the spatial coordinates of each corresponding positioning landmark.

[0012] The spatial coordinates of the location landmarks are optimized to obtain the optimized coordinates of the location landmarks;

[0013] Based on the optimized coordinates of all location landmarks, a location landmark map is obtained;

[0014] Project each frame of point cloud data on the local path into the horizontal direction to generate a line segment diagram;

[0015] By associating all the line segment diagrams, an environmental structure diagram can be obtained;

[0016] The location landmark map and the environmental structure map are combined to form a semantic map.

[0017] Secondly, a robot synchronous semantic localization and mapping system based on human-like principles is proposed, including:

[0018] The data acquisition module is used to acquire environmental images and spatial structure point cloud data of the environment for each frame;

[0019] The location landmark acquisition module is used to identify the observation lines of each frame of environmental image and obtain the corresponding location landmarks; it divides all observation lines into clusters, grouping observation lines belonging to the same location landmark into one cluster; it analyzes the observation lines in each cluster using the least squares method to determine the intersection points of the observation lines in each cluster, which are the spatial coordinates of each corresponding location landmark; it optimizes the spatial coordinates of the location landmarks to obtain the optimized coordinates of the location landmarks; and it obtains the location landmark map based on the optimized coordinates of all location landmarks.

[0020] The environment structure map acquisition module is used to project each frame of point cloud data on the local path into the horizontal direction to generate a line segment map; and to associate all the line segment maps to obtain the environment structure map.

[0021] The semantic map acquisition module is used to combine the location landmark map and the environmental structure map to form a semantic map.

[0022] Thirdly, an electronic device is proposed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps described in the robot synchronous semantic localization and mapping method based on humanoid thinking.

[0023] Fourthly, a computer-readable storage medium is proposed for storing computer instructions, which, when executed by a processor, complete the steps described in the humanoid robot synchronous semantic localization and mapping method.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. This invention mimics human self-localization and spatial mapping models, acquiring targets with high-level semantics from the environment as localization landmarks; acquiring spatial structure point cloud data of the environment, and abstracting and simplifying the point cloud to construct an information-dense semantic map. Localization landmarks serve as the basis for the robot to determine its pose, and their properties directly affect the localization effect. This invention uses a camera to capture semantic objects in the environment, and integrates multi-view observation results to estimate the spatial position of semantic objects, forming localization landmarks. Semantic objects of different types and locations together constitute the localization landmark map, allowing the robot to locate itself based on the type and position of surrounding landmarks. Since mobile robot navigation typically uses planar maps, this invention, to more concisely represent the environmental structure, abstracts the acquired point cloud into line segments on a plane. These line segments represent obstacles that the robot is not allowed to cross, forming an environmental structure map. Finally, the localization landmark map and the environmental structure map together form an abstract semantic map, achieving simultaneous semantic localization and spatial mapping for the robot, and the constructed semantic map is closer to the real environment map.

[0026] 2. This invention categorizes observed lines into clusters based on the angular change of observed lines in consecutive frames. Observed lines with angular changes less than a set value are grouped into a single cluster and arranged chronologically. Newly generated clusters of observed lines for the same type of target are merged with historical clusters one by one. The residuals of the least squares intersections are then calculated, and the magnitude of the residuals determines whether they are associated with the same group. This solves the problem that observations of the same positioning landmark may not be completely continuous along the path due to factors such as occlusion, lighting differences, and instability of the target detector, resulting in discontinuous observation lines and inaccurate cluster division. It also eliminates redundant intersections, enabling accurate acquisition of the spatial coordinates of the positioning path.

[0027] 3. The present invention also optimizes the spatial coordinates of the positioning landmarks, so that the overall observation error and mileage error of the optimized positioning landmarks are minimized.

[0028] 4. In constructing the environmental structure diagram, this invention uses the center of the local path corresponding to each line segment diagram as a seed to construct a Voronoi polygon; deletes line segments that exceed the center of the polygon to obtain retained line segments; and associates each retained line segment to obtain the environmental structure diagram. This not only preserves the environmental structure information more completely, but also makes the line segments more continuous and complete.

[0029] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0031] Figure 1 This is a block diagram of the method disclosed in Example 1;

[0032] Figure 2 This is a schematic diagram of the operation and observation of the robot disclosed in Example 1;

[0033] Figure 3 This is the original point cloud image disclosed in Example 1;

[0034] Figure 4 The grid map disclosed in Example 1;

[0035] Figure 5 This is the skeleton extraction diagram disclosed in Example 1;

[0036] Figure 6 The line segment diagram disclosed in Example 1;

[0037] Figure 7The Voronoi polygon disclosed in Example 1;

[0038] Figure 8 This refers to the laser grid map disclosed in Example 1;

[0039] Figure 9 The wheel-type odometer track and the laser odometer track disclosed in Example 1;

[0040] Figure 10 The orientation of the fisheye camera disclosed in Example 1;

[0041] Figure 11 This is a circular view obtained by the fisheye camera disclosed in Example 1;

[0042] Figure 12 The circular view disclosed in Embodiment 1 is unfolded into a rectangular view;

[0043] Figure 13 The results of generating road markers for the wheeled odometer disclosed in Example 1;

[0044] Figure 14 The loop closure detection and optimized wheel odometer road marker results are shown in Example 1.

[0045] Figure 15 The results of generating road markers using the laser odometry disclosed in Example 1;

[0046] Figure 16 This is the environmental structure diagram based on the optimized trajectory disclosed in Example 1;

[0047] Figure 17 This refers to the semantic map generated based on the optimized trajectory disclosed in Example 1;

[0048] Figure 18 A semantic map generated for the entire experimental site disclosed in Example 1. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0051] Example 1

[0052] In this embodiment, a robot synchronous semantic localization and mapping method based on humanoid concepts is disclosed, such as... Figure 1 As shown, it includes:

[0053] Acquire environmental images and spatial structure point cloud data for each frame;

[0054] The environmental images in each frame are identified to obtain the observation lines of the corresponding positioning landmarks;

[0055] All observed lines are clustered, and observed lines belonging to the same location landmark are grouped into one cluster;

[0056] By using the least squares method, the observed lines in each cluster are analyzed to determine the intersection points of the observed lines in each cluster, which are the spatial coordinates of each corresponding positioning landmark.

[0057] The spatial coordinates of the location landmarks are optimized to obtain the optimized coordinates of the location landmarks;

[0058] Based on the optimized coordinates of all location landmarks, a location landmark map is obtained;

[0059] Project each frame of point cloud data on the local path into the horizontal direction to generate a line segment diagram;

[0060] By associating all the line segment diagrams, an environmental structure diagram can be obtained;

[0061] The location landmark map and the environmental structure map are combined to form a semantic map.

[0062] Among them, the location markers come from semantic objects in the environment, and these semantic objects need to have the properties of being easy to identify, having a fixed spatial location, and existing for a long time.

[0063] In this embodiment, the process of acquiring environmental images using a camera mounted on the robot and obtaining the observation line of the positioning landmark based on the environmental map is as follows:

[0064] S21: Recognize the environmental image and obtain the pixel coordinates of the location landmarks in the image.

[0065] An object detector is used to identify the environmental image, resulting in the semantic object y. j The detection bounding box, the pixel coordinates of the center point of the detection bounding box [u I ,v I ] T , which are the pixel coordinates of the landmark in the image, where u I ,v I These represent the horizontal and vertical pixel coordinates of the center point of the detection box, respectively.

[0066] S22: Transform pixel coordinates into observation vectors in the camera coordinate system.

[0067] The pixel coordinates of the landmark in the image are determined using the camera's intrinsic parameters. I v I ] TTransformed into the observation vector G = [u] in the camera coordinate system c v c w c ] T u c v c w c K represents the components of a unit vector in the camera coordinate system along the x, y, and z axes. Different camera models obtain the observation vectors of semantic objects in different ways. For a typical pinhole camera, its intrinsic parameter matrix is ​​K:

[0068]

[0069] S23: Transform the observation vector from the camera coordinate system to the world coordinate system, and translate it to the world coordinates of the camera to obtain the observation line of the positioning landmark.

[0070] Transform the observation vector G from the camera coordinate system to the world coordinate system, and then translate it to the camera's world coordinates to obtain an observation line l. i，j The observed line is represented by a vector of the form [x, y, z, u, v, w], where x, y, z represent the three components of the spatial coordinates of the camera's optical center when the observed line is formed, and u, v, w represent the three components of the direction vector of the line in the world coordinate system.

[0071] Camera at different poses for semantic object y j Observational straight line cluster L j All observed lines are clustered together, with lines belonging to the same location landmark grouped into one cluster. The least squares method is then used to analyze the observed lines within each cluster, determining the intersection points of these lines as the spatial coordinates of the corresponding location landmark.

[0072]

[0073] Where I represents an identity matrix of dimension three, (l i，j ) (1：3) Represents vector l i，j The first three components, (l i，j ) (4：6) Represents vector l i，j The last three components.

[0074] Location landmarks include location information indicating their spatial coordinates, as well as category information of their corresponding semantic objects. These location landmarks scattered in space together constitute a location landmark map.

[0075] Since the camera moves continuously, the observation angle of the same semantic object changes very little between adjacent frames, and the semantic objects are sparsely distributed in the environment, the angle change relationship can be used to associate different observed lines in adjacent frames, forming a cluster of observed lines, such as... Figure 2 As shown. However, due to factors such as occlusion, poor lighting, and instability of the target detector, observations of the same location landmark may not be completely continuous along the path. For example... Figure 2 In the diagram, the observation lines marked with an "×" are missing for some reason, causing discontinuities between the preceding and following observation lines. Therefore, the cluster of observation lines L corresponding to the same semantic object... j It may be split into multiple contiguous subclusters L j，1 L j，2 Each sub-cluster will also correspond to a least-squares intersection point. To obtain more accurate location landmarks and eliminate redundant intersections, these sub-clusters need to be correlated. Based on the angular change of the observed lines in consecutive frames, the observed lines are clustered. Observed lines with angular changes less than a set value are grouped into one cluster, resulting in observed line sub-clusters. For observed line sub-clusters obtained from observations of the same type of target, they are arranged in chronological order. Newly generated observed line sub-clusters are merged with historical observed line sub-clusters one by one, and then the residuals of the least-squares intersection points are calculated. The magnitude of the residuals determines whether they are correlated into the same group. The same type of target here refers to objects identified and marked as the same type by the target detector. These objects are pre-designated as location landmarks. When calculating the spatial coordinates of the location landmarks, the observed line sub-clusters obtained from observations of the same type of target refer to observed line sub-clusters obtained from observations of the same type of location landmark.

[0076] Specifically, in this embodiment, the observed straight lines are clustered based on the angle change of the observed straight lines in consecutive frames. Observed straight lines with an angle change of less than a set value are grouped into one cluster to obtain an observed straight line sub-cluster. According to the time sequence of environmental image acquisition, the observed straight line sub-cluster is divided into historical observed straight line sub-cluster and newly generated observed straight line sub-cluster. Among them, the cluster containing the observed straight lines generated from the environmental image at the current moment is the newly generated observed straight line sub-cluster, and the clusters before the newly generated observed straight line sub-cluster are historical observed straight line sub-clusters.

[0077] The intersection points of each observed line in the historical observed line sub-cluster are calculated using the least squares method;

[0078] The newly generated cluster of observation lines obtained from observations of the same type of target is merged one by one with the historical cluster of observation lines, and the intersection point of each observation line in the merged cluster is calculated by the least squares method.

[0079] Calculate the residual between the intersection points of each observed line in the historical observed line sub-cluster and the intersection points of each observed line in the merged cluster;

[0080] When the residual is greater than or equal to the set value, the newly generated observation line sub-cluster cannot be grouped with the historical observation line sub-cluster.

[0081] When the residual is less than the set value, the newly generated observation line sub-cluster is grouped with the historical observation line sub-cluster into a group, which is the observation line cluster of the same positioning landmark.

[0082] In order to minimize the coordinate error of the acquired location landmarks and thus make the mapping more accurate, this embodiment also optimizes the spatial coordinates of the location landmarks through the location landmark optimization equation to obtain the optimized coordinates of the location landmarks.

[0083] The localization landmark optimization equation is constructed by performing least squares calculations on the transformation relationship error of the robot odometry neighbor poses and the error between the observed line and the localization landmark.

[0084] Specifically: Using the robot's pose as a node, edges are generated between neighboring poses to construct a pose graph. Each edge in the graph represents the transformation relationship between the two poses connected by the edge. The proximity relationship is represented by a set C consisting of positive integers. For position x... i and the positive integer c∈C represents x i With x i+c Connections. The number of elements in set C determines the edge density in the pose graph, and the numerical value of the elements determines the range of proximity relationships.

[0085] Based on the pose map, an error equation for the transformation relationship of the robot's odometry neighboring poses is constructed. The error equation for the transformation relationship of the robot's odometry neighboring poses is:

[0086]

[0087] Among them, e od，i This represents the error in the transformation relationship between neighboring poses in the robot odometry system. This represents the initial pose, which is treated as a constant during optimization. x represents i The corresponding pose inverse transformation.

[0088] The error equation between the observed straight line and the positioning landmark is:

[0089]

[0090] Among them, e ob，i，j To observe the error between the straight line and the positioning landmark.

[0091] The least squares calculation is performed on the transformation relationship error of the robot odometry neighbor poses and the error between the observed straight line and the loop positioning landmark to obtain the positioning landmark optimization equation. Specifically, the positioning landmark optimization equation involving positioning landmark y and odometry pose x is as follows:

[0092]

[0093] Where, x * y * The optimal solution to the optimization problem is represented by the robot's optimal pose and the spatial coordinates of the local landmarks, respectively. α is the odometry confidence factor. If the odometry used has high accuracy, the value of this factor can be increased to improve the optimization algorithm's attention to odometry error and avoid excessive adjustments to the pose provided by the odometry.

[0094] The camera pose is provided by the odometry, and the accumulation of errors over a long period will render the map unusable. Optimization and loop closure detection can effectively eliminate accumulated errors; therefore, this embodiment also performs loop closure detection on local landmarks. In traditional SLAM, each frame contains a large number of low-level features, providing rich information for optimization and loop closure detection. However, in the map generation mode described above, the amount of information in each frame is very sparse, and the local landmarks in the map are also very sparse. Therefore, it is necessary to adjust the optimization and loop closure detection strategies. Due to the scarcity of observation data and the lack of sufficient mutual constraints between observation data, simply adjusting the camera pose with the goal of reducing the error between observed lines and landmarks will transfer the observation errors eliminated in least squares to the camera pose. The camera pose information comes from the odometry, and the changes in pose provided by the odometry have very small errors within a local range, which can provide sufficient constraints for the camera pose.

[0095] Before optimizing the spatial coordinates of the localization landmarks, the error in the observation part stems from the accumulation of odometry errors and noise during the observation process, while the odometry error is zero. When optimization begins, the robot's pose is readjusted to reduce the observation error, causing the odometry error to no longer be zero. The greater the reduction in observation error, the more the robot's pose is adjusted, and the larger the odometry error will be. After optimization, a state of minimum overall balance between observation error and odometry error will be achieved.

[0096] In the location landmark map, different types of landmarks are scattered in space. The organizational relationships between landmarks and their types and positions form unique information, which provides matching information for loop closure detection. This embodiment also performs loop closure detection on the landmarks. According to the generation order of the landmarks, the location landmark map is divided into two parts: an active map and a historical map. The active map starts from the current position of the camera and adds the most recently generated landmarks in reverse chronological order according to spatial coordinates until the number of landmarks in the active map reaches a specified number or the size reaches a specified size. The historical map refers to the map composed of landmarks not included in the active map. Because the error of the odometer gradually accumulates along the path, the error between the relative positional relationship between the landmarks and the actual relative positional relationship in the environment is small within a local range of the path. Therefore, based on the organizational relationship between the landmarks in the active map, the corresponding set of landmarks can be matched in the historical map. The matched loop closure landmarks are then data-associated and optimized and corrected.

[0097] In this embodiment, each frame of point cloud data on the local path is projected horizontally to generate a line segment map, and then all line segment maps are associated to obtain an environmental structure map.

[0098] Specifically, the environmental structure does not participate in localization during mapping; it only serves as an obstacle in the semantic map to delineate passable areas. Therefore, the acquisition and processing of point clouds can be more flexible and open, better representing the environmental structure. This embodiment uses a line segment diagram to represent the abstract environmental structure, where line segments represent insurmountable obstacles in the environment, used to constrain the robot's range of motion. The line segment diagram is derived from the abstraction of the point cloud. Ground and portions exceeding the robot's height are removed from a frame of the point cloud, retaining only the portions affecting the robot's movement. The spatial path is divided into multiple local paths; the point clouds on the local paths are projected horizontally to generate a grid map. The occupied portions in the grid map are segmented to obtain a frame of the line segment diagram. Figures 3-6 It demonstrates the transformation process from point cloud to line segment.

[0099] Since environmental structures do not participate in localization, the correlation between line segment maps relies solely on the pose provided by the odometry. When the odometry is inadequate, observations from a distant location and the current location may not match, and only the environmental structures acquired at the current location can be trusted. This often results in significant overlap and misalignment between line segment maps, leading to discrepancies. Figure 10 The data was scattered and unusable. To address this, this system only retains line segments near each collection point, ensuring reliable observation data along the collection path and eliminating overlapping and misalignment of line segments.

[0100] Because the sampling points are discretely distributed on a plane, any point in the plane can find one or more sampling points closest to it; conversely, for each sampling point, there exists a region where all points within this region are no closer to other sampling points than to that sampling point itself. Using methods such as... Figure 7 The Voronoi polygons were used to define the region. A series of dots were used as seeds to generate Voronoi polygons, where each dot corresponds to a convex polygon, and all points inside each polygon are closest to the dot corresponding to that polygon. Voronoi polygons were then generated on the plane using the acquisition points as seeds. Each polygon corresponds to the area of ​​observation data retained for each acquisition point; line segments exceeding the range of the polygons associated with each acquisition point were deleted.

[0101] Therefore, in this embodiment, when associating all line segment diagrams to obtain the environmental structure diagram;

[0102] First, the line segment diagram is cropped using Voronoi polygons to obtain the retained line segments;

[0103] Then, the preserved line segments are associated to obtain the environmental structure diagram.

[0104] Single-frame point clouds often suffer from view loss due to obstacle occlusion, while consecutive multi-frame point clouds can compensate for the lost view. Although odometry has errors, line segment maps of closely spaced frames have good overlap, so multiple frame point clouds along a short path can be merged to form a local line segment map. Using the center of each short path segment as a seed, Voronoi polygons are generated, and finally, line segments are clipped. This method not only preserves environmental structure information more completely but also makes the line segments more continuous and complete.

[0105] Therefore, the process of cropping the line segment diagram using Voronoi polygons in this embodiment is as follows:

[0106] Using the center point of the local path corresponding to the line segment diagram as a seed, construct the Voronoi polygon;

[0107] Delete the line segments that extend beyond the center polygon to obtain the retained line segments.

[0108] After the camera pose is optimized, the pose of the point cloud acquisition is also adjusted accordingly. However, historical point cloud information has been deleted, making it impossible to regenerate a new line segment map. Since the changes within the local path before and after camera pose optimization are minimal, the changes in the aforementioned line segment maps are also small. Therefore, this system retains all line segment maps, which only retain the line segment representation of the local environmental structure. This results in a small data volume that does not burden the system. When the camera pose is adjusted, only the pose of the line segment map is adjusted accordingly, thereby updating the global line segment map.

[0109] Finally, the location landmark map and the environmental structure map are combined to form a semantic map.

[0110] The method disclosed in this embodiment mimics the human self-localization and spatial mapping model to perform synchronous semantic localization and mapping for robots. Humans use semantic information in space for mapping and self-localization; semantic objects include landmark information for localization and environmental structures that obstruct passage. Landmarks are used for human self-localization, while environmental structures restrict the range of human movement. Mimicking human localization and spatial description behavior, this method uses visual sensors and object detection tools to generate landmarks, which are represented by points with spatial location information and semantic labels. An environmental structure sensor is used to acquire obstacle information; the point cloud obstacle information obtained by the environmental structure sensor is abstracted into straight line segments on a plane. Ultimately, synchronous semantic localization and spatial mapping of the robot are achieved.

[0111] The method disclosed in this embodiment was verified through experiments.

[0112] An office corridor was selected as the robotics experimental environment, where doors, safety indicator lights, and fire extinguishers can be used as semantic objects. A dataset was collected in the corridor, including two lighting scenarios: daylight and artificial light. Each frame of data collected included wheel odometer data, 2D laser odometer data, fisheye camera and pinhole camera images, as well as color images and point clouds from an RGBD camera, and laser grid maps for both scenarios. Figure 8 A laser grid map of the experimental site was displayed. Figure 9 The image shows the odometer tracks numbered 30 to 1130 under daylight conditions. The dashed lines represent the laser odometer track, and the dotted lines represent the wheel odometer track. It can be seen that the wheel odometer experienced significant drift after one round trip. The odometer information source for generating positioning landmarks can be either a wheel odometer or a laser odometer. Wheel odometers have a large cumulative error; therefore, loop closure detection and optimization will be performed in subsequent loop closure detection experiments.

[0113] When a robot traverses a narrow corridor, it is difficult to observe semantic objects located on both sides. Fisheye cameras, with their wide field of view, are more conducive to observation; therefore, a fisheye camera was used in the experiment. However, the circular field of view generated by a fisheye camera is not suitable for target detection. Therefore, in this embodiment, the fisheye camera's optical axis is placed vertically upwards, so that the scene images around the camera fall evenly on the periphery of the circular view. Finally, the circular view is cylindrically unfolded to obtain a rectangular view suitable for the target detection model, as shown below. Figures 11-12 As shown.

[0114] Images captured by fisheye cameras suffer from severe distortion. After cylindrical unfolding, the horizontal observation angle of the semantic object is significantly different from the x-coordinate v of the pixel coordinates at the center of its detection box. IA linear relationship is established (the panoramic view is cylindrically unfolded along the negative x-axis, with the direction corresponding to the image center as the positive x-axis). Due to the limitations of fisheye lens usage and cylindrical unfolding, a planar model is more suitable; therefore, the pitch component w of the observation vector is used. c If set to zero, then:

[0115]

[0116] A reciprocating trajectory (daylight hours 30-1130) was selected from the dataset for testing, such as... Figures 13-15 The results of generating localization landmark maps using two different odometer data sets are shown. Short black arrows represent observed straight lines, and circular and square markers represent two different types of high-level semantic localization landmarks distributed along the odometer trajectory. The landmarks in both maps have not yet undergone loop closure detection for data association; therefore, points observed before and after are independent. Comparison reveals that due to significant drift in the wheeled odometer, the generated landmark positions for the same semantic object differ considerably before and after the observation. In contrast, the landmark positions generated by the laser odometer are closer, consistent with reality. When loop closures exist, loop closure detection first associates the relevant landmarks, and then the landmark positions and camera pose are adjusted using a localization landmark optimization equation to obtain a globally consistent map. After loop closure detection and pose optimization, both the wheeled odometer trajectory and the generated landmarks are closer to reality.

[0117] An environmental structure diagram is generated based on the loop-complete trajectory, such as... Figure 16 As shown, line segments represent obstacles, similar in shape to a grid map, but with significantly reduced data volume. By fusing the location landmark map and the environmental structure map, a complete semantic map is generated, such as... Figure 17 As shown. Figure 18 The semantic map generated for the entire experimental site is shown. The final semantic map matches the actual situation, verifying the accuracy of the method disclosed in this embodiment.

[0118] Example 2

[0119] In this embodiment, a robot synchronous semantic localization and mapping system based on humanoid concepts is disclosed, including:

[0120] The data acquisition module is used to acquire environmental images and spatial structure point cloud data of the environment for each frame;

[0121] The location landmark acquisition module is used to identify the observation lines of each frame of environmental image and obtain the corresponding location landmarks; it divides all observation lines into clusters, grouping observation lines belonging to the same location landmark into one cluster; it analyzes the observation lines in each cluster using the least squares method to determine the intersection points of the observation lines in each cluster, which are the spatial coordinates of each corresponding location landmark; it optimizes the spatial coordinates of the location landmarks to obtain the optimized coordinates of the location landmarks; and it obtains the location landmark map based on the optimized coordinates of all location landmarks.

[0122] The environment structure map acquisition module is used to project each frame of point cloud data on the local path into the horizontal direction to generate a line segment map; and to associate all the line segment maps to obtain the environment structure map.

[0123] The semantic map acquisition module is used to combine the location landmark map and the environmental structure map to form a semantic map.

[0124] Example 3

[0125] In this embodiment, an electronic device is disclosed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps described in the humanoid robot synchronous semantic localization and mapping method disclosed in Embodiment 1.

[0126] Example 4

[0127] In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps described in the humanoid robot synchronous semantic localization and mapping method disclosed in Embodiment 1.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A robot synchronous semantic localization and mapping method based on humanoid thinking, characterized in that, include: Acquire environmental images and spatial structure point cloud data for each frame; The environmental images in each frame are identified to obtain the observation lines of the corresponding positioning landmarks; All observed lines are clustered, and observed lines belonging to the same location landmark are grouped into one cluster; By using the least squares method, the observed lines in each cluster are analyzed to determine the intersection points of the observed lines in each cluster, which are the spatial coordinates of each corresponding positioning landmark. The spatial coordinates of the location landmarks are optimized to obtain the optimized coordinates of the location landmarks; Based on the optimized coordinates of all location landmarks, a location landmark map is obtained; Project each frame of point cloud data on the local path into the horizontal direction to generate a line segment diagram; By associating all the line segment diagrams, an environmental structure diagram can be obtained; The location landmark map and the environmental structure map are combined to form a semantic map.

2. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 1, characterized in that, The system identifies environmental images and obtains the pixel coordinates of location landmarks within the images. Transform pixel coordinates into observation vectors in the camera coordinate system; The observation vector is transformed from the camera coordinate system to the world coordinate system and translated to the world coordinates of the camera to obtain the observation line of the positioning landmark.

3. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 1, characterized in that, Based on the change in angle of the observed line in the previous and next frames, the observed lines were divided into clusters. The observed lines with an angle change of less than a set value were divided into a cluster to obtain the observed line sub-cluster. The intersection points of each observed line in the historical observed line sub-cluster are calculated using the least squares method; The newly generated cluster of observation lines obtained from observations of the same type of target is merged with the historical cluster of observation lines one by one, and the intersection point of each observation line in the merged cluster is calculated by the least squares method. Calculate the residual between the intersection points of each observed line in the historical observed line sub-cluster and the intersection points of each observed line in the merged cluster; When the residual is greater than or equal to the set value, the newly generated observation line sub-cluster cannot be grouped with the historical observation line sub-cluster. When the residual is less than the set value, the newly generated observation line sub-cluster is grouped with the historical observation line sub-cluster.

4. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 1, characterized in that, The spatial coordinates of the location landmarks are optimized by using the location landmark optimization equation to obtain the optimized coordinates of the location landmarks; Among them, the localization landmark optimization equation is constructed by performing least squares calculation on the transformation relationship error of the robot odometry neighbor poses and the error between the observed line and the localization landmark.

5. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 4, characterized in that, Using robot poses as nodes, edges are generated between neighboring poses to construct a pose graph. Each edge in the graph represents the transformation relationship between the two poses connected by the edge. Based on the pose diagram, the transformation relationship error of the robot odometry neighbor poses is obtained.

6. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 1, characterized in that, When associating all line segment diagrams to obtain the environment structure diagram; First, the line segment diagram is cropped using Voronoi polygons to obtain the retained line segments; Then, the preserved line segments are associated to obtain the environmental structure diagram.

7. The robot synchronous semantic localization and mapping method based on humanoid thinking as described in claim 6, characterized in that, The process of cropping a line segment diagram using Voronoi polygons is as follows: Using the center point of the local path corresponding to the line segment diagram as a seed, construct the Voronoi polygon; Delete the line segments that extend beyond the center polygon to obtain the retained line segments.

8. A robot synchronous semantic localization and mapping system based on humanoid design principles, characterized in that: include: The data acquisition module is used to acquire environmental images and spatial structure point cloud data of the environment for each frame; The location landmark acquisition module is used to identify the observation lines of each frame of environmental image and obtain the corresponding location landmarks; it divides all observation lines into clusters, grouping observation lines belonging to the same location landmark into one cluster; it analyzes the observation lines in each cluster using the least squares method to determine the intersection points of the observation lines in each cluster, which are the spatial coordinates of each corresponding location landmark; it optimizes the spatial coordinates of the location landmarks to obtain the optimized coordinates of the location landmarks; and it obtains the location landmark map based on the optimized coordinates of all location landmarks. The environment structure map acquisition module is used to project each frame of point cloud data on the local path into the horizontal direction to generate a line segment map; and to associate all the line segment maps to obtain the environment structure map. The semantic map acquisition module is used to combine the location landmark map and the environmental structure map to form a semantic map.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, they complete the steps of the robot synchronous semantic localization and mapping method based on humanoid thinking as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the robot synchronous semantic localization and mapping method based on humanoid thinking as described in any one of claims 1-7.

Images