Semantic map relocalization method and system based on multi-semantic object as main feature

Abstract

This invention provides a semantic map relocalization method and system based on multiple semantic objects as primary features. The method includes collecting information to build an environmental grid map and mapping it to the primary semantic object and its nearby secondary semantic objects to generate a semantic map; initializing a particle swarm; randomly distributing multiple particles in the semantic map and assigning the same weight value to each particle; predicting the pose of each particle in the particle swarm; and performing relocalization based on the prediction results. Specifically, this involves calculating the weight values ​​of each particle in the particle set, extracting environmental semantics and remapping them, comparing them with the semantic map, and updating the weight values ​​a second time based on the comparison results to obtain the robot's precise pose value. This invention achieves better relocalization results by matching semantic objects with the constructed semantic map.

Description

Technical Field

[0001] This invention belongs to the field of robot mapping, localization and navigation, specifically involving a semantic map relocalization method and system based on multi-semantic objects as the main feature. Background Technology

[0002] Classical localization algorithms are mostly based on the AMCL algorithm. However, in similar environments or when the environment changes drastically, particle divergence occurs, causing the robot to fail to locate accurately. Due to insufficient prior information, the robot often needs a lot of movement to accurately locate its initial global position. This process requires continuously issuing movement commands to the robot to keep it moving until the particle set converges.

[0003] In addition, single-distance localization means that the robot's understanding of the surrounding environment is still based on distance perception, while single-visual localization, although it can provide additional dimensions of environmental perception, is prone to misidentification, which affects localization accuracy. Summary of the Invention

[0004] This invention proposes a semantic map relocalization method and system based on multiple semantic objects as the main feature. Combining a lidar sensor and a depth camera, the semantic object with the most frequent occurrences of a category during the localization process is identified as the main semantic object and used as the primary feature for localization. This effectively reduces the impact of misidentification on localization. The system also assists with secondary semantic object information by matching the acquired visually detected object category and distance information with the constructed semantic map, thereby assisting the lidar localization method to achieve better localization results.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] Collect information to build an environmental grid map and then map it with the main semantic object and its nearby secondary semantic objects to generate a semantic map;

[0007] Initialize the particle swarm by randomly scattering multiple particles in the semantic map and setting the same weight value for each particle.

[0008] Predicting the pose of each particle in a particle swarm;

[0009] Relocation is performed based on the prediction results. Specifically, the weight values ​​of each particle in the particle set are calculated, environmental semantics are extracted and remapped, and then compared with the semantic map.

[0010] After updating the weight values ​​a second time based on the comparison results, the robot's precise pose value is obtained.

[0011] Furthermore, the semantic objects of the same type that appear most frequently are designated as primary semantic objects, and the remaining identified semantic objects are designated as secondary semantic objects.

[0012] Furthermore, the semantic map includes semantic information, which includes the categories of primary semantic objects and secondary semantic objects, and the relative positional relationships between each primary semantic object and its nearby secondary semantic objects.

[0013] Furthermore, the robot's motion at a set time is obtained, and the motion in the robot's coordinate system is transformed to the world coordinate system. Using the particle state estimate from the previous moment and the transformed motion, the pose prediction of each particle in the particle swarm is achieved. The calculation formula is as follows:

[0014]

[0015] Where, x t ,y t ,θ t Let x be the particle pose at time t. t-1 ,y t-1 ,θ t-1 Let V be the particle pose at time t-1, V be the robot's velocity at this time, Δt be the time interval, and Δθ be the position of the particle. t This represents the robot's rotation angle.

[0016] Furthermore, the relocalization mapping method is as follows: the point cloud of the center part of the detection box of the main semantic object is mapped, and the point cloud of the secondary semantic object is processed by taking half of the original detection length based on the center of the detection box. The point cloud cluster closest to the robot after point cloud filtering and coordinate transformation is mapped onto the two-dimensional plane to obtain the mapping result.

[0017] Furthermore, if there is an overlap in the comparison results, the particle weight values ​​are concentrated in the overlapping area, while the particle weight values ​​in other areas are reduced, until the particles converge to obtain the accurate pose value.

[0018] If there is no overlapping part, the robot's possible pose is determined based on the relative distance information between the primary and secondary semantic objects, and the robot's precise pose value is inferred.

[0019] Furthermore, the method for inferring the robot's precise pose value when there are no overlapping parts is as follows: the detected object category is compared with the object category in the environmental grid map containing semantic information. During the localization process, the main semantic object is used as the main localization reference point. The distance of the robot relative to the main semantic object is determined by combining the scanning distance of the lidar sensor, resulting in multiple possible solutions. Then, some possible solutions are eliminated based on the information of secondary semantic objects near the main semantic object to obtain the robot's precise pose value.

[0020] A semantic map relocalization device based primarily on multi-semantic objects, comprising:

[0021] The map acquisition module is used to collect information to build an environmental raster map and generate a semantic map by mapping it with the main semantic object and its nearby secondary semantic objects.

[0022] The particle swarm generation module is used to initialize the particle swarm, randomly scattering multiple particles in the semantic map and setting the same weight setting value for each particle.

[0023] The particle pose prediction module is used to predict the pose of each particle in the particle swarm.

[0024] The relocation module is used to relocate based on the prediction results. Specifically, it calculates the weight values ​​of each particle in the particle set, extracts environmental semantics and remaps them, and compares them with the semantic map.

[0025] The robot precise pose value acquisition module is used to obtain the robot's precise pose value after updating the weight value a second time based on the comparison results.

[0026] A semantic map relocalization device with multiple semantic objects as the main feature includes a processor and a memory for storing a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of any of the semantic map relocalization methods described above with multiple semantic objects as the main feature.

[0027] A computer storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the above-mentioned semantic map relocation methods with multiple semantic objects as the main features.

[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0029] The semantic map relocalization method and system based on multiple semantic objects as primary features provided by this invention employs a two-level localization process. This requires dividing the map into primary and secondary semantic objects during the mapping process and recording their relative positions. LiDAR distance information provides the reference positions of the primary and secondary semantic objects relative to the robot, thereby achieving accurate localization. Without this division, the robot would need to perform a global search, comparing semantic object information region by region, which is time-consuming. However, the semantic map relocalization method based on multiple semantic objects as primary features provided by this invention, by pre-determining the location points of the primary semantic objects, effectively eliminates some regional solutions. It directly obtains a definitive solution through local comparison based on surrounding secondary semantic object information and LiDAR sensor distance information, thus reducing the time consumed in the localization process. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and are intended to explain the invention, but do not constitute an undue limitation thereof. In the drawings:

[0031] Figure 1 This is a flowchart of the method of the present invention.

[0032] Figure 2 This is the environmental grid map of the present invention.

[0033] Figure 3 This is the semantic map of the present invention.

[0034] Figure 4 This is the semantic map optimized for this invention.

[0035] Figure 5 This is a segmentation diagram of the GrabCut algorithm of this invention.

[0036] Figure 6 This is a diagram showing the semantic object detection results of this invention.

[0037] Figure 7 This is a diagram showing the semantic mapping result of the present invention.

[0038] Figure 8 This is a primary and secondary semantic object information diagram of the present invention.

[0039] Figure 9 This is a diagram illustrating the particle convergence process of the method of the present invention.

[0040] Figure 10 This is a comparison chart of the average weights of particles in this invention. Detailed Implementation

[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0042] First Embodiment

[0043] like Figure 1 The diagram shown is a flowchart of the method of this invention. This invention provides a semantic map relocalization method based on multiple semantic objects as primary features, comprising the following steps:

[0044] Step S1: Collect information to build an environmental grid map and map it with the main semantic object and its nearby secondary semantic objects to generate a semantic map;

[0045] Step S2: Initialize the particle swarm by randomly distributing multiple particles in the semantic map and setting the same weight value for each particle.

[0046] Step S3: Predict the pose of each particle in the particle swarm.

[0047] Step S4: Relocalize based on the prediction results. Specifically, calculate the weight values ​​of each particle in the particle set, extract the environmental semantics and remap it, and compare it with the semantic map.

[0048] Step S5: After updating the weight values ​​a second time based on the comparison results, obtain the robot's precise pose value.

[0049] Compared to current semantic object recognition and localization methods, the semantic map relocalization method based on multiple semantic objects as primary features provided by this invention employs a two-stage localization process. This requires dividing the robot into primary and secondary semantic objects during the mapping process and recording their relative positions. LiDAR distance information provides the reference positions of the primary and secondary semantic objects relative to the robot, thereby achieving precise localization. Without this division, the robot would need to perform a global search, comparing semantic object information region by region, which is time-consuming. In contrast, the semantic map relocalization method based on multiple semantic objects as primary features, by pre-determining the location points of the primary semantic objects, effectively eliminates some possible solutions. It directly compares the surrounding secondary semantic object information and LiDAR sensor distance information to obtain a definitive solution, thus reducing the time consumed during the localization process.

[0050] In step S1 of the present invention, as follows Figure 2 As shown, this is the environmental grid map of the present invention. The robot platform moves indoors under the control of the main controller. The lidar sensor collects the distance between the object and the robot platform and the orientation angle between the object and the robot platform in real time and transmits them to the main controller. The main controller processes the distance between the object and the robot platform and the orientation angle between the object and the robot platform through the Gmapping algorithm to obtain the environmental grid map.

[0051] Unlike the AMCL method, such as Figure 3 As shown, this is the semantic map of the present invention. The present invention detects objects by a depth camera and maps them with the environmental grid map obtained by a lidar sensor to generate a semantic map, classifies objects into primary semantic objects and secondary semantic objects, and records the relative positional relationship between primary semantic objects and secondary semantic objects.

[0052] In step S1, the environmental grid map established using the Gmapping algorithm is loaded for localization. Based on the principle of the likelihood domain observation model, the distance value dist of the nearest primary or secondary semantic objects in the loaded environmental grid map is calculated, and the closest distance between the robot pose and the distributed particles is selected as the dist value for subsequent applications.

[0053] The formula for calculating the probability value p(x) of each point in the current frame laser likelihood map is:

[0054]

[0055] Where σ is the standard deviation of the specified measurement noise.

[0056] Semantic maps can provide the categories of primary and secondary semantic objects, as well as the relative positions of the primary semantic object and its nearby secondary semantic objects. In the subsequent localization process, the detected object categories can be compared with the known object categories in the semantic map. The primary semantic object is used as the main localization reference point, and the distance of the robot relative to the primary semantic object is determined by combining the scanning distance of the LiDAR sensor, resulting in multiple possible solutions. Finally, the possible localization solutions are eliminated based on the information of the secondary semantic objects next to the primary semantic object, and the precise pose value of the robot is obtained.

[0057] In this invention, as shown in Table 1 below, there is a table of semantic object quantity statistics and classification. In a certain room, semantic objects include cabinets, chairs, trash cans and doors. The number of cabinets is 7, the number of chairs is 2, the number of trash cans is 2 and the number of doors is 1. Cabinets are classified as primary semantic objects, and chairs, trash cans and doors are classified as secondary semantic objects.

[0058] Table 1 Primary and Secondary Semantic Object Information Table

[0059] Semantic objects cabinet Chair garbage can Door quantity 7 2 2 1 Primary and secondary classification host Second-rate Second-rate Second-rate

[0060] As shown in Table 2 below, this is the information table of primary and secondary semantic objects. Specifically, there are secondary semantic objects, a chair and a trash can, near the primary semantic object ①. The distance between the chair and the primary semantic object ① is 1.10m, and the distance between the trash can and the primary semantic object ① is 0.75m. There are secondary semantic objects, a chair and a trash can, near the primary semantic object ②. The distance between the trash can and the primary semantic object ② is 0.85m, and the distance between the door and the primary semantic object ② is 1.90m.

[0061] Table 2 Primary and Secondary Semantic Object Information Table

[0062]

[0063] In step S1, in order to ensure that the principle of positioning based on the primary semantic object remains effective even when the environment changes, this invention categorizes the most frequently occurring objects of the same type as primary semantic objects and the remaining identified objects as secondary semantic objects. When the surrounding environment changes, the principle of positioning based on the primary semantic object remains effective because the number of primary semantic objects far exceeds that of secondary semantic objects. This allows for the acquisition of the robot's approximate pose. Then, the robot's precise pose can be further inferred based on the information of non-primary semantic objects identified in the surrounding environment, thus achieving precise positioning of the robot. If the primary and secondary semantic objects are not categorized, the robot will simply compare the identified object information with the semantic map, resulting in positioning failure and the inability to find a position point that coincides with the robot's actual pose.

[0064] like Figure 4 As shown, this is the optimized semantic map of the present invention. The mapping shapes of primary and secondary semantic objects are simplified, and shapes similar to those of the primary or secondary semantic objects are used to represent them. Minimum bounding rectangles and minimum bounding circles are used to represent primary and secondary semantic objects. For example, a trash can is cylindrical, so its minimum bounding circle represents the trash can; a table is cuboid, so its minimum bounding rectangle represents the table. The center points of the primary and secondary semantic objects in the environmental raster map are extracted, and the object information is mapped to the location information in the environmental raster map to generate an environmental raster map containing semantic information, forming a set of object semantic mapping points.

[0065] During the robot's movement, coordinate transformation and pose estimation are performed:

[0066] The robot's pose in the world coordinate system is calculated based on the coordinates of primary and secondary semantic objects in the robot coordinate system and the corresponding semantic objects in the grid coordinate system. Keyframes are selected based on the primary and secondary semantic objects detected by the depth camera, and the semantic object point cloud in the camera coordinate system is extracted. After point cloud filtering and coordinate transformation, the coordinates of the center of the primary and secondary semantic objects in the robot coordinate system are obtained, and coordinate transformation is performed.

[0067]

[0068] Where A is the raster precision, and in this embodiment of the invention, A = 0.05, (X g ,Y g (X) represents the coordinates of the semantic object in the raster coordinate system. W ,Y W () represents the coordinates of the semantic object in the world coordinate system. Transforming the camera coordinate system to the world coordinate system yields the formula in the world coordinate system:

[0069]

[0070] Among them, (X) r ,Y r Let be the coordinates of the center of the semantic object in the robot's coordinate system. After simplification, we obtain the possible poses of the robot at time t. T w,r Let be the relative transformation matrix from the robot to the world coordinate system.

[0071] In step S2 of this invention, the particle set is initialized on the main control unit. The initial pose in the map is unknown. M particles are randomly scattered in the environmental grid map, and the weight of each particle is set to the same value. The particle set χ0 is initialized in the laser map with a weight of 1 / M.

[0072] Step 2 initializes the particle swarm. The particle swarm at time t is:

[0073]

[0074] Wherein, the particle swarm size is M, and when t = 0, it represents the initial particle set at time 0. Let represent the j-th particle at time t.

[0075] In step S3 of this invention, the main controller uses sensor data (such as odometer readings) of relevant motion quantities acquired by the robot itself, combined with the robot's motion model, to update the particle set. It calculates the robot's motion quantity over a certain period using the obtained sensor data, performs a coordinate transformation to convert the motion quantity in the robot coordinate system to the world coordinate system, and uses the particle state estimate from the previous moment and the motion quantity combined with the motion model to predict the pose of each particle in the particle set, thus obtaining the predicted particle set χ. t Predict the pose of each particle in the particle swarm.

[0076] Predicting particle set χ based on odometry model t The formula is:

[0077]

[0078] Where, x t ,y t ,θ t Let x be the particle pose at time t. t-1 ,y t-1 ,θ t-1 Let V be the particle pose at time t-1, V be the robot's velocity at this time, Δt be the time interval, and Δθ be the position of the particle. t This represents the robot's rotation angle.

[0079] In step S4 of this invention, during relocalization, laser matching is performed by combining the likelihood domain observation model and the beam model. The depth camera acquires a color image and a depth image from the current viewpoint. The pre-trained SSD target detection model is used to detect and identify the color image to obtain the detection result. Figure 5 As shown, the GrabCut segmentation algorithm is used to remove the background. Figure 6 As shown, the detection bounding boxes obtained by the object detection model are used to replace the user-drawn object bounding boxes. After obtaining the object segmentation map of the color image under the viewpoint, the pixel coordinates of the object in the image coordinate system are transformed to the camera coordinate system by combining its corresponding depth map information and depth camera calibration parameters, and then semantic mapping is performed. In order to reduce the computational load of point cloud processing, the point cloud of the center part of the main semantic object detection box is mapped; for secondary semantic objects, half of the original detection length is taken as the reference for point cloud processing, that is, the detection box center is used as the reference. Figure 6 The semantic point cloud is re-extracted from half of the detection box. The point cloud cluster closest to the robot after filtering and coordinate transformation is mapped onto a two-dimensional plane to obtain the mapping result.

[0080] For the identified semantic objects, the point cloud at the center of the detection box is mapped in two dimensions to obtain a semantic mapping map reflecting the mapping result, such as... Figure 7 The diagram shown is a semantic mapping diagram of the present invention.

[0081] If a primary semantic object is identified from the current perspective, calculate the center coordinates of the primary semantic object in the semantic mapping map, and calculate the distances from the centers of other secondary semantic objects near the primary semantic object to the center of the primary semantic object.

[0082] In the detection results, the semantic objects of the same category that are detected most frequently are taken as the primary semantic objects. Based on the current semantic map and the semantic information previously obtained from the semantic map, i.e., the primary and secondary semantic object information table, points that simultaneously contain the primary semantic object in the semantic map are first selected. In large scenes, the primary and secondary semantic objects are unevenly distributed and scattered, and some objects may be very close together while others are too far apart. Therefore, a circle with a certain radius (chosen according to the actual size of the environment) is drawn with the center of the primary semantic object as the center. In the first embodiment of this invention, a radius of 2.0 meters is selected. If the center point coordinates of the secondary semantic objects are located inside the circle, the primary semantic object and the nearby secondary semantic objects are considered as a set, such as... Figure 8The diagram shown illustrates the primary and secondary semantic object information map of this invention. Using the primary semantic object as a baseline, nearby secondary semantic objects are searched, and the distances between them are calculated and stored in the semantic information, i.e., the primary and secondary semantic object information table. The inclusion of other non-primary semantic objects within the primary semantic object's viability increases its visibility, facilitating object recognition and localization in long-distance, large-scale environments. The primary and secondary semantic object information table provides a reference for the localization process. During localization, if a primary or secondary semantic object is identified, it can be matched against the established primary and secondary semantic object information table to determine if its location matches the environmental grid map. If they match, precise localization can be achieved using both the primary and secondary semantic object information table and the environmental grid map. If they do not match, it indicates a change in the environment, requiring further assessment based on the detected primary semantic object information. If the distances between primary semantic objects are not significantly different from the distances recorded in the information table, the robot's position can be calculated based on the positional differences.

[0083] When the robot identifies the primary semantic object and the secondary semantic object through the object detection model, it can infer the position of the primary semantic object and the current position of the robot by comparing the distance of the robot relative to the primary semantic object and the distance between the primary and secondary semantic objects with the semantic information obtained in step 1.

[0084] Preferably, in step 5, particles are evenly distributed across the map. When the robot identifies the primary semantic object and surrounding secondary semantic objects, the particle weights are updated based on the matching probability between the semantic information acquired by the depth camera and the semantic map. First, semantic information is extracted from the indoor environment using the depth camera, and the particle weights are updated using a visual localization method. When the robot identifies the primary and secondary semantic objects, the positions of the primary semantic object and the current robot can be inferred by combining the target detection model and the global pre-positioning method of the primary semantic object and surrounding secondary semantic object information tables. At the robot pose estimation point, particles are distributed using a Gaussian distribution, with particles closer to the robot pose estimation point having a higher weight. After particle resampling, particles with lower weights are removed, and particles with higher weights are copied. KLD sampling is used to dynamically adjust the number of particles, improving particle utilization and reducing particle redundancy. Particles are then redistributed according to their weight proportions, such as... Figure 9 The diagram shows the particle convergence process of the method of the present invention. Through repeated iterations, the precise pose value of the robot is obtained, and the robot's localization is achieved.

[0085] In one embodiment of the present invention, an experiment was conducted in an indoor environment using the semantic map relocalization method based on multiple semantic objects as the main feature proposed by the present invention. The experimental equipment included a robot platform, a main controller, a lidar sensor, and a depth camera. The main controller, lidar sensor, and depth camera were mounted on the robot platform. The processor was sequentially connected to the robot platform chassis, lidar sensor, and depth camera via wired connections.

[0086] The main control unit is a Core i9 mini host; it is compact, convenient, and suitable for mounting on a robot platform to process detected object information in real time.

[0087] The robot platform uses the SCOUT MINI intelligent mobile chassis, which is highly stable and can maintain high stability when turning or accelerating / decelerating.

[0088] The lidar sensor used is the Sick Lms111 single-line lidar, which provides accurate distance information with a resolution of 0.5° and a maximum detection distance of 20 meters.

[0089] The depth camera used is Microsoft's Kinect V2 depth camera, which can detect the depth information of objects and provide a reference when building maps.

[0090] In this specific embodiment, such as Figure 10 The figure shown is a comparison of the average particle weights of the present invention and the traditional AMCL localization method. The semantic map relocalization method provided by the present invention, which is based on multi-semantic objects as the main feature, improves localization by 68.75% compared with the traditional AMCL localization method, and improves localization speed by 65.96% when the environment changes.

[0091] Second Embodiment

[0092] This invention also provides a semantic map relocation device based on multiple semantic objects as the main feature, comprising:

[0093] The map acquisition module is used to collect information to build an environmental raster map and then map it with the main semantic object and its nearby secondary semantic objects to generate a semantic map.

[0094] The particle swarm generation module is used to initialize the particle swarm, randomly scattering multiple particles in the semantic map and setting the same weight setting value for each particle.

[0095] The particle pose prediction module is used to predict the pose of each particle in the particle swarm.

[0096] The relocation module is used to relocate based on the prediction results. Specifically, it calculates the weight values ​​of each particle in the particle set, extracts environmental semantics and remaps them, and compares them with the semantic map.

[0097] The robot precise pose value acquisition module is used to obtain the robot's precise pose value after updating the weight value a second time based on the comparison results.

[0098] Third Embodiment

[0099] A semantic map relocalization device based on multiple semantic objects as the main feature includes a processor and a memory for storing a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the semantic map relocalization method based on multiple semantic objects as the main feature described above.

[0100] The memory in this embodiment of the invention is used to store various types of data to support the operation of a semantic map relocalization device with multiple semantic objects as its primary feature. Examples of such data include any computer program used to operate on a semantic map relocalization device with multiple semantic objects as its primary feature.

[0101] The semantic map relocalization method based on multiple semantic objects as the main feature disclosed in this invention can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the semantic map relocalization method based on multiple semantic objects as the main feature can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in this invention's embodiments. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention's embodiments can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically a memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the semantic map relocalization method based on multiple semantic objects as the main feature provided in this invention's embodiments.

[0102] In an exemplary embodiment, the semantic map relocalization device with multiple semantic objects as its main feature can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0103] It is understood that memory can be volatile or non-volatile, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0104] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A semantic map relocation method with multi-semantic object as main feature, characterized in that, The method comprises the following steps: Collecting information to establish an environmental grid map and mapping it with primary semantic objects and secondary semantic objects near the primary semantic objects to generate a semantic map; Initializing a particle swarm, randomly distributing a plurality of particles in the semantic map, and setting the same weight setting value for each particle; Predicting the pose of each particle in the particle swarm; According to the prediction result, the particle weight value in each particle in the particle swarm is calculated, the environmental semantics is extracted, and the mapping is performed again, and the mapping result is compared with the semantic map; According to the comparison result, the weight value is updated again to obtain the accurate pose value of the robot; If there is an overlapping part in the comparison result, the particle weight value is aggregated to the overlapping part region, and the particle weight value in other regions is reduced, and finally the particle converges to obtain the accurate pose value; If there is no overlapping part, the possible pose of the robot itself is judged according to the relative distance information between the primary and secondary semantic object information, and the accurate pose value of the robot is inferred. The method for obtaining the accurate pose value of the robot when there is no overlapping part is: comparing the detected object category with the object category in the environmental grid map containing semantic information, taking the primary semantic object as the main positioning reference point in the positioning process, and combining the scanning distance of the laser radar sensor to determine the distance of the robot relative to the primary semantic object to obtain a plurality of possible solutions, and then excluding part of the possible solutions according to the secondary semantic object information near the primary semantic object to obtain the accurate pose value of the robot.

2. The semantic map repositioning method with multiple semantic objects as the main feature according to claim 1, characterized in that: The same type of semantic object with the highest frequency is taken as the primary semantic object, and the rest of the recognized semantic objects are taken as the secondary semantic objects.

3. The semantic map repositioning method with multiple semantic objects as the main feature according to claim 1, characterized in that: The semantic map includes semantic information, and the semantic information is the category of the primary semantic object and the secondary semantic object, and the relative position relationship between each primary semantic object and the secondary semantic object near it.

4. The semantic map repositioning method with multiple semantic objects as the main feature according to claim 1, characterized in that: The motion amount of the robot in a set time is obtained, the motion amount in the robot coordinate system is converted to the world coordinate system, the particle state estimation value at the last time and the converted motion amount are used to realize the pose prediction of each particle in the particle swarm, and the calculation formula is: wherein, is the robot's movement speed at the moment, is the robot's movement speed at the moment, is the robot's movement speed at the moment, is the interval time, is the robot's rotation angle.

5. The semantic map repositioning method with multiple semantic objects as the main feature according to claim 1, characterized in that: The mapping method during repositioning is: mapping the center part of the primary semantic object detection box point cloud, taking the center of the detection box as the reference to take 1 / 2 of the original detection length for point cloud processing, filtering and coordinate transforming the point cloud, and mapping the point cloud closest to the robot to a two-dimensional plane to obtain the mapping result.

6. A semantic map relocation apparatus with multi-semantic object as a main feature, characterized in that, It comprises: A map acquisition module for collecting information to establish an environmental grid map and mapping it with primary semantic objects and secondary semantic objects near the primary semantic objects to generate a semantic map; A particle swarm establishment module for initializing a particle swarm, randomly distributing a plurality of particles in the semantic map, and setting the same weight setting value for each particle; A particle pose prediction module is configured to predict the pose of each particle in the particle group; A repositioning module is configured to reposition according to the prediction result, specifically, to calculate the weight value of each particle in the particle group, extract the environmental semantics and re-map, and compare with the semantic map; A robot precise pose value acquisition module is configured to acquire the precise pose value of the robot after the weight value is updated for the second time according to the comparison result; In the robot precise pose value acquisition module, if there is an overlapping part in the comparison result, the particle weight value is aggregated to the overlapping part region, and the particle weight value of other regions is reduced, and finally the particle converges to obtain the precise pose value; If there is no overlapping part, the possible pose of the robot itself is determined according to the relative distance information between the primary and secondary semantic object information, and the precise pose value of the robot is inferred; In the robot precise pose value acquisition module, if there is no overlapping part, the precise pose value of the robot is acquired in the following way: the detected object category is compared with the object category in the semantic information containing environmental grid map, the primary semantic object is taken as the main positioning reference point in the positioning process, the distance of the robot relative to the primary semantic object is determined by combining the scanning distance of the laser radar sensor, a plurality of possible solutions are obtained, and then the precise pose value of the robot is obtained by excluding part of the possible solutions according to the secondary semantic object information near the primary semantic object.

7. A semantic map relocalization device with multi-semantic object as main feature, characterized in that: The computer program is executed by the processor, and the steps of the semantic map repositioning method with the multi-semantic object as the main feature in any one of claims 1-5 are implemented.

8. A computer storage medium storing a computer program, wherein the computer storage medium comprises: The computer program is executed by the processor, and the steps of the semantic map repositioning method with the multi-semantic object as the main feature in any one of claims 1-5 are implemented.

Images