Positioning method, device and storage medium for mobile robot

Abstract

The application discloses a positioning method and device of a mobile robot and a storage medium. The method comprises the following steps: performing semantic point cloud extraction on an inverse perspective mapping image to obtain a point cloud set, and performing clustering on the point cloud set to obtain a plurality of point cloud clusters; extracting a geometric feature corresponding to each point cloud cluster to obtain a geometric feature value corresponding to each point cloud cluster respectively; detecting whether the geometric feature value meets a preset geometric condition, and screening a point cloud cluster whose geometric feature value meets the preset geometric condition as a usable semantic point cloud; and performing repositioning on the mobile robot based on the usable semantic point cloud, and if the repositioning fails, performing repositioning by using other repositioning strategies, wherein the other repositioning strategies perform repositioning based on environment perception data collected by other types of sensors. The repositioning means is enriched in a changing environment, the repositioning success rate is significantly improved, the long-term stable operation of the mobile robot in a complex environment is ensured, and the positioning capability of the mobile robot is improved.

Description

Technical Field

[0001] This application relates to the field of robot positioning technology, and in particular to a positioning method, device and storage medium for a mobile robot. Background Technology

[0002] Indoor navigation for mobile robots includes active navigation such as laser navigation and visual navigation, as well as passive navigation using base station networking technologies such as Wireless Fidelity (WIFI), Ultra Wide Band (UWB), and Bluetooth.

[0003] Among these, laser navigation developed earlier and is therefore more widely used. However, single-line lasers provide less information and are highly dependent on the environment, often requiring the use of reflectors or reflective stickers in complex environments for navigation and positioning. Multi-line lasers generate large amounts of data, necessitating the processing of massive point cloud data and placing high demands on the performance of the robot platform. Furthermore, laser sensors are more expensive than visual sensors. Navigation methods such as Wi-Fi, UWB, and Bluetooth have emerged recently due to their lower cost and high scalability, but in indoor applications, these methods are susceptible to obstruction by debris and signal interference, and perform poorly in scenarios requiring high precision. Visual navigation has developed rapidly recently and its applications are becoming more widespread. A typical example is QR code navigation, which is widely used in intelligent warehousing and logistics, but QR codes are not allowed to be affixed in shelving areas or main running areas.

[0004] Therefore, improving the positioning capability of mobile robots is a technical problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides at least one positioning method, device, and storage medium for a mobile robot.

[0006] The first aspect of this application provides a method for localizing a mobile robot. The method includes: acquiring a currently acquired inverse perspective mapping image; extracting semantic point clouds from the inverse perspective mapping image to obtain a point cloud set; clustering the point cloud set to obtain multiple point cloud clusters; extracting geometric features corresponding to each point cloud cluster to obtain geometric feature values ​​corresponding to each point cloud cluster; detecting whether the geometric feature values ​​meet preset geometric conditions; selecting point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds; relocalizing the mobile robot based on the usable semantic point clouds; if relocalization fails, using other relocalization strategies for relocalization, wherein the other relocalization strategies are based on environmental perception data acquired by other types of sensors for relocalization.

[0007] In one embodiment, extracting the geometric features corresponding to each point cloud cluster to obtain the geometric feature values ​​corresponding to each point cloud cluster includes: performing principal component analysis on each point cloud cluster to obtain the principal direction feature value and the secondary direction feature value of each point cloud cluster; detecting whether the geometric feature values ​​meet preset geometric conditions, and filtering point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds, including: detecting whether the magnitudes of the principal direction feature value and the secondary direction feature value are both greater than a preset feature value threshold; if so, filtering the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds; and / or, detecting whether the ratio between the principal direction feature value and the secondary direction feature value is greater than a preset ratio threshold; if so, filtering the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds; and / or, detecting whether the difference between the principal direction feature value and the secondary direction feature value is greater than a preset difference threshold; if so, filtering the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds.

[0008] In one embodiment, the relocation strategies are ordered in descending order of priority and include a QR code relocation strategy, a visual semantic relocation strategy, and a laser relocation strategy. The method further includes: executing a QR code relocation strategy, which relocates based on a QR code image acquired by a downward-looking camera; if the QR code relocation strategy fails, then executing a visual semantic relocation strategy based on an inverse perspective mapping image acquired by a surround-view camera; if the relocation fails, then using other relocation strategies, which relocate based on environmental perception data acquired by other types of sensors, including: if no usable semantic point cloud is found in the inverse perspective mapping image, or if no effective pose is calculated based on the usable semantic point cloud, then the visual semantic relocation strategy fails, and a laser relocation strategy is executed based on a laser point cloud acquired by a lidar.

[0009] In one embodiment, after relocalizing the mobile robot, the method further includes: acquiring multiple navigation and localization strategies, wherein different navigation and localization strategies perform navigation and localization based on environmental perception data collected by different sensors; simultaneously executing multiple navigation and localization strategies, and fusing the localization results of multiple navigation and localization strategies to obtain the real-time pose of the mobile robot.

[0010] In one embodiment, the navigation and positioning strategy includes a visual positioning strategy, an odometry positioning strategy, and a laser positioning strategy. The visual positioning strategy performs navigation and positioning based on visual images acquired by a camera, the odometry positioning strategy performs navigation and positioning based on odometry information, and the laser positioning strategy performs navigation and positioning based on point cloud data acquired by a lidar. The real-time pose of the mobile robot is obtained by fusing the positioning results of multiple navigation and positioning strategies, including: detecting whether the visual positioning strategy obtains a positioning result with accuracy meeting a preset accuracy condition; if the visual positioning strategy obtains a positioning result with accuracy meeting the preset accuracy condition, the positioning result of the laser positioning strategy is ignored, and the positioning results of the visual positioning strategy and the odometry positioning strategy are fused to obtain the real-time pose of the mobile robot; if the visual positioning strategy fails to obtain a positioning result with accuracy meeting the preset accuracy condition within a consecutive preset time period, the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot.

[0011] In one embodiment, the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot. This includes: detecting whether the accuracy of the positioning result of the visual positioning strategy is lower than a minimum accuracy threshold; if the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold, then the positioning results of the odometry positioning strategy and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot; otherwise, the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot.

[0012] In one embodiment, the visual positioning strategy includes a QR code positioning strategy and / or a visual semantic positioning strategy. The QR code positioning strategy performs navigation positioning based on a QR code image captured by a downward-facing camera, while the visual semantic positioning strategy performs navigation positioning based on an inverse perspective mapping image captured by a surround-view camera. The positioning results of the visual positioning strategy and the odometry positioning strategy are fused to obtain the real-time pose of the mobile robot, including: in response to the downward-facing camera capturing a QR code image, detecting whether the timestamp of the QR code image capture is later than the real-time pose obtained by fusion at the previous moment; if so, then the positioning result corresponding to the QR code image is combined with the positioning result of the odometry positioning strategy. The system performs fusion to obtain the real-time pose of the mobile robot; otherwise, it ignores the localization result corresponding to the QR code image; and / or, in response to the inverse perspective mapping image acquired by the surround-view camera, it acquires the real-time pose obtained from the previous fusion, the odometry information from the previous time to the current time, and the pre-constructed semantic map; it calculates the current predicted pose based on the real-time pose obtained from the previous fusion and the odometry information; it extracts the semantic point cloud contained in the inverse perspective mapping image, matches the semantic point cloud and the semantic map based on the current predicted pose to obtain the semantically matched pose; and it fuses the semantically matched pose with the current predicted pose to obtain the real-time pose of the mobile robot.

[0013] In one embodiment, matching the semantic point cloud and the semantic map based on the current predicted pose to obtain a semantic matching pose includes: acquiring a semantic probability raster map obtained by rasterizing the semantic map; projecting the semantic point cloud onto the semantic probability raster map based on the current predicted pose to obtain projection points; acquiring the probability value corresponding to the projection point in the semantic probability raster map; if the probability value is greater than a preset probability threshold, then the projection point is taken as a valid semantic point; matching the valid semantic point with the point cloud in the semantic map to obtain an initial matching pose; projecting the semantic point cloud onto the semantic probability raster map based on the initial matching pose; calculating the accuracy corresponding to the initial matching pose based on the probability value of each projection point corresponding to the semantic point cloud in the semantic probability raster map; if the accuracy is greater than a preset accuracy threshold, then the initial matching pose is taken as the semantic matching pose.

[0014] The second aspect of this application provides a localization device for a mobile robot. The device includes: a semantic extraction module, used to acquire a currently collected inverse perspective mapping image, extract semantic point clouds from the inverse perspective mapping image to obtain a point cloud set, and cluster the point cloud set to obtain multiple point cloud clusters; a feature extraction module, used to extract the geometric features corresponding to each point cloud cluster to obtain the geometric feature values ​​corresponding to each point cloud cluster; a point cloud filtering module, used to detect whether the geometric feature values ​​meet preset geometric conditions, and filter the point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds; and a relocalization module, used to relocalize the mobile robot based on the usable semantic point clouds. If the relocalization fails, other relocalization strategies are used for relocalization, and the other relocalization strategies are based on environmental perception data collected by other types of sensors.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the aforementioned positioning method for a mobile robot.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the aforementioned positioning method for a mobile robot.

[0017] The above scheme first performs relocalization using visual semantic information. During visual semantic relocalization, the currently detected visual point cloud is clustered and geometric features are extracted. The extracted geometric feature values ​​are used to determine whether the point cloud cluster can be used for relocalization. If reliable visual semantic features are observed, they are used for localization calculation. Otherwise, environmental perception data collected by other types of sensors is used for relocalization. This not only enriches the relocalization methods in changing environments but also significantly improves the relocalization success rate, ensuring the long-term stable operation of the mobile robot in complex environments and improving the mobile robot's localization capability.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a positioning method for a mobile robot, as shown in an exemplary embodiment of this application; Figure 3This is a schematic diagram of a mobile robot shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the relative pose of the IPM image coordinate system and the vehicle body coordinate system in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the extracted semantic information as shown in an exemplary embodiment of this application; Figure 6 This is a schematic diagram illustrating semantic and laser multilayer point cloud map, as shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating a semantic and laser multilayer probabilistic grid map, as shown in an exemplary embodiment of this application; Figure 8 This is a schematic diagram illustrating the distinction between valid and invalid semantic points, as shown in an exemplary embodiment of this application. Figure 9 This is a schematic diagram of a factor map for multi-sensor data fusion positioning, illustrating an exemplary embodiment of this application; Figure 10 This is a schematic diagram illustrating the effect of multi-sensor data fusion positioning, as shown in an exemplary embodiment of this application. Figure 11 This is a block diagram illustrating a positioning device for a mobile robot, as shown in an exemplary embodiment of this application; Figure 12 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 13 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] The positioning method for mobile robots provided in the embodiments of this application will be described below.

[0025] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of an exemplary embodiment of this application. The implementation environment may include a mobile robot 110 and a server 120, which are interconnected.

[0026] The mobile robot 110 can be an automated guided vehicle (AGV), a sweeping robot, an industrial robot, or other autonomous mobile robot (AMR), and this application does not limit it to this.

[0027] The mobile robot 110 is equipped with a vision camera to acquire visual images of the scene to be located. The number of vision cameras can be one or more, and the types of vision cameras can include downward-looking cameras, and / or surround-looking cameras, and / or monocular cameras, and / or multi-view cameras, etc. The number and type of vision cameras are flexibly set according to the actual application scenario.

[0028] Optionally, the mobile robot 110 may also be equipped with other types of sensors such as LiDAR and depth cameras to collect other types of environmental perception data.

[0029] Of course, the mobile robot 110 may also include a mobile chassis, which may include a motion controller, motor, battery, embedded computer, odometer, gyroscope, etc. This application does not limit this.

[0030] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0031] In one example, server 120 can calculate the pose of mobile robot 110 based on the inverse perspective mapping image obtained from mobile robot 110, and server 120 can generate navigation instructions for mobile robot 110 based on the pose of mobile robot 110 so that mobile robot 110 can perform navigation movement.

[0032] In one example, a client application for a target application is installed and running on the mobile robot 110. This target application may be an application that provides positioning and navigation functions. The pose of the mobile robot 110 is calculated using this target application, and navigation is performed based on the pose of the mobile robot 110. The server 120 may be a backend server for this target application, used to provide backend services to the client of the target application.

[0033] The mobile robot positioning method provided in this application embodiment can be executed by a mobile robot 110, such as the client of the target application installed and running in the mobile robot 110, or by a server 120, or by the interaction and cooperation between the mobile robot 110 and the server 120, that is, some steps of the method are executed by the mobile robot 110 and other steps are executed by the server 120.

[0034] It should be noted that the mobile robot positioning method of this application can be applied to any navigation scenario, such as the shelf docking scenario of industrial robots and the floor cleaning scenario of sweeping robots, and this application does not limit it.

[0035] Please see Figure 2 , Figure 2 This is a flowchart illustrating a mobile robot localization method as shown in an exemplary embodiment of this application, such as... Figure 2 As shown, the mobile robot localization method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain the currently acquired inverse perspective mapping image, extract semantic point clouds from the inverse perspective mapping image to obtain a point cloud set, and cluster the point cloud set to obtain multiple point cloud clusters.

[0036] Inverse Perspective Mapping (IPM) is the inverse process of perspective mapping. It uses the camera's angle, height, and other positional information to establish a three-dimensional coordinate system, eliminates perspective effects, and obtains a top view of the scene.

[0037] IPM images can be generated from images captured by one or more cameras.

[0038] For example, a mobile robot is equipped with multiple surround-view cameras. Each surround-view camera captures environmental images from different angles. The environmental images captured by each surround-view camera are projected onto the IPM image coordinate system to obtain an IPM image.

[0039] Then, semantic point cloud extraction is performed on the IPM image to obtain a point cloud set.

[0040] Semantic point cloud is a set of discrete points in the mobile robot coordinate system (hereinafter referred to as vehicle coordinate system) calculated from the image feature pixels contained in the IPM image.

[0041] For example, semantic segmentation of IPM images can be performed to obtain image features belonging to environmental semantics (such as parking space lines, and / or lane lines, and / or indicator lines, etc.).

[0042] After identifying the pixels in the IPM image that belong to the environmental semantics, the pixels in the IPM image that belong to the environmental semantics are transformed into the vehicle coordinate system to obtain semantic points. The point cloud set corresponding to the IPM image is obtained by combining all semantic points.

[0043] Then, the point cloud set is clustered to obtain multiple point cloud clusters.

[0044] The clustering algorithms used include, but are not limited to, density-based clustering of applications with noise (DBSCAN), and / or K-means clustering, and / or Euclidean cluster extraction. This application does not limit the specific clustering algorithm used.

[0045] Step S220: Extract the geometric features corresponding to each point cloud cluster to obtain the geometric feature values ​​corresponding to each point cloud cluster.

[0046] Principal Component Analysis (PCA), and / or line fitting, and / or plane fitting methods, and / or calculation of the minimum bounding rectangle, can be used to extract the geometric features corresponding to the point cloud clusters. This application does not limit the specific method of extracting geometric features.

[0047] Among them, geometric features are used to describe the geometric information of point cloud clusters, such as the length, and / or width, and / or shape, and / or orientation of point cloud clusters. This application does not limit the specific type of extracted geometric features.

[0048] Taking PCA decomposition to obtain geometric features as an example, PCA uses the idea of ​​dimensionality reduction to transform multiple indicators into a few comprehensive indicators. Performing PCA decomposition on a point cloud cluster yields eigenvalues ​​for at least one principal direction corresponding to the point cloud cluster, which describe the distribution pattern of the point cloud cluster in the principal direction.

[0049] Step S230: Detect whether the geometric feature values ​​meet the preset geometric conditions, and filter the point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds.

[0050] By detecting whether the geometric feature values ​​meet the preset geometric conditions, it is possible to determine whether the shape, and / or size, and / or orientation of the point cloud clusters meet the requirements, thereby filtering out usable semantic point clouds and improving the accuracy of subsequent localization calculations.

[0051] For example, the size of a point cloud cluster (such as length, and / or width, and / or area) is obtained based on geometric feature values. The structural salience of the point cloud cluster is calculated based on the size, and point cloud clusters with structural salience higher than a threshold are selected as usable semantic point clouds. The structural salience of a point cloud cluster can be determined based on the size relationship between the size and a preset size threshold, and / or the size relationship between individual point cloud clusters.

[0052] For example, if the size of a point cloud cluster is greater than a preset size threshold, it is determined that the point cloud cluster has high structural significance; and / or, after sorting each point cloud cluster in descending order according to size, the top N point cloud clusters are determined to have high structural significance.

[0053] For example, the orientation of a point cloud cluster is obtained based on geometric feature values. It is then determined whether the orientation is a preset orientation. If so, the corresponding point cloud cluster is used as a usable semantic point cloud.

[0054] For example, based on geometric feature values, the shape of the point cloud cluster (such as lines, and / or rectangles, and / or circles, etc.) is obtained. It is then determined whether the shape is a preset shape. If so, the corresponding point cloud cluster is used as a usable semantic point cloud.

[0055] Of course, the above filtering methods can also be combined to obtain usable semantic point clouds through comprehensive filtering.

[0056] The preset geometric conditions used to filter available semantic point clouds can be flexibly set according to the actual application scenario, and this application does not impose any restrictions on them.

[0057] Step S240: Relocalize the mobile robot based on the available semantic point cloud. If relocalization fails, other relocalization strategies are used for relocalization. Other relocalization strategies are based on environmental perception data collected by other types of sensors for relocalization.

[0058] After obtaining the available semantic point cloud, the mobile robot is relocalized based on the available semantic point cloud.

[0059] The pose calculation methods used for relocalization include, but are not limited to, the Iterative ClosestPoint (ICP) algorithm and / or the Perspective-n-Point (PnP) algorithm. The pose calculation method used can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0060] If relocalization based on the inverse perspective mapping image fails, relocalization is performed using environmental perception data collected by other types of sensors, such as laser point clouds collected by LiDAR, and / or QR codes collected by the downward-looking camera, and / or collected WIFI, and / or collected Bluetooth information, etc., and relocalization is performed again.

[0061] This application first performs relocalization using visual semantic information. During visual semantic relocalization, the currently detected visual point cloud is clustered and geometric features are extracted. The extracted geometric feature values ​​are used to determine whether the point cloud cluster can be used for relocalization. As long as reliable visual semantic features are observed, they can be used for localization estimation. Otherwise, environmental perception data collected by other types of sensors is used for relocalization. This not only enriches the relocalization methods in changing environments but also significantly improves the relocalization success rate, ensuring the long-term stable operation of the mobile robot in complex environments and improving the localization capability of the mobile robot.

[0062] The following describes some embodiments of this application in detail.

[0063] Build a priori map to facilitate the relocalization and real-time localization of the mobile robot.

[0064] For example, regarding mobile robots in a warehouse, please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of a mobile robot illustrated in an exemplary embodiment of this application, as shown below. Figure 3 As shown, the sensors used by the mobile robot for environmental perception data acquisition include three surround-view cameras and a LiDAR. In the warehouse environment, the mobile robot uses odometry and LiDAR to construct a laser map. During the laser map construction process, it uses the three surround-view cameras and estimated pose projections to generate IPM images, thus obtaining a semantic map.

[0065] Of course, other types of prior maps can also be constructed, such as QR code maps. The specific map type can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0066] It should be noted that, in order to ensure positioning accuracy, different prior maps need to be aligned, and coordinates on different maps can be converted to each other, such as aligning laser maps, semantic maps, and QR code maps.

[0067] To illustrate the process of constructing a semantic map, consider this example: In a warehouse environment, markers such as storage location lines and lane lines are often deployed. These semantic features can be used to assist in localization. Specifically, images are acquired from each surround-view camera, and each image is projected into an IPM image. The IPM image is a top-down planar image obtained by stitching and correcting the ground view images from the three surround-view cameras, eliminating perspective distortion and facilitating ground feature processing.

[0068] Specifically, let the IPM image coordinate system be {ipm}, the robot's body coordinate system be {b}, and the camera coordinate system of the i-th surround-view camera be {ci}. The pixel value of the image acquired by the three surround-view cameras transformed into the IPM image coordinate system is denoted as... U and V are the column index and row index of the pixel, respectively, and the third dimension being 0 indicates a two-dimensional planar point. The scale of each pixel value in the IPM image is r. The transformation relationship between the coordinates in the IPM image coordinate system and the coordinates in the vehicle body coordinate system is shown in Formula 1 below: (Formula 1) in, These are coordinates in the vehicle coordinate system; This is the relative pose of the IPM image coordinate system relative to the vehicle coordinate system, i.e., the transformation matrix from the IPM image coordinate system to the vehicle coordinate system. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating the relative pose of the IPM image coordinate system and the vehicle body coordinate system in an exemplary embodiment of this application. The IPM image coordinate system corresponds to the blue coordinate axis, and the vehicle body coordinate system corresponds to the red coordinate axis.

[0069] The transformation relationship between coordinates in the vehicle coordinate system and coordinates in the camera coordinate system is shown in Formula 2 below: (Formula 2) in, These are the coordinates in the camera coordinate system. K represents the extrinsic parameter of the i-th surround-view camera relative to the vehicle body; K represents the intrinsic parameter of the i-th surround-view camera.

[0070] That is, each pixel in the IPM image coordinate system can be used to find the pixel position in different camera coordinate systems using Formula 1 and Formula 2.

[0071] In Formula 2 In the obtained camera coordinate system, if the z-axis value is less than zero, it means that the pixel is behind the camera coordinate system and is not within the camera's field of view, so the pixel is ignored; otherwise, the pixel is retained.

[0072] The average pixel value of the overlapping area of ​​multiple camera projections is taken as the pixel value of the IPM image. For example, the average pixel value corresponding to the same pixel position under three panoramic cameras is taken as the pixel value of the corresponding pixel position in the IPM image coordinate system, so as to realize the stitching of multiple cameras and obtain the IPM image.

[0073] Then, semantic information is extracted from the IPM image, and a semantic map is constructed based on the extracted semantic information.

[0074] For examples, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the extracted semantic information in an exemplary embodiment of this application, such as... Figure 5 As shown, the semantic information of the IPM image is extracted using a Bilateral Segmentation Network (BiSeNet), and the resulting red line segment is the semantic information extracted from the current IPM image.

[0075] In addition, during the mapping process, the matching results of wheel odometry, laser point cloud, and local laser map can be used to calculate the pose of the mobile robot at the current moment. Among them, wheel odometry provides short-term pose increment prediction by performing inter-frame matching between the current laser point cloud frame and the constructed local laser map to correct the accumulated error of odometry and obtain the pose at the current moment.

[0076] Of course, other methods can also be used to calculate the pose at the current moment, such as relying solely on odometer data and the pose at the previous moment to calculate the pose at the current moment. This application does not limit this method.

[0077] For example, let the pose of the IPM image relative to the world coordinate system (denoted by w) at the current time t be denoted by . This represents the transformation relationship between the vehicle coordinate system and the world coordinate system. The point cloud coordinates obtained by transforming the semantic information in the IPM image to the world coordinate system are then... This can be expressed as Formula 3 below: (Formula 3) The physical meaning of each symbol is explained in the aforementioned embodiments and will not be repeated here.

[0078] The semantic point cloud corresponding to the semantic information obtained from the segmentation of each IPM image is transformed into the world coordinate system using Formula 3. The transformation results of the semantic point clouds of all IPM images are accumulated to construct a semantic map.

[0079] Furthermore, a semantic probability raster map can be constructed by refreshing the semantic map in a rasterized manner. The semantic probability raster map includes multiple grids, and each grid stores a probability value. The probability value is used to represent the probability that a semantic feature exists at the corresponding location of the grid.

[0080] Similarly, a laser map can be constructed by converting the point cloud output by the LiDAR into a world coordinate system. Furthermore, a laser probabilistic raster map can be constructed by rasterizing and refreshing the laser map. This probabilistic raster map consists of multiple grids, each storing a probability value that represents the probability of an obstacle existing at the corresponding location of that grid.

[0081] For examples, please refer to Figure 6 , Figure 6 This is a schematic diagram of a semantic and laser multi-layer point cloud map shown in an exemplary embodiment of this application, wherein the black dots are laser point clouds, and the purple part is lane line semantic point cloud segmented from the IPM image stitched by multiple surround-view cameras. The semantic map and the laser map share the same world coordinate system to form the final multi-layer prior map.

[0082] Then, during mobile robot localization, the aforementioned multi-layered prior map is loaded, and the semantic map and laser map in the multi-layered prior map are converted into a semantic probabilistic raster map and a laser probabilistic raster map, respectively. (See [link to relevant documentation]). Figure 7 , Figure 7 This is a schematic diagram of a semantic and laser multilayer probability grid map shown in an exemplary embodiment of this application, wherein the orange part is the semantic probability grid map and the black part is the laser probability grid map.

[0083] The mobile robot is located based on the map described above. Mobile robot localization includes relocalization and real-time localization.

[0084] Relocalization refers to redetermining the pose of a mobile robot. When the initial pose of a mobile robot is completely unknown or the current pose estimate is unreliable, the relocalization process of the mobile robot is initiated.

[0085] For example, if the mobile robot loses its location due to occlusion, missing features, or violent movement during startup or navigation, the mobile robot's relocation process will be initiated.

[0086] Real-time localization refers to continuously calculating the pose after obtaining the initial pose from relocalization.

[0087] The relocation process of the mobile robot will be explained next.

[0088] In some implementations, multiple relocation strategies are pre-set, the priority of each relocation strategy is determined, and the relocation strategies are executed in descending order of priority until relocation is successful.

[0089] For example, each relocation strategy is ordered in descending order of priority as follows: QR code relocation strategy, visual semantic relocation strategy, and laser relocation strategy.

[0090] The QR code relocation strategy is based on the QR code image captured by the downward-looking camera for relocation; the visual semantic relocation strategy is based on the inverse perspective mapping image captured by the surround-view camera for relocation, that is, the mobile robot is relocated based on the relocation steps recorded in steps S210 to S240; the laser relocation strategy is based on the point cloud data captured by the lidar for relocation.

[0091] The relocalization steps include: executing a QR code relocalization strategy; if the QR code relocalization is successful, the initial pose of the mobile robot is output; if the QR code relocalization fails, executing a visual semantic relocalization strategy; if the visual semantic relocalization is successful, the initial pose of the mobile robot is output; if the visual semantic relocalization fails, executing a laser relocalization strategy and outputting the initial pose of the mobile robot.

[0092] For example, a QR code relocation strategy is executed, which relocates the QR code image captured by the downward-looking camera. If the QR code relocation strategy fails, a visual semantic relocation strategy is executed based on the inverse perspective mapping image captured by the surround-view camera. If no usable semantic point cloud is found in the inverse perspective mapping image, or no effective pose is calculated based on the usable semantic point cloud, the visual semantic relocation strategy fails, and a laser relocation strategy is executed based on the laser point cloud captured by the lidar.

[0093] Specifically, the system first determines whether the downward-facing camera has detected a valid QR code image. If a valid QR code image is detected, the robot's pose is directly read from the QR code image. If no valid QR code image is detected, a visual semantic relocalization strategy is executed. This strategy calculates the robot's pose based on the inverse perspective mapping image acquired by the surround-view camera. If both the QR code relocalization strategy and the visual semantic relocalization strategy fail, a laser relocalization strategy is executed. This strategy acquires the current laser point cloud and matches it with the laser probabilistic grid map to obtain the relocalization result.

[0094] Of course, in addition to the examples above, other combinations of relocation strategies can also be used. For example, a visual semantic relocation strategy can be executed first, and if it fails, a laser relocation strategy can be executed; or, a visual semantic relocation strategy can be executed first, and if it fails, a QR code relocation strategy can be executed; if the QR code relocation fails, a laser relocation strategy can be executed last. The specific combination of relocation strategies can be flexibly set according to the actual application scenario, and this application does not limit it in this regard.

[0095] Furthermore, the priority among various relocation strategies can be preset based on experience or calculated flexibly; this application does not impose any limitations on this.

[0096] For example, record the relocation strategy used for each successful relocation of the mobile robot, obtain the number of successes for each relocation strategy, and set the priority of each relocation strategy based on the number of successes. The more successes, the higher the priority.

[0097] For example, if a mobile robot performs relocalization after losing its localization, the last valid pose before the localization loss is obtained. Based on a prior map, the density of different types of environmental feature data within a preset range of the valid pose is detected. According to the density of different environmental feature data, the priority of the relocalization strategy associated with the environmental feature data is set; the higher the density, the higher the priority of the associated relocalization strategy. For example, the higher the density of a QR code image, the higher the priority of the QR code relocalization strategy; the higher the density of semantic information (such as parking space lines, lane lines, etc.), the higher the priority of the visual semantic relocalization strategy.

[0098] The specific implementation process of visual semantic relocation is explained.

[0099] If semantic information is detected in the IPM image, semantic point cloud extraction is performed on the inverse perspective mapping image to obtain a point cloud set, and the point cloud set is clustered to obtain multiple point cloud clusters. Then, the geometric features corresponding to each point cloud cluster are extracted to obtain the geometric feature values ​​corresponding to each point cloud cluster, and usable semantic point clouds are selected based on the geometric feature values.

[0100] An example is given to illustrate how to filter available semantic point clouds.

[0101] For example, the step of extracting geometric feature values ​​includes: performing principal component analysis on each point cloud cluster to obtain the principal direction feature values ​​and secondary direction feature values ​​for each point cloud cluster.

[0102] For example, let a point cloud cluster be denoted as {Ci}. Performing PCA decomposition on the point cloud cluster yields the principal eigenvalues ​​and secondary eigenvalues, denoted as... , .

[0103] Example 1: Detect whether the magnitudes of the main direction feature value and the secondary direction feature value are both greater than the preset feature value threshold. If so, filter the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds.

[0104] The preset feature value threshold is determined based on the size of semantic information (such as parking space lines, lane lines, etc.) existing in the positioning environment.

[0105] For example, if the principal direction eigenvalue and secondary eigenvalues All are greater than the preset feature value threshold. If the corresponding point cloud cluster has sufficient length in both the primary and secondary directions, it indicates that the point cloud cluster is a large-scale, stable marker, rather than segmentation noise or small spots. Therefore, the point cloud cluster is selected as a usable semantic point cloud.

[0106] Example 2: Detect whether the ratio between the main direction feature value and the secondary direction feature value is greater than a preset ratio threshold. If so, filter the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds.

[0107] Similarly, the preset ratio threshold is determined based on the size of semantic information (such as parking space lines, lane lines, etc.) existing in the positioning environment.

[0108] For example, if the principal direction eigenvalue and secondary eigenvalues The ratio between them is greater than the preset ratio threshold. If the corresponding point cloud cluster has a linear structure and conforms to the shape definition of semantic information (such as parking space lines, lane lines, etc.), then the point cloud cluster is selected as a usable semantic point cloud.

[0109] Example 3: Detect whether the difference between the main direction feature value and the secondary direction feature value is greater than a preset difference threshold. If so, filter the point cloud clusters corresponding to the geometric feature values ​​as usable semantic point clouds.

[0110] Similarly, the preset difference threshold is determined based on the size of semantic information (such as parking space lines, lane lines, etc.) existing in the positioning environment.

[0111] For example, if the principal direction eigenvalue and secondary eigenvalues The difference between them is greater than the preset difference threshold. If the corresponding point cloud cluster has a linear structure and conforms to the shape definition of semantic information (such as parking space lines, lane lines, etc.), then the point cloud cluster is selected as a usable semantic point cloud.

[0112] Alternatively, the available semantic point clouds can be comprehensively screened by combining the above embodiments 1 and / or 2 and / or 3, and this application does not limit this.

[0113] Of course, other methods can also be used for point cloud filtering, such as filtering point cloud clusters based on the number of points contained in the clusters. This application does not limit this method.

[0114] For example, after obtaining multiple point cloud clusters, the clusters with a number of points exceeding a preset threshold are filtered out. Clusters of dots, The value can be limited according to the semantic information type, such as The value is 50. If the number of point clouds in a point cloud cluster is less than... If the condition is not met, skip the point cloud cluster; otherwise, perform relocalization based on the point cloud cluster, such as constructing a kdtree for the point clouds in each point cloud cluster and matching it with the prior map to obtain the pose of the mobile robot.

[0115] For example, first filter out points whose point cloud count exceeds a preset threshold. The point cloud cluster is then decomposed using PCA to obtain the principal direction feature value and the secondary direction feature value. If both feature values ​​are greater than the preset feature value threshold, the point cloud cluster is used for relocalization.

[0116] The relocalization calculation process is illustrated with an example: A semantic probability grid map is acquired. Within a preset pose search range, a pose search is performed according to preset displacement and angle step sizes. Based on the currently searched pose, the available semantic point cloud is projected onto the semantic probability grid map, resulting in a point cloud projection result. Then, based on the probability values ​​of the corresponding grid cells in the point cloud projection result, the pose matching score is calculated; the higher the probability value, the higher the matching score. The pose with the highest score is selected as the initial pose of the mobile robot.

[0117] The real-time positioning process of the mobile robot will be explained next.

[0118] In some embodiments, after repositioning the mobile robot, the method further includes the following steps S250 to S260.

[0119] Step S250: Obtain multiple navigation and positioning strategies. Different navigation and positioning strategies perform navigation and positioning based on environmental perception data collected by different sensors.

[0120] For example, navigation and positioning strategies include visual positioning strategies, and / or odometry positioning strategies, and / or laser positioning strategies. Visual positioning strategies are based on visual images captured by cameras for navigation and positioning, odometry positioning strategies are based on odometry information for navigation and positioning, and laser positioning strategies are based on point cloud data collected by lidar for navigation and positioning.

[0121] Step S260: Simultaneously execute multiple navigation and positioning strategies, fuse the positioning results of multiple navigation and positioning strategies, and obtain the real-time pose of the mobile robot.

[0122] For example, during the localization process of a mobile robot, three threads are simultaneously activated: QR code localization, visual semantic localization, and laser localization. The localization results of each thread are then fused to obtain the real-time pose of the mobile robot.

[0123] For example, the navigation and positioning strategies include visual positioning strategies, odometry positioning strategies, and laser positioning strategies; in step S260, the positioning results of multiple navigation and positioning strategies are fused to obtain the real-time pose of the mobile robot, including the following steps S261 to S264.

[0124] Step S261: Detect whether the visual positioning strategy has obtained a positioning result whose accuracy meets the preset accuracy conditions.

[0125] Whether a visual positioning strategy can obtain a positioning result with an accuracy that meets a preset accuracy condition can be determined based on whether it can acquire usable visual information; and / or whether the accuracy of the pose output by the visual positioning strategy is greater than a preset accuracy threshold can be determined based on whether the visual positioning strategy can obtain a positioning result with an accuracy that meets a preset accuracy condition. This application does not limit this.

[0126] Step S262: If the visual positioning strategy obtains a positioning result whose accuracy meets the preset accuracy condition, then proceed to step S263; if the visual positioning strategy fails to obtain a positioning result whose accuracy meets the preset accuracy condition within a consecutive preset time period, then proceed to step S264.

[0127] Step S263: Ignore the positioning results of the laser positioning strategy, and fuse the positioning results of the visual positioning strategy and the odometry positioning strategy to obtain the real-time pose of the mobile robot.

[0128] If the visual positioning strategy can achieve effective positioning results, the positioning results of the laser positioning strategy can be skipped due to the higher accuracy of visual observation.

[0129] Taking a visual positioning strategy, including a QR code positioning strategy and / or a visual semantic positioning strategy, as an example, the QR code positioning strategy performs navigation positioning based on the QR code image acquired by the downward-looking camera, and the visual semantic repositioning strategy performs navigation positioning based on the inverse perspective mapping image acquired by the surround-view camera. The calculation process of real-time pose is illustrated as follows: In step S263, the positioning results of the visual positioning strategy and the odometry positioning strategy are fused to obtain the real-time pose of the mobile robot, including: executing the QR code positioning strategy and / or the visual semantic positioning strategy to obtain the positioning result of the visual positioning strategy, and fusing the positioning results of the visual positioning strategy and the odometry positioning strategy to obtain the real-time pose of the mobile robot.

[0130] Among them, the odometry positioning strategy obtains the pose change recorded by the odometry, and predicts the pose at the current moment based on the pose change and the real-time pose at the previous moment.

[0131] It should be noted that before performing pose calculation, it can be determined whether the pose change recorded by the odometer is greater than a preset change threshold, and / or whether the time difference between the previous moment and the current moment of pose calculation is greater than a preset time threshold. If so, the odometer positioning strategy is executed to predict the pose at the current moment.

[0132] The positioning results of the visual positioning strategy and the odometry positioning strategy are fused to obtain the real-time pose of the mobile robot.

[0133] QR code positioning strategy: In response to the downward-facing camera capturing a QR code image, it checks whether the capture timestamp of the QR code image is later than the real-time pose obtained by fusion at the previous moment; if so, it fuses the positioning result corresponding to the QR code image with the positioning result of the odometry positioning strategy to obtain the real-time pose of the mobile robot; if not, it ignores the positioning result corresponding to the QR code image.

[0134] Visual semantic localization strategy: In response to the inverse perspective mapping image acquired by the surround-view camera, the system obtains the real-time pose fused from the previous moment, the odometry information from the previous moment to the current moment, and the pre-constructed semantic map; the system calculates the current predicted pose based on the real-time pose fused from the previous moment and the odometry information; the system extracts the semantic point cloud contained in the inverse perspective mapping image, matches the semantic point cloud and the semantic map based on the current predicted pose to obtain the semantically matched pose; the system fuses the semantically matched pose with the current predicted pose to obtain the real-time pose of the mobile robot.

[0135] The current moment refers to the moment when the current inverse perspective mapping image is acquired.

[0136] The methods for matching and calculating poses between semantic point clouds and semantic maps include, but are not limited to, ICP algorithm, and / or PnP algorithm, and / or factor graph model localization algorithm, etc., and this application does not limit them.

[0137] Preferably, in order to improve the accuracy of visual semantic pose calculation, effective semantic points are selected from the semantic point cloud and matched with the point cloud in the semantic map.

[0138] For example, matching the semantic point cloud and the semantic map based on the current predicted pose to obtain the semantic matching pose includes: obtaining the semantic probability raster map obtained after rasterizing the semantic map; projecting the semantic point cloud onto the semantic probability raster map based on the current predicted pose to obtain the projection point; obtaining the probability value corresponding to the projection point in the semantic probability raster map; if the probability value is greater than a preset probability threshold, the projection point is taken as a valid semantic point.

[0139] If the probability value is greater than the preset probability threshold, it is determined that there is a high probability of semantic information in the positioning environment. Therefore, the projection point is regarded as a valid semantic point.

[0140] For example, using the real-time pose obtained from the previous time step and the odometry information from the previous time step to the current time step, the current predicted pose of the mobile robot can be calculated as follows: .according to The semantic point cloud segmented from the IPM image is projected onto a semantic probability raster map. If the probability value of a certain semantic point after projection is greater than a preset probability threshold... If the value is greater than 0.2, the semantic point is considered valid; otherwise, it is considered invalid.

[0141] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the distinction between valid and invalid semantic points, as shown in an exemplary embodiment of this application. Figure 8 As shown, the yellow part represents the valid semantic points obtained from the current IPM image segmentation, and the pink part represents the invalid semantic points. Only the valid semantic points are used to match the semantic map.

[0142] Here's an example to illustrate the matching process: A kdtree, denoted as KD, is constructed from the point cloud in the semantic map. The valid semantic points detected in the current IPM image are traversed, and the nearest neighbor matching point is queried in KD: if the nearest neighbor distance is greater than the maximum distance threshold (e.g., greater than 10cm), it is skipped; otherwise, it is determined to match a valid semantic point, resulting in a matching point pair. The residual equation is constructed based on the spatial locations of all matching point pairs, as shown in Formula 4 below. ri=(T*pbi- pMi)(Formula 4) Where ri represents the residual of the i-th matching point pair; T represents the pose to be estimated; pbi represents the coordinates of the effective semantic point in the current IPM image that has a nearest neighbor that matches the semantic map (in the vehicle coordinate system); pMi represents the coordinates of the point in the semantic map that can match the effective semantic point (in the map coordinate system).

[0143] The above residual represents the deviation between the effective semantic points under the vehicle body projected onto the world coordinate system and the corresponding points on the map, assuming the current pose is T.

[0144] The optimal robot pose T can be solved by minimizing the sum of squared residuals of all matching point pairs.

[0145] Optionally, the accuracy of the pose obtained by matching with the semantic map can be calculated, and pose calculation results with low accuracy can be filtered out.

[0146] For example, valid semantic points are matched with point clouds in the semantic map to obtain an initial matching pose; the semantic point cloud is projected onto the semantic probability grid map based on the initial matching pose; the accuracy corresponding to the initial matching pose is calculated based on the probability value of each projection point of the semantic point cloud in the semantic probability grid map; if the accuracy is greater than a preset accuracy threshold, the initial matching pose is used as the semantic matching pose.

[0147] For example, the average probability value of each projection point is calculated, and the accuracy corresponding to the initial matching pose is calculated based on the average value. The larger the average value, the higher the accuracy corresponding to the initial matching pose.

[0148] For example, the number of projection points with a statistical probability value greater than a preset probability threshold is used to calculate the accuracy of the initial matching pose. The larger the number, the higher the accuracy of the initial matching pose.

[0149] Of course, other methods can also be used to calculate the accuracy of the initial matching pose, and this application does not limit this.

[0150] If the accuracy is greater than the preset accuracy threshold If so, the initial matching pose will be used as the semantic matching pose.

[0151] For example, if the accuracy is greater than 0.25, the initial matched pose is used as the semantic matched pose. The semantic matched pose is fused with the current predicted pose and judged as the positioning result obtained by the visual positioning strategy with an accuracy that meets the preset accuracy condition (such as assigning the effective visual observation mark position to true). At this time, other navigation and positioning strategies can be skipped directly (such as skipping the laser positioning strategy), and the pose obtained by fusing the semantic matched pose with the current predicted pose is used as the real-time pose of the mobile robot.

[0152] The above embodiments can use QR codes or visual semantic information for relocation, enriching the relocation methods in changing environments. Moreover, it is not necessary to deploy QR codes in the entire environment; only a small number of QR codes need to be deployed in places without laser features and visual semantic features.

[0153] In addition, throughout the positioning process, QR codes or visual semantic information are continuously detected. As long as reliable visual features are observed, they can be used for positioning estimation. Unlike traditional solutions that require the use of semantic features to extract corner features and generate descriptors, this method uses probability values ​​to distinguish between valid and invalid semantic information from the currently detected semantic point cloud, thereby improving the matching accuracy of semantic features.

[0154] Step S264: Fuse the positioning results of the visual positioning strategy, odometry positioning strategy and laser positioning strategy to obtain the real-time pose of the mobile robot.

[0155] If the visual positioning strategy cannot achieve effective positioning results, the positioning results of the laser positioning strategy can be integrated to improve the accuracy and robustness of the mobile robot's operation.

[0156] For example, in step S264, the positioning results of the visual positioning strategy, the odometry positioning strategy and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot, including the following steps S2641 to S2644.

[0157] Step S2641: Detect whether the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold.

[0158] The accuracy of the positioning results of the visual positioning strategy can be determined based on the quality of the acquired visual images and / or the degree of matching with the prior map.

[0159] For example, if the visual positioning strategy includes a visual semantic positioning strategy, the accuracy corresponding to the initial matching pose can be calculated by using the probability value of each projection point corresponding to the semantic point cloud in the semantic probability grid map in the aforementioned embodiment.

[0160] Step S2642: If the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold, then proceed to step S2643; otherwise, proceed to step S2644.

[0161] Step S2643: Fuse the positioning results of the odometry positioning strategy and the laser positioning strategy to obtain the real-time pose of the mobile robot.

[0162] If the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold, then the visual positioning strategy has no effective visual observation, and only the positioning results of the odometry positioning strategy and the laser positioning strategy are fused for positioning.

[0163] For example, the minimum accuracy threshold is 0.1. If the accuracy of the positioning result of the visual positioning strategy is less than 0.1, then only the laser and the odometer will be fused for positioning.

[0164] Step S2644: Fuse the positioning results of the visual positioning strategy, odometry positioning strategy and laser positioning strategy to obtain the real-time pose of the mobile robot.

[0165] If the accuracy of the positioning result of the visual positioning strategy is not lower than the minimum accuracy threshold, then the positioning results of the visual positioning strategy, the odometry positioning strategy and the laser positioning strategy will be fused.

[0166] For example, if the visual positioning strategy yields a positioning result with an accuracy that meets a preset accuracy condition (e.g., a valid visual observation marker corresponding to the visual semantic positioning strategy is detected and assigned a value of true), then the laser matching result is skipped. If no valid visual observation is obtained within a preset time period (e.g., within 5 seconds), the positioning results of the visual positioning strategy, odometry positioning strategy, and laser positioning strategy are fused for positioning. During this transition period, only odometry is used for pose estimation. In the above fusion process, if the accuracy corresponding to the initial matching pose of the visual semantic positioning is greater than 0.1 and less than 0.25, then the positioning result of the visual positioning strategy participates in the fusion; otherwise, only the positioning results of the odometry positioning strategy and the laser positioning strategy are fused for positioning.

[0167] The above embodiments can determine whether to fuse laser matching during the positioning process based on the reliability of the identified QR code and visual semantic information, without requiring external settings for the positioning mode. When valid visual observations exist, laser matching observations are not fused; otherwise, visual, laser, and odometry information are fused for pose estimation, improving the accuracy and robustness of the mobile robot's operation.

[0168] Of course, besides the fusion positioning method of multiple navigation positioning strategies exemplified above, other methods can also be used to fuse multiple navigation positioning strategies. For example, multiple navigation positioning strategies may include QR code positioning strategy, and / or visual semantic positioning strategy, and / or odometer positioning strategy, and / or laser positioning strategy. The implementation methods of each navigation positioning strategy are described in the foregoing embodiments. Fusion weights are set according to the accuracy of the positioning results corresponding to each navigation positioning strategy to fuse the various positioning results; alternatively, fusion weights corresponding to each navigation positioning strategy are pre-set, and the various positioning results are fused according to the fusion weights. This application does not limit the specific fusion method.

[0169] Optionally, factor plots can be constructed based on data from different sensors. Please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of a factor map for multi-sensor data fusion positioning, as illustrated in an exemplary embodiment of this application. Figure 9 As shown, the image optimization constraints include: The relative pose constraint between adjacent local visual images is that the change in the pose of the mobile robot between adjacent local visual images is less than a preset change threshold. The visual images can be IPM images and / or QR code images. The visual odometry is obtained by matching the poses of two adjacent visual images and is used as the relative pose constraint between the two visual images. And / or, the absolute pose constraint between the global visual image and the visual prior map. The global visual image refers to the visual image acquired after the pose change of the mobile robot relative to the previous effective pose exceeds a preset change threshold. The visual prior map can be the semantic probability grid map and / or semantic map constructed in the aforementioned embodiments. And / or, relative pose constraints between odometry data of adjacent global visual images; And / or, the absolute pose constraint obtained by matching the laser point cloud frame with the laser prior map, wherein the laser prior map may be the laser probability grid map and / or laser map constructed in the aforementioned embodiments; And / or, relative pose constraints calculated between laser point cloud frames, or between visual image frames, or between the odometry data corresponding to the laser point cloud frame and the nearest visual image frame, or relative pose constraints calculated between the odometry data corresponding to the local visual image and the global visual image. And / or, relative pose constraints between poses calculated from adjacent laser point cloud frames, where the change in the pose of the mobile robot between adjacent laser point cloud frames is less than a preset change threshold, and pose matching is performed between two adjacent laser point cloud frames to obtain a laser odometry, which is then used as a relative pose constraint between the two laser point cloud frames.

[0170] Based on the residual terms corresponding to the above optimization constraint factors, the pose of the mobile robot is calculated with the goal of minimizing the residual.

[0171] In multi-sensor fusion localization, the weight of the laser-matched pose obtained by matching the laser point cloud with the laser prior map is higher than the weight of the semantic-matched pose obtained by matching the semantic point cloud with the semantic prior map. The weight of the semantic-matched pose increases with the accuracy of semantic matching, but its maximum value is still lower than that of the laser-matched pose. When the accuracy of the laser-matched pose exceeds a preset threshold, the weight between the laser and visual optimization variables increases, as does the weight of the laser-matched pose obtained through global laser matching, thus ensuring the accuracy of pose estimation. When the accuracies of both semantic-matched poses and the laser-matched pose are below a minimum threshold, the weight between the laser and visual optimization variables decreases, while the weights of the laser and prior map, and the weights of semantic information and the prior map are set to zero, entering odometry localization mode.

[0172] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the effect of multi-sensor data fusion positioning in an exemplary embodiment of this application, such as... Figure 10 As shown, the green point cloud is the point cloud matched between the current laser and the laser prior map, and the yellow point cloud is the point cloud matched between the current IPM image and the semantic prior map. The fused pose trajectory is shown in the blue lines of the image. The overall shape is very smooth, ensuring the robustness of localization in changing environments.

[0173] Figure 11 This is a block diagram illustrating a positioning device for a mobile robot, as shown in an exemplary embodiment of this application. Figure 11 As shown, the exemplary positioning device 1100 for the mobile robot includes: The semantic extraction module 1110 is used to acquire the currently acquired inverse perspective mapping image, extract semantic point clouds from the inverse perspective mapping image to obtain a point cloud set, and cluster the point cloud set to obtain multiple point cloud clusters. The feature extraction module 1120 is used to extract the geometric features corresponding to each point cloud cluster, and obtain the geometric feature values ​​corresponding to each point cloud cluster. The point cloud filtering module 1130 is used to detect whether the geometric feature values ​​meet the preset geometric conditions, and to filter point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds. The relocalization module 1140 is used to relocalize the mobile robot based on the available semantic point cloud. If the relocalization fails, other relocalization strategies are used for relocalization. These other relocalization strategies are based on environmental perception data collected by other types of sensors.

[0174] It should be noted that the mobile robot positioning device and the mobile robot positioning method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the mobile robot positioning device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0175] Please see Figure 12 , Figure 12 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 1200 includes a memory 1210 and a processor 1220. The processor 1220 executes program instructions stored in the memory 1210 to implement the steps in any of the above-described embodiments of the mobile robot positioning method. In a specific implementation scenario, the electronic device 1200 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 1200 may also include mobile devices such as laptops and tablets, without limitation.

[0176] Specifically, processor 1220 controls itself and memory 1210 to implement the steps in any of the above-described embodiments of the mobile robot positioning method. Processor 1220 may also be referred to as a Central Processing Unit (CPU). Processor 1220 may be an integrated circuit chip with signal processing capabilities. Processor 1220 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 1220 may be implemented using integrated circuit chips.

[0177] Please see Figure 13 , Figure 13This is a schematic diagram illustrating the structure of a computer-readable storage medium according to an exemplary embodiment of this application. The computer-readable storage medium 1300 stores program instructions 1310 that can be executed by a processor. The program instructions 1310 are used to implement the steps in any of the above-described embodiments of the positioning method for mobile robots.

[0178] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0179] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A positioning method for a mobile robot, characterized in that, The method includes: The inverse perspective mapping image is acquired, semantic point cloud extraction is performed on the inverse perspective mapping image to obtain a point cloud set, and the point cloud set is clustered to obtain multiple point cloud clusters; Extract the geometric features corresponding to each point cloud cluster to obtain the geometric feature values ​​corresponding to each point cloud cluster; Detect whether the geometric feature values ​​meet preset geometric conditions, and filter point cloud clusters whose geometric feature values ​​meet the preset geometric conditions as usable semantic point clouds; The mobile robot is relocalized based on the available semantic point cloud. If the relocalization fails, other relocalization strategies are adopted for relocalization. These other relocalization strategies are based on environmental perception data collected by other types of sensors.

2. The method according to claim 1, characterized in that, The step of extracting the geometric features corresponding to each point cloud cluster to obtain the geometric feature values ​​corresponding to each point cloud cluster includes: Principal component analysis was performed on each point cloud cluster to obtain the principal directional eigenvalues ​​and secondary directional eigenvalues ​​of each point cloud cluster. The step of detecting whether the geometric feature values ​​satisfy preset geometric conditions and filtering point cloud clusters whose geometric feature values ​​satisfy the preset geometric conditions as usable semantic point clouds includes: If the magnitudes of the primary direction feature value and the secondary direction feature value are both greater than a preset feature value threshold, then the point cloud clusters corresponding to the geometric feature values ​​are selected as usable semantic point clouds. And / or, detect whether the ratio between the main direction feature value and the secondary direction feature value is greater than a preset ratio threshold; if so, filter the point cloud cluster corresponding to the geometric feature value as a usable semantic point cloud. And / or, detect whether the difference between the primary direction feature value and the secondary direction feature value is greater than a preset difference threshold. If so, filter the point cloud cluster corresponding to the geometric feature value as a usable semantic point cloud.

3. The method according to claim 1, characterized in that, The relocation strategies include QR code relocation strategies, visual semantic relocation strategies, and laser relocation strategies; the method further includes: The QR code relocation strategy is executed, which relocates the QR code based on the QR code image captured by the downward-facing camera. If the QR code relocation strategy fails to relocate, the visual semantic relocation strategy is executed based on the inverse perspective mapping image acquired by the surround-view camera. If relocation fails, other relocation strategies will be used for relocation. These other relocation strategies are based on environmental perception data collected by other types of sensors, including: If no usable semantic point cloud is found in the inverse perspective mapping image, or if no valid pose is calculated based on the usable semantic point cloud, the visual semantic relocalization strategy fails to relocalize, and the laser relocalization strategy is executed based on the laser point cloud acquired by the lidar.

4. The method according to claim 1, characterized in that, After repositioning the mobile robot, the method further includes: Multiple navigation and positioning strategies are acquired, and different navigation and positioning strategies perform navigation and positioning based on environmental perception data collected by different sensors. Simultaneously execute multiple navigation and positioning strategies, and fuse the positioning results of the multiple navigation and positioning strategies to obtain the real-time pose of the mobile robot.

5. The method according to claim 4, characterized in that, The navigation and positioning strategy includes a visual positioning strategy, an odometer positioning strategy, and a laser positioning strategy. The visual positioning strategy is based on visual images captured by a camera for navigation and positioning. The odometer positioning strategy is based on odometer information for navigation and positioning. The laser positioning strategy is based on point cloud data collected by a lidar for navigation and positioning. The real-time pose of the mobile robot is obtained by fusing the positioning results of multiple navigation and positioning strategies, including: Detect whether the visual positioning strategy yields a positioning result whose accuracy meets a preset accuracy condition; If the visual positioning strategy obtains a positioning result whose accuracy meets the preset accuracy condition, then the positioning result of the laser positioning strategy is ignored, and the positioning results of the visual positioning strategy and the odometry positioning strategy are fused to obtain the real-time pose of the mobile robot. If the visual positioning strategy fails to obtain a positioning result that meets the preset accuracy condition within a consecutive preset time period, the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot.

6. The method according to claim 5, characterized in that, The step of fusing the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy to obtain the real-time pose of the mobile robot includes: Detect whether the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold; If the accuracy of the positioning result of the visual positioning strategy is lower than the minimum accuracy threshold, the positioning results of the odometry positioning strategy and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot. Otherwise, the positioning results of the visual positioning strategy, the odometry positioning strategy, and the laser positioning strategy are fused to obtain the real-time pose of the mobile robot.

7. The method according to claim 5, characterized in that, The visual positioning strategy includes a QR code positioning strategy and / or a visual semantic positioning strategy. The QR code positioning strategy performs navigation positioning based on the QR code image captured by the downward-looking camera, and the visual semantic repositioning strategy performs navigation positioning based on the inverse perspective mapping image captured by the surround-view camera. The step of fusing the positioning results of the visual positioning strategy and the odometry positioning strategy to obtain the real-time pose of the mobile robot includes: In response to the downward-facing camera capturing a QR code image, it is detected whether the timestamp of the QR code image capture is later than the real-time pose obtained by fusion at the previous moment; if so, the positioning result corresponding to the QR code image is fused with the positioning result of the odometry positioning strategy to obtain the real-time pose of the mobile robot; if not, the positioning result corresponding to the QR code image is ignored. And / or, in response to the surround-view camera acquiring an inverse perspective mapping image, the system acquires the real-time pose obtained from the previous moment fusion, the odometry information from the previous moment to the current moment, and the pre-constructed semantic map; calculates the current predicted pose based on the real-time pose obtained from the previous moment fusion and the odometry information; extracts the semantic point cloud contained in the inverse perspective mapping image, matches the semantic point cloud and the semantic map based on the current predicted pose to obtain a semantically matched pose; and fuses the semantically matched pose with the current predicted pose to obtain the real-time pose of the mobile robot.

8. The method according to claim 7, characterized in that, The step of matching the semantic point cloud and the semantic map based on the current predicted pose to obtain a semantic matching pose includes: Obtain the semantic probability raster map obtained after rasterizing the semantic map, and project the semantic point cloud onto the semantic probability raster map based on the current predicted pose to obtain the projection points; Obtain the probability value corresponding to the projection point in the semantic probability grid map. If the probability value is greater than a preset probability threshold, then the projection point is regarded as a valid semantic point. The effective semantic points are matched with the point cloud in the semantic map to obtain the initial matching pose; Based on the initial matching pose, the semantic point cloud is projected onto the semantic probability grid map, and the accuracy corresponding to the initial matching pose is calculated based on the probability value of each projection point corresponding to the semantic point cloud in the semantic probability grid map. If the accuracy is greater than a preset accuracy threshold, then the initial matching pose is used as the semantic matching pose.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images