Positioning method based on visual semantics, mobile robot and storage medium

Abstract

The invention discloses a positioning method based on visual semantics, a mobile robot and a storage medium, and the method comprises the steps: extracting semantic point clouds of a positioning identifier in a current visual image, carrying out the clustering of the semantic point clouds, obtaining a point cloud cluster, extracting point clouds in a plurality of local regions in the point cloud cluster, and obtaining a plurality of local point cloud sets; calculating a global direction based on all point clouds in the point cloud cluster; calculating a local direction based on the point cloud in the local point cloud set; if the included angle between the local direction and the global direction is smaller than a preset threshold value, linear equations are constructed based on the local point cloud set, and the vertex point cloud of the positioning identifier is determined by using the intersection points among the multiple linear equations; and matching the current vertex point cloud with the vertex point cloud in the semantic map to calculate the pose of the mobile robot. According to the method, the more accurate local point cloud set can be selected to construct the linear equation, and the pose calculation is performed by taking the vertex as the main part, so that the anti-interference capability is stronger, and the calculation amount is reduced on the premise of ensuring the accuracy of the pose calculation.

Description

Technical Field

[0001] This application relates to the field of mobile robot navigation technology, and in particular to a visual semantic-based localization method, a mobile robot, and a storage medium. Background Technology

[0002] Currently, indoor navigation for ground mobile robots is mainly divided into active navigation such as laser navigation and visual navigation, and passive navigation using base station networking technologies such as Wireless Fidelity (WIFI), Ultra Wide Band (UWB), and Bluetooth.

[0003] Among them, wireless network communication technology navigation has significant limitations and poor positioning accuracy; single-line laser navigation is highly dependent on the environment, while multi-line laser navigation requires processing a large amount of point cloud data, resulting in a large amount of computation, and laser radar is expensive; while visual navigation is cheaper than laser navigation and has high navigation accuracy, but if visual navigation accuracy is to be guaranteed, there must be sufficient visual markers in the environment (such as QR codes and textures), and a large amount of computing resources are still required to analyze the images, resulting in high computational complexity. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one location method based on visual semantics, a mobile robot, and a storage medium.

[0005] The first aspect of this application provides a localization method based on visual semantics. The method includes: acquiring a pre-constructed semantic map, and acquiring a visual image of the scene to be localized captured by a mobile robot at the current moment; wherein, a localization marker is set in the scene to be localized, and the semantic map contains vertex point clouds of the localization marker; extracting semantic information of the localization marker from the visual image to obtain a semantic point cloud, clustering the semantic point cloud to obtain a point cloud cluster, and extracting point clouds from multiple local regions in the point cloud cluster to obtain multiple local point cloud sets; determining the direction of the localization marker based on the distribution characteristics of all point clouds in the point cloud cluster to obtain a global direction; and determining the direction of the localization marker based on the distribution characteristics of the point clouds in the multiple local point cloud sets to obtain multiple local directions; if the angle between the local direction and the global direction is less than a preset threshold, constructing a straight line equation based on the local point cloud set, and using the intersection points between the multiple straight line equations to determine the vertex of the localization marker to obtain the vertex point cloud corresponding to the current moment; matching the vertex point cloud corresponding to the current moment with the vertex point cloud in the semantic map, and calculating the pose of the mobile robot based on the successfully matched vertex point cloud.

[0006] In one embodiment, the vertex of the positioning identifier is determined by the intersection of multiple linear equations to obtain the vertex point cloud corresponding to the current time. This includes: calculating the included angle between each linear equation; if the included angle between two linear equations satisfies the included angle threshold condition corresponding to the positioning identifier, then the intersection of the two linear equations is calculated to obtain candidate intersection points; traversing each candidate intersection point, if the distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier, and / or the proportion of pixels belonging to the positioning identifier between the two candidate intersection points satisfies the proportion condition corresponding to the positioning identifier, then the two candidate intersection points are taken as vertices to obtain the vertex point cloud corresponding to the current time.

[0007] In one embodiment, determining whether the distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier includes: obtaining the length of the straight line segment of the positioning identifier; calculating the distance between the two candidate intersection points; and if the difference between the distance and the length of the straight line segment of the positioning identifier is less than a preset distance difference threshold, then the two candidate intersection points are determined to satisfy the distance condition corresponding to the positioning identifier.

[0008] In one embodiment, the visual image is a top view; determining whether the proportion of pixels belonging to the positioning identifier between two candidate intersections meets the proportion condition corresponding to the positioning identifier includes: transforming the two candidate intersections in reverse perspective to the bottom of the top view, traversing the pixels of the straight line between the two candidate intersections; determining whether the traversed pixels belong to the positioning identifier, calculating the proportion of pixels belonging to the positioning identifier, and obtaining a first proportion; if the first proportion is greater than a preset first threshold, it is determined that the two candidate intersections meet the proportion condition corresponding to the positioning identifier.

[0009] In one embodiment, the visual image is a top view; the method further includes: transforming the two determined vertices in reverse perspective to the top view, obtaining the normal vector of the line where the two determined vertices are located, and traversing a preset number of pixels along the normal vector; determining whether the traversed pixels belong to the positioning identifier, and calculating the proportion of pixels belonging to the positioning identifier to obtain a second proportion; if the second proportion is greater than a preset second threshold, then determining other vertices based on the normal vector and the two determined vertices.

[0010] In one embodiment, the mobile robot is equipped with multiple vision cameras; acquiring the visual image of the scene to be located by the mobile robot at the current moment includes: acquiring the image of the scene to be located by each vision camera to obtain multiple initial images; transforming each initial image inverse perspective to the top view coordinate system to obtain a top view; and using the top view as the visual image corresponding to the scene to be located at the current moment.

[0011] In one embodiment, constructing a straight line equation based on a local point cloud set includes: projecting points in the local point cloud set onto a local direction, determining the center point of the local point cloud set based on the projection result, and constructing a straight line equation through the center point using the global direction as the straight line direction vector.

[0012] In one embodiment, the semantic map also contains semantic point clouds containing other types of semantic information such as location identifiers, and semantic point clouds containing other types of semantic information are extracted from the visual image; the pose of the mobile robot is calculated based on the successfully matched vertex point clouds, including: matching semantic point clouds of the same type between the visual image and the semantic map to obtain other types of semantic point clouds that are successfully matched; setting weight parameters based on the number of successfully matched vertex point clouds; constructing a weighted residual equation based on the weight parameters, the successfully matched vertex point clouds, and the other types of semantic point clouds that are successfully matched; and solving the weighted residual equation to obtain the pose of the mobile robot.

[0013] The second aspect of this application provides a positioning device based on visual semantics. The device includes: a data acquisition module for acquiring a pre-constructed semantic map and a visual image of the scene to be positioned captured by a mobile robot at the current moment; wherein, positioning markers are set in the scene to be positioned, and the semantic map contains vertex point clouds of the positioning markers; a semantic extraction module for extracting semantic information of the positioning markers from the visual image to obtain semantic point clouds, clustering the semantic point clouds to obtain point cloud clusters, and extracting point clouds from multiple local regions in the point cloud clusters to obtain multiple local point cloud sets; a direction calculation module for determining the direction of the positioning markers based on the distribution characteristics of all point clouds in the point cloud clusters to obtain a global direction; and determining the direction of the positioning markers based on the distribution characteristics of the point clouds in the multiple local point cloud sets to obtain multiple local directions; a vertex determination module for constructing a straight line equation based on the local point cloud set if the angle between the local direction and the global direction is less than a preset threshold, and determining the vertex of the positioning markers using the intersection points between the multiple straight line equations to obtain the vertex point cloud corresponding to the current moment; and a pose calculation module for matching the vertex point cloud corresponding to the current moment with the vertex point clouds in the semantic map, and calculating the pose of the mobile robot based on the successfully matched vertex point clouds.

[0014] A third aspect of this application provides a mobile robot, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the aforementioned visual semantic-based localization method.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the aforementioned location method based on visual semantics.

[0016] The above scheme obtains semantic point clouds by extracting semantic information of the positioning markers from the visual image acquired at the current moment. These semantic point clouds are then clustered to form point cloud clusters. Point clouds within multiple local regions of each point cloud cluster are extracted to obtain multiple local point cloud sets. The direction of the positioning markers is determined based on the distribution characteristics of all points in the point cloud clusters, yielding the global direction. The direction of the positioning markers is also determined based on the distribution characteristics of the points in each of the multiple local point cloud sets, yielding multiple local directions. If the angle between the local direction and the global direction is less than a preset threshold, a straight line equation is constructed based on the local point cloud sets. The intersection points of these equations are used to determine the vertices of the positioning markers, thus obtaining the vertex points corresponding to the current moment. The algorithm matches the vertex point cloud corresponding to the current moment with the vertex point cloud in the semantic map. Based on the successfully matched vertex point cloud, the pose of the mobile robot is calculated. Since the local orientation of the local point cloud set is verified based on the global orientation, a more accurate local point cloud set can be selected to construct the straight line equation, avoiding interference from other non-localization marker straight line features of the point cloud. Furthermore, the pose calculation is based on vertices, which can reflect the appearance features of the localization marker and are not affected by image downsampling, image quality, etc., thus having stronger anti-interference ability and improving the accuracy of pose calculation. In addition, the overall algorithm does not involve complex image analysis, has low computational load, and high localization efficiency.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a visual semantic-based localization method as shown in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the IPM image coordinate system and the vehicle body coordinate system in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the extraction of semantic information from a top view, as shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating point cloud matching in an exemplary embodiment of this application; Figure 6 This is a block diagram illustrating a visual semantic-based positioning device in an exemplary embodiment of this application; Figure 7This is a schematic diagram of the structure of a mobile robot shown in an exemplary embodiment of this application; Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] The following describes the location method based on visual semantics provided in the embodiments of this application.

[0024] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of an exemplary embodiment of this application. The implementation environment may include a mobile robot 110 and a server 120, which are interconnected.

[0025] The mobile robot 110 may be an automated guided vehicle (AGV), a sweeping robot, an industrial robot, etc., and this application does not limit it to this.

[0026] The mobile robot 110 is equipped with a vision camera to capture visual images of the scene to be located. The number of vision cameras can be one or more, depending on the specific application scenario.

[0027] Optionally, the mobile robot 110 is also equipped with environmental perception devices such as LiDAR and depth cameras to collect other types of environmental perception data.

[0028] Of course, the mobile robot 110 may also include a mobile chassis, which includes a motion controller, motor, battery, embedded computer, odometer, gyroscope, etc. This application does not limit this.

[0029] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0030] In one example, server 120 can calculate the pose of mobile robot 110 based on the visual image obtained from mobile robot 110, and server 120 can generate navigation instructions for mobile robot 110 based on the pose of mobile robot 110 so that mobile robot 110 can perform navigation movement.

[0031] In one example, a client application for a target application is installed and running on the mobile robot 110. This target application may be an application that provides positioning and navigation functions. Using the target application, the pose of the mobile robot 110 is calculated based on the acquired visual images, and the mobile robot 110 navigates and moves according to its pose. The server 120 may be a backend server for the target application, used to provide background services to the client of the target application.

[0032] The mobile robot positioning method provided in this application embodiment can be executed by a mobile robot 110, such as the client of the target application installed and running in the mobile robot 110, or by a server 120, or by the interaction and cooperation between the mobile robot 110 and the server 120, that is, some steps of the method are executed by the mobile robot 110 and other steps are executed by the server 120.

[0033] It should be noted that the mobile robot positioning method of this application can be applied to any navigation scenario, such as the shelf docking scenario of industrial robots and the floor cleaning scenario of sweeping robots, and this application does not limit it.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating a visual semantic-based localization method as shown in an exemplary embodiment of this application, such as... Figure 2 As shown, the localization method based on visual semantics includes at least steps S210 to S250, which are described in detail below: Step S210: Obtain a pre-built semantic map and a visual image of the scene to be localized by the mobile robot at the current moment; wherein, the scene to be localized is equipped with localization markers and the semantic map contains vertex point clouds of localization markers.

[0035] In the scene to be located, the location of the corresponding scene area is marked by setting a location identifier.

[0036] For example, location markers include warehouse location marks, directional arrow marks, etc. Location markers can be set on the ground or on a wall, and this application does not limit them.

[0037] A semantic map is pre-constructed for the scene to be located, and the semantic map stores the vertex point cloud of the location markers in the scene. Of course, in addition to storing the vertex point cloud of the location markers, other types of point clouds that can reflect the appearance features of the location markers can also be stored. For example, the point cloud corresponding to the lines of the location markers can also be stored. This application does not limit this.

[0038] The semantic map construction process is illustrated below: Visual images periodically collected as the mobile robot moves within the scene to be localized are acquired to obtain key image frames. The robot's pose is estimated using the matching results of wheel odometry and / or laser point cloud frames with the laser map, and the corresponding pose is labeled for each key image frame. Semantic information of the localization markers in the key image frames is extracted to obtain various types of semantic point clouds, and the vertex point clouds of the localization markers are determined from the extracted semantic point clouds. The specific steps for extracting semantic information and determining vertex point clouds can be found in the following embodiments. Then, a semantic map is constructed based on the pose, vertex point clouds, and / or other types of semantic point clouds for each key image frame.

[0039] During the localization process, the visual images of the scene to be localized by the mobile robot at the current moment are acquired. Subsequently, based on the visual images acquired at the current moment and the pre-built semantic map, the pose of the mobile robot at the current moment is calculated.

[0040] Step S220: Extract semantic information of localization markers in the visual image to obtain semantic point cloud, cluster the semantic point cloud to obtain point cloud cluster, and extract point clouds from multiple local regions in the point cloud cluster to obtain multiple local point cloud sets.

[0041] First, semantic information of the location markers is extracted from the visual images acquired at the current moment to obtain a semantic point cloud.

[0042] Specifically, a pre-trained semantic extraction network is used to extract semantics from visual images, outputting semantic point clouds corresponding to localization labels. The semantic extraction network can be implemented based on network structures such as BiSeNet (Bilateral Segmentation Network), and / or PSPNet (Pyramid Scene Parsing Network), and / or Convolutional Neural Network (CNN), and this application does not impose any limitations on this.

[0043] Then, the semantic point clouds are clustered to obtain one or more point cloud clusters. The clustering can be based on Euclidean distance, instance segmentation, or region growing algorithms; this application does not limit the specific clustering method.

[0044] Optionally, if the number of points in a point cloud cluster is less than a preset threshold (e.g., 50), then the point cloud cluster is ignored.

[0045] Furthermore, point clouds within multiple local regions of a point cloud cluster are extracted to obtain multiple local point cloud sets, where point clouds within the same local region are relatively close to each other. This can be achieved by calculating the distance between point clouds within the cluster and grouping point clouds with distances less than a preset distance threshold into the same local region, thus obtaining a local point cloud set; alternatively, multiple local regions can be divided first, and point clouds within the same local region can be identified to obtain a local point cloud set. This application does not limit the specific method of dividing the local point cloud set.

[0046] Step S230: Determine the direction of the positioning marker based on the distribution characteristics of all point clouds in the point cloud cluster to obtain the global direction; and determine the direction of the positioning marker based on the distribution characteristics of point clouds in multiple local point cloud sets to obtain multiple local directions.

[0047] Each point cloud cluster is traversed. For the currently traversed point cloud cluster, the distribution characteristics of all points in the cluster are extracted, and the direction of the positioning marker is determined based on the distribution characteristics of all points in the cluster.

[0048] In addition, multiple local point cloud sets corresponding to the point cloud cluster are obtained, and the directions of the positioning markers are determined based on the distribution characteristics of the point clouds in the multiple local point cloud sets, thus obtaining multiple local directions.

[0049] For example, the direction of the localization marker can be obtained by analyzing the distribution characteristics of the point cloud using the Principal Components Analysis (PCA) algorithm. Of course, in addition to the PCA algorithm, the direction of the localization marker can also be determined by the Minimum Bounding Rectangle (MBR), the Hough Transform, or deep learning-based methods, etc. This application does not limit the specific methods used.

[0050] Step S240: If the angle between the local direction and the global direction is less than a preset threshold, then construct a straight line equation based on the local point cloud set, use multiple straight line equations to determine the vertex of the positioning marker, and obtain the vertex point cloud corresponding to the current time.

[0051] If the angle between the local direction and the global direction is less than a preset threshold, it is determined that the direction belongs to the location identifier. A straight line equation is constructed based on the local point cloud set. Multiple straight line equations can be constructed from multiple local point cloud sets that meet the above conditions.

[0052] Determine the intersection points between multiple linear equations, and then determine the vertices of the positioning markers based on these intersection points to obtain the vertex point cloud corresponding to the current time.

[0053] For example, the intersection points between line equations can be directly used as vertices; alternatively, the intersection points between line equations can be verified, and the verified intersection points can be used as vertices. This can be done by verifying the intersection points between line equations based on the angle information between each line equation, and / or the positional relationship between the intersection points, and / or the image features corresponding to the intersection points in the visual image, thereby determining the vertices belonging to the positioning markers and obtaining the vertex point cloud corresponding to the positioning markers perceived by the mobile robot at the current moment.

[0054] Step S250: Match the vertex point cloud corresponding to the current moment with the vertex point cloud in the semantic map, and calculate the pose of the mobile robot based on the successfully matched vertex point cloud.

[0055] After obtaining the vertex point cloud corresponding to the current moment, it is matched with the vertex point cloud in the semantic map. Based on the matching result, the relative pose relationship between the two is determined, and the pose of the mobile robot at the current moment is calculated based on the relative pose relationship.

[0056] The visual semantic-based localization method provided in this application obtains a semantic point cloud by extracting semantic information of the localization marker in the visual image acquired at the current time. The semantic point cloud is then clustered to obtain a point cloud cluster. Point clouds within multiple local regions of each point cloud cluster are extracted to obtain multiple local point cloud sets. The direction of the localization marker is determined based on the distribution characteristics of all points in the point cloud cluster, resulting in a global direction. Furthermore, the direction of the localization marker is determined based on the distribution characteristics of the points in each of the multiple local point cloud sets, resulting in multiple local directions. If the angle between the local direction and the global direction is less than a preset threshold, a linear equation is constructed based on the local point cloud sets. The intersection points of these linear equations are used to determine the vertices of the localization marker, resulting in the current direction. The algorithm calculates the pose of the mobile robot based on the vertex point cloud corresponding to the previous time step. Since the local orientation of the local point cloud set is verified based on the global orientation, a more accurate local point cloud set can be selected to construct the straight line equation, avoiding interference from other non-localization marker straight line features of the point cloud. Furthermore, the pose calculation is based on vertices, which can reflect the appearance features of the localization marker and are not affected by image downsampling, image quality, etc., thus having stronger anti-interference ability and improving the accuracy of pose calculation. In addition, the overall algorithm does not involve complex image analysis, has low computational load, and high localization efficiency.

[0057] The following describes some embodiments of this application in detail.

[0058] In some implementations, semantic maps can be constructed and localization calculations can be performed based on images directly acquired by a vision camera. Alternatively, an inverse perspective transformation can be performed on the images acquired by the vision camera to obtain a top view, and semantic maps can be constructed and localization calculations can be performed based on the top view to eliminate the interference caused by perspective distortion to image processing.

[0059] For example, a mobile robot is equipped with multiple vision cameras. When performing real-time localization of the mobile robot, the images captured by each vision camera of the scene to be localized are obtained to form an initial image. Each initial image is then subjected to inverse perspective transformation to obtain a top view. The top view is then used as the visual image corresponding to the scene to be localized at the current moment.

[0060] The process of performing inverse perspective transformation on the acquired image is explained in detail: Each vision camera simultaneously captures multiple images, which are then projected into an inverse perspective mapping image (IPM) to obtain a top view.

[0061] Specifically, let the IPM image coordinate system be {IPM}, the mobile robot's body coordinate system be {b}, and the camera coordinate system be {c}. The coordinate system of a pixel in the IPM image after transforming from the mobile robot's camera coordinate system is: Where U represents the column number of the pixel in the IPM image, V represents the row number of the pixel in the IPM image, and the scale of each pixel in the IPM is r. For a schematic diagram of the IPM image coordinate system and the vehicle body coordinate system, please refer to... Figure 3 The transformation relationship between pixel coordinates of the IPM image and coordinates in the vehicle coordinate system is shown in Formula 1 below: (Formula 1) in, This represents the pixel's coordinates in the vehicle coordinate system; This indicates the relative pose between the pre-calibrated IPM image coordinate system and the vehicle body coordinate system.

[0062] The coordinate transformation relationship from the vehicle coordinate system to the camera coordinate system is shown in Formula 2 below: (Formula 2) in, This represents the coordinates of a pixel in the camera coordinate system of the i-th visual camera; K represents the pre-calibrated extrinsic parameters of the i-th visual camera; K represents the pre-calibrated intrinsic parameters of the visual camera.

[0063] Using Formulas 1 and 2 above, each pixel in the IPM image coordinate system can be found to have pixel coordinates in the camera coordinate system corresponding to different visual cameras.

[0064] In some implementations, during the semantic map construction and localization calculation process, if formula 2 is used... If the z-axis of the calculated pixel in the camera coordinate system is less than zero (i.e., behind the camera coordinate system), then the pixel is ignored. Otherwise, the average of the corresponding pixel values ​​at the same pixel position under multiple visual cameras is used as the pixel value at the corresponding pixel position in the IPM image to generate the IPM image and obtain the top view.

[0065] Then, the top view can be used as the visual image acquired at that moment, and semantic information can be extracted from the visual image.

[0066] For examples, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the extraction of semantic information from an IPM image, as shown in an exemplary embodiment of this application. Figure 4As shown, BiSeNet is used to extract location-based semantic information from IPM images. Of course, other types of semantic information can also be extracted, such as the semantic information of arrows indicating road directions. Different semantic information can be represented using different RGB values, such as... Figure 4 The semantic information of the storage location identifier is represented by red pixels.

[0067] When performing real-time localization of a mobile robot, after extracting the semantic point cloud corresponding to the visual image collected at the current moment, the linear features of the localization marker are obtained based on the semantic point cloud, and the vertices of the localization marker are obtained based on the linear features.

[0068] In some implementations, step S240 involves constructing a straight line equation based on a local point cloud set, including steps S2411 to S2412.

[0069] Step S2411: Project the points in the local point cloud set onto the local direction to obtain the center point of the local point cloud set.

[0070] By projecting points onto a local direction, the multidimensional point cloud is reduced to a one-dimensional line, but the distribution information in the local direction is preserved. The center point of the local point cloud set can be calculated using the four maximum and minimum values ​​after projection, or the center point of the local point cloud set can be obtained by calculating the average of all projected values.

[0071] Step S2412: Use the global direction as the line direction vector and construct the line equation through the center point.

[0072] Then, using the global direction as the straight line direction vector, construct the straight line equation corresponding to the local point cloud set through the center point. This straight line equation represents the overall direction of the point cloud.

[0073] In addition to constructing the linear equation based on the global direction and the projection center point, we can also use the average value of all point coordinates as the center point of the local point cloud set, or calculate the maximum and minimum values ​​of the point cloud in each direction to obtain a bounding box, and use the geometric center of the bounding box as the center point of the local point cloud set. Furthermore, we can also use the local direction as the direction vector of the line to construct the linear equation corresponding to the local point cloud set through the center point.

[0074] Of course, in addition to the point-to-point method of constructing the line equation as shown in the example above, other methods can be used to construct the line equation. For example, the coordinates of each point in the local point cloud set can be combined to perform line fitting to construct the line equation. This application does not limit the specific method of constructing the line equation.

[0075] Traverse other local point cloud sets, and construct multiple line equations for each local point cloud set whose angle between the local direction and the global direction is less than a preset threshold. Then, determine the vertex of the positioning marker based on the multiple line equations to determine the vertex point cloud corresponding to the current time.

[0076] In some implementations, step S240 uses multiple straight line equations to determine the vertices of the positioning markers and obtains the vertex point cloud corresponding to the current time, including steps S2421 to S2423.

[0077] Step S2421: Calculate the angle between the equations of each line.

[0078] Step S2422: If the included angle between the two line equations satisfies the included angle threshold condition corresponding to the positioning mark, then calculate the intersection point between the two line equations to obtain the candidate intersection point.

[0079] Different positioning identifiers correspond to different included angle threshold conditions, and the included angle threshold conditions are set according to the specific appearance of the positioning identifier.

[0080] For example, if the location marker is composed of multiple vertical lines, it is necessary to determine whether the angle between two line equations is close to 90°. If so, it indicates that the angle between the two line equations meets the angle threshold condition corresponding to the location marker. At this time, the intersection point between the two line equations is further calculated to obtain the candidate intersection point.

[0081] By iterating through all the line equations, the intersection points between the line equations that satisfy the included angle threshold condition are selected, resulting in multiple candidate intersection points.

[0082] Step S2423: Traverse each candidate intersection point. If the distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier, and / or the proportion of pixels belonging to the positioning identifier between the two candidate intersection points satisfies the proportion condition corresponding to the positioning identifier, then take the two candidate intersection points as vertices to obtain the vertex point cloud corresponding to the current time.

[0083] For the two candidate intersection points currently being traversed, determine whether the candidate intersection point is a vertex based on one or more of the following criteria: Condition 1: The distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier; Condition 2: The proportion of pixels belonging to the positioning identifier between two candidate intersection points satisfies the proportion condition corresponding to the positioning identifier.

[0084] The judgment process for the above judgment conditions will be explained in detail: Condition 1: Determine whether the distance between two candidate intersection points meets the distance condition corresponding to the positioning identifier, including: obtaining the straight line segment length of the positioning identifier; calculating the distance between the two candidate intersection points. If the difference between the distance and the straight line segment length of the positioning identifier is less than the preset distance difference threshold, then the two candidate intersection points are determined to meet the distance condition corresponding to the positioning identifier.

[0085] Different types of positioning markers require different lengths of straight line segments for conditional judgments.

[0086] Taking the storage location marker as a positioning marker as an example, when the mobile robot docks with the storage location, the ground is marked with the storage location marker. The length of the storage location marker is L and the width is W. If the difference between the distance between two candidate intersection points and L or W is less than the preset distance difference threshold (such as 5cm), it is determined that the two candidate intersection points meet condition one.

[0087] Condition 2: Determine whether the proportion of pixels belonging to the positioning identifier between two candidate intersections meets the proportion condition corresponding to the positioning identifier, including: the visual image is a top view, the two candidate intersections are transformed to the bottom of the top view through inverse perspective, and the pixels of the straight line between the two candidate intersections are traversed; it is determined whether the traversed pixels belong to the positioning identifier, and the proportion of pixels belonging to the positioning identifier is calculated to obtain the first proportion; if the first proportion is greater than the preset first threshold, it is determined that the two candidate intersections meet the proportion condition corresponding to the positioning identifier.

[0088] Transform the candidate intersection points into the top view through inverse perspective. Since semantic recognition has been performed on the top view, pixels in the top view are marked as belonging to the positioning identifier. The pixels of the line containing the two candidate intersection points are traversed sequentially, and it is determined whether the traversed pixels belong to the positioning identifier. The proportion of pixels belonging to the positioning identifier between the two candidate intersection points to the total number of pixels between the two candidate intersection points is calculated to obtain the first proportion. If the first proportion is greater than the preset first threshold (e.g., 40%), it is determined that the two candidate intersection points satisfy condition two.

[0089] The proportion of semantic pixels between vertices is used to confirm whether they are real vertex features, thereby improving the accuracy of the obtained vertices.

[0090] If two candidate intersection points satisfy condition one and / or condition two above, then these two candidate intersection points are taken as vertices to obtain the vertex point cloud corresponding to the current time.

[0091] The number of judgment conditions used to determine the vertex can be flexibly selected according to the actual application. For example, when detecting the positioning accuracy of the current scene, if the current scene is a high-precision positioning scene such as device docking or cargo picking and placing, more judgment conditions are used; if the current scene is a low-precision positioning scene or there are few available computing resources, fewer judgment conditions are used.

[0092] Of course, in addition to the judgment conditions shown in the above embodiments, other judgment conditions can also be used, and the specific conditions can be flexibly set according to the appearance characteristics of the positioning mark. This application does not limit them.

[0093] In addition, other vertices can be deduced based on the already determined vertices, and the specific deduction method is determined according to the appearance of the positioning marker.

[0094] For example, if the visual image is a top view, the two determined vertices are transformed in reverse perspective to the bottom of the top view, the normal vectors of the lines containing the two determined vertices are obtained, and a preset number of pixels are traversed along the normal vectors; it is determined whether the traversed pixels belong to the positioning markers, and the proportion of pixels belonging to the positioning markers is counted to obtain the second proportion; if the second proportion is greater than the preset second threshold, other vertices are determined based on the normal vectors and the two determined vertices.

[0095] Taking a storage location marker as an example, a storage location marker consists of multiple edges. Let p1 and p2 be the two vertices of one of the shorter edges of an identified storage location marker, and let n be the normal vector of the line segment connecting p1 and p2. In the IPM image coordinate system, if we start from p1 or p2 and walk L / r pixels along n, and if the number of pixels belonging to the storage location marker is greater than 40% of the total number of pixels on the longer edge (i.e., L / r), then the other vertices are determined based on the normal vector n and the coordinates of the two already determined vertices. Here, L is the length of the longer edge of the storage location marker, and r is the scale of each pixel in the IPM image coordinate system.

[0096] Then, the pose of the mobile robot is determined based on the vertex point cloud corresponding to each vertex at the current moment and the vertex point cloud in the semantic map.

[0097] Specifically, the vertex point cloud corresponding to each vertex determined at the current moment and the vertex point cloud in the semantic map are both transformed to the world coordinate system. The vertex point cloud at the current moment is traversed to determine the nearest neighbor point in the semantic map. If the distance between the nearest neighbor point and the current traversed point is less than a preset distance threshold (such as 10cm), the match is considered successful. Otherwise, the current traversed point is skipped and the next point is traversed.

[0098] The pose of the mobile robot is calculated by combining the poses of all successfully matched vertex points in the cloud, for example, based on the poses of the vertex point cloud in the vehicle coordinate system at the current moment and the poses of the vertex point cloud in the semantic map.

[0099] Of course, in addition to vertex point clouds, semantic maps can also contain semantic point clouds containing other types of semantic information such as location identifiers. Semantic point clouds containing other types of semantic information can be extracted from visual images, such as line point clouds, arrow point clouds, etc. Step S250 calculates the pose of the mobile robot based on the successfully matched vertex point clouds, including steps S251 to S253.

[0100] Step S251: Match the semantic point clouds of the same type between the visual image and the semantic map to obtain other types of semantic point clouds that have been successfully matched.

[0101] For example, the line point cloud extracted from the visual image is matched with the line point cloud in the semantic map, and the arrow point cloud extracted from the visual image is matched with the arrow point cloud in the semantic map. The matching steps are similar to the vertex point cloud matching steps, and will not be described in detail here.

[0102] Step S252: Set weight parameters based on the number of successfully matched vertex point clouds, and construct a weighted residual equation based on the weight parameters, successfully matched vertex point clouds, and other types of successfully matched semantic point clouds.

[0103] Let the current pose of the mobile robot be T. Calculate the pose difference between the semantic point cloud extracted from the visual image at the current moment and the corresponding semantic point cloud in the semantic map, based on the pose T, for each type of semantic point cloud that is successfully matched. Obtain the residual equation and set the weight parameters corresponding to each type of semantic point cloud to obtain the weighted residual equation.

[0104] The weight parameters of the residual equation are set based on the number of successfully matched vertex point clouds. The more successfully matched vertex point clouds there are, the greater the weight of the residual terms corresponding to the vertex point clouds in the weighted residual equation, so that the pose of the mobile robot is calculated mainly based on the vertex point clouds.

[0105] By using different fusion weights when calculating the matching results, and increasing the residual weight of the vertex semantic information when a sufficient amount of vertex semantic information is detected, the accuracy of pose estimation is guaranteed.

[0106] Step S253: Based on the weighted residual equation, the pose of the mobile robot is calculated.

[0107] The pose of the mobile robot is calculated by minimizing the value of the weighted residual equation.

[0108] Next, the location identifier will include the storage location identifier (such as...). Figure 4 Using the example shown (and the directional arrow), we will explain in detail a specific application scenario of mobile robot positioning.

[0109] During the mapping process, the pose of the mobile robot is estimated using the matching results of wheel odometry and / or laser point cloud frames with the laser map. The IPM image acquired at time t and the pose of the mobile robot estimated at time t are denoted as follows: Let w represent the world coordinate system. Then, the point cloud coordinates obtained by transforming the semantic information in the IPM image to the world coordinate system are... The semantic point cloud is obtained. For the specific calculation formula, please refer to Formula 3 below: (Formula 3) For the interpretation of each element, please refer to Formula 1 above, which will not be repeated here.

[0110] Then, based on the semantic point cloud, the straight line containing each storage location line of the storage location identifier is obtained and the vertex of the storage location identifier is obtained to construct a semantic map. The specific steps include the following steps a1 to a8.

[0111] Step a1: Semantic point cloud clustering.

[0112] Specifically, the semantic point clouds detected in the acquired visual images are clustered. If the number of points in a point cloud cluster is less than a preset threshold, the point cloud cluster is skipped; otherwise, a k-dimensional tree (kd-tree) is constructed for the points in the point cloud cluster.

[0113] Step a2: Calculate the global direction.

[0114] Specifically, let a point cloud cluster be {Ci}. Perform PCA decomposition on the point cloud cluster {Ci} to obtain two principal directions (denoted as eigenvectors), denoted as... and The two main directions mentioned above are taken as global directions.

[0115] Step a3: Calculate the local orientation.

[0116] Specifically, iterate through the points cij in {Ci}, and using cij as the center, find points within a preset range from cij in the kd-tree of the point cloud cluster {Ci}. Save these points to the set {Sij} to obtain a local point cloud set. The points in the local point cloud set form a small plane, and PCA decomposition is performed on this plane to obtain two principal directions. and The two main directions mentioned above are taken as local directions.

[0117] It should be noted that the goal of PCA is to find a new set of orthogonal bases (i.e., principal directions) that maximizes the variance of the data projection onto these new bases. The PCA decomposition above yields two principal directions: the first principal direction is the direction with the largest data variance, and the second principal direction is the direction orthogonal to the first and has the second largest variance. The number of principal directions is specifically determined based on the appearance of the positioning marker, and this application does not limit this.

[0118] Step a4: Determine the direction of the storage location marker.

[0119] Specifically, if or and or If the included angle between them is less than a preset included angle threshold (e.g., 5°), then it is considered... or It refers to the direction indicated by the storage location marker.

[0120] Step a5: Construct the equation of the straight line.

[0121] Specifically, move the points in {Sij} to... and Perform projection, and use the four maximum and minimum values ​​after projection to calculate the center point zij of the plane corresponding to {Sij}. or Let be the direction vector of the line, and construct the equation of the line lij through the center point zij.

[0122] Then, continue traversing the remaining points cik in the point cloud cluster {Ci}. If a point is in the set {Sij}, skip it directly; otherwise, repeat steps a3 to a5 (the corresponding local point cloud set is {Sik}) to obtain the line equation lik. If the line equation lik and the line equation lij are not parallel, and the distance between them is greater than a preset distance threshold (e.g., 30cm), then the line equation lik and the line equation lij are considered not to be the same reservoir line, and the new line lik is added to the line set {li}; otherwise, skip it directly. Use this method to calculate the line equations of other reservoir lines.

[0123] When solving the equation of a straight line, skipping points that have already been visited speeds up the process of solving individual straight line equations.

[0124] Step a6: Determine candidate intersection points.

[0125] Specifically, iterate through any two line equations in the line set {li}, calculate whether the included angle between the two line equations is close to 90 degrees. If so, calculate the intersection point of the two line equations. The specific calculation method for the intersection point is shown in Formula 4 below: (Formula 4) Here, a1, b1, c1 and a2, b2, c2 are the coefficients of the two line equations, respectively, and the coordinates of the intersection point (x, y) are obtained. All intersection points are added to the candidate intersection point set {Sc}.

[0126] Step a7: Determine the vertices.

[0127] Specifically, the length of the storage location identifier is L and the width is W. Traverse any two candidate intersection points in {Sc}. If the difference between the distance between the two candidate intersection points and W is less than 5cm, then transform the coordinates of the two candidate intersection points to the IPM image coordinate system using the above formula 3. Traverse the pixels of the straight line between the two candidate intersection points and determine whether the pixel is a pixel of the storage location identifier. If the number of pixels belonging to the storage location identifier between the two candidate intersection points accounts for 40% of the total number of pixels between the two candidate intersection points, then the two candidate intersection points are considered to be the two vertices of the short side of the storage location identifier.

[0128] Then, calculate the vertices of the longer side based on the vertices of the shorter side.

[0129] Let the two vertices of a short side be p1=(p1x,p1y) and p2=(p2x,p2y). Then the direction of the line containing the two vertices is (slx,sly)=(p2x-p1x,p2y-p1y) / |len|, where |len| is the length of the line segment formed by p1 and p2. The normal vector of the line formed by p1 and p2 is then n=(-sly,slx). In the IPM image coordinate system, starting from p1 or p2 along n, traverse L / r pixels. If the number of pixels traversed is greater than 40% of the total number of pixels on the long side (L / r), then calculate the long side vertex. Otherwise, continue traversing L / r pixels in the opposite direction of n to check if the above condition is met. The calculation method for long side vertices p3 and p4 is shown in Formula 5 below: (Formula 5) The detected vertices are saved and an index number is added to each vertex. If a vertex is detected in the next frame that is too close to a saved vertex, it is considered to be the same vertex, and the average of the two observed vertices is used as the vertex coordinates of the indexed vertex.

[0130] Interfering storage location lines are eliminated by using the ratio of pixels that satisfy the semantic label between two intersection points. At the same time, the other two vertices of the storage location that meet the conditions are obtained by using the short storage location line and the normal vector of the storage location line, thereby improving the efficiency and accuracy of vertex calculation.

[0131] Step a8: Generate a semantic map.

[0132] Specifically, all vertices are stored in a hash table using semantic identifier 3. After projecting the pixel coordinates of all visual images that are identified as location lines and directional arrows into the world coordinate system to obtain point clouds, the point clouds are downsampled using 2r voxels. Then, semantic identifiers 1 and 2 are stored in the hash table respectively to obtain a semantic map.

[0133] Among them, the point cloud of other types of semantic information can be determined based on the calculated vertices. For example, the storage location vertices can be used to filter the storage location semantic information that meets the appearance requirements of the storage location identifier, thereby generating point clouds of other types of semantic information such as storage location lines and directional arrows. Alternatively, points marked as other types of semantic information during semantic information extraction can be directly used as point clouds of other types of semantic information. This application does not limit this.

[0134] During the localization process, the mobile robot is located based on a semantic map, specifically including the following steps b1 to b4.

[0135] Step b1: Load the semantic map.

[0136] Specifically, the semantic map loads three types of semantic information: storage location lines, directional arrows, and storage location vertices, and constructs three corresponding kd-trees, denoted as KD1, KD2, and KD3, respectively.

[0137] Step b2: Switch to semantic navigation mode and detect the library position vertex.

[0138] Specifically, during warehouse location docking, the system switches to semantic navigation mode, using the pose from the previous moment and the odometer data up to the current moment to predict the pose at the current moment, denoted as... Formula 3 is used to transform the semantic information of the storage location marker and direction arrow detected in the IPM image at the current time to the world coordinate system, and the storage location marker vertex in the IPM image at the current time is calculated using the method of calculating the storage location marker vertex in steps a1 to a7, so as to obtain the vertex point cloud.

[0139] During operation on the main road, the visual camera does not need to extract any additional features and uses laser positioning and / or wheel odometers for positioning. When entering the storage location for docking, it switches to semantic navigation mode to ensure the robustness of positioning during operation on the main road and the accuracy of positioning during docking.

[0140] Step b3: Point cloud matching.

[0141] Specifically, let N1, N2, and N3 be the number of point clouds of the storage location line, the number of point clouds of the directional arrows, and the number of point clouds of the vertex detected by the IPM image at the current moment.

[0142] Traverse the point cloud of the storage line detected by the IPM image at the current time, and search for the nearest neighbor in KD1. If the distance between the nearest neighbor and the nearest neighbor is greater than 10cm, skip it; otherwise, the match is successful. If the number of traversed points is greater than N1, traverse the point cloud of the directional arrows detected by the IPM image at the current time, and search for the nearest neighbor in KD2. If the distance between the nearest neighbor and the nearest neighbor is greater than 10cm, skip it; otherwise, the match is successful. If the number of traversed point clouds is greater than N1+N2, traverse the point cloud of the vertex detected by the IPM image at the current time, and search for the nearest neighbor in KD3. If the distance between the nearest neighbor and the nearest neighbor is greater than 20cm, skip it; otherwise, the match is successful.

[0143] For example, please see Figure 5 , Figure 5 This is a schematic diagram illustrating point cloud matching in an exemplary embodiment of this application, as shown below. Figure 5 As shown, red represents the movement path of the mobile robot, purple dots represent the semantic point clouds of various types detected in real time, such as purple dots representing the semantic point clouds of various types detected in the real-time acquired IPM image, and green dots representing the semantic point clouds of various types in the semantic map. The two are matched to perform pose calculation.

[0144] Step b4: Construct the weighted residual equation and solve for the pose based on the weighted residual equation.

[0145] Specifically, let the three semantic point clouds in the matching be denoted as {PM1}, {PM2}, and {PM3}, respectively. The residual equation for the matching points of each type of semantic point cloud is shown in the following formula 6: (Formula 6) in, This represents the residual of the semantic point cloud with type label i, where i can be 1, 2, or 3, representing three types of semantic information: storage location lines, direction arrows, and storage location vertices, respectively. This represents the weight parameter with type label i, generally speaking. Let T be the pose of the mobile robot to be optimized. This represents the coordinates of a point in the IPM image at the current moment with type label i and a matching point with the semantic map, in the vehicle coordinate system. Indicating semantic map with The coordinates of the matched points in the semantic map coordinate system.

[0146] Based on Formula 6, a residual equation for each type of semantic information is constructed. By combining the residual equations corresponding to each type of semantic information and the corresponding weight parameters, a weighted residual equation is obtained.

[0147] In this context, the number of matching point clouds of the storage location line in the IPM image at the current moment is denoted as CN1, the number of matching point clouds of the directional arrows is denoted as CN2, and the number of matching point clouds of the vertices is denoted as CN3. Since the number of storage location vertices is the average value of observations over multiple frames and is not affected by downsampling, when CN3 is greater than a preset threshold (e.g., 2), the weight of the residual corresponding to the storage location vertex is increased. For example, the weight of the residual corresponding to the storage location vertex is amplified by 1000 times. That is, when the number of matching storage location vertices is greater than the preset threshold, the pose calculation depends on the coordinates of the matching storage location vertex pairs, thereby improving the reliability of the pose calculation.

[0148] Then, with the goal of minimizing the value of the weighted residual equation, the value of T is calculated, and the pose of the mobile robot is obtained.

[0149] The calculated T can be used directly as the pose of the mobile robot, or the pose can be fused with the pose predicted based on odometry data or lidar data to obtain the pose of the mobile robot, thus ensuring the positioning accuracy of the mobile robot when entering the storage location.

[0150] Figure 6 This is a block diagram illustrating a visual semantic-based positioning device as shown in an exemplary embodiment of this application. Figure 6 As shown, the exemplary visual semantic-based positioning device 600 includes: The data acquisition module 610 is used to acquire a pre-built semantic map and to acquire the visual image of the scene to be located by the mobile robot at the current moment; wherein, the scene to be located is set with positioning markers, and the semantic map contains vertex point clouds of positioning markers. The semantic extraction module 620 is used to extract semantic information of the positioning markers in the visual image to obtain a semantic point cloud, cluster the semantic point cloud to obtain a point cloud cluster, and extract the point cloud in multiple local regions in the point cloud cluster to obtain multiple local point cloud sets. The orientation calculation module 630 is used to determine the orientation of the positioning identifier based on the distribution characteristics of all point clouds in the point cloud cluster to obtain the global orientation; and to determine the orientation of the positioning identifier based on the distribution characteristics of point clouds in multiple local point cloud sets to obtain multiple local orientations. The vertex determination module 640 is used to construct a straight line equation based on the local point cloud set if the angle between the local direction and the global direction is less than a preset threshold, and use the intersection of multiple straight line equations to determine the vertex of the positioning marker to obtain the vertex point cloud corresponding to the current time. The pose calculation module 650 is used to match the vertex point cloud corresponding to the current moment with the vertex point cloud in the semantic map, and calculate the pose of the mobile robot based on the successfully matched vertex point cloud.

[0151] It should be noted that the visual semantic-based positioning device and the visual semantic-based positioning method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the visual semantic-based positioning device provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0152] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a mobile robot in an exemplary embodiment of this application. The mobile robot 110 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-described embodiments of the visual semantic-based localization method.

[0153] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described embodiments of the visual semantic-based localization method. Processor 102 may also be referred to as a Central Processing Unit (CPU). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 102 may be implemented using integrated circuit chips.

[0154] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the above-described embodiments of the visual semantic-based localization method.

[0155] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0156] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A localization method based on visual semantics, characterized in that, The method includes: The system acquires a pre-constructed semantic map and a visual image of the scene to be located captured by the mobile robot at the current moment; wherein the scene to be located is equipped with a location marker, and the semantic map contains a vertex point cloud of the location marker. Semantic point cloud is obtained by extracting the semantic information of the positioning identifier in the visual image, and point cloud cluster is obtained by clustering the semantic point cloud. Point cloud sets are obtained by extracting point clouds in multiple local regions of the point cloud cluster. The direction of the positioning identifier is determined based on the distribution characteristics of all point clouds in the point cloud cluster to obtain a global direction; and the direction of the positioning identifier is determined based on the distribution characteristics of point clouds in the multiple local point cloud sets to obtain multiple local directions. If the angle between the local direction and the global direction is less than a preset threshold, then a straight line equation is constructed based on the local point cloud set, and the vertices of the positioning identifier are determined by the intersections of multiple straight line equations to obtain the vertex point cloud corresponding to the current time. The vertex point cloud corresponding to the current moment is matched with the vertex point cloud in the semantic map, and the pose of the mobile robot is calculated based on the successfully matched vertex point cloud.

2. The method according to claim 1, characterized in that, The step of determining the vertices of the positioning marker by using the intersection points of multiple straight line equations to obtain the vertex point cloud corresponding to the current time includes: Calculate the angle between the equations of the lines; If the included angle between two line equations satisfies the included angle threshold condition corresponding to the positioning identifier, then the intersection point between the two line equations is calculated to obtain the candidate intersection point; Traverse each candidate intersection point. If the distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier, and / or the proportion of pixels belonging to the positioning identifier between the two candidate intersection points satisfies the proportion condition corresponding to the positioning identifier, then take the two candidate intersection points as vertices to obtain the vertex point cloud corresponding to the current time.

3. The method according to claim 2, characterized in that, Determining whether the distance between two candidate intersection points satisfies the distance condition corresponding to the positioning identifier includes: Obtain the length of the straight line segment of the positioning marker; Calculate the distance between two candidate intersection points. If the difference between the distance and the length of the straight line segment of the positioning identifier is less than a preset distance difference threshold, then the two candidate intersection points are determined to satisfy the distance condition corresponding to the positioning identifier.

4. The method according to claim 2, characterized in that, The visual image is a top view; Determining whether the proportion of pixels belonging to the positioning identifier between two candidate intersection points satisfies the proportion condition corresponding to the positioning identifier includes: Transform the two candidate intersection points into the top view by reverse perspective transformation, and then iterate through the pixels of the line between the two candidate intersection points. Determine whether the traversed pixels belong to the location identifier, and calculate the proportion of pixels belonging to the location identifier to obtain the first proportion; If the first proportion is greater than the preset first threshold, then it is determined that the two candidate intersections satisfy the proportion condition corresponding to the positioning identifier.

5. The method according to claim 2, characterized in that, The visual image is a top view; the method further includes: Transform the two determined vertices into the top view through reverse perspective, obtain the normal vector of the line containing the two determined vertices, and traverse along the normal vector for a preset number of pixels; Determine whether the traversed pixels belong to the location identifier, and calculate the proportion of pixels belonging to the location identifier to obtain the second proportion; If the second proportion is greater than the preset second threshold, then other vertices are determined based on the normal vector and the two already determined vertices.

6. The method according to claim 1, characterized in that, The mobile robot is equipped with multiple vision cameras; acquiring the visual images of the scene to be localized by the mobile robot at the current moment includes: Acquire images of the scene to be localized from each visual camera to obtain multiple initial images; Transform each initial image inverse perspective to the top view coordinate system to obtain the top view; The top view is used as the visual image corresponding to the scene to be located at the current moment.

7. The method according to claim 1, characterized in that, The construction of the straight line equation based on the local point cloud set includes: The points in the local point cloud set are projected onto the local direction, and the center point of the local point cloud set is determined based on the projection result. Using the global direction as the direction vector of the line, construct the equation of the line through the center point.

8. The method according to claim 1, characterized in that, The semantic map also contains semantic point clouds containing other types of semantic information of the positioning identifier, and the semantic point cloud containing the other types of semantic information is extracted from the visual image; the pose of the mobile robot is calculated based on the successfully matched vertex points, including: Match the visual image and the semantic map with semantic point clouds of the same type to obtain other types of semantic point clouds that have been successfully matched. A weighted residual equation is constructed based on the number of successfully matched vertex point clouds, the number of successfully matched vertex point clouds, and other types of successfully matched semantic point clouds. The pose of the mobile robot is calculated based on the weighted residual equation.

9. A mobile robot, characterized in that, The mobile robot includes a memory and a processor, the processor being used to execute program instructions stored in the memory to implement the steps in the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images