Robot positioning method, device and equipment and storage medium

Abstract

The embodiment of the invention provides a robot positioning method, device and equipment and a storage medium, and relates to the technical field of robot positioning, and the method comprises the steps: collecting environment information of a robot working environment through a sensor, and converting an environment space point corresponding to the environment information into a robot coordinate system to obtain a space point coordinate; according to the environmental information and the spatial point coordinates, calculating semantic probabilities and category probabilities corresponding to the environmental spatial points, and performing probability fusion on the semantic probabilities and the category probabilities to obtain semantic features; geometric descriptors of the environment space points are obtained, and feature fusion is carried out on the geometric descriptors and the semantic features to obtain fusion features; and constructing a robot positioning model according to the fused features, and predicting the predicted position of the robot through the robot positioning model. According to the invention, effective utilization of multi-source sensor data can be realized, so that the robot can still maintain high-precision and high-reliability positioning effects in typical airport scenes such as illumination variation, large-area homogenized areas and crowded people.

Description

Technical Field

[0001] This invention relates to the field of robot positioning technology, and more specifically, to a robot positioning method, apparatus, device, and storage medium. Background Technology

[0002] Existing service robots or inspection robots typically use LiDAR to construct two-dimensional or three-dimensional point cloud maps for positioning, or rely on visual features for positioning. However, in special scenarios such as airports, there are areas with glass curtain walls or large smooth surfaces. Laser reflection or sparse data can lead to insufficient point cloud features. Moreover, the high density of people in airports can easily cause the loss of visual features, thus affecting the stability and accuracy of positioning. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a robot positioning method, apparatus, device and storage medium to solve the problems of poor stability and accuracy of existing positioning methods.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a robot localization method, comprising: The robot collects environmental information about its working environment using sensors, and then converts the corresponding environmental spatial points into the robot coordinate system to obtain the spatial point coordinates. Based on the environmental information and the spatial point coordinates, calculate the semantic probability and category probability corresponding to the environmental spatial point, and perform probability fusion on the semantic probability and category probability to obtain semantic features; Obtain the geometric descriptor of the environmental spatial point, and fuse the geometric descriptor and the semantic feature to obtain the fused feature; A robot localization model is constructed based on the fused features, and the robot's predicted position is predicted using the robot localization model.

[0005] In an optional implementation, the step of calculating the semantic probability and category probability corresponding to the environmental information based on the environmental information and the spatial point coordinates includes: Obtain image information of the environmental spatial points and map the pixels corresponding to the image information to K-class semantic probability vectors; The point cloud information of the environmental spatial points is obtained, and the point cloud information is segmented through a point cloud semantic segmentation network to obtain the category probability of each point cloud.

[0006] In an optional implementation, the step of probabilistically fusing the semantic probability and the category probability to obtain semantic features includes: The semantic probability vector and the point cloud semantics are probabilistically fused using Bayesian fusion to obtain the fusion probability. The fusion probability is normalized to obtain the joint semantic probability distribution of the point cloud; The semantic features are determined based on the joint semantic probability distribution.

[0007] In an optional implementation, the step of fusing the geometric descriptor and the semantic features to obtain fused features includes: The geometric descriptor and the semantic feature are aligned by linear projection to obtain aligned geometric features and aligned semantic features. The alignment geometric features and alignment semantic features are assigned corresponding weights, and the alignment geometric features and alignment semantic features are fused according to the weights to obtain the fused features.

[0008] In an optional implementation, the step of constructing a robot localization model based on the fused features includes: The observation model of the robot is constructed based on the fusion features; Generate corresponding prediction and update models based on the observation model; The robot localization model is constructed based on the observation model, the prediction model, and the update model.

[0009] In an optional implementation, the step of predicting the robot's predicted position using the robot localization model includes: Obtain the pre-integrated odometer term and uncertainty of the sensor; The pre-integrated odometry term, the uncertainty, and the fusion feature are input into the robot localization model, and the predicted position of the robot is output.

[0010] In an optional implementation, the step of predicting the robot's predicted position using the robot localization model further includes: The observation position of the robot is determined by the observation model based on the pre-integrated odometry term, the uncertainty, and the fusion characteristics. The predicted position of the robot is obtained by predicting and updating the observed position using the prediction model and the update model.

[0011] In a second aspect, embodiments of the present invention provide a robot positioning device, comprising: The information acquisition module is used to collect environmental information of the robot's working environment through sensors, and convert the environmental spatial points corresponding to the environmental information into the robot coordinate system to obtain the spatial point coordinates; The feature extraction module is used to calculate the semantic probability and category probability corresponding to the environmental spatial point based on the environmental information and the spatial point coordinates, and to perform probability fusion of the semantic probability and the category probability to obtain semantic features; The feature fusion module is used to obtain the geometric descriptor of the environmental spatial point, and to fuse the geometric descriptor and the semantic feature to obtain the fused feature; The position prediction module is used to construct a robot localization model based on the fused features, and to predict the robot's position using the robot localization model.

[0012] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor can execute the machine-executable instructions to implement the robot localization method described in the first aspect.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robot localization method described in the first aspect.

[0014] The present invention provides a robot localization method, apparatus, device, and storage medium that integrates geometric and semantic features to effectively utilize multi-source sensor data. This enables the robot to maintain high accuracy and reliability in typical airport scenarios such as changing lighting, large homogeneous areas, and dense crowds, thereby meeting the long-term stable operation requirements of airport services and management.

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A block diagram of an electronic device provided by an embodiment of the present invention is shown; Figure 2 A flowchart illustrating a robot localization method provided by an embodiment of the present invention is shown; Figure 3 A flowchart illustrating a probability acquisition method provided by an embodiment of the present invention is shown; Figure 4 A frame structure diagram of a robot positioning device provided in an embodiment of the present invention is shown.

[0018] icon: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication module; 400 - Robot positioning device; 401 - Information acquisition module; 402 - Feature extraction module; 403 - Feature fusion module; 404 - Position prediction module. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] Please refer to Figure 1 This is a block diagram of an electronic device 100. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0023] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0024] The processor 120 is used to read / write data or programs stored in the memory 110 and to perform corresponding functions.

[0025] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through the network, and to send and receive data through the network.

[0026] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0027] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a robot localization method provided in this embodiment. The method includes: S201. Collect environmental information of the robot's working environment through sensors, and convert the environmental spatial points corresponding to the environmental information into the robot coordinate system to obtain the spatial point coordinates.

[0028] Sensors can include lidar, visible light cameras, depth cameras, and inertial measurement units. Each sensor is mounted on the robot to collect information about the robot's surrounding environment.

[0029] Suppose a certain sensor s The spatial point measured in its own coordinate system is The homogeneous transformation matrix from the sensor to the robot base is: , Then the coordinates of this point in the base coordinate system are: This allows us to obtain the spatial coordinates of environmental points in the robot's coordinate system.

[0030] If the uncertainty of sensor measurement is expressed as a covariance matrix Given that the uncertainty in the base system is approximately obtained through a linearized Jacobian transform: Jacobi When external reference It inherently contains uncertainties (in terms of parameter vectors) In other words, covariance When the sensor attitude / position error contributes additionally to the position uncertainty, it is through Superimposed, among which Therefore, the output of the sensor acquisition module is paired data. .

[0031] S202. Based on the environmental information and the spatial point coordinates, calculate the semantic probability and category probability corresponding to the environmental spatial point, and perform probability fusion of the semantic probability and the category probability to obtain semantic features.

[0032] Environmental information images can be acquired using visible light cameras and depth cameras, and environmental information point clouds can be acquired using LiDAR. Then, for the same spatial point in the environmental information, the semantic probability corresponding to the environmental information image and the category probability corresponding to the environmental information point cloud can be obtained respectively. Finally, the semantic probability and the category probability are fused to obtain semantic features.

[0033] S203. Obtain the geometric descriptor of the environmental spatial point, and fuse the geometric descriptor and the semantic feature to obtain the fused feature.

[0034] Geometric descriptors are a core concept in computer vision, graphics, and pattern recognition. They refer to sets or vectors of values ​​extracted mathematically to quantify the inherent features of geometric objects (such as points, lines, surfaces, 3D models, and image regions). Essentially, they transform abstract geometric shapes, structures, or spatial relationships into computable and comparable "digital signatures," thereby enabling the identification, matching, classification, or retrieval of geometric objects. By fusing the geometric descriptors of points in the environmental space with their corresponding semantic features, a fused feature for each point in the environmental space is obtained.

[0035] S204. Construct a robot localization model based on the fusion features, and predict the robot's predicted position using the robot localization model.

[0036] The fusion features reflect the characteristics of environmental spatial points in multiple dimensions. Then, a robot localization model is constructed based on the fusion features, so that the robot can combine the multi-angle features of the robot's surrounding environment to predict the robot's position.

[0037] This embodiment achieves effective utilization of multi-source sensor data by fusing geometric and semantic features, enabling the robot to maintain high accuracy and reliability in typical airport scenarios such as changing lighting, large homogeneous areas, and dense crowds, thereby meeting the long-term stable operation requirements of airport services and management.

[0038] Please refer to Figure 3 In one embodiment, step S202 includes: step S2021-step S2022.

[0039] S2021. Obtain the image information of the environmental spatial points, and map the pixels corresponding to the image information to K-class semantic probability vectors; S2022. Obtain the point cloud information of the environmental spatial points, and segment the point cloud information through a point cloud semantic segmentation network to obtain the category probability of each point cloud.

[0040] Let the input image be Pixels are mapped to... through a convolutional network K Class semantic probability vector ,satisfy Point cloud semantics is obtained for each point through a point cloud semantic segmentation network. Category probability , among which, point This is the point to be transformed into the robot coordinate system.

[0041] For the same point in the environmental space, i.e., when point Corresponding projection to pixels At that time, Bayesian fusion is used for probabilistic fusion: ,in" "This is element-wise multiplication; after normalization, we get the point." Joint semantic probability distribution The semantic feature is represented by the class with the highest probability or the probability vector itself, denoted as . ,in, d For the semantic embedding dimension, in order to filter semantic features, semantic confidence can also be returned simultaneously. .

[0042] This embodiment acquires image information and point cloud information of environmental spatial points respectively, then maps the pixels corresponding to the image information to K-class semantic probability vectors, segments the point cloud information through a point cloud semantic segmentation network to obtain the class probability of each point cloud, and then fuses the semantic probability vector and the class probability to obtain semantic features. This not only utilizes the high-resolution texture of the image, but also takes into account the three-dimensional structure of the point cloud, thus reducing the risk of single-channel misclassification from a probabilistic perspective.

[0043] In one embodiment, the step of fusing the geometric descriptor and the semantic features to obtain fused features includes: The geometric descriptor and the semantic feature are aligned by linear projection to obtain aligned geometric features and aligned semantic features. The alignment geometric features and alignment semantic features are assigned corresponding weights, and the alignment geometric features and alignment semantic features are fused according to the weights to obtain the fused features.

[0044] Point The geometric descriptor (e.g., local normal, curvature, or point cloud depth descriptor) is denoted as Semantic embedding is First, domain alignment is achieved through linear projection or a small feedforward network: ,in , , These are trainable parameters. Construct the fused representation:

[0045] Among them, weight Determined by the attention (quality assessment) mechanism: First, the scores of the two channels are calculated. Then, normalized weights are obtained in the form of softmax.

[0046] When using one-dimensional weights, it can be simplified to:

[0047] Here For learning parameters, the score may include a confidence factor (e.g., multiplied by semantic confidence). Or a measure of inverse variance based on observed residuals.

[0048] After semantic fusion, point cloud registration is also required. In the point cloud registration stage, the weights can be calculated using a weighted registration cost:

[0049] in For the current frame point, For the corresponding point on the map, the weight With fusion features Related (e.g.) ),function ( ) is the robust loss (e.g., Huber). The constant 𝜆 represents the degree of semantic inconsistency (based on category label or embedding distance), and controls the regularization strength of semantic consistency.

[0050] This embodiment incorporates geometric error and semantic consistency into the registration objective, improving the stability of registration under semantic conflicts or repetitive textures.

[0051] In one implementation, the step of constructing a robot localization model based on the fused features includes: The observation model of the robot is constructed based on the fusion features; Generate corresponding prediction and update models based on the observation model; The robot localization model is constructed based on the observation model, the prediction model, and the update model.

[0052] The observation model formed by the fusion features is as follows: residual Observing Jacobi .

[0053] The prediction model is:

[0054] in .

[0055] The updated model is as follows:

[0056]

[0057] To achieve model adaptability, the covariance is measured. With fusion weight Employing online estimation rules, for example, through exponentially weighted residual estimation:

[0058]

[0059] in (0, 1) is the attenuation factor.

[0060] Anomalies were eliminated using the Mahalanobis distance test: ( Chi-square quantiles, For the residual dimension, When the significance level is reached, the corresponding observation weights are reduced or removed. If factor graph optimization is used, the weighted residuals and robustness costs are equivalently minimized within the sliding window, using semantic and geometric factors as constraints.

[0061] In one embodiment, the step of predicting the robot's predicted position using the robot localization model includes: Obtain the pre-integrated odometer term and uncertainty of the sensor; The pre-integrated odometry term, the uncertainty, and the fusion feature are input into the robot localization model, and the predicted position of the robot is output.

[0062] It can acquire pre-integrated odometer terms generated by inertial measurement equipment. Its uncertainty and covariance The fused features are input together into the robot localization model so that the localization model outputs the robot's predicted position. The pre-integration here can serve as a short-term prior when visual / point cloud failures caused by glass and illumination occur temporarily, which is crucial for filter convergence.

[0063] In one embodiment, the step of predicting the robot's predicted position using the robot localization model further includes: The observation position of the robot is determined by the observation model based on the pre-integrated odometry term, the uncertainty, and the fusion characteristics. The predicted position of the robot is obtained by predicting and updating the observed position using the prediction model and the update model.

[0064] In one embodiment, the method further includes: Provides real-time pose estimation for robots to external users. With location-based reliability metrics (such as the main diagonal or determinant of posterior covariance), and reused observations for incremental map maintenance.

[0065] The map representation of semantic anchors uses Gaussian parameterization: each semantic anchor... k by position average With covariance This indicates that when new measurements of the anchor point are observed... (with covariance) When using Bayesian update:

[0066] Semantic anchors refer to fixed semantic elements that serve as long-term stable references during the localization process. Their positions and attributes are relatively stable in the environment, providing additional constraints for the robot.

[0067] This incremental update enables the semantic map to automatically converge and correct itself based on the observation confidence level during long-term operation, avoiding full map reconstruction and improving maintenance efficiency.

[0068] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of a robot localization device is given below. Please refer to [link / reference]. Figure 4 , Figure 4This is a functional block diagram of a robot positioning device provided in an embodiment of the present invention. It should be noted that the basic principle and technical effects of the robot positioning device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The robot positioning device 400 includes: The information acquisition module 401 is used to acquire environmental information of the robot's working environment through sensors, and convert the environmental spatial points corresponding to the environmental information into the robot coordinate system to obtain the spatial point coordinates. The feature extraction module 402 is used to calculate the semantic probability and category probability corresponding to the environmental spatial point based on the environmental information and the spatial point coordinates, and to perform probability fusion of the semantic probability and the category probability to obtain semantic features; The feature fusion module 403 is used to obtain the geometric descriptor of the environmental spatial point, and to fuse the geometric descriptor and the semantic feature to obtain the fused feature. The position prediction module 404 is used to construct a robot localization model based on the fused features and predict the robot's predicted position through the robot localization model.

[0069] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory shown may be stored in or embedded in the robot's operating system (OS), and can be accessed by... Figure 1 The processor executes the commands. Meanwhile, the data and program code required to execute these modules can be stored in memory.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0071] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0072] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot localization method, characterized in that, include: The robot collects environmental information about its working environment using sensors, and then converts the corresponding environmental spatial points into the robot coordinate system to obtain the spatial point coordinates. Based on the environmental information and the spatial point coordinates, calculate the semantic probability and category probability corresponding to the environmental spatial point, and perform probability fusion on the semantic probability and category probability to obtain semantic features; Obtain the geometric descriptor of the environmental spatial point, and fuse the geometric descriptor and the semantic feature to obtain the fused feature; A robot localization model is constructed based on the fused features, and the robot's predicted position is predicted using the robot localization model.

2. The robot localization method according to claim 1, characterized in that, The step of calculating the semantic probability and category probability corresponding to the environmental information based on the environmental information and the spatial point coordinates includes: Obtain image information of the environmental spatial points and map the pixels corresponding to the image information to K-class semantic probability vectors; The point cloud information of the environmental spatial points is obtained, and the point cloud information is segmented through a point cloud semantic segmentation network to obtain the category probability of each point cloud.

3. The robot localization method according to claim 2, characterized in that, The step of probabilistically fusing the semantic probability and the category probability to obtain semantic features includes: The semantic probability vector and the point cloud semantics are probabilistically fused using Bayesian fusion to obtain the fusion probability. The fusion probability is normalized to obtain the joint semantic probability distribution of the point cloud; The semantic features are determined based on the joint semantic probability distribution.

4. The robot localization method according to claim 1, characterized in that, The step of fusing the geometric descriptor and the semantic features to obtain fused features includes: The geometric descriptor and the semantic feature are aligned by linear projection to obtain aligned geometric features and aligned semantic features. The alignment geometric features and alignment semantic features are assigned corresponding weights, and the alignment geometric features and alignment semantic features are fused according to the weights to obtain the fused features.

5. The robot localization method according to claim 1, characterized in that, The step of constructing a robot localization model based on the fused features includes: The observation model of the robot is constructed based on the fusion features; Generate corresponding prediction and update models based on the observation model; The robot localization model is constructed based on the observation model, the prediction model, and the update model.

6. The robot localization method according to claim 5, characterized in that, The step of predicting the robot's predicted position using the robot localization model includes: Obtain the pre-integrated odometer term and uncertainty of the sensor; The pre-integrated odometry term, the uncertainty, and the fusion feature are input into the robot localization model, and the predicted position of the robot is output.

7. The robot localization method according to claim 6, characterized in that, The step of predicting the robot's predicted position using the robot localization model further includes: The observation position of the robot is determined by the observation model based on the pre-integrated odometry term, the uncertainty, and the fusion characteristics. The predicted position of the robot is obtained by predicting and updating the observed position using the prediction model and the update model.

8. A robot positioning device, characterized in that, include: The information acquisition module is used to collect environmental information of the robot's working environment through sensors, and convert the environmental spatial points corresponding to the environmental information into the robot coordinate system to obtain the spatial point coordinates; The feature extraction module is used to calculate the semantic probability and category probability corresponding to the environmental spatial point based on the environmental information and the spatial point coordinates, and to perform probability fusion of the semantic probability and the category probability to obtain semantic features; The feature fusion module is used to obtain the geometric descriptor of the environmental spatial point, and to fuse the geometric descriptor and the semantic feature to obtain the fused feature; The position prediction module is used to construct a robot localization model based on the fused features, and to predict the robot's position using the robot localization model.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor to implement the robot localization method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot localization method as described in any one of claims 1-7.

Images