Indoor repositioning method, self-moving device, storage medium and computer device

Abstract

The application provides an indoor repositioning method, a self-moving device, a storage medium and a computer device. The method comprises: calculating a three-dimensional visual descriptor of a target key frame; the three-dimensional visual descriptor is used for indicating three-dimensional structure features and three-dimensional color features; determining a registration key frame from a plurality of historical key frames according to the three-dimensional visual descriptor of the target key frame and the three-dimensional visual descriptors of the plurality of historical key frames; and calculating pose information of the self-moving device based on pose information of the registration key frame and a pose transformation matrix of the target key frame and the registration key frame. The application reduces the limitations and inaccuracies caused by changes in observation angles in related visual repositioning methods, and can achieve high-precision indoor repositioning without external markers or auxiliary positioning devices.

Description

Technical Field

[0001] This application relates to the field of indoor robot positioning technology, and in particular to indoor repositioning methods, self-moving devices, computer-readable storage media, and computer devices. Background Technology

[0002] Self-moving devices need to accurately know their location in the environment when performing tasks. Indoor positioning methods mainly rely on sensors such as vision or LiDAR, but each of these methods has its own limitations.

[0003] Vision-based relocalization algorithms typically use traditional bag-of-words models. The problem with this is that different angles from which images are taken will produce different descriptors for the same spatial point. Vision-based methods can also rely on external signage. The drawback of these algorithms is that commercial deployment requires external signage, such as QR codes, in locations where mobile devices are used. While LiDAR-based indoor relocalization algorithms can provide relatively accurate initial positioning information, they rely on other indoor auxiliary devices (such as UWB or Wi-Fi), which also increases the system's complexity and cost.

[0004] Based on this, this application provides an indoor relocation method, a self-moving device, a computer-readable storage medium, and a computer device to improve related technologies. Summary of the Invention

[0005] The purpose of this application is to provide an indoor relocation method, a self-moving device, a computer-readable storage medium, and a computer device.

[0006] The objective of this application is achieved through the following technical solution:

[0007] In a first aspect, this application provides an indoor relocation method applied to a self-moving device that operates within a designated area, the method comprising:

[0008] Calculate the three-dimensional visual descriptor of the target keyframe; the target keyframe is obtained based on the target radar point cloud information and target image information collected by the self-mobile device, and the three-dimensional visual descriptor is used to indicate the three-dimensional structural features and three-dimensional color features;

[0009] Based on the 3D visual descriptor of the target keyframe and the 3D visual descriptors of multiple historical keyframes, a registration keyframe is determined from multiple historical keyframes; the historical keyframes are obtained based on historical radar point cloud information and historical image information collected by the self-moving device.

[0010] Based on the pose information of the registration keyframe and the pose transformation matrix between the target keyframe and the registration keyframe, the pose information of the self-moving device is calculated.

[0011] In some embodiments, the process of constructing the target keyframe includes:

[0012] The target radar point cloud information is processed to remove distortion according to the timestamp of the target image information;

[0013] The distortion-free target radar point cloud information is projected onto the target image information to determine the image pixels corresponding to multiple target points in the target radar point cloud information, so as to obtain the color information corresponding to multiple target points in the target radar point cloud information.

[0014] Using the color information corresponding to multiple target points in the target radar point cloud information, the target radar point cloud information is converted into target color point cloud information, which serves as the target keyframe.

[0015] In some embodiments, the process of constructing the historical keyframes includes:

[0016] The historical radar point cloud information is processed to remove distortion according to the timestamps of the historical image information;

[0017] The distortion-free historical radar point cloud information is projected onto the historical image information to determine the image pixels corresponding to multiple historical points in the historical radar point cloud information, so as to obtain the color information corresponding to multiple historical points in the historical radar point cloud information.

[0018] By utilizing the color information corresponding to multiple historical points in the historical radar point cloud information, the historical radar point cloud information is converted into historical color point cloud information.

[0019] The historical color point cloud information of multiple frames in the world coordinate system is accumulated into a single accumulated color point cloud information, which is used as the historical key frame.

[0020] In some embodiments, calculating the 3D visual descriptor of the target keyframe includes:

[0021] One or more corner points are determined from multiple target points within the target keyframe, with each corner point corresponding to three planes and three edges;

[0022] For each corner point, the included angle between every two edges corresponding to the corner point is calculated to obtain three angles corresponding to the corner point. The three angles corresponding to the corner point and the three color values ​​corresponding to the corner point are combined to form a six-dimensional vector to obtain a three-dimensional visual descriptor. The three color values ​​corresponding to the corner point are obtained based on the color information of the target points in the three planes corresponding to the corner point.

[0023] In some embodiments, determining one or more corner points from a plurality of target points within the target keyframe includes:

[0024] Calculate the unit normal vector of the plane containing multiple target points within the target keyframe;

[0025] For each target point within the target keyframe, find multiple nearest neighbors of the target point from among the multiple target points within the target keyframe, and use the unit normal vectors of the multiple nearest neighbors to cluster the multiple nearest neighbors to obtain the clustering result of the target point, thereby taking the target point with a clustering result of 3 as a corner point.

[0026] In some embodiments, calculating the included angle between each pair of edges corresponding to the corner point to obtain the three angles corresponding to the corner point includes:

[0027] For the three planes corresponding to the corner point, the points on the planes are projected onto the normal vectors of the other two planes respectively, resulting in two sets of projection values ​​corresponding to the planes.

[0028] When both sets of projection values ​​corresponding to one of the planes include both positive and negative values, the corner point is determined to be a first type of corner point, and corner points other than the first type of corner points are designated as second type of corner points.

[0029] Calculate the included angle between each pair of edges corresponding to the corner point to determine the angles corresponding to each pair of edges, thus obtaining the three angles corresponding to the corner point;

[0030] Specifically, for the first type of corner point, a plane whose two sets of projection values ​​both include positive and negative values ​​is designated as the first plane; two edges within the first plane are designated as the first edge and the second edge; and an edge outside the first plane is designated as the third edge. For the first edge and the second edge, the difference between 360 degrees and the angle between these two edges is taken as the angle corresponding to these two edges; for the first edge and the third edge, the angle between these two edges is taken as the angle corresponding to these two edges; and for the second edge and the third edge, the angle between these two edges is taken as the angle corresponding to these two edges.

[0031] For the second type of corner point, the included angle between any two edges is taken as the angle corresponding to those two edges.

[0032] In some embodiments, in the six-dimensional vector:

[0033] The first three dimensions are three angles, and these three angles are arranged in descending order;

[0034] The last three dimensions consist of three color values, and the order of these three color values ​​corresponds to the order of the angles corresponding to the two edges in the corresponding plane.

[0035] In some embodiments, determining the registration keyframe from multiple historical keyframes based on the 3D visual descriptor of the target keyframe and the 3D visual descriptors of multiple historical keyframes includes:

[0036] For each 3D visual descriptor within the target keyframe, candidate 3D visual descriptors whose distance to the target is less than the target distance are determined from the 3D visual descriptors in the 3D visual descriptor map, and the score of the corresponding historical keyframe of the candidate 3D visual descriptor is added; the 3D visual descriptor map is generated using the 3D visual descriptors of multiple historical keyframes.

[0037] The top N historical keyframes with the highest scores are selected as candidate keyframes, and the registration keyframes are determined from the N candidate keyframes; N is a positive integer.

[0038] In some embodiments, determining the registration keyframe from N candidate keyframes includes:

[0039] For each candidate keyframe, the target keyframe and the candidate keyframe are registered to obtain the pose transformation matrix between the target keyframe and the candidate keyframe.

[0040] Perform pose transformation on the target keyframe according to the pose transformation matrix;

[0041] Calculate the residual value between the target keyframe and the candidate keyframe after pose transformation, and select the candidate keyframe with the smallest residual value as the registration keyframe.

[0042] Secondly, this application provides a self-moving device, which includes a lidar, a vision sensor, and a control module, wherein the control module is used to perform any of the methods described above.

[0043] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above.

[0044] Fourthly, this application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the above methods.

[0045] This application provides an indoor repositioning method, a self-moving device, a computer-readable storage medium, and a computer device. Utilizing target radar point cloud information and target image information collected by the self-moving device, a 3D visual descriptor of the target keyframe is calculated. By comparing this descriptor with the 3D visual descriptors of stored historical keyframes, the most suitable registration keyframe for current observation can be identified. Using the known pose information of the registration keyframe, combined with the pose transformation matrix between the target keyframe and the registration keyframe, the current pose information of the self-moving device can be accurately calculated. By combining radar point cloud information and image information to construct a 3D visual descriptor, structural and visual information are effectively utilized, reducing the limitations and inaccuracies caused by changes in the observation angle in related visual repositioning methods. Furthermore, only radar point cloud information and image information are required, and the self-moving device only needs to be equipped with a lidar and a visual sensor. This enables high-precision indoor repositioning without the need for external markers or auxiliary positioning equipment, reducing the overall cost and complexity of the system. Attached Figure Description

[0046] This application will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 This is a flowchart illustrating an indoor relocation method provided in an embodiment of this application.

[0048] Figure 2 This is a schematic diagram of a process for constructing a target keyframe provided in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of a process for constructing historical keyframes provided in an embodiment of this application.

[0050] Figure 4 This is a flowchart illustrating the calculation of a 3D visual descriptor for a target keyframe, provided in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram of a process for obtaining three angles corresponding to a corner point, provided in an embodiment of this application.

[0052] Figure 6 This is a top view of a designated area provided in an embodiment of this application.

[0053] Figure 7 This is a schematic diagram of a process for determining registration keyframes from multiple historical keyframes, provided in an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of a process for determining a registration keyframe from N candidate keyframes, provided in an embodiment of this application.

[0055] Figure 9This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0058] With the rapid development of self-moving devices (e.g., mobile robots), indoor relocation technology has become a very important research area. Self-moving devices need to accurately know their location in the environment to perform tasks such as navigation, searching, or specific task processing. Related indoor positioning methods mainly rely on sensors such as vision or LiDAR, but these methods each have certain limitations.

[0059] Vision-based relocalization algorithms typically use traditional bag-of-words (BOB) models. These algorithms extract two-dimensional descriptors from the image to describe feature points and then determine whether two images observe the same spatial location by matching the degree of overlap between feature points. However, vision-based descriptors have a problem: the image capture angle matters. Observing the same object from different angles yields different visual information, resulting in different descriptors for the same spatial point. Therefore, extracting only two-dimensional descriptors from the image is easily affected by the viewing angle and can easily fail.

[0060] Vision-based methods can also rely on external markers, typically by deploying QR codes or other markers to help identify location information at the location where the mobile device is powered on. The current location of the mobile device is calculated by finding the location of the marker and the relative position of the mobile device to the marker. The disadvantage of this type of algorithm is that it requires external markers for commercial deployment, such as placing QR codes in the places where the mobile device is used, and the deployment personnel need to have a certain level of understanding of the mobile device.

[0061] Indoor relocation algorithms based on lidar typically rely on indoor UWB or Wi-Fi, similar to BeiDou or GPS outdoors, to provide a relatively accurate initial value for lidar relocation. Then, the lidar information is used for more refined matching. While providing relatively accurate initial positioning information, its dependence on other indoor auxiliary devices (such as UWB or Wi-Fi) increases system complexity and cost. Furthermore, relying solely on lidar information for refined matching has limitations, especially in structurally complex or feature-lacking indoor environments.

[0062] See Figure 1 , Figure 1 This is a flowchart illustrating an indoor relocation method provided in an embodiment of this application.

[0063] In order to improve the relevant technology, this application provides an indoor relocation method for use with a self-moving device that operates in a designated area. The method includes steps S101 to S103.

[0064] Step S101: Calculate the three-dimensional visual descriptor of the target keyframe; the target keyframe is obtained based on the target radar point cloud information and target image information collected by the self-moving device, and the three-dimensional visual descriptor is used to indicate the three-dimensional structural features and three-dimensional color features.

[0065] Step S102: Based on the three-dimensional visual descriptor of the target keyframe and the three-dimensional visual descriptor of multiple historical keyframes, determine the registration keyframe from multiple historical keyframes; the historical keyframes are obtained based on historical radar point cloud information and historical image information collected by the self-moving device.

[0066] Step S103: Based on the pose information of the registration keyframe and the pose transformation matrix between the target keyframe and the registration keyframe, calculate the pose information of the self-moving device.

[0067] Self-moving devices, such as mobile robots and automated guided vehicles (AGVs), can complete specific tasks or move within a specific area without direct human intervention. The designated area can be various indoor environments, such as hotels, shopping malls, factory workshops, warehouses, and hospital corridors. In hotels or shopping malls, self-moving devices can be, for example, cleaning robots, delivery robots, or navigation robots. In factory workshops, self-moving devices can be mobile robots used in automated production lines, whose tasks can include raw material transportation, parts assembly, or product packaging. In warehouses, self-moving devices can be automated guided vehicles (AGVs) used for goods handling and inventory management. In hospital corridors, self-moving devices can be cleaning robots, delivery robots, or navigation robots used for medicine delivery, food delivery, or cleaning. Indoor relocation methods are used to determine the precise position and orientation (or attitude) of self-moving devices in indoor environments. These methods are specifically designed for indoor environments because navigation using BeiDou or GPS is typically not feasible indoors.

[0068] Radar point cloud information (including target radar point cloud information and historical radar point cloud information) is a set of data captured by radar (e.g., LiDAR). It can represent the three-dimensional coordinates of various points in the physical environment and accurately describe the structural details of the environment. Image information (including target image information and historical image information) is image data captured by a camera (or camera module) on the self-moving device. Image information provides a visual representation of the environment, including information such as color and texture. Target keyframes can be acquired at one or a series of specific moments (e.g., the current moment, or a specific time period before the current moment). In this embodiment, target keyframes are determined, for example, by target radar point cloud information and target image information. Historical keyframes can be acquired and stored before the current moment, for example, determined by historical radar point cloud information and historical image information. Registration keyframes are selected by comparing the three-dimensional visual descriptors of target keyframes and historical keyframes, choosing the most suitable historical keyframe as the reference object for the current observation. Pose information describes the position and orientation of the self-moving device in three-dimensional space. The pose transformation matrix is ​​used to represent the transformation from one pose to another, including translation and rotation information, and can be represented as a 4*4 matrix for example.

[0069] A 3D visual descriptor is a data structure that describes the visual and spatial information of a target keyframe in detail by combining 3D structural and color features. 3D visual descriptors provide a detailed understanding of a scene, aiding in matching and identifying specific locations within the environment. For example, 3D visual descriptors can be extracted based on the 3D structural and color features of a triangular pyramid (tetrahedron), leveraging the invariant properties of the pyramid's structure in 3D space, exhibiting scale invariance—meaning the vertex attributes of the pyramid remain unchanged regardless of the viewing distance. Furthermore, because the shape of the pyramid remains consistent across different viewpoints, the 3D visual descriptor also exhibits viewpoint invariance. This means that regardless of the viewing angle, the extracted 3D visual descriptor effectively reflects the characteristics of the same spatial structure, improving upon the inconsistency issues caused by differences in viewing angles in related methods. By comprehensively utilizing spatial structure and color information, 3D visual descriptors significantly improve the accuracy and robustness of indoor relocation, enabling self-moving devices to achieve precise relocation relying solely on their own sensor information.

[0070] In the above embodiments, target radar point cloud information and target image information collected by the self-moving device are used to calculate a 3D visual descriptor for the target keyframe. The 3D visual descriptor integrates 3D structural features and 3D color features, providing a detailed visual representation of the environment. By comparing it with the 3D visual descriptors of stored historical keyframes, the registration keyframe most suitable as the reference object for the current observation can be identified. Using the known pose information of the registration keyframe, combined with the pose transformation matrix between the target keyframe and the registration keyframe, the current pose information of the self-moving device can be accurately calculated. This process achieves high-precision indoor repositioning and only requires the participation of radar point cloud information and image information. It only requires the self-moving device to be equipped with LiDAR and a visual sensor, without the need for external markers or auxiliary positioning equipment.

[0071] This indoor relocation method provides a high-precision, low-cost positioning solution for self-moving devices in indoor environments lacking BeiDou or GPS services. It enables high-precision indoor relocation without the need for external markers or auxiliary positioning devices. By combining radar point cloud information and image information to construct a 3D visual descriptor, it effectively utilizes structural and visual information, improving the accuracy and robustness of relocation. It reduces the limitations and inaccuracies caused by changes in viewing angle in related visual relocation methods, and decreases reliance on external markers and auxiliary positioning devices, thereby reducing the overall cost and complexity of the system.

[0072] This application does not limit the method of constructing the target keyframe. In some embodiments, the process of constructing the target keyframe may include:

[0073] The target radar point cloud information is projected onto the target image information to determine the image pixels corresponding to multiple target points in the target radar point cloud information, so as to obtain the color information corresponding to multiple target points in the target radar point cloud information.

[0074] Using the color information corresponding to multiple target points in the target radar point cloud information, the target radar point cloud information is converted into target color point cloud information, which serves as the target keyframe.

[0075] See Figure 2 , Figure 2 This is a schematic diagram of a process for constructing a target keyframe provided in an embodiment of this application.

[0076] Since the points in a single frame of radar point cloud data may not have been acquired at the same time or in the same pose, and radar points measure the distance between the object and the radar, the coordinate systems of different points may be different, potentially causing distortion in the radar point cloud data. To improve this, it is necessary to calculate the pose changes of the radar during the acquisition process and compensate for this motion at the corresponding points; that is, to perform distortion correction processing on the radar point cloud information to correct these distortions. In some embodiments, the process of constructing the target keyframe may include steps S201 to S203.

[0077] Step S201: Perform distortion correction processing on the target radar point cloud information according to the timestamp of the target image information.

[0078] Step S202: Project the distortion-free target radar point cloud information onto the target image information, determine the image pixels corresponding to multiple target points in the target radar point cloud information, and obtain the color information corresponding to multiple target points in the target radar point cloud information.

[0079] Step S203: Using the color information corresponding to multiple target points in the target radar point cloud information, the target radar point cloud information is converted into target color point cloud information, which is used as the target keyframe.

[0080] A timestamp is a marker that records the time the data was captured. Timestamps allow for the synchronization of data from different sources, such as radar point cloud information and image information, ensuring that they describe the environmental state at the same moment. Projection is the process of mapping one dataset (such as radar point cloud information) onto another dataset (such as image information). Through projection, points in a 3D point cloud (i.e., radar point cloud information) can be mapped to pixels in a 2D image (i.e., image pixels). Target points can be some or all of the points in the target radar point cloud information. Color point cloud information not only contains the 3D coordinates of the points but also the color information of each point. Color information can be represented as RGB color information, HSI color information, CMYK color information, grayscale information, etc.

[0081] In the above embodiments, the process of constructing the target keyframe first involves distortion correction of the target radar point cloud information to correct distortions caused by changes in the pose of the mobile device during radar scanning. This step ensures that the point cloud data accurately reflects the structure of the physical world. Subsequently, the distortion-corrected target radar point cloud information is projected onto the target image information. This operation is synchronized via timestamps to ensure that the point cloud and the image describe the scene at the same moment. Through this projection, the corresponding image pixels can be determined for each target point in the target radar point cloud information, thereby obtaining the color information of these target points. Finally, combining the color information, the target radar point cloud information is transformed into target color point cloud information. The target color point cloud information is used as the target keyframe, providing an environmental description combining three-dimensional structural features and three-dimensional color features for subsequent indoor relocation.

[0082] The above embodiments, by combining the precise structural information of the target radar point cloud with the specific color information of the target image, provide more comprehensive data support for indoor relocation by constructing target keyframes compared to simple two-dimensional image information. Furthermore, distortion correction and precise projection of the point cloud into the image further improve data accuracy, ensuring that the relocation algorithm can operate reliably in complex indoor environments.

[0083] This application does not limit the method of constructing historical keyframes, which can be similar to the method of constructing target keyframes. In some embodiments, the process of constructing the historical keyframe may include: projecting historical radar point cloud information onto the historical image information, determining the image pixels corresponding to multiple historical points in the historical radar point cloud information to obtain color information corresponding to multiple historical points in the historical radar point cloud information; and using the color information corresponding to multiple historical points in the historical radar point cloud information to convert the historical radar point cloud information into historical color point cloud information, which serves as the historical keyframe.

[0084] In other embodiments, distortion correction is performed before projection. The process of constructing the historical keyframe may include: performing distortion correction on the historical radar point cloud information according to the timestamp of the historical image information; projecting the distortion-corrected historical radar point cloud information onto the historical image information, determining the image pixels corresponding to multiple historical points in the historical radar point cloud information to obtain the color information corresponding to the multiple historical points in the historical radar point cloud information; and using the color information corresponding to the multiple historical points in the historical radar point cloud information to convert the historical radar point cloud information into historical color point cloud information, which serves as the historical keyframe.

[0085] See Figure 3 , Figure 3 This is a schematic diagram of a process for constructing historical keyframes provided in an embodiment of this application.

[0086] To increase the point cloud density of a local map, N frames of historical color point cloud information in the world coordinate system can be accumulated into a single accumulated color point cloud information, forming a historical keyframe. In some other embodiments, the process of constructing the historical keyframe may include steps S301 to S304.

[0087] Step S301: Perform distortion correction processing on the historical radar point cloud information according to the timestamp of the historical image information.

[0088] Step S302: Project the distortion-free historical radar point cloud information onto the historical image information, determine the image pixels corresponding to multiple historical points in the historical radar point cloud information, and obtain the color information corresponding to multiple historical points in the historical radar point cloud information.

[0089] Step S303: Using the color information corresponding to multiple historical points in the historical radar point cloud information, convert the historical radar point cloud information into historical color point cloud information.

[0090] Step S304: Accumulate the historical color point cloud information of multiple frames in the world coordinate system into an accumulated color point cloud information, which serves as the historical key frame.

[0091] In practical applications, steps S301 to S303 can be executed multiple times to obtain multiple frames of historical color point cloud information. Then, in step S304, the historical key frames are obtained by accumulating the multiple frames of historical color point cloud information. The historical key frames generated in this way significantly improve the density and detail level of the point cloud, which is beneficial for detailed reconstruction of the environment and feature recognition, making the indoor relocation process more accurate and reliable.

[0092] In some embodiments, the accumulation process in step S304 may include the following steps. First, it is necessary to ensure that all historical color point cloud information to be accumulated is in the same world coordinate system. This means that if this historical color point cloud information is obtained in different local coordinate systems (such as the sensor coordinate systems of multiple self-moving devices, or the device coordinate systems of the same self-moving device in different poses), it needs to be transformed to a unified world coordinate system using a coordinate transformation formula. This transformation may, for example, depend on the position and orientation information of the sensors and be accomplished through sensor fusion technology, SLAM (Simultaneous Localization and Mapping), or other localization methods. Second, for each frame of historical color point cloud information, distortion correction processing can be performed before transformation to the world coordinate system. Then, the distortion-corrected and coordinate-transformed frames of historical color point cloud information are accumulated, which is essentially merging them in three-dimensional space. This process needs to consider the overlapping areas between the frames of historical color point cloud information. Various methods can be used to handle the overlapping areas, such as taking the average value, selecting the nearest point, or using a more complex fusion algorithm to optimize the final point cloud representation. Accumulating multiple frames of historical color point cloud data may result in some areas having excessively dense point clouds while others are too sparse. To optimize the density distribution of the point cloud, downsampling (e.g., voxelization) or upsampling (e.g., interpolation) can be performed. Furthermore, averaging or fusion algorithms can be used to ensure consistency in color information after accumulation, avoiding color deviations caused by factors such as lighting changes. After completing these steps, the resulting accumulated color point cloud information serves as a historical keyframe, providing a detailed environmental model that can be used for subsequent indoor relocalization.

[0093] In some embodiments, an M-pyramid is formed at each corner point. The calculation of the 3D visual descriptor of the target keyframe (i.e., step S101) may include: determining one or more corner points from multiple target points within the target keyframe, each corner point corresponding to M planes and M edges, where M is an integer greater than 2; for each corner point, calculating the included angle between every two edges corresponding to the corner point to obtain the M angles corresponding to the corner point; constructing a 2M-dimensional vector from the M angles and the M color values ​​corresponding to the corner point to obtain a 3D visual descriptor; the M color values ​​corresponding to the corner point are obtained based on the color information of the target points in the M planes corresponding to the corner point. M can be 3, 4, 5, 6, 8, etc. Generally, the larger the value of M, the fewer corner points in space that meet the conditions. When the value of M is too large, such corner points may not exist in the environment. In some embodiments, finding all corner points within the target keyframe can form a corner point set. In subsequent corner point-related calculations, the corner points in the corner point set can be traversed to perform relevant calculation steps.

[0094] See Figure 4 , Figure 4 This is a flowchart illustrating the calculation of a 3D visual descriptor for a target keyframe, provided in an embodiment of this application.

[0095] In some embodiments, M=3, and a triangular pyramid (i.e., a tetrahedron) is formed at each corner point. The calculation of the three-dimensional visual descriptor of the target keyframe (i.e., step S101) may include steps S401 to S402.

[0096] Step S401: Determine one or more corner points from multiple target points within the target keyframe, with each corner point corresponding to three planes and three edges.

[0097] Step S402: For each corner point, calculate the included angle between every two edges corresponding to the corner point to obtain three angles corresponding to the corner point. Combine the three angles corresponding to the corner point and the three color values ​​corresponding to the corner point to form a six-dimensional vector to obtain a three-dimensional visual descriptor. The three color values ​​corresponding to the corner point are obtained based on the color information of the target points in the three planes corresponding to the corner point.

[0098] As can be seen in the above embodiments, each corner point can constitute a three-dimensional visual descriptor. For example, if there are 30 corner points within a target keyframe, then 30 three-dimensional visual descriptors can be extracted from that keyframe. In this embodiment, the constructed six-dimensional vector can be directly used as a three-dimensional visual descriptor, or the six-dimensional vector can be encoded, and the encoded six-dimensional vector can be used as a three-dimensional visual descriptor.

[0099] A corner point is a point in three-dimensional space, such as located at the intersection of two or more planes. It has distinct geometric characteristics and can be used for image processing and 3D modeling. An edge, in geometry, refers to the line of intersection of two faces. In this embodiment, M = 3, meaning the corner point is located at the intersection of three planes and connected to three edges. An included angle refers to the angle between any two edges corresponding to the corner point, and its value ranges from 0 degrees to 180 degrees.

[0100] In the embodiments of this application, the included angle between each pair of edges corresponding to a corner point includes three included angles. It should be noted that the three angles corresponding to a corner point can be the three included angles themselves, or they can be calculated based on the three included angles using a specified rule.

[0101] In the above embodiment, corner points with significant geometric features are identified from multiple target points within the target keyframe. These corner points are located at the intersection of three intersecting planes and connected to three edges. For each corner point, three angles are obtained by calculating the included angle between every two edges connected to the corner point. These three angles reflect the shape characteristics of the spatial structure surrounding the corner point. Simultaneously, based on the color information of the target points on the three planes where the corner point is located, three corresponding color values ​​are extracted, for example, by averaging or sampling. These three angles and three color values ​​are combined into a six-dimensional vector, forming a three-dimensional visual descriptor that not only contains the spatial structure information of the corner point but also integrates color information, providing rich feature data for indoor relocation. By combining the geometric features of the corner points with color information, the generated three-dimensional visual descriptor provides highly detailed and accurate environmental feature information for the self-moving device, which is beneficial to improving the accuracy of relocation. In addition, the use of the three-dimensional visual descriptor makes the relocation process more robust to environmental changes, maintaining good performance even under changing lighting conditions or partial obstruction of the field of view.

[0102] To determine one or more corner points from multiple target points within a target keyframe, various methods can be employed. For example, image recognition can be used based on target image information to identify one or more corner points. Alternatively, target keyframe / target radar point cloud information / target image information can be input into a corresponding corner detection model to identify one or more corner points. The corner detection model can be trained on an initial model using a training set; this initial model may be, for example, a machine learning-based model.

[0103] In other embodiments, determining one or more corner points from multiple target points within the target keyframe (i.e., step S401) may include: calculating the unit normal vector of the plane containing the multiple target points within the target keyframe; for each target point within the target keyframe, finding multiple nearest neighbors of the target point from the multiple target points within the target keyframe, and clustering the multiple nearest neighbors using the unit normal vectors of the multiple nearest neighbors to obtain the clustering result of the target point, thereby taking the target point with a clustering result of 3 as a corner point.

[0104] For any target point within a target keyframe, its multiple nearest neighbors refer to the K nearest target points. This application does not limit the method for finding multiple nearest neighbors; for example, the KD-tree algorithm can be used. A KD-tree is a structure used to organize points within a point cloud and can be directly accessed from publicly available libraries. In some embodiments, a KD-tree is constructed for multiple target points within the target keyframe, and the multiple nearest neighbors of each target point are found by traversing the multiple target points within the target keyframe using the KD-tree. Besides the KD-tree algorithm, this application can also employ other methods for finding multiple nearest neighbors, such as the Ball Tree algorithm, Brute-force Search algorithm, Locality-Sensitive Hashing (LSH) algorithm, Annoy (Approximate Nearest Neighbors Oh Yeah) algorithm, and FAISS (Facebook AI Similarity Search) algorithm. In practical applications, appropriate methods can be flexibly selected to find multiple nearest neighbors of a target point within a target keyframe, based on the specific application scenario, dataset size and dimensions, and specific requirements for query speed and accuracy. This achieves efficient and accurate nearest neighbor search, thereby improving the performance and efficiency of target keyframe processing. For each target point, the number of nearest neighbors searched can be, for example, 10, 30, 50, 100, 1000, etc., and this application does not impose any limit on this.

[0105] The length of a unit normal vector is 1. In three-dimensional space, the unit normal vector of any plane is perpendicular to that plane and is used to describe the orientation of the plane. Clustering is a method of grouping data points such that points within the same group are more similar than points in different groups. In the embodiments of this application, clustering is used to divide data points into different groups based on the unit normal vectors of their nearest neighbors.

[0106] In the above embodiments, corner points are identified by analyzing multiple target points within the target keyframe. First, the unit normal vector of the plane containing each target point is calculated, providing directional information about the surface where each point is located. Next, for each target point within the target keyframe, its multiple nearest neighbors are found, and their unit normal vectors are analyzed. Clustering methods are then used to classify these neighbors. When the multiple nearest neighbors of a target point can be clustered into three clearly separate groups, this indicates that the target point is located at the intersection of three different surfaces, meeting the corner point condition in this embodiment, and the target point is identified as a corner point. This process relies on geometric analysis and data clustering techniques. By accurately identifying corner points in space, it provides a foundation for constructing a 3D visual descriptor. By accurately identifying and utilizing the stable and easily identifiable feature of corner points, the accuracy of relocation and map construction is significantly improved. As a salient feature in the environment, corner points exhibit high stability and reliability in identification from different perspectives, making the relocation process more robust, especially in complex or dynamically changing indoor environments.

[0107] See Figure 5 and Figure 6 , Figure 5 This is a flowchart illustrating a method for obtaining three angles corresponding to a corner point, as provided in an embodiment of this application. Figure 6 This is a top view of a designated area provided in an embodiment of this application.

[0108] In some embodiments, in step S402, calculating the included angle between each pair of edges corresponding to the corner point to obtain the three angles corresponding to the corner point may include steps S501 to S503.

[0109] Step S501: For the three planes corresponding to the corner point, project the points on the planes onto the normal vectors of the other two planes respectively to obtain two sets of projection values ​​corresponding to the planes.

[0110] Step S502: When both sets of projection values ​​corresponding to one of the planes simultaneously include positive and negative values, the corner point is determined to be a first type of corner point, and the corner points other than the first type of corner points are designated as second type of corner points.

[0111] Step S503: Calculate the included angle between each pair of edges corresponding to the corner point to determine the angles corresponding to each pair of edges, and obtain the three angles corresponding to the corner point.

[0112] Specifically, for the first type of corner point, a plane whose two sets of projection values ​​simultaneously include both positive and negative values ​​is designated as the first plane. Two edges within the first plane are designated as the first edge and the second edge, and an edge outside the first plane is designated as the third edge. For the first and second edges, the difference between 360 degrees and the angle between these two edges is taken as the corresponding angle. Similarly, for the second and third edges, the angle between them is taken as the corresponding angle. For the second type of corner point, the angle between any two edges is taken as the corresponding angle.

[0113] Projection values ​​are obtained by projecting a point on one plane onto the normal vector of another plane, and can be used to determine the relative positional relationship between the planes. The classification of corner points into Class I and Class II corner points is based on the projection values. When the projection value of a corner point on a given plane contains both positive and negative values, the corner point is classified as a Class I corner point (e.g., ...). Figure 6 (As shown by the solid red dots); Classify corner points other than those in the first category as corner points in the second category (e.g., ...). Figure 6 (As shown by the solid green dot).

[0114] Generally, corner points are located at the edges of objects or the edges of space. Object edge corner points are divided into two categories: those located within a larger plane (ground, wall, or tabletop) and those located outside of a larger plane. For object edge corner points located within a larger plane, if all the corresponding interior angles of the object are minor angles (minor angles, i.e., angles less than 180 degrees), the corner point is a first-category corner point; if the corresponding interior angle of the object has a major angle (major angle, i.e., angles greater than 180 degrees), the corner point is a second-category corner point. Similarly, for object edge corner points located outside a larger plane, if all the corresponding interior angles of the object are minor angles, the corner point is a second-category corner point; if the corresponding interior angle of the object has a major angle, the corner point is a first-category corner point. Space edge corner points are also divided into two categories: if all the corresponding interior angles of the space are minor angles, the corner point is a first-category corner point; if the corresponding interior angle of the space has a major angle, the corner point is a second-category corner point.

[0115] From geometry, we know that the angle between any two edges is less than 180 degrees. For a first-type angle point, the angle between the first and second edges in the first plane is 360 - α (where α is the angle between the first and second edges). Since α is less than 180 degrees, 360 - α is greater than 180 degrees. The angle between the first and third edges is less than 180 degrees; the angle between the second and third edges is also less than 180 degrees. For a second-type angle point, the angle between any two edges is less than 180 degrees.

[0116] In calculating the angle between any two edges, we can first calculate the vector of each edge. Since the edge is the intersection of two planes, the edge is perpendicular to the unit normal vectors of the corresponding two planes. The vector of the edge can be calculated from the unit normal vectors of the corresponding two planes.

[0117] In the above embodiments, corner points within the target keyframe are identified and classified using a unique geometric analysis method. First, for each corner point, its three constituent planes are analyzed. By projecting points on these planes onto the normal vectors of the other two planes, two sets of projection values ​​are generated. The distribution of these projection values ​​(whether they contain both positive and negative values) is used to determine the type of corner point: if the projection values ​​of a plane contain both positive and negative values, the corner point is classified as a first-type corner point; otherwise, it is classified as a second-type corner point. Based on spatial geometric characteristics, among the three planes A, B, and C corresponding to the same corner point, if the projection value of plane A onto plane B contains both positive and negative values, then the projection value of plane A onto plane C also contains both positive and negative values. Subsequently, the geometric characteristics of the corner point are determined by calculating the included angle between each pair of edges corresponding to the corner point. For first-type corner points, edges associated with the "first plane" are specially processed. By calculating the difference between 360 degrees and the included angle between two edges within the first plane, the corresponding angles for these two edges are determined, thereby highlighting their special spatial relationship. For the second type of corner point, the included angle between every two edges is directly calculated. This process generates a six-dimensional vector containing three angles and corresponding color values, serving as a three-dimensional visual descriptor for the corner point. The aforementioned corner point classification method is not only applicable to accurate indoor relocation but also to applications such as 3D modeling and augmented reality; this application does not impose any limitations on these applications.

[0118] This application does not limit the feature ordering in the six-dimensional vector. For example, it can use structural features in the first three dimensions and color features in the last three dimensions, or it can use color features in the first three dimensions and structural features in the last three dimensions, or it can use a combination of structural and color features. This application does not set any limitations in this regard. In the three-dimensional structural features, the three angles can be arranged in descending order or in ascending order. In the three-dimensional color features, the three color values ​​can be arranged in descending order; or they can be arranged in descending order of the angles corresponding to two edges in the corresponding plane, or they can be arranged in ascending order of the angles corresponding to two edges in the corresponding plane, and when the angles are the same, they can be arranged in descending order of lightness.

[0119] In some embodiments, in the six-dimensional vector: the first three dimensions can be three angles, and these three angles are arranged in descending order; the last three dimensions can be three color values, and the order of these three color values ​​corresponds to the order of the angles corresponding to the two edges in the corresponding plane.

[0120] In the above embodiment, a six-dimensional vector is constructed to describe the features of corner points within the target keyframe. The construction of the six-dimensional vector considers the three-dimensional structural features and three-dimensional color features of the corner points. Specifically, firstly, the angles corresponding to every two edges of the three edges formed by the three planes containing the corner point are determined, and these three angles serve as the three-dimensional structural features. Next, based on the color information of the target points within the three planes containing the corner point, the color values ​​of the three planes are extracted as the three-dimensional color features. When constructing the six-dimensional vector, the order of the structural and color features is not limited; different arrangement methods can be selected according to specific application requirements, such as sorting by angle size or color brightness, thereby increasing the flexibility and adaptability of the vector description.

[0121] In some cases, the order of color values ​​corresponds to the order of angles. That is, after determining the order of the three angles, the color values ​​can be arranged according to the order of the corresponding angles. Taking the three planes corresponding to the corner point as P1, P2, and P3 as an example, assuming that the angles corresponding to the two edges in the P1, P2, and P3 planes are α, β, and γ respectively, and satisfying α > β > γ, and assuming that the corresponding color values ​​of the P1, P2, and P3 planes are C1, C2, and C3 respectively, then the six-dimensional vector (three-dimensional visual descriptor) corresponding to the corner point can be represented as [α, β, γ, C1, C2, C3].

[0122] The above embodiments, by combining three-dimensional structural features and three-dimensional color features, construct a six-dimensional vector that provides a comprehensive description of corner points, enhancing the distinguishability of features and facilitating more accurate corner point identification and matching in complex environments. Allowing different feature sorting methods makes the construction of the six-dimensional vector more flexible, enabling optimization of feature representation according to the needs of the application scenario, thereby improving the efficiency and accuracy of feature matching. Furthermore, selecting a feature sorting method suitable for the current application can optimize the data processing flow, reduce computational complexity, and accelerate processing speed, especially when processing large amounts of data (e.g., when the number of corner points is large).

[0123] The embodiments of this application do not limit the method of calculating the three-dimensional visual descriptor of historical keyframes. For example, it can be similar to the method of calculating the three-dimensional visual descriptor of target keyframes, and will not be described in detail here.

[0124] See Figure 7 , Figure 7 This is a schematic diagram of a process for determining registration keyframes from multiple historical keyframes, provided in an embodiment of this application.

[0125] The embodiments of this application do not limit the method of determining the registration key frame. In some embodiments, the step of determining the registration key frame from multiple historical key frames based on the three-dimensional visual descriptor of the target key frame and the three-dimensional visual descriptor of multiple historical key frames (i.e., step S102) may include steps S601 to S602.

[0126] Step S601: For each 3D visual descriptor within the target keyframe, determine candidate 3D visual descriptors whose distance to the target is less than the target distance from the 3D visual descriptors in the 3D visual descriptor map, and add the score of the corresponding historical keyframe of the candidate 3D visual descriptor; the 3D visual descriptor map is generated using the 3D visual descriptors of multiple historical keyframes.

[0127] Step S602: Select the top N historical keyframes with the highest scores as candidate keyframes, and determine the registration keyframe from the N candidate keyframes; N is a positive integer. N can be, for example, 1, 2, 3, 5, 10, 20, 30, etc., and this application does not impose any limitation on this. When N=1, the historical keyframe with the highest score can be directly selected as the candidate keyframe.

[0128] In the above embodiments, the distance can be Euclidean distance. The target distance can be a pre-set distance threshold or calculated based on a specific algorithm. Assume there are two 3D visual descriptors, B1 and B2. B1 is a 3D visual descriptor within a target keyframe, and B2 is a 3D visual descriptor within a 3D visual descriptor map. B1 = [α1, β1, γ1, C11, C21, C31], β2 = [α2, β2, γ2, C12, C22, C32]. Then the Euclidean distance between them is:

[0129]

[0130] In the six-dimensional vector B1, the first three terms α1, β1, and γ1 in [α1, β1, γ1, C11, C21, C31] are the three angles corresponding to the corner point S1 of a three-dimensional visual descriptor within the target keyframe. That is, the angles corresponding to each pair of edges formed by the three planes (let's say X1, X2, X3) where S1 is located. Let's assume α1 is located in plane X1, β1 is located in plane X2, and γ1 is located in plane X3. The last three terms C11, C21, and C31 in [α1, β1, γ1, C1, C21, C31] are the three color values ​​corresponding to the corner point S1, where C11, C21, and C31 are the color values ​​corresponding to planes X1, X2, and X3, respectively. In the six-dimensional vector B2, the first three terms α2, β2, and γ2 in [α2, β2, γ2, C12, C22, C32] are the three angles corresponding to the corner point S2 of a three-dimensional visual descriptor map. That is, the angles corresponding to each pair of edges formed by the three planes (let's say Y1, Y2, Y3) where S2 is located. Let's assume α2 is located in plane Y1, β2 is located in plane Y2, and γ2 is located in plane Y3. The last three terms C12, C22, and C32 in [α1, β1, γ1, C11, C21, C31] are the three color values ​​corresponding to the corner point S2, where C12, C22, and C32 are the color values ​​corresponding to planes Y1, Y2, and Y3, respectively.

[0131] A 3D visual descriptor map is, for example, a dataset containing all 3D visual descriptors from one or more historical keyframes. It allows for searching and matching of currently observed 3D visual descriptors (i.e., 3D visual descriptors within the target keyframe) to aid in localization and identification. This application does not limit the number of historical keyframes used to generate the 3D visual descriptor map; it can be, for example, 3, 5, 10, 30, 100, 1000, etc. In some embodiments, when generating the 3D visual descriptor map, the 3D visual descriptors of all historical keyframes can be stored as the 3D visual descriptor map, along with the index of the historical keyframe to which each 3D visual descriptor belongs. When a candidate 3D visual descriptor is found, the historical keyframe can be located based on the index.

[0132] The score is a quantitative indicator used to characterize the priority of historical keyframes when registering target keyframes. A higher score indicates that the historical keyframe is used more frequently for registration with the target keyframe. Increasing the score of the historical keyframe corresponding to a candidate 3D visual descriptor aims to improve the priority of that historical keyframe as a registration keyframe. The score increment can be the same or different each time; for example, each score increment can be one point, meaning one point is added to the score of the corresponding historical keyframe. For instance, for a 3D visual descriptor X within the target keyframe, if Y candidate 3D visual descriptors with a distance less than the target distance are found from the 3D visual descriptor map, the scores of the historical keyframes corresponding to these Y candidate 3D visual descriptors can be increased by one point. Here, Y can be, for example, 0, 1, 2, 3, etc., and this application does not impose any limitations on this. It should be noted that Y=0 means that for one of the 3D visual descriptors, no candidate 3D visual descriptors with a distance less than the target distance were found from the 3D visual descriptors in the 3D visual descriptor map. This is generally caused by the sparseness of historical keyframes.

[0133] In the above embodiments, each 3D visual descriptor within the target keyframe is used to find a match in the 3D visual descriptor map, for example, through traversal. This process involves calculating the distance between each 3D visual descriptor within the target keyframe and the 3D visual descriptors in the 3D visual descriptor map, and identifying all candidate 3D visual descriptors whose distance is less than a preset threshold (i.e., the target distance). For each such candidate 3D visual descriptor found, the corresponding historical keyframe receives a score increase. In this way, the scores of all historical keyframes are accumulated, and the top N historical keyframes with the highest scores are ultimately selected as candidate keyframes. Among these candidate keyframes, the final registration keyframe can be further determined according to specified criteria (such as score ranking or other rules). This registration keyframe is a historical keyframe used to register the current target keyframe, providing a reference object for the current device pose. Selecting the top N historical keyframes with the highest scores as candidate keyframes effectively narrows the search range, reduces computational load, and thus optimizes processing speed and response time.

[0134] See Figure 8 , Figure 8 This is a schematic diagram of a process for determining a registration keyframe from N candidate keyframes, provided in an embodiment of this application.

[0135] This application does not limit the method of determining the registration keyframe from N candidate keyframes. In some embodiments, in step S602, determining the registration keyframe from N candidate keyframes includes steps S701 to S703.

[0136] Step S701: For each candidate keyframe, register the target keyframe with the candidate keyframe to obtain the pose transformation matrix between the target keyframe and the candidate keyframe.

[0137] Step S702: Perform pose transformation on the target keyframe according to the pose transformation matrix.

[0138] Step S703: Calculate the residual value between the target keyframe and the candidate keyframe after pose transformation, and take the candidate keyframe with the smallest residual value as the registration keyframe.

[0139] In step S701, to register the target keyframe and candidate keyframes, RANSAC (Random Sample Consensus) can be used to calculate the pose transformation matrix between the two keyframes to reduce misregistration interference. RANSAC is used to estimate the parameters of a mathematical model from a set of data containing outliers. During the registration process, RANSAC can effectively identify outliers (misregistrations) in the registration point pairs, thereby reducing the interference of these outliers on the calculation of the pose transformation matrix.

[0140] In some possible implementations, the residual values ​​between the target keyframe and the candidate keyframe can be, for example, point-to-plane residual values ​​or point-to-line residual values. During registration, the point-to-plane residual value measures the vertical distance from a target point in the target keyframe to the corresponding plane within the candidate keyframe, and can be used to evaluate the fit between the target point and the plane. The point-to-line residual value measures the distance from a target point to the nearest corresponding line within the candidate keyframe, and can be used to evaluate the consistency between the target point and the linear feature. Compared to the complex global optimization process, calculating the point-to-plane or point-to-line residual values ​​is relatively simple and can be executed quickly, thereby reducing computational complexity and improving processing speed.

[0141] In computer vision and image processing, registration refers to the spatial alignment of two or more datasets (such as images or point clouds) to match similar regions or feature points. The pose transformation matrix is ​​a matrix describing rotation and translation from one coordinate system to another, used to achieve spatial alignment between datasets. During registration, residual values ​​characterize the differences between the registered target data points and reference data points. The magnitude of the residual value measures the registration accuracy. The candidate keyframe with the smallest residual value has the highest registration accuracy with the target keyframe and is selected as the registration keyframe. By accurately calculating the pose transformation matrix and selecting the candidate keyframe with the smallest residual value as the registration keyframe, the relocalization accuracy of mobile devices in complex environments can be significantly improved. Using residual values ​​as the basis for selecting the best registration keyframe improves the algorithm's robustness to environmental noise and incomplete data. By selecting from a limited number of candidate keyframes, the amount of data requiring in-depth analysis is reduced, thereby optimizing overall computational efficiency and processing speed.

[0142] In some embodiments, the step of calculating the pose information of the self-moving device based on the pose information of the registration keyframe and the pose transformation matrix between the target keyframe and the registration keyframe (i.e., step S103) may include: using NDT+ICP to adjust the parameters of the pose transformation matrix between the target keyframe and the registration keyframe (calculated in step S701) to obtain an adjusted pose transformation matrix; and calculating the pose information of the self-moving device based on the pose information of the registration keyframe and the adjusted pose transformation matrix.

[0143] Among them, NDT (Normal Distribution Transform) divides the space into several small regions and assumes that the point cloud distribution within each small region follows a normal distribution. The NDT algorithm finds the optimal registration by maximizing the similarity of these normal distributions. ICP (Iterative Closest Point) achieves accurate registration by iteratively finding the closest point pairs between two sets of point clouds and minimizing the distance between these point pairs. After the initial calculation of the pose transformation, the pose parameters are further adjusted to improve the accuracy and reliability of the registration.

[0144] In the above embodiments, during the registration of target keyframes and candidate keyframes, the RANSAC algorithm is first used to calculate the pose transformation matrix between the two keyframes. RANSAC effectively eliminates interference from erroneous registration points by randomly selecting a set of sample points and iteratively calculating model parameters, thus improving the accuracy and reliability of registration. The random sampling mechanism of the RANSAC algorithm makes pose estimation highly resistant to noise and outliers, thereby enhancing the robustness of the entire system. After obtaining the preliminary pose transformation matrix, if the candidate keyframe is determined to be the registration keyframe, the NDT and ICP algorithms are then used to refine the pose transformation matrix of the corresponding registration keyframe. The NDT algorithm coarsely adjusts the pose by optimizing the normal distribution similarity in space, while the ICP algorithm further refines the pose by minimizing the distance between the nearest point pairs in the point cloud. This series of processes ultimately produces an accurate pose transformation matrix, which accurately reflects the spatial relationship between the target keyframe and the registration keyframe. Based on the known pose information of the registration keyframe and the adjusted pose transformation matrix, the current pose information of the mobile device can be accurately calculated. The pose refinement process combining NDT and ICP algorithms can effectively handle complex and changing environments, especially achieving high-precision relocalization even when there are many similar features in the environment. Although RANSAC, NDT, and ICP algorithms involve iterative calculations, the computation process can be optimized by intelligently selecting sample points and iterative strategies, reducing unnecessary computation. In particular, once a relatively accurate pose transformation matrix is ​​initially determined using RANSAC, the number of iterations for NDT and ICP can be relatively reduced, thereby improving overall computational efficiency.

[0145] This application also provides a self-moving device, which includes a lidar, a vision sensor, and a control module, wherein the control module is used to execute any of the above methods.

[0146] Among them, the visual sensor can be, for example, a camera (or a camera module).

[0147] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above methods.

[0148] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the methods described above.

[0149] The computer program product may be in the form of a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of this application is not limited thereto, and the computer program product may be in any combination of one or more computer-readable media.

[0150] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement any of the methods described above.

[0151] See Figure 9 , Figure 9 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0152] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0153] like Figure 9 As shown, the computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, processor 120, and communication interface 130 are connected via internal connection paths.

[0154] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0155] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0156] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0157] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0158] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0159] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0160] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0161] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0162] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0163] It should be noted that although some embodiments of this application use mobile robots as examples, this application can be applied to other self-moving devices, such as AGVs, drones, etc., and this application does not set any limitations on them.

[0164] It is understood that the specific examples in this specification are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0165] It is understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0166] It is understood that the various implementation methods described in this specification can be implemented individually or in combination, and this application does not limit them.

[0167] Unless otherwise stated, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this specification. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0168] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0170] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0172] In addition, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0173] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The above are merely specific embodiments described in this specification, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this specification should be included within the scope of protection of this specification. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An indoor repositioning method, characterized in that, Applied to a self-moving device that operates within a specified area, the method includes: Calculate the three-dimensional visual descriptor of the target keyframe; the target keyframe is obtained based on the target radar point cloud information and target image information collected by the self-mobile device, and the three-dimensional visual descriptor is used to indicate the three-dimensional structural features and three-dimensional color features; Based on the 3D visual descriptor of the target keyframe and the 3D visual descriptors of multiple historical keyframes, a registration keyframe is determined from multiple historical keyframes; the historical keyframes are obtained based on historical radar point cloud information and historical image information collected by the self-moving device. Based on the pose information of the registration keyframe and the pose transformation matrix between the target keyframe and the registration keyframe, the pose information of the self-moving device is calculated. The 3D visual descriptor for the calculated target keyframe includes: One or more corner points are determined from multiple target points within the target keyframe, with each corner point corresponding to three planes and three edges; For each corner point, the included angle between every two edges corresponding to the corner point is calculated to obtain three angles corresponding to the corner point. The three angles corresponding to the corner point and the three color values ​​corresponding to the corner point are combined to form a six-dimensional vector to obtain a three-dimensional visual descriptor. The three color values ​​corresponding to the corner point are obtained based on the color information of the target points in the three planes corresponding to the corner point.

2. The indoor relocation method according to claim 1, characterized in that, The process of constructing the target keyframe includes: The target radar point cloud information is processed to remove distortion according to the timestamp of the target image information; The distortion-free target radar point cloud information is projected onto the target image information to determine the image pixels corresponding to multiple target points in the target radar point cloud information, so as to obtain the color information corresponding to multiple target points in the target radar point cloud information. Using the color information corresponding to multiple target points in the target radar point cloud information, the target radar point cloud information is converted into target color point cloud information, which is used as the target keyframe.

3. The indoor repositioning method according to claim 1, characterized in that, The process of constructing the historical keyframes includes: The historical radar point cloud information is processed to remove distortion according to the timestamps of the historical image information; The distortion-free historical radar point cloud information is projected onto the historical image information to determine the image pixels corresponding to multiple historical points in the historical radar point cloud information, so as to obtain the color information corresponding to multiple historical points in the historical radar point cloud information. The historical radar point cloud information is converted into historical color point cloud information by using the color information corresponding to multiple historical points in the historical radar point cloud information. The historical color point cloud information of multiple frames in the world coordinate system is accumulated into a single accumulated color point cloud information, which is used as the historical key frame.

4. The indoor repositioning method according to claim 1, characterized in that, Determining one or more corner points from multiple target points within the target keyframe includes: Calculate the unit normal vector of the plane containing multiple target points within the target keyframe; For each target point within the target keyframe, find multiple nearest neighbors of the target point from among the multiple target points within the target keyframe, and use the unit normal vectors of the multiple nearest neighbors to cluster the multiple nearest neighbors to obtain the clustering result of the target point, thereby taking the target point with a clustering result of 3 as a corner point.

5. The indoor repositioning method according to claim 1, characterized in that, The step of calculating the included angle between each pair of edges corresponding to the corner point to obtain the three angles corresponding to the corner point includes: For the three planes corresponding to the corner point, the points on the planes are projected onto the normal vectors of the other two planes respectively, resulting in two sets of projection values ​​corresponding to the planes. When both sets of projection values ​​corresponding to one of the planes include both positive and negative values, the corner point is determined to be a first type of corner point, and corner points other than the first type of corner points are designated as second type of corner points. Calculate the included angle between each pair of edges corresponding to the corner point to determine the angles corresponding to each pair of edges, thus obtaining the three angles corresponding to the corner point; Specifically, for the first type of corner point, a plane whose two sets of projection values ​​simultaneously include both positive and negative values ​​is designated as the first plane; two edges within the first plane are designated as the first edge and the second edge; and an edge outside the first plane is designated as the third edge. For the first edge and the second edge, the difference between 360 degrees and the angle between these two edges is taken as the angle corresponding to these two edges; for the first edge and the third edge, the angle between these two edges is taken as the angle corresponding to these two edges; and for the second edge and the third edge, the angle between these two edges is taken as the angle corresponding to these two edges. For the second type of corner point, the included angle between any two edges is taken as the angle corresponding to those two edges.

6. The indoor repositioning method according to claim 1, characterized in that, In the six-dimensional vector: The first three dimensions are three angles, and these three angles are arranged in descending order; The last three dimensions consist of three color values, and the order of these three color values ​​corresponds to the order of the angles corresponding to the two edges in the corresponding plane.

7. The indoor repositioning method according to claim 1, characterized in that, The step of determining the registration keyframe from multiple historical keyframes based on the 3D visual descriptor of the target keyframe and the 3D visual descriptors of multiple historical keyframes includes: For each 3D visual descriptor within the target keyframe, candidate 3D visual descriptors whose distance to the target is less than the target distance are determined from the 3D visual descriptors in the 3D visual descriptor map, and the score of the corresponding historical keyframe of the candidate 3D visual descriptor is added; the 3D visual descriptor map is generated using the 3D visual descriptors of multiple historical keyframes. The top N historical keyframes with the highest scores are selected as candidate keyframes, and the registration keyframes are determined from the N candidate keyframes; N is a positive integer.

8. The indoor repositioning method according to claim 7, characterized in that, Determining the registration keyframe from N candidate keyframes includes: For each candidate keyframe, the target keyframe and the candidate keyframe are registered to obtain the pose transformation matrix between the target keyframe and the candidate keyframe. Perform pose transformation on the target keyframe according to the pose transformation matrix; Calculate the residual value between the target keyframe and the candidate keyframe after pose transformation, and select the candidate keyframe with the smallest residual value as the registration keyframe.

9. A self-moving device, characterized in that, The self-moving device includes a lidar, a vision sensor, and a control module, the control module being used to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

11. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

Images