Repositioning method, device, equipment, medium and product

Abstract

The invention discloses a repositioning method and device, equipment, a medium and a product, and relates to the technical field of machine vision, and the method comprises the steps: responding to a preset condition, triggering repositioning, and obtaining the current frame information and historical frame information of an environment map; determining the pose and speed estimation value of the current frame of the environment map according to the current frame information and the historical frame information; visual inertia joint initialization is carried out based on the pose and the speed estimation value of the current frame to obtain a relocation result of the environment map, relocation is triggered under the condition that a preset condition is met, timely intervention can be carried out when a system accumulative error reaches a certain degree, the information of the current frame is still available, and user experience is improved. And the conditions of overlong repositioning time, position jump after repositioning, even incapability of repositioning and the like caused by repositioning when the positioning is completely lost are also avoided, repositioning can be realized without inputting more computing resources, and the computing resource consumption in the repositioning process is reduced, so that the real-time requirement is met.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a repositioning method, device, equipment, medium and product. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) technology is a technology that allows a robot to move in an unknown environment, estimate its own motion trajectory by observing the environment, and construct a map of the environment. Therefore, SLAM is a set of technologies for estimating relative motion. Its trajectory error will gradually increase over time. The graph optimization method it uses is sensitive to the initial value. When the error of a certain state quantity increases to a certain extent, it may cause the system error to diverge rapidly, resulting in the problem of complete loss of positioning. The re-positioning method is to retrieve the current position or correct the trajectory error in time by matching the current observation with the historical map before the positioning is lost or completely lost.

[0003] The relocalization method in the prior art mainly extracts ORB (OrientedFAST and Rotated BRIEF) feature points and descriptors from the image frame when tracking fails, and matches the bag-of-words vector of the current frame with all the data in the image database. However, the extraction of ORB feature points and the calculation of descriptors are very time-consuming, making it difficult to meet real-time requirements.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a relocation method, device, equipment, medium and product, aiming to reduce the consumption of computing resources during the relocation process to meet real-time requirements.

[0006] To achieve the above object, the present application provides a relocation method, the method comprising:

[0007] In response to meeting a preset condition, triggering relocalization, obtaining current frame information and historical frame information of the environment map;

[0008] Determine the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information;

[0009] Visual-inertial joint initialization is performed based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

[0010] In one embodiment, before the step of triggering relocation in response to satisfying a preset condition, the step further includes:

[0011] Obtaining system status parameters of the environmental map;

[0012] Performing an error check on the system state parameters to determine whether a target event occurs;

[0013] If the target event occurs, and the time interval between the current frame and the last time the relocation was triggered exceeds a preset time threshold, it is determined that the preset condition is met.

[0014] In one embodiment, the system state parameter includes at least one of the current bias values ​​of the accelerometer and the gyroscope, the relative displacement or relative speed change between two adjacent frames in the static state of the machine, the relative displacement or relative speed change between two adjacent frames in the motion mode of the machine, the ratio of the external points to the feature points, the absolute value of the height, and the absolute value of the speed, and the target event includes at least one of the following:

[0015] The current bias values ​​of the accelerometer and the gyroscope exceed a first preset threshold;

[0016] The relative displacement or relative speed change between two adjacent frames of the machine in a stationary state exceeds a second preset threshold;

[0017] The relative displacement or relative speed change between two adjacent frames in the motion mode of the machine exceeds a third preset threshold;

[0018] The proportion of the external points to the feature points exceeds a fourth preset threshold;

[0019] The absolute value of the height exceeds a fifth preset threshold;

[0020] The absolute value of the speed exceeds a sixth preset threshold.

[0021] In one embodiment, the current frame information includes feature points in the current frame, the historical frame information includes map points in the historical frames, and the step of determining the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information includes:

[0022] Traversing the feature points in the current frame, and determining a map point among the map points that matches the feature point;

[0023] Acquire the two-dimensional information of the feature point and the three-dimensional information of the map points among the map points that match the feature point;

[0024] The posture of the current frame is solved according to the two-dimensional information and the three-dimensional information, and the speed estimation value is determined according to the posture of the current frame.

[0025] In one embodiment, the historical frame information also includes previous frame information, and the step of solving the posture of the current frame according to the two-dimensional information and the three-dimensional information, and determining the speed estimation value according to the posture of the current frame includes:

[0026] According to the two-dimensional information and the three-dimensional information, the pose of the current frame is calculated by a perspective n-point algorithm;

[0027] Updating the current frame information according to the posture of the current frame to obtain updated current frame information;

[0028] The current relative displacement is determined based on the previous frame information and the updated current frame information, and the velocity estimation value is calculated according to the current relative displacement.

[0029] In one embodiment, the relocation result includes target bias values ​​of an accelerometer and a gyroscope, and the step of performing visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain the relocation result of the environment map includes:

[0030] Clearing a preset number of historical frame information stored in the sliding window, taking the current frame as the first frame, and resetting the initialization state to uninitialized;

[0031] When the sliding window accumulates to the 10th frame, visual-inertial joint initialization is performed according to the pose and velocity estimation values ​​of the current frame, and the target bias values ​​of the accelerometer and gyroscope are determined, wherein the target bias values ​​of the accelerometer and gyroscope are used to optimize the pose of the image frame within the sliding window.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a relocation device, the relocation device comprising:

[0033] A response module, used to trigger relocation in response to meeting a preset condition, and obtain current frame information and historical frame information of the environment map;

[0034] A solution module, used for determining the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information;

[0035] The initialization module is used to perform visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

[0036] In addition, to achieve the above objectives, the present application also proposes a relocation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the relocation method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the relocation method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the relocation method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] In response to meeting the preset conditions, relocation is triggered, and the current frame information and historical frame information of the environment map are obtained; the pose and velocity estimate of the current frame of the environment map are determined according to the current frame information and the historical frame information; visual-inertial joint initialization is performed based on the pose and velocity estimate of the current frame to obtain the relocation result of the environment map. Relocation is triggered when the preset conditions are met, and timely intervention can be made when the system cumulative error reaches a certain level, so that the current frame information is still available, and the situation that relocation time is too long, position jumps after relocation, or even relocation cannot be performed when positioning is completely lost is avoided. The pose and velocity estimate of the current frame are determined according to the current frame information and the historical frame information, and then visual-inertial joint initialization is performed to obtain the target bias values ​​of the accelerometer and the gyroscope. Relocation can be achieved without investing more computing resources, which reduces the computing resource consumption in the relocation process, thereby meeting the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the relocation method of the present application;

[0044] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the relocation method of the present application;

[0045] Figure 3A schematic diagram of a system error detection process in an embodiment of the present application;

[0046] Figure 4 A flowchart of the third embodiment of the relocation method of the present application is provided;

[0047] Figure 5 A schematic diagram of a relocation algorithm flow chart according to an embodiment of the present application;

[0048] Figure 6 This is a schematic diagram of the module structure of the relocation device according to an embodiment of the present application;

[0049] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the relocation method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: in response to meeting the preset conditions, triggering repositioning, obtaining the current frame information and historical frame information of the environment map; determining the pose and velocity estimate of the current frame of the environment map according to the current frame information and the historical frame information; performing visual-inertial joint initialization based on the pose and velocity estimate of the current frame to obtain the repositioning result of the environment map, triggering repositioning when the preset conditions are met, and timely intervention can be made when the system cumulative error reaches a certain level, so that the current frame information is still available, and it also avoids the situation that repositioning when the positioning is completely lost will cause the repositioning time to be too long, the position jump after repositioning, or even the inability to reposition, etc., the pose and velocity estimate of the current frame are determined by the current frame information and the historical frame information, and then visual-inertial joint initialization is performed to obtain the target bias values ​​of the accelerometer and the gyroscope, so that repositioning can be achieved without investing more computing resources, reducing the computing resource consumption in the repositioning process, thereby meeting the real-time requirements.

[0054] In this embodiment, for the convenience of description, the following description is made with the relocation device as the execution subject.

[0055] SLAM (Simultaneous Localization and Mapping), when a robot moves in an unknown environment, it estimates its own motion trajectory by observing the environment and builds a map of the environment. Therefore, SLAM is a set of technologies for estimating relative motion, and its trajectory error will gradually increase over time. The graph optimization method is sensitive to the initial value. When the error of a certain state quantity increases to a certain extent, it may cause the system error to diverge rapidly and the positioning to be completely lost. The re-positioning method is to retrieve the current positioning or correct the trajectory error in time by matching the current observation with the historical map before the positioning is lost or completely lost.

[0056] ORB_SLAM is a classic visual SLAM solution. The relocalization method extracts ORB feature points and descriptors from the image frame, and matches the BoW (bag of words vector) of the current frame with all the data in the image database. However, ORB feature point extraction and descriptor calculation are very time-consuming, and it is difficult for embedded devices to meet real-time requirements. And as the number of maps increases, the database will become larger and larger, and the matching time will also increase.

[0057] This application provides a solution, which aims to solve the problem of gradually increasing or even diverging trajectory errors during positioning and mapping by proposing a SLAM fast repositioning technology based on FAST feature points. Through the repositioning technology, the trajectory error is corrected in time and intervention is made to avoid the situation where the error gradually amplifies during the optimization process and the final positioning is completely lost. In addition, the solution in this embodiment consumes less computing resources and can be applied to embedded devices to meet real-time requirements.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a relocation device, etc. The following takes the relocation device as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, the present application embodiment provides a relocation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the relocation method of the present application.

[0060] In this embodiment, the relocation method includes steps S10 to S30:

[0061] Step S10, in response to meeting a preset condition, triggering relocation, and obtaining current frame information and historical frame information of the environment map;

[0062] Exemplarily, in the embodiment of the present application, the repositioning system is mainly composed of three modules: a system error detection module, a current posture solution module, and a system state reset module. The system error detection module comprehensively determines whether a repositioning operation is required by measuring the quality of feature point reprojection errors, the absolute value of accelerometer and gyroscope bias, and the relative displacement, relative speed, and relative angle change between two adjacent frames.

[0063] For example, since SLAM is a set of technologies for estimating relative motion, its trajectory error will gradually increase over time. The graph optimization method has a certain sensitivity to the initial value. When the error of a certain state quantity increases to a certain extent, it may cause the system error to diverge rapidly and the positioning to be completely lost. Through the repositioning method in the embodiment of the present application, the current positioning can be retrieved or the trajectory error can be corrected in time by matching the current observation with the historical map before the positioning is lost or completely lost.

[0064] Step S20, determining the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information;

[0065] Furthermore, after triggering relocalization to obtain current frame information and historical frame information, the position and velocity estimation values ​​of the current frame of the environment map can be determined according to the current frame information and historical frame information.

[0066] Exemplarily, in the embodiment of the present application, the current pose solution module uses the optical flow method to track the matching relationship between the map points (3D) in the historical map and the FAST feature points (2D) of the current frame, and estimates the position and angle of the current frame using the PnP (Perspective-n-Point) algorithm based on the matching relationship between the 3D points and the 2D points. The system state reset module then resets the pose of the current frame to the PnP solution result, and clears the historical map information.

[0067] For example, the PnP algorithm is a commonly used algorithm in computer vision to solve the problem of estimating the camera pose from the correspondence between 2D image points and 3D space points. Given a set of points in the 3D world coordinate system and their corresponding projection points on the 2D image plane, the goal of the PnP algorithm is to find the position and orientation of the camera relative to these 3D points, that is, to determine the camera's external parameter matrix (including rotation and translation).

[0068] Exemplarily, the PnP solution result is used to update the current frame information, and then the average speed is calculated according to the relative displacement of two adjacent frames to be used as the speed estimation value of the current frame.

[0069] Step S30, performing visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

[0070] Furthermore, after determining the pose and speed estimation values ​​of the current frame of the environment map according to the current frame information and the historical frame information, visual-inertial joint initialization can be performed based on the pose and speed estimation values ​​of the current frame to obtain the relocation result of the environment map.

[0071] Exemplarily, the repositioning result includes target bias values ​​of an accelerometer and a gyroscope.

[0072] Exemplarily, the step of performing visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain a relocalization result of the environment map includes:

[0073] Clearing a preset number of historical frame information stored in the sliding window, taking the current frame as the first frame, and resetting the initialization state to uninitialized;

[0074] When the sliding window accumulates to the 10th frame, visual-inertial joint initialization is performed according to the pose and velocity estimation values ​​of the current frame, and the target bias values ​​of the accelerometer and gyroscope are determined, wherein the target bias values ​​of the accelerometer and gyroscope are used to optimize the pose of the image frame within the sliding window.

[0075] Exemplarily, the current frame information is updated using the PnP solution result, and the average speed is calculated based on the relative displacement of two adjacent frames as the speed estimate of the current frame. The first 10 frames of information saved in the sliding window are all cleared, the current frame is taken as the first frame, and the initialization state is reset to uninitialized. When the window accumulates to the 10th frame, visual-inertial joint optimization is performed to estimate the bias values ​​of the accelerometer and gyroscope and optimize the posture of all image frames in the window.

[0076] For example, accelerometers and gyroscopes are components of IMU (Inertial Measurement Unit). The accelerometer is used to measure linear acceleration along three orthogonal axes (usually x, y, and z axes). By integrating the acceleration data, the velocity change and displacement can be estimated; the gyroscope is used to measure the angular velocity (rotation rate) around three orthogonal axes. By integrating the angular velocity data, the changes in attitude angles, such as pitch, yaw, and roll, can be obtained.

[0077] Exemplarily, the bias of an accelerometer and a gyroscope refers to the non-zero measurement value output by the sensor in an ideal static state, that is, without any external linear acceleration or angular velocity. These biases are inherent sources of error in the IMU (inertial measurement unit). If not corrected, they will lead to accumulated positioning and attitude estimation errors over time. Therefore, in applications requiring high-precision navigation and attitude estimation, it is very important to accurately estimate and correct these biases. The bias values ​​of accelerometers and gyroscopes can be accurately estimated and corrected by the repositioning method in the embodiment of the present application.

[0078] Exemplarily, the objective function r optimized by combining the image frames and IMU data in the sliding window specifically includes two parts: the first part is the residual of the pre-integration of the IMU data between two consecutive image frames and the true value; the second part is the residual of the coordinates obtained after the 3D information of the map point is reprojected back to the two-dimensional image coordinate system and the coordinates of the three-dimensional point actually observed in the image coordinate system. The two parts of the constraints are added together to form the total objective function, and the system state value is optimized by the least squares method to minimize the objective function. The system state value at this time is the optimal solution.

[0079] Through the above scheme, this embodiment specifically triggers relocation in response to meeting preset conditions, obtains current frame information and historical frame information of the environment map; determines the pose and velocity estimate of the current frame of the environment map according to the current frame information and historical frame information; performs visual-inertial joint initialization based on the pose and velocity estimate of the current frame to obtain the relocation result of the environment map, triggers relocation when the preset conditions are met, and can intervene in time when the system cumulative error reaches a certain level, so that the current frame information is still available, and avoids the situation that relocation when positioning is completely lost will cause too long relocation time, position jump after relocation, or even inability to relocate, etc., determines the pose and velocity estimate of the current frame according to the current frame information and historical frame information, and then performs visual-inertial joint initialization to obtain the target bias values ​​of the accelerometer and gyroscope, so that relocation can be achieved without investing more computing resources, reducing the computing resource consumption in the relocation process, thereby meeting real-time requirements.

[0080] Based on the first embodiment of the present application, a second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0081] On this basis, please refer to Figure 2 Before step S10, the relocation method further includes steps S01 to S03:

[0082] Step S01: obtaining system status parameters of the environment map;

[0083] Step S02: performing an error check on the system state parameters to determine whether a target event occurs;

[0084] Step S03: If the target event occurs, and the time interval between the current frame and the last time the relocation was triggered exceeds a preset time threshold, it is determined that the preset condition is met.

[0085] Exemplarily, the system state parameters include current bias values ​​of the accelerometer and the gyroscope, relative displacement or relative speed change between two adjacent frames in the machine's stationary state, relative displacement or relative speed change between two adjacent frames in the machine's motion mode, the proportion of feature points occupied by external points, at least one of the absolute value of height and the absolute value of speed.

[0086] Exemplarily, the target event includes at least one of the following:

[0087] The current bias values ​​of the accelerometer and the gyroscope exceed a first preset threshold;

[0088] The relative displacement or relative speed change between two adjacent frames of the machine in a stationary state exceeds a second preset threshold;

[0089] The relative displacement or relative speed change between two adjacent frames in the motion mode of the machine exceeds a third preset threshold;

[0090] The proportion of the external points to the feature points exceeds a fourth preset threshold;

[0091] The absolute value of the height exceeds a fifth preset threshold;

[0092] The absolute value of the speed exceeds a sixth preset threshold.

[0093] Reference Figure 3 , Figure 3 Schematic diagram of the system error detection process in the embodiment of the present application. Figure 3 As shown, the system state parameters are checked through the system error detection module, specifically including: checking whether the accelerometer bias and the gyroscope bias exceed the first preset threshold; checking whether the relative displacement or relative speed change of two adjacent frames in the static state of the machine exceeds the second preset threshold; checking whether the relative displacement or relative speed change of two adjacent frames in the motion mode of the machine exceeds the third preset threshold; checking whether the proportion of feature points occupied by external points (points with reprojection errors greater than the threshold) exceeds the fourth preset threshold; checking whether the absolute value of the height exceeds the fifth preset threshold; checking whether the absolute value of the speed exceeds the sixth preset threshold. If any of the above items is met, and the time interval between the current frame and the last triggering of repositioning exceeds the preset time threshold, repositioning is triggered; if any of the above items is not met, or the time interval between the current frame and the last triggering of repositioning does not exceed the preset time threshold, the algorithm is terminated.

[0094] Exemplarily, the system error detection method in the embodiment of the present application uses feature point reprojection errors and relative displacements of adjacent frames, etc. This allows the system to enter the relocation module earlier and intervene in the system cumulative error earlier, so that the optical flow tracking information is still available. At the same time, it also avoids the situation where relocation when positioning is completely lost will result in too long relocation time, position jump after relocation, or even inability to relocate.

[0095] It should be noted that the preset thresholds in the embodiments of the present application can be set according to empirical values ​​or determined according to actual needs, and the embodiments of the present application do not specifically limit this.

[0096] This embodiment adopts the above scheme, specifically by obtaining system status parameters; checking the system status parameters through the system error detection module to determine whether the target event occurs; if the target event occurs, and the time interval between the current frame and the last time the repositioning is triggered exceeds the preset time threshold, it is determined that the preset condition is met, and the system status parameters are checked through the system error detection module to trigger repositioning when the preset condition is met. When the system cumulative error reaches a certain level, timely intervention can be made, so that the current frame information is still available, and the situation that repositioning when the positioning is completely lost, which will cause the repositioning time to be too long, the position jump after repositioning, or even the inability to reposition, etc., is avoided.

[0097] Based on any of the foregoing embodiments of the present application, a third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as any of the foregoing embodiments can be referred to the above introduction and will not be described in detail later. Figure 4 , step S20 also includes steps S201 to S203:

[0098] Step S201: traversing the feature points in the current frame, and determining the map points among the map points that match the feature points;

[0099] Step S202: Acquire the two-dimensional information of the feature point and the three-dimensional information of the map point that matches the feature point among the map points;

[0100] Step S203: Solve the posture of the current frame according to the two-dimensional information and the three-dimensional information, and determine the speed estimation value according to the posture of the current frame.

[0101] Exemplarily, the current frame information includes feature points in the current frame, and the historical frame information includes map points in the historical frames.

[0102] Exemplarily, the historical frame information also includes previous frame information, and the step of solving the posture of the current frame according to the two-dimensional information and the three-dimensional information, and determining the speed estimation value according to the posture of the current frame includes:

[0103] According to the two-dimensional information and the three-dimensional information, the pose of the current frame is calculated by a perspective n-point algorithm;

[0104] Updating the current frame information according to the posture of the current frame to obtain updated current frame information;

[0105] The current relative displacement is determined based on the previous frame information and the updated current frame information, and the velocity estimation value is calculated according to the current relative displacement.

[0106] Reference Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a relocation algorithm flow according to an embodiment of the present application, such as Figure 5 As shown in the figure, after entering the relocalization, the FAST feature points of the current image frame are traversed, and the feature point ID is used to determine whether it can be matched with the map point in the historical map. The 3D information of the successfully matched map point and the 2D information of the FAST feature point are recorded. The PnP algorithm is used to solve the pose of the current frame.

[0107] The PnP algorithm solves the following objective function through the least squares method:

[0108]

[0109] Among them, T cw represents the pose of the current frame, ρ represents the 3D information of the map point, and u represents the 2D information of the corresponding feature point. represents the set of all successfully matched map points, and χ represents the variable to be optimized T cw .

[0110] Exemplarily, in the embodiment of the present application, the PnP solution result is used to update the current frame information, and the average speed is calculated according to the relative displacement of two adjacent frames as the speed estimation value of the current frame. In other embodiments, different algorithms such as EPnP (Efficient PnP) or MLPnP (Maximum Likelihood Perspective-n-Point) can be used to replace the PnP algorithm, or the pose estimated by other sensors besides the camera can be used, such as GNSS (Global Navigation Satellite System) and RTK (Real-Time Kinematic).

[0111] Exemplarily, compared with ORB_SLAM, the relocation method in the embodiment of the present application does not need to construct a word bag, and only uses the feature point information extracted by the optical flow method. That is, in SLAM, there is no need to invest too much additional computing resources in the relocation module. This ensures that the SLAM algorithm can meet the real-time requirements even on embedded devices with insufficient computing resources.

[0112] This embodiment adopts the above scheme, specifically by traversing the feature points in the current frame, determining the map points in the map points that match the feature points; obtaining the two-dimensional information of the feature points and the three-dimensional information of the map points in the map points that match the feature points; solving the posture of the current frame according to the two-dimensional information and the three-dimensional information, and determining the speed estimation value according to the posture of the current frame. The relocation method in this embodiment does not need to extract ORB feature points from the image, calculate descriptors, match bag of words, etc. Only the matching relationship of the feature points extracted by the optical flow method is used, and PnP calculation is performed according to the two-dimensional information and the three-dimensional information, which can save a lot of computing resources.

[0113] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the relocation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0114] This application also provides a repositioning device, please refer to Figure 6 , the relocation device comprises:

[0115] A response module 10 is used to trigger relocation in response to meeting a preset condition, and obtain current frame information and historical frame information of the environment map;

[0116] A solution module 20, used to determine the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information;

[0117] The initialization module 30 is used to perform visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

[0118] The relocation device provided by the present application adopts the relocation method in the above embodiment to solve the technical problem of relocation. Compared with the prior art, the beneficial effects of the relocation device provided by the present application are the same as the beneficial effects of the relocation method provided by the above embodiment, and other technical features in the relocation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0119] The present application provides a relocation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the relocation method in the above-mentioned embodiment 1.

[0120] Reference below Figure 7 , which shows a schematic diagram of the structure of a relocation device suitable for implementing the embodiment of the present application. The relocation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The relocation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Figure 7 As shown, the relocation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the relocation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the relocation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a relocation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0122] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0123] The relocation device provided by the present application adopts the relocation method in the above embodiment to solve the technical problem of relocation. Compared with the prior art, the beneficial effects of the relocation device provided by the present application are the same as the beneficial effects of the relocation method provided by the above embodiment, and other technical features in the relocation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0124] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0126] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the relocation method in the above-mentioned embodiment.

[0127] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0128] The computer-readable storage medium may be included in the relocation device; or may exist independently without being assembled into the relocation device.

[0129] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the repositioning device, the repositioning device: in response to meeting a preset condition, triggers repositioning and obtains the current frame information and historical frame information of the environment map; determines the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information; performs visual-inertial joint initialization based on the pose and velocity estimation value of the current frame to obtain the repositioning result of the environment map. When the preset condition is met, repositioning is triggered, and timely intervention can be made when the system cumulative error reaches a certain level, so that the current frame information is still available, and the situation that repositioning when the positioning is completely lost will cause the repositioning time to be too long, the position jump after repositioning, or even the inability to reposition is avoided. The pose and velocity estimation value of the current frame is determined by the current frame information and the historical frame information, and then the visual-inertial joint initialization is performed to obtain the target bias value of the accelerometer and the gyroscope. Repositioning can be achieved without investing more computing resources, reducing the computing resource consumption in the repositioning process, thereby meeting the real-time requirements.

[0130] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0133] The readable storage medium provided by the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned relocation method, which can solve the technical problem of relocation. Compared with the prior art, the beneficial effect of the computer-readable storage medium provided by the present application is the same as the beneficial effect of the relocation method provided by the above-mentioned embodiment, and will not be repeated here.

[0134] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned relocation method when executed by a processor.

[0135] The computer program product provided by the present application can solve the technical problem of relocation. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the relocation method provided by the above embodiment, which will not be described in detail here.

[0136] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A relocation method, characterized in that: The method comprises: In response to meeting a preset condition, triggering relocalization, obtaining current frame information and historical frame information of the environment map; Determine the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information; Visual-inertial joint initialization is performed based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

2. The method according to claim 1, characterized in that In response to satisfying a preset condition, before the step of triggering relocation, the step further includes: Obtaining system status parameters of the environmental map; Performing an error check on the system state parameters to determine whether a target event occurs; If the target event occurs, and the time interval between the current frame and the last time the relocation was triggered exceeds a preset time threshold, it is determined that the preset condition is met.

3. The method according to claim 2, characterized in that The system state parameter includes at least one of the current bias values ​​of the accelerometer and the gyroscope, the relative displacement or relative speed change between two adjacent frames in the static state of the machine, the relative displacement or relative speed change between two adjacent frames in the motion mode of the machine, the ratio of the external points to the feature points, the absolute value of the height and the absolute value of the speed, and the target event includes at least one of the following: The current bias values ​​of the accelerometer and the gyroscope exceed a first preset threshold; The relative displacement or relative speed change between two adjacent frames of the machine in a stationary state exceeds a second preset threshold; The relative displacement or relative speed change between two adjacent frames in the motion mode of the machine exceeds a third preset threshold; The proportion of the external points to the feature points exceeds a fourth preset threshold; The absolute value of the height exceeds a fifth preset threshold; The absolute value of the speed exceeds a sixth preset threshold.

4. The method according to claim 1, characterized in that The current frame information includes feature points in the current frame, the historical frame information includes map points in the historical frames, and the step of determining the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information includes: Traversing the feature points in the current frame, and determining a map point among the map points that matches the feature point; Acquire the two-dimensional information of the feature point and the three-dimensional information of the map points among the map points that match the feature point; The posture of the current frame is solved according to the two-dimensional information and the three-dimensional information, and the speed estimation value is determined according to the posture of the current frame.

5. The method according to claim 4, characterized in that The historical frame information also includes previous frame information, and the steps of solving the posture of the current frame according to the two-dimensional information and the three-dimensional information, and determining the speed estimation value according to the posture of the current frame include: According to the two-dimensional information and the three-dimensional information, the pose of the current frame is calculated by a perspective n-point algorithm; Updating the current frame information according to the posture of the current frame to obtain updated current frame information; The current relative displacement is determined based on the previous frame information and the updated current frame information, and the velocity estimation value is calculated according to the current relative displacement.

6. The method according to claim 1, characterized in that The relocation result includes target bias values ​​of an accelerometer and a gyroscope, and the step of performing visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain the relocation result of the environment map includes: Clearing a preset number of historical frame information stored in the sliding window, taking the current frame as the first frame, and resetting the initialization state to uninitialized; When the sliding window accumulates to the 10th frame, visual-inertial joint initialization is performed according to the pose and velocity estimation values ​​of the current frame, and the target bias values ​​of the accelerometer and gyroscope are determined, wherein the target bias values ​​of the accelerometer and gyroscope are used to optimize the pose of the image frame within the sliding window.

7. A repositioning device, characterized in that: The device comprises: A response module, used to trigger relocation in response to meeting a preset condition, and obtain current frame information and historical frame information of the environment map; A solution module, used for determining the pose and velocity estimation value of the current frame of the environment map according to the current frame information and the historical frame information; The initialization module is used to perform visual-inertial joint initialization based on the pose and velocity estimation values ​​of the current frame to obtain a relocation result of the environment map.

8. A relocation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the relocation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the relocation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the relocation method according to any one of claims 1 to 6 are implemented.

Images