Relocation method and device, electronic equipment and storage medium

Abstract

The invention discloses a repositioning method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a first image frame currently shot by a first camera and a second image frame currently shot by a second camera; if the electronic equipment currently meets the relocation condition, respectively acquiring a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame from the image word bag; acquiring first pose information of the first camera according to a target pose algorithm, the first image frame, the first key frame and pose information carried by the first key frame; acquiring second pose information of a second camera according to a target pose algorithm, the second image frame, the second key frame and pose information carried by the second key frame; and if the first pose information and the second pose information meet the target pose condition, determining the current pose information of the electronic equipment based on the first pose information or the second pose information. Therefore, the repositioning accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a repositioning method, device, electronic device and storage medium. Background Art

[0002] Simultaneous Localization and Mapping (SLAM): The robot starts moving from an unknown position in an unknown environment, locates itself according to the position and map during the movement, and builds an incremental map based on its own positioning to achieve autonomous positioning and navigation of the robot.

[0003] However, in the related technologies, the robustness of the methods for autonomous positioning and navigation of robots is still not high enough, and there are often problems of tracking loss and relocalization failure. Currently, how to improve the accuracy of relocalization after tracking loss has become an urgent problem to be solved. Summary of the invention

[0004] The present application proposes a relocation method, device, electronic device and storage medium to improve the accuracy of relocation.

[0005] In a first aspect, an embodiment of the present application provides a repositioning method, which is applied to an electronic device, wherein the electronic device includes at least a first camera and a second camera, and the method includes: obtaining a current frame image taken by the first camera as a first image frame, and a current frame image taken by the second camera as a second image frame; if the electronic device currently meets the repositioning conditions, obtaining key frame images matching the first image frame and the second image frame from an image word bag, respectively, to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame, wherein the image word bag includes multiple key frame images, and the multiple key frame images are based on the first camera and the second camera during the process of the electronic device moving along the target path. The captured image is determined, and each key frame image carries the posture information of the first camera or the second camera corresponding to when the key frame image is captured; according to the target posture algorithm, the first image frame, the first key frame and the posture information carried by the first key frame, the current posture information of the first camera is obtained as the first posture information; according to the target posture algorithm, the second image frame, the second key frame and the posture information carried by the second key frame, the current posture information of the second camera is obtained as the second posture information; if the first posture information and the second posture information meet the target posture condition, the current posture information of the electronic device is determined based on the first posture information or the second posture information.

[0006] In a second aspect, an embodiment of the present application provides a repositioning device, which is applied to an electronic device, wherein the electronic device includes at least a first camera and a second camera, and the device includes: a current frame acquisition module, a key frame acquisition module, a first pose acquisition module, a second pose acquisition module, and a repositioning module. The current frame acquisition module is used to acquire the current frame image taken by the first camera as the first image frame, and the current frame image taken by the second camera as the second image frame; the key frame acquisition module is used to acquire key frame images matching the first image frame and the second image frame from the image word bag, respectively, if the electronic device currently meets the repositioning conditions, to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame, wherein the image word bag includes a plurality of key frame images, and the plurality of key frame images are determined based on the images taken by the first camera and the second camera during the movement of the electronic device along the target path, and each of the key frame images carries the time when the key frame image is taken. The corresponding posture information of the first camera or the second camera; a first posture acquisition module, used to obtain the current posture information of the first camera as the first posture information according to the target posture algorithm, the first image frame, the first key frame and the posture information carried by the first key frame; a second posture acquisition module, used to obtain the current posture information of the second camera as the second posture information according to the target posture algorithm, the second image frame and the second key frame and the posture information carried by the second key frame; a repositioning module, used to determine the current posture information of the electronic device based on the first posture information or the second posture information if the first posture information and the second posture information meet the target posture condition.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the above method.

[0009] In the solution provided by the present application, the current frame images collected by the first camera and the second camera are fully utilized, as well as the key frame images corresponding to the current frame images collected by each camera, and the postures of the first camera and the second camera are calculated respectively. Moreover, after the postures of the two cameras are subjected to a consistency check for mutual constraints, the current posture information of the electronic device is relocated based on the first posture information or the second posture information after the consistency check. That is, the relocation result is verified, which can improve the precision and accuracy of the relocation to a certain extent, thereby improving the reliability of the relocation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of the system architecture of a relocation system 10 provided in an embodiment of the present application is shown.

[0012] Figure 2 A schematic diagram showing the placement angles of the dual cameras according to an embodiment of the present application is shown.

[0013] Figure 3 A schematic flow chart of a relocation method provided in an embodiment of the present application is shown.

[0014] Figure 4 A schematic diagram of the flow of a positioning method provided in an embodiment of the present application is shown.

[0015] Figure 5 A schematic diagram of a map building process provided by an embodiment of the present application is shown.

[0016] Figure 6 A schematic flow chart of a relocation method provided in another embodiment of the present application is shown.

[0017] Figure 7 Shows Figure 6 A flowchart of the sub-steps of step S390 in one implementation manner.

[0018] Figure 8 A schematic diagram showing the walking route of the robot dog during the test scheme of this application.

[0019] Fig. 9 It is a block diagram of a relocation device provided according to an embodiment of the present application.

[0020] Fig.10It is a block diagram of an electronic device according to an embodiment of the present application for executing the relocation method according to an embodiment of the present application.

[0021] Fig.11 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing the relocation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings of the present application, multiple operations that appear in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The sequence numbers of operations such as S110, S120, etc. are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. And, the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0024] At present, robot-related technologies and industries are developing rapidly, and more diverse robot forms and smarter robot functions are gradually entering the lives of modern people. Robot dogs are a new type of robot form, which has strong mobility and can achieve tasks such as security patrols, transporting items, and emotional companionship. At the same time, robot dogs are equipped with a variety of sensors, such as radar, cameras, IMU, ultrasound, depth sensors, etc., and cooperate with various perception, positioning, navigation, and control methods to realize intelligent functions such as autonomous control, perception, and interaction of robot dogs. However, the repositioning of the robot after tracking is lost and repositioning fails is still not accurate enough. Therefore, how to improve the accuracy of repositioning after tracking is lost has become an urgent problem to be solved.

[0025] The inventor proposes a relocation method, device, electronic device and storage medium. The relocation method provided by the embodiment of the present application is described in detail below.

[0026] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture of a repositioning system 10 provided in an embodiment of the present application. The repositioning system 10 is applied to an electronic device 100 , and the electronic device 100 includes at least a first camera 101 and a second camera 102 .

[0027] The electronic device 100 may be a robot, an intelligent monitoring device, an intelligent tracking device, an unmanned driving device, an intelligent mobile device or a remote sensing mobile device, etc., which have autonomous positioning and navigation. For example, the electronic device 100 may be a robot dog, which is a robot form with advantages such as flexibility and strong mobility, and can achieve tasks such as security patrol, transporting items, and emotional companionship. Both the first camera 101 and the second camera 102 are cameras that can simultaneously capture color images and depth images.

[0028] Exemplarily, the first camera 101 and the second camera 102 may be Realsense D455 cameras. The Realsense D455 camera consists of an RGB camera, two infrared (IR) cameras, and an IR transmitter. The RGB camera outputs a color image, and the two IR cameras may output a dense depth map aligned with the color image. The field of view (FOV) of the Realsense D455 camera is 90° horizontally and 65° vertically.

[0029] In such Figure 1In the repositioning system 10 shown, the first camera 101 is a left camera, and the second camera 102 is a right camera. In the following embodiments, the left camera involved can be understood as the first camera 101, and the right camera involved can be understood as the second camera 102. However, it should be understood that "left", "right", "first", and "second" are only exemplary descriptions for distinction. In other embodiments of the present disclosure, the first camera 101 can be a right camera, and the second camera 102 can be a left camera, and the present application does not limit this. In other words, Figure 1 The placement of the camera in the repositioning system 10 shown is only an exemplary description. Depending on the type of electronic device 100 and the camera configuration space, there may be a variety of placement methods, and the present disclosure does not limit this.

[0030] In this application, in order to maximize the superposition of the two camera fields of view, the following can be used: Figure 2 The first camera 101 and the second camera 102 are placed at the angle shown in the figure. For the first camera 101 and the second camera 102 which are both placed vertically, their viewing angles are both 65°, corresponding to Figure 2 When placed, the leftmost sight line of the first camera 110 can be parallel to the rightmost sight line of the second camera 120. At this time, the two cameras can obtain the maximum field of view, that is, 130°, corresponding to Figure 3 There is a narrow common viewing area between the first camera 101 and the second camera 102. According to the above angle design, it can be determined that the angle between the first camera 101 and the second camera 102 is 115°, corresponding to Figure 3 In other words, the first camera 101 and the second camera 102 are placed vertically side by side at an angle of 115°, and the fields of view of the two cameras are 130° in the horizontal direction and 90° in the vertical direction. This achieves the maximum superposition of the fields of view of the two cameras, effectively increasing the field of view of the electronic device 100, providing more sufficient image feature points for the subsequent visual positioning algorithm, thereby improving the accuracy of positioning.

[0031] Furthermore, the first camera 101 and the second camera 102 support multi-camera hardware synchronization, and the first camera 101 and the second camera 102 can be connected by a wire, and the same pulse signal is used to trigger the two cameras to expose simultaneously, thereby realizing hardware synchronization of multiple cameras. After the hardware synchronization setting, the image input into the subsequent positioning algorithm is the image taken at the same time. Thus, the additional error caused by the inconsistent shooting time of multiple cameras is avoided.

[0032] In some embodiments, the electronic device 100 may further include more cameras in addition to the first camera 101 and the second camera 102, and the fields of view of each camera may not overlap as much as possible to obtain a larger field of view. For example, more cameras may be used to obtain a maximum field of view of 360°.

[0033] After placing the first camera 101 and the second camera 102 in the above manner, the two cameras can be calibrated for internal and external parameters respectively, and then used for the positioning algorithm and re-positioning algorithm mentioned in this application. The present disclosure does not limit the camera calibration method.

[0034] In the relocation method disclosed in the present application, the electronic device 100 obtains the current frame image captured by the first camera 101 as the first image frame, and the current frame image captured by the second camera 102 as the second image frame; if the electronic device 100 currently meets the relocation conditions, key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame. The image word bag includes multiple key frame images, and the multiple key frame images are determined based on the images captured by the first camera and the second camera during the movement of the electronic device 100 along the target path. Each key frame image carries the corresponding posture information of the first camera 101 or the second camera 102 when the key frame image is taken; according to the target posture algorithm, the first image frame, the first key frame and the posture information carried by the first key frame, the current posture information of the first camera 101 is obtained as the first posture information; according to the target posture algorithm, the second image frame, the second key frame and the posture information carried by the second key frame, the current posture information of the second camera 102 is obtained as the second posture information; if the first posture information and the second posture information meet the target posture condition, the current posture information of the electronic device 100 is determined based on the first posture information or the second posture information.

[0035] Please refer to Figure 3 , Figure 3 A schematic flow chart of a relocation method provided in an embodiment of the present application is applied to an electronic device, wherein the electronic device includes at least a first camera and a second camera. Figure 3 The relocation method provided in the embodiment of the present application is described in detail. The relocation method may include the following steps:

[0036] Step S210: obtaining a current frame image captured by the first camera as a first image frame, and obtaining a current frame image captured by the second camera as a second image frame.

[0037] In this embodiment, the first image frame is a color image of the current frame taken by the first camera, and the second image frame is a color image of the current frame taken by the second camera. Since the camera continuously takes image frames during the motion process, the relative position relationship between the camera and each object in the image frame is related to the motion trajectory of the camera. Therefore, the position information of the camera can be determined by analyzing the image frames taken continuously by the camera, that is, the positioning of the camera can be realized, and then the real-time positioning and navigation of the movement of the electronic device can be realized.

[0038] In some embodiments, the present application may be based on Figure 4 The positioning method corresponding to the positioning process implements positioning of the electronic device. Among them, the left-eye image frame includes a color image and a depth image, and the right-eye image frame also includes a color image and a depth image. First, the electronic device can use the depth image output by the first camera and the depth image output by the second camera to construct a point cloud. Among them, the three-dimensional space points corresponding to the two depth images can be merged to obtain a point cloud of three-dimensional feature points. Next, the terminal device uses a plane detection algorithm to extract plane information from the point cloud, and filters out a specified plane (such as a ground plane) based on the extracted plane information. Then, the terminal device can calculate the transformation matrix based on the normal vector and gravity vector of the specified plane to achieve alignment of the first camera coordinate system with the world coordinate system. If the specified plane does not exist in the scene, the terminal device can return to the step of obtaining a reference depth image, re-acquire the reference depth image, and perform a judgment process on whether a specified plane exists.

[0039] It can be understood that based on the pre-obtained internal and external parameter calibration results, the transformation matrix between the first camera coordinate system and the second camera coordinate system can be obtained. In this case, the transformation matrix between the second camera coordinate system and the world coordinate system can also be obtained to achieve alignment among the first camera coordinate system, the second camera coordinate system, and the world coordinate system.

[0040] Furthermore, after the alignment of the camera coordinate system and the world coordinate system is completed, positioning can be achieved based on feature extraction and matching of the color image frames of the left and right eyes. Specifically, first, for the positioning initialization stage, the electronic device can determine the posture of the first camera when it initially captures the color image. It should be understood that the posture of the camera when capturing an image referred to in the present disclosure refers to the posture in the world coordinate system. On the one hand, the terminal device can determine the three-dimensional feature points corresponding to the initial frame color image captured by the first camera, and the three-dimensional feature points are feature points in the first camera coordinate system. On the other hand, the initial rotation matrix and the initial translation vector can be set. For example, the initial rotation matrix is ​​a unit matrix, and the initial translation vector is [0,0,0]. After determining the three-dimensional feature points corresponding to the initial frame color image, the initial rotation matrix and the initial translation vector, the positioning initialization in the first camera coordinate system is completed.

[0041] Next, combined with the transformation matrix between the first camera coordinate system and the world coordinate system determined in the coordinate system alignment stage, the positioning initialization result in the first camera coordinate system can be converted into the positioning initialization result in the world coordinate system, that is, the position and posture of the first camera when capturing the initial frame color image is determined.

[0042] Secondly, for the real-time positioning stage, the terminal device can combine the initial pose determined in the positioning initialization stage to obtain the pose of the current frame in real time. In this process, the features of the second camera can be transferred to the coordinate system of the first camera, and the pose can be solved in conjunction with the features of the first camera to complete the pose prediction of the current frame and obtain the pose to be optimized when the first camera captures the color image of the current frame.

[0043] Finally, for the posture optimization stage, the terminal device can optimize the optimized posture when the first camera captures the current frame color image to obtain the target posture. Among them, the optimization can be carried out in a way of tight coupling of left and right eye constraints. Specifically, the matching feature points between the two images in the first color image set are obtained, and the reprojection errors of the matching feature points between the two images in the first color image set are determined. The first color image set is composed of the first image frame captured by the first camera and the first n frames of color images captured by the first camera; the matching feature points between the two images in the second color image set are obtained, and the reprojection errors of the matching feature points between the two images in the second color image set are determined in combination with the transformation matrix between the first camera coordinate system of the first camera and the second camera coordinate system of the second camera. The second color image set is composed of the second image frame captured by the second camera and the first n frames of color images captured by the second camera; based on the reprojection errors of the matching feature points between the two images in the first color image set and the reprojection errors of the matching feature points between the two images in the second color image set, the posture to be optimized when the first camera captures the current frame color image is optimized to determine the target posture when the first camera captures the current frame color image; wherein n is a positive integer. In this way, by establishing error constraints for different cameras respectively and then combining these error constraints to optimize the pose, the accuracy and robustness of positioning can be improved. In addition, the establishment of the error constraints disclosed in the present invention relies on the matching feature points between the current frame and the previous n frames, that is, the scheme takes into account the correlation between adjacent frames, further improving the accuracy and robustness of positioning.

[0044] Step S220: If the electronic device currently meets the repositioning conditions, key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame. The image word bag includes multiple key frame images, and the multiple key frame images are determined based on the images taken by the first camera and the second camera during the movement of the electronic device along the target path. Each of the key frame images carries the corresponding position information of the first camera or the second camera when the key frame image was taken.

[0045] It is understandable that in actual applications, even if the aforementioned positioning methods with high accuracy and robustness are used, various environmental problems may cause electronic devices to lose their positioning during visual positioning. Among them, positioning loss can be understood as when using sensors (such as cameras) for positioning, the robot positioning algorithm fails due to the overly complex environment, and the confidence (the degree of credibility of the algorithm for judging the current position) is too low. In this case, it is believed that the robot has lost its way and cannot continue to move. Robots here include but are not limited to unmanned devices, smart mobile devices, remote sensing mobile devices, etc., and the following reasons for the loss of positioning of electronic devices in actual applications include but are not limited to: the driving wheel slips or idles due to the convolution of obstacles on the ground being too smooth or the driving wheel slips or idles due to obstacles; the current operating environment has changed significantly compared to the environment when the map was constructed; mirror reflections or regular obstacles in narrow spaces interfere; the robot is manually pushed when it is turned off or the battery is exhausted, and it is not at the designated power-on point when it is turned on again.

[0046] In the case of electronic device positioning loss, it will have a great impact on the subsequent autonomous navigation of the electronic device. Therefore, in the positioning process of the electronic device, it is possible to detect in real time whether the electronic device currently meets the repositioning conditions, and then reposition the electronic device in time if it meets the repositioning conditions, so as to ensure the subsequent accurate positioning and movement of the electronic device. Among them, the repositioning conditions include at least one of the following conditions: the positioning of the electronic device based on the first image frame and the second image frame fails; the cumulative positioning error in the process of historical positioning of the electronic device is higher than the preset error.

[0047] In some embodiments, the repositioning condition is the failure to position the electronic device based on the first image frame and the second image frame. Specifically, when the electronic device tracks the historical image frame of the first image frame relative to the previous frame captured by the first camera and the historical image frame of the second image frame relative to the previous frame captured by the second camera, if there are no feature points in the first image frame that match the historical image frame of the previous frame captured by the first camera, or the feature points in the first image frame that match the historical image frame of the previous frame captured by the first camera are less than a first number, or there are no feature points in the second image frame that match the historical image frame of the previous frame captured by the second camera, or the feature points in the second image frame that match the historical image frame of the previous frame captured by the second camera are less than a first number, it is determined that the tracking process of the first image frame relative to the previous frame captured by the first camera and the tracking process of the second image frame relative to the previous frame captured by the second camera has failed, and then it is determined that the electronic device currently meets the repositioning condition.

[0048] In other embodiments, the repositioning condition is that the cumulative positioning error in the process of historical positioning of the electronic device is higher than a preset error. Specifically, the electronic device can determine that when the number of frames between the first image frame or the second image frame and the historical image frame of the last repositioning is greater than a second number, it determines that the cumulative error in the historical tracking process is higher than the preset condition, and further determines that the electronic device currently meets the repositioning condition.

[0049] Based on this, after determining that the electronic device currently meets the relocation conditions, the visual word bag model method can be used to achieve the relocation of the electronic device. Among them, the visual word bag model uses the visual features in the image as "words" and uses the frequency differences of visual words to classify or similarity detect images. It is a technology that uses a visual dictionary to convert a single image into a sparse vector. This method can process a large number of image sets.

[0050] The first key frame is a key frame image with the highest similarity to the first image frame in the image word bag, and the second key frame is a key frame image with the highest similarity to the second image frame in the image word bag.

[0051] Therefore, before step S220, the key frame map (i.e., mapping) and dictionary construction can be performed in advance for the current environment in which the electronic device is located. Specifically, the electronic device is controlled to move according to the target path in the current environment in which it is located, and during the movement, the first camera and the second camera are controlled to shoot the environment image frame according to the target frame rate to obtain multiple environment image frames, and the map is constructed based on the multiple environment image frames. Among them, the target frame rate is the same as the frame rate output by the visual positioning, for example, both are 30 frames per second (Frame Per Second, fps). It can be understood that since the camera is in the process of moving with the electronic device, in general, the similarity of objects in the image frames captured within a period of time is high, and there is generally a large difference between the image frames separated by a period of time, so using all the image frames captured by the camera for visual simultaneous positioning and mapping will generate a lot of redundant calculations. Based on this, the image frames captured by the camera in sequence can be screened, and only some of the image frames are used for visual simultaneous positioning and mapping. The image frames for visual simultaneous positioning and mapping calculation are called key frames, and the key frames characterize the key features of the environment captured by the camera. Each filtered key frame represents some key features of the object captured by the camera. The image frames captured by the camera can be filtered according to some specific rules, and the key frames in the image frames captured by the camera can be determined in turn, and then the camera pose corresponding to each key frame can be determined in turn.

[0052] Specifically, see Figure 5 , Figure 5A schematic diagram of the process of mapping and dictionary construction is shown, when the electronic device is moving in the current environment in which it is located. The environment image frames can be continuously collected by binocular cameras (i.e., the first camera and the second camera) to obtain multiple left-eye image frames and multiple right-eye image frames. At the same time, the IMU sensor continuously collects the posture information of the electronic device; further, the posture of the electronic device determined based on the left-eye image frame and the right-eye image frame can be optimized in combination with the sensor data collected by the IMU, thereby obtaining the posture information of the electronic device (i.e., 6 degrees of freedom). At the same time, the key frame can be determined from the continuously collected left-eye image frames and right-eye image frames; the key frame is determined from the input video stream, and the generated rules are generally related to time, for example, the time interval between two adjacent key frames is greater than the preset duration, and is related to the posture difference of the previous key frame, for example, the posture difference of the posture corresponding to the two adjacent key frames is greater than the specified threshold. According to the set threshold, the key frame generation conditions can be determined, which include but are not limited to one of the following conditions: the acquisition time of the current frame image and the acquisition time of the adjacent key frame are greater than the target duration; the posture difference between the posture corresponding to the current frame image and the posture corresponding to the adjacent key frame is greater than the specified threshold. In this way, it can be ensured that the generated key frames will not be too dense, maintain a certain spacing, and save the memory of the electronic device. Furthermore, the image frames determined to be key frames are saved in the map database.

[0053] Furthermore, based on the multiple keyframes finally obtained and the depth information corresponding to each keyframe, a point cloud map of the current environment can be constructed; wherein, each left-eye keyframe has a corresponding right-eye keyframe. At the same time, the feature points of each keyframe can be extracted. The feature points include key points and descriptors corresponding to the key points, and the image descriptors of the left-eye keyframe and the right-eye keyframe are cascaded and merged into the descriptor of the spliced ​​image, and written into the Bow dictionary. It should be noted that dataBase stores two copies of the keyframe information, one from the left image and the other from the right image. The Bow dictionary is a tree structure, and the written image descriptor is the spliced ​​description information. The Bow dictionary is the image word bag mentioned above.

[0054] Step S230: According to the target pose algorithm, the first image frame, the first key frame and the pose information carried by the first key frame, the current pose information of the first camera is obtained as the first pose information.

[0055] In this embodiment, the pose information carried by the first key frame can be understood as the pose information of the first camera is known when the first camera shoots the first key frame; it can also be understood as the position information of the image feature points on the first key frame in the world coordinate system is known. Therefore, after obtaining the first image frame containing only two-dimensional information, the target pose algorithm, the first image frame, the first key frame and the pose information carried by the first key frame can be used to determine the current pose information of the first camera when the first camera shoots the first image frame.

[0056] Step S240: According to the target pose algorithm, the second image frame, the second key frame and the pose information carried by the second key frame, obtain the current pose information of the second camera as the second pose information.

[0057] In this embodiment, the specific implementation of step S240 is similar to the principle of step S230, and the details can be found in the aforementioned steps, which will not be described again.

[0058] Step S250: If the first posture information and the second posture information meet the target posture condition, the current posture information of the electronic device is determined based on the first posture information or the second posture information.

[0059] In this embodiment, since there is a rigid body transformation relationship between the first camera and the second camera, the current posture information of the second camera can be directly obtained according to the relative position relationship between the first camera and the second camera and the first posture information, that is, the current posture information of the second camera can be inferred as the third posture information. Similarly, the current posture information of the first camera can also be obtained according to the relative position relationship between the first camera and the second camera and the second posture information, that is, the current posture information of the first camera can be inferred as the fourth posture information. Then, based on the posture error between the third posture information and the second posture information, and the posture error between the fourth posture information and the first posture information, it is further determined whether the target posture condition is met.

[0060] Specifically, if the posture error between the third posture information and the second posture information is less than the target threshold, and the posture error between the fourth posture information and the first posture information is less than the target threshold, it indicates that the camera postures of the left eye camera and the right eye camera determined by the current frame image taken by the left eye camera at this time are consistent with the camera postures of the left eye camera and the right eye camera determined by the current frame image taken by the right eye camera, and then it can be determined that the first posture information and the second posture information meet the target posture condition. In the case where the first posture information and the second posture information meet the target posture condition, the current posture information of the electronic device is determined based on the first posture information or the second posture information. That is to say, in the case where the first posture information and the second posture information are both accurate, since the first camera and the second camera are both fixed on the electronic device, the current posture information of the electronic device can be determined according to the first posture information and the relative position relationship between the first camera and the electronic device; or, the current posture information of the electronic device can also be determined according to the second posture information and the relative position relationship between the second camera and the electronic device.

[0061] For example, PoseL represents the first pose information, PoseR represents the second pose information, and the external reference T RL Represents the relative position relationship between the first camera and the second camera. At this time, the third position information PoseR' can be obtained as PoseL*T RL , the fourth pose information PoseL' is PoseR*T RL If the difference between PoseL and PoseL' is less than the aforementioned target threshold, and the difference between PoseR and PoseR' is less than the aforementioned target threshold, it is determined that the first pose information and the second pose information meet the target pose condition.

[0062] Optionally, if the posture error between the third posture information and the second posture information is greater than or equal to the target threshold, or the posture error between the fourth posture information and the first posture information is greater than or equal to the target threshold, it indicates that the camera posture determined based on the current frame image taken by the left camera or the camera posture determined based on the current frame image taken by the right camera is inaccurate, that is, the repositioning effect will be inaccurate at this time. Therefore, it can be determined that the first posture information and the second posture information do not meet the target posture conditions.

[0063] It can be seen that the present application makes full use of the current frame images captured by the first camera and the second camera respectively, as well as the key frame images corresponding to the current frame images captured by each camera, to calculate the poses of the first camera and the second camera respectively, and to perform a mutual constrained consistency check on the poses of the two cameras, that is, to check the repositioning results, so as to improve the reliability of the repositioning results.

[0064] In other embodiments, if the electronic device currently meets the relocation condition and at least one of the key frame images matching the first image frame and the second image frame is not obtained from the image word bag, the electronic device is controlled to rotate the target angle and return to the step of obtaining the current frame image shot by the first camera as the first image frame and the current frame image shot by the second camera as the second image frame, until the key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag. In other words, when the electronic device currently meets the relocation condition and the first key frame or the second key frame cannot be found from the image word bag, the electronic device is controlled to rotate in situ to retake a new first image frame and a new second image frame, and then the corresponding first key frame and the second key frame are re-searched in the image word bag based on the new first image frame and the new second image frame, so as to prepare for the subsequent relocation process of the electronic device. In addition, by controlling the electronic device to rotate in situ, it is possible to avoid the occurrence of problems such as collision or rollover of the electronic device with other objects in the process of retaking the new first image frame and the new second image frame by controlling the electronic device to move when the electronic device is already in a state of lost or inaccurate positioning.

[0065] In some embodiments, in order to further improve the recall rate of key frame search and adapt to changes in the environment relative to the key frame map, the image descriptors stored in the dictionary can be updated regularly. Specifically, in the visual positioning process, key frames need to be continuously generated. New key frames can be inserted into or replace historical key frames. The condition for insertion is that the position difference between the new key frame and the historical key frame is large; the condition for replacement is that the image descriptor difference between the new key frame and the historical key frame is large, but the pose difference is small; through the above two conditions, the key frame can be updated for a long time, and the dataBase memory space composed of the key frame is limited. This method is conducive to adapting to environmental changes and further improving the success rate of key frame image search. In this way, regular updates to the image descriptors stored in the dictionary help improve the accuracy of pose determination during repositioning and positioning.

[0066] In this implementation, the current frame images captured by the first camera and the second camera, as well as the key frame images corresponding to the current frame images captured by each camera, are fully utilized to calculate the poses of the first camera and the second camera, respectively. Furthermore, the poses of the two cameras are subjected to a mutual constraint consistency check, that is, the relocation results are checked, which can improve the precision and accuracy of the relocation to a certain extent, thereby improving the reliability of the relocation results. In addition, the first camera and the second camera placed at a specific angle are used for image acquisition, which has a larger field of view, obtains more sufficient visual features, provides more sufficient and stable features and matching relationships, and provides sufficient constraints for the back end of the positioning algorithm; in the case of less environmental textures or more repeated textures, the recall rate of relocation can be improved compared to the single camera solution.

[0067] Please refer to Figure 6 , Figure 6 A flow chart of a relocation method provided in another embodiment of the present application is applied to an electronic device, wherein the electronic device includes at least a first camera and a second camera. Figure 6 The relocation method provided in the embodiment of the present application is described in detail. The relocation method may include the following steps:

[0068] Step S310: obtaining a current frame image captured by the first camera as a first image frame, and obtaining a current frame image captured by the second camera as a second image frame.

[0069] Step S320: If the electronic device currently meets the repositioning conditions, key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame. The image word bag includes multiple key frame images, and the multiple key frame images are determined based on the images taken by the first camera and the second camera during the movement of the electronic device along the target path. Each of the key frame images carries the corresponding position information of the first camera or the second camera when the key frame image was taken.

[0070] In this embodiment, the specific implementation of step S310 to step S320 can refer to the content of the above-mentioned embodiment, which will not be repeated here.

[0071] Step S330: extracting feature points in the first image frame to obtain a first feature point set.

[0072] In this embodiment, the feature extraction algorithm used for any image frame includes but is not limited to the FAST feature point detection algorithm, the DOG feature point detection algorithm, the Harris feature point detection algorithm, the SIFT feature point detection algorithm, the SURF feature point detection algorithm, etc. Correspondingly, the feature descriptors corresponding to the feature points include but are not limited to the BRIEF feature point descriptor, the BRISK feature point descriptor, the FREAK feature point descriptor, etc.

[0073] It is understandable that different feature point detection algorithms have different detection effects in different texture scenes. Therefore, according to the texture of the scene in actual application, the feature point detection algorithm that matches the current scene can be automatically selected to perform feature point extraction to achieve the best feature point extraction effect. For example, for strong texture scenes, the FAST feature point detection algorithm and the BRIEF feature point descriptor can be used for feature extraction; for weak texture scenes, the DOG feature point detection algorithm and the FREAK feature point descriptor can be used for feature extraction.

[0074] Step S340: Acquire a feature point set corresponding to the first key frame as a second feature point set.

[0075] Step S350: Determine depth information corresponding to each second feature point in the second feature point set based on the current position information of the first camera.

[0076] It can be understood that, since the current position information of the first camera in the world coordinate system is known, the depth information of the second feature point included in the first key frame can be determined.

[0077] Step S360: Acquire feature points in the first feature point set that match feature points in the second feature point set to obtain a plurality of matching feature point pairs.

[0078] In this embodiment, the matching feature points between each pair of images are 2D-2D (two-dimensional-two-dimensional) feature points. The terminal device can use a feature point matching algorithm to determine the matching feature points between the first key frame and the first image frame, wherein the feature point matching algorithm is not limited in this disclosure.

[0079] Step S370: According to the target pose algorithm, the multiple matching feature point pairs, and the depth information corresponding to each second feature point in the multiple matching feature point pairs, obtain the current pose information of the first camera as the first pose information.

[0080] In this embodiment, multiple matching feature point pairs can be input into the RANSAC algorithm for iterative calculation, and the current pose information of the first camera can be calculated using the N-point perspective projection (PnP) algorithm. Specifically, the homography matrix homography of the first image frame relative to the first key frame is calculated, and the homography matrix homography can be decomposed into the rotation matrix Rrelocalize and the translation vector Trelocalize through the decomposition algorithm in the inertial measurement unit (IMU), that is, the current pose information of the first camera when the first image frame is captured.

[0081] Step S380: According to the target pose algorithm, the second image frame, the second key frame and the pose information carried by the second key frame, obtain the current pose information of the second camera as the second pose information.

[0082] In this embodiment, the principle of calculating the second pose information in step S370 is similar to the principle of calculating the first pose information in the aforementioned steps. Please refer to the contents of the aforementioned related steps for calculating the first pose information, which will not be repeated here.

[0083] Step S390: If the first posture information and the second posture information meet the target posture condition, determine the current posture information of the electronic device based on the first posture information or the second posture information.

[0084] In some embodiments, when the first pose information and the second pose information meet the target pose condition, refer to Figure 7 , step S390 may include the following contents from step S391 to step S393:

[0085] Step S391: obtaining a feature point set corresponding to the second image frame as a third feature point set, wherein the third feature point set includes a plurality of feature points extracted from the second image frame.

[0086] In this embodiment, the third feature point set can be extracted in the process of positioning based on the second image frame, and the feature point set corresponding to the previously extracted second image frame can be directly obtained in step S381. Of course, the method of extracting feature points is the same as the method of extracting features for the first image, and the details can be referred to the contents in the aforementioned steps, which will not be repeated here.

[0087] Step S392: Determine, from the first feature point set and the third feature point set, an image frame corresponding to a feature point set whose feature points meet the target distribution condition as a target image frame.

[0088] In some embodiments, a feature point set containing a greater number of feature points than the target number and having the largest distribution variance value corresponding to the feature points can be determined from the first feature point set and the third feature point set as the target feature point set; an image frame corresponding to the target feature point set is obtained as the target image frame. It can be understood that the more feature points there are, the more information in the image they contain; the larger the distribution variance value corresponding to the feature points, the more uniform the distribution of the feature points, and at this time, the feature point matching can more accurately express the geometric relationship of the space. In extreme cases, for example, if all feature points are concentrated at one point, the camera's position and posture cannot be calculated. In other words, if the distribution of feature points is too concentrated, it will affect the accuracy of SLAM; that is, an image frame with more feature points and a more uniform distribution of feature points is selected from the first image frame and the second image frame as the aforementioned target image frame, and the accuracy and precision of the camera's position and posture information calculated based on the target image frame are relatively high.

[0089] Step S393: Acquire the posture information determined based on the target image frame as the current posture information of the electronic device.

[0090] Optionally, if the target image frame is the first image frame, the current posture information of the electronic device is determined according to the relative position relationship between the first camera and the electronic device and the first posture information.

[0091] Optionally, if the target image frame is the second image frame, the current posture information of the electronic device is determined according to the relative position relationship between the second camera and the electronic device and the second posture information.

[0092] See also Figure 8 , taking the electronic device as a robot dog as an example, control the robot dog to walk along a rectangle with a length of 5m and a width of 5m, and turn on the relocation function. Figure 8 Repositioning was performed at the 8 five-pointed star landmarks shown, and at the same time, the actual posture (x, y, θ) was recorded by a caliper. It can be understood that by recording the posture of the repositioning output when the robot dog passed the 8 landmarks and comparing it with the actual posture, the accuracy of the repositioning method disclosed in the present application can be evaluated.

[0093] Eight groups of test data were recorded, and the test results are shown in Table 1.

[0094] Table 1

[0095] mark Relocation output True pose error 1 73cm,30cm,11° 70cm,30cm,5° 3cm / 6° 2 170cm,36cm,8° 172cm,41cm,11° 7cm / 3° 3 323cm,75cm,51° 325cm,74cm,40° 2.5cm / 11° 4 319cm,222cm,147° 321cm,221cm,152° 3cm / 5° 5 138cm,294cm,139° 132cm,291cm,141° 8cm / 2° 6 78cm,431cm,35° 85cm,436cm,39° 11cm / 4° 7 422cm,281cm,189° 424cm,280cm,183° 3cm / 6° 8 268cm,318cm,153° 272cm,321cm,138° 6cm / 5°

[0096] It can be seen from Table 1 that, by using the repositioning method of the embodiment of the present disclosure, the positioning distance accuracy is basically within 0.5%, and the rotation accuracy is within 5° on average. It can be seen that the solution of the embodiment of the present disclosure can achieve the effect of high positioning accuracy.

[0097] In this embodiment, the current frame images collected by the first camera and the second camera are fully utilized, and the key frame images corresponding to the current frame images collected by each camera are respectively calculated, and the poses of the first camera and the second camera are respectively calculated, and the poses of the two cameras are mutually constrained by consistency verification, that is, the relocation results are verified, which can improve the precision and accuracy of the relocation to a certain extent, and thus improve the reliability of the relocation results. In addition, the first camera and the second camera placed at a specific angle are used for image acquisition, which has a larger field of view, obtains more sufficient visual features, provides more sufficient and stable features and matching relationships, and provides sufficient constraints for the back end of the positioning algorithm; in the case of less environmental texture or more repeated textures, the recall rate of relocation can be improved compared with the single camera solution. In addition, according to the distribution of feature points in the two current frame images used to calculate the first pose information and the second pose information, the pose information with relatively higher precision and accuracy is selected from the first pose information and the second pose information to calculate the current pose information of the electronic device, so as to further ensure the accuracy of the relocation result of the electronic device.

[0098] Please refer to Fig. 9 , which shows a structural block diagram of a repositioning device 400 provided by an embodiment of the present application, which is applied to an electronic device, and the electronic device at least includes a first camera and a second camera. The device 400 may include: a current frame acquisition module 410, a key frame acquisition module 420, a first pose acquisition module 430, a second pose acquisition module 440, and a repositioning module 450.

[0099] The current frame acquisition module 410 is used to acquire the current frame image taken by the first camera as a first image frame, and the current frame image taken by the second camera as a second image frame.

[0100] The key frame acquisition module 420 is used to obtain key frame images matching the first image frame and the second image frame from the image word bag if the electronic device currently meets the repositioning conditions, and obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame. The image word bag includes multiple key frame images, and the multiple key frame images are determined based on the images taken by the first camera and the second camera during the movement of the electronic device along the target path. Each of the key frame images carries the corresponding position information of the first camera or the second camera when the key frame image was taken.

[0101] The first pose acquisition module 430 is used to acquire the current pose information of the first camera as the first pose information according to the target pose algorithm, the first image frame, the first key frame and the pose information carried by the first key frame.

[0102] The second posture acquisition module 440 is used to acquire the current posture information of the second camera as the second posture information according to the target posture algorithm, the second image frame, the second key frame and the posture information carried by the second key frame.

[0103] The repositioning module 450 is used to determine the current posture information of the electronic device based on the first posture information or the second posture information if the first posture information and the second posture information meet the target posture condition.

[0104] In some embodiments, the repositioning device 400 may further include: a third posture inference module, a fourth posture inference module, and a posture judgment module. The third posture inference module may be used to obtain the current posture information of the second camera as the third posture information according to the relative position relationship between the first camera and the second camera and the first posture information before determining the current posture information of the electronic device based on the first posture information or the second posture information if the first posture information and the second posture information meet the target posture condition. The fourth posture inference module may be used to obtain the current posture information of the first camera as the fourth posture information according to the relative position relationship between the first camera and the second camera and the second posture information. The posture judgment module may be used to determine that the first posture information and the second posture information meet the target posture condition if the posture error between the third posture information and the second posture information is less than the target threshold, and the posture error between the fourth posture information and the first posture information is less than the target threshold.

[0105] In some embodiments, the first pose acquisition module 430 may include: a first feature point set acquisition unit, a second feature point set acquisition unit, a depth information acquisition unit, a feature point matching unit, and a pose acquisition unit. The first feature point set acquisition unit may be used to extract feature points in the first image frame to obtain a first feature point set. The second feature point set acquisition unit may be used to obtain a feature point set corresponding to the first key frame as a second feature point set. The depth information acquisition unit may be used to determine the depth information corresponding to each second feature point in the second feature point set based on the current pose information of the first camera. The feature point matching unit may be used to obtain feature points in the first feature point set that match those in the second feature point set to obtain a plurality of matching feature point pairs. The pose acquisition unit may be used to obtain the current pose information of the first camera as the first pose information according to the target pose algorithm, the plurality of matching feature point pairs, and the depth information corresponding to each second feature point in the plurality of matching feature point pairs.

[0106] In this manner, the repositioning module 450 may include: a third feature point set acquisition unit, an image frame determination unit, and a repositioning unit. The third feature point set acquisition unit may be used to acquire a feature point set corresponding to the second image frame as a third feature point set, wherein the third feature point set includes a plurality of feature points extracted from the second image frame. The image frame determination unit may be used to determine, from the first feature point set and the third feature point set, an image frame corresponding to a feature point set whose feature points meet a target distribution condition as a target image frame. The repositioning unit may be used to acquire posture information determined based on the target image frame as the current posture information of the electronic device.

[0107] In some embodiments, the image frame determination unit can be specifically used to: determine, from the first feature point set and the third feature point set, a feature point set whose number of feature points is greater than the target number and whose distribution variance value corresponding to the feature points is the largest, as the target feature point set; and obtain an image frame corresponding to the target feature point set as the target image frame.

[0108] In some embodiments, the repositioning module 450 can be specifically used to: if the target image frame is the first image frame, determine the current posture information of the electronic device based on the relative position relationship between the first camera and the electronic device and the first posture information; if the target image frame is the second image frame, determine the current posture information of the electronic device based on the relative position relationship between the second camera and the electronic device and the second posture information.

[0109] In some embodiments, the repositioning condition includes at least one of the following conditions: failure to position the electronic device based on the first image frame and the second image frame; a cumulative positioning error in a historical positioning process for the electronic device is higher than a preset error.

[0110] In some embodiments, the relocation device 400 may further include: a rotation control module. The rotation control module may be used to control the electronic device to rotate to a target angle if the electronic device currently meets the relocation conditions and at least one of the key frame images matching the first image frame and the second image frame is not obtained from the image word bag, and return to the step of obtaining the current frame image captured by the first camera as the first image frame and the current frame image captured by the second camera as the second image frame, until the key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0112] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0113] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0114] In summary, the current frame images collected by the first camera and the second camera, as well as the key frame images corresponding to the current frame images collected by the first camera and the second camera, are fully utilized to calculate the poses of the first camera and the second camera, and the poses of the two cameras are mutually constrained for consistency verification, that is, the relocation results are verified, which can improve the precision and accuracy of the relocation to a certain extent, thereby improving the reliability of the relocation results. In addition, the first camera and the second camera placed at a specific angle are used for image acquisition, which has a larger field of view, obtains more sufficient visual features, provides more sufficient and stable features and matching relationships, and provides sufficient constraints for the back end of the positioning algorithm; in the case of less environmental texture or more repeated textures, the recall rate of relocation can be improved compared with the single camera solution. In addition, according to the distribution of feature points in the two current frame images used to calculate the first pose information and the second pose information, the pose information with relatively higher precision and accuracy is selected from the first pose information and the second pose information to calculate the current pose information of the electronic device, so as to further ensure the accuracy of the relocation results of the electronic device. .

[0115] The following will be combined Fig.10 An electronic device provided by the present application is described.

[0116] Reference Fig.10 , Fig.10 The structural block diagram of an electronic device 500 provided in an embodiment of the present application is shown, and the above method provided in an embodiment of the present application can be executed by the electronic device 500. The electronic device can be an electronic terminal with a data processing function, and the electronic terminal includes but is not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, and smart home devices, etc.; of course, the electronic device can also be a server, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0117] The electronic device 500 in the embodiment of the present application may include one or more of the following components: a processor 501, a memory 502, and one or more applications, wherein the one or more applications may be stored in the memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0118] The processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 502, and calling data stored in the memory 502. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 501 and implemented separately through a communication chip.

[0119] The memory 502 may include a random access memory (RAM) or a read-only memory (ROM). The memory 502 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the electronic device 500 during use.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0121] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0122] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0123] Please refer to Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0124] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in an appropriate form.

[0125] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the steps in the above-mentioned method embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A relocation method, characterized in that: Applied to an electronic device, the electronic device includes at least a first camera and a second camera, and the method includes: Acquire a current frame image captured by the first camera as a first image frame, and acquire a current frame image captured by the second camera as a second image frame; If the electronic device currently meets the relocation condition, key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame, wherein the image word bag includes a plurality of key frame images, and the plurality of key frame images are determined based on images captured by the first camera and the second camera during the movement of the electronic device along the target path, and each of the key frame images carries the position information of the first camera or the second camera corresponding to when the key frame image was captured; According to the target pose algorithm, the first image frame, the first key frame, and the pose information carried by the first key frame, obtaining current pose information of the first camera as first pose information; According to the target pose algorithm, the second image frame, the second key frame, and the pose information carried by the second key frame, obtaining current pose information of the second camera as second pose information; If the first posture information and the second posture information meet the target posture condition, the current posture information of the electronic device is determined based on the first posture information or the second posture information.

2. The method according to claim 1, characterized in that Before determining the current posture information of the electronic device based on the first posture information or the second posture information if the first posture information and the second posture information meet the target posture condition, the method further includes: According to the relative position relationship between the first camera and the second camera and the first posture information, obtaining the current posture information of the second camera as the third posture information; According to the relative position relationship between the first camera and the second camera and the second posture information, obtaining current posture information of the first camera as fourth posture information; If the posture error between the third posture information and the second posture information is less than the target threshold, and the posture error between the fourth posture information and the first posture information is less than the target threshold, it is determined that the first posture information and the second posture information meet the target posture condition.

3. The method according to claim 1, characterized in that The step of obtaining the current pose information of the first camera as the first pose information according to the target pose algorithm, the first image frame, the first key frame, and the pose information carried by the first key frame includes: Extracting feature points from the first image frame to obtain a first feature point set; Acquire a set of feature points corresponding to the first key frame as a second set of feature points; Determine, based on the current position information of the first camera, depth information corresponding to each second feature point in the second feature point set; Acquire feature points in the first feature point set that match those in the second feature point set to obtain a plurality of matching feature point pairs; According to the target pose algorithm, the multiple matching feature point pairs, and the depth information corresponding to each second feature point in the multiple matching feature point pairs, current pose information of the first camera is acquired as the first pose information.

4. The method according to claim 3, characterized in that The determining, based on the first posture information or the second posture information, current posture information of the electronic device includes: Acquire a feature point set corresponding to the second image frame as a third feature point set, wherein the third feature point set includes a plurality of feature points extracted from the second image frame; Determine, from the first feature point set and the third feature point set, an image frame corresponding to a feature point set whose feature points meet a target distribution condition, as a target image frame; Acquire posture information determined based on the target image frame as current posture information of the electronic device.

5. The method according to claim 4, characterized in that The step of determining, from the first feature point set and the third feature point set, an image frame corresponding to a feature point set that contains feature points and meets the target distribution condition as a target image frame includes: From the first feature point set and the third feature point set, determine a feature point set whose number of feature points is greater than the target number and whose distribution variance values ​​corresponding to the feature points are the largest, as the target feature point set; An image frame corresponding to the target feature point set is obtained as the target image frame.

6. The method according to claim 4, characterized in that The determining, based on the first posture information or the second posture information, current posture information of the electronic device includes: If the target image frame is the first image frame, determining the current posture information of the electronic device according to the relative position relationship between the first camera and the electronic device and the first posture information; If the target image frame is the second image frame, the current posture information of the electronic device is determined according to the relative position relationship between the second camera and the electronic device and the second posture information.

7. The method according to any one of claims 1 to 5, characterized in that: The relocation condition includes at least one of the following conditions: Positioning of the electronic device based on the first image frame and the second image frame fails; The accumulated positioning error during the historical positioning of the electronic device is higher than the preset error.

8. The method according to any one of claims 1 to 5, further characterized in that: The method further comprises: If the electronic device currently meets the repositioning conditions and at least one of the key frame images matching the first image frame and the second image frame is not obtained from the image word bag, the electronic device is controlled to rotate the target angle and return to the step of obtaining the current frame image taken by the first camera as the first image frame and the current frame image taken by the second camera as the second image frame until key frame images matching the first image frame and the second image frame are respectively obtained from the image word bag.

9. A repositioning device, characterized in that: Applied to an electronic device, the electronic device includes at least a first camera and a second camera, and the device includes: A current frame acquisition module, used to acquire a current frame image taken by the first camera as a first image frame, and a current frame image taken by the second camera as a second image frame; A key frame acquisition module, used for acquiring key frame images matching the first image frame and the second image frame from the image word bag, respectively, if the electronic device currently meets the relocation condition, to obtain a first key frame corresponding to the first image frame and a second key frame corresponding to the second image frame, wherein the image word bag includes a plurality of key frame images, and the plurality of key frame images are determined based on images captured by the first camera and the second camera during the movement of the electronic device along the target path, and each of the key frame images carries the position information of the first camera or the second camera corresponding to when the key frame image was captured; A first pose acquisition module, used to acquire current pose information of the first camera as first pose information according to a target pose algorithm, the first image frame, the first key frame and pose information carried by the first key frame; A second posture acquisition module, configured to acquire current posture information of the second camera as second posture information according to the target posture algorithm, the second image frame, the second key frame, and the posture information carried by the second key frame; A repositioning module is used to determine the current posture information of the electronic device based on the first posture information or the second posture information if the first posture information and the second posture information meet the target posture condition.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 8.

Images