Method for executing repositioning, positioning device and computer readable storage medium

Abstract

The embodiment of the invention provides a method for executing repositioning, a positioning device and a computer readable storage medium. The method comprises the following steps: executing SLAM (Simultaneous Localization and Mapping) to obtain a plurality of first key frames, and sensing first environment sensing information corresponding to each first key frame when each first key frame is obtained; when the positioning device executes the repositioning mechanism, sensing current environment sensing information, and at least comparing the current environment sensing information with the first environment sensing information corresponding to each first key frame to find out at least one second key frame from the plurality of first key frames; and performing a relocation mechanism based on the at least one second key frame.

Description

Technical Field

[0001] This invention relates to a positioning mechanism, and more particularly to a method, positioning device, and computer-readable storage medium for performing repositioning. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a technology that enables a positioning device to simultaneously locate itself and build a map of the environment in an unknown environment. However, as the exploration range of the positioning device expands, the size of the SLAM map built by the positioning device also continues to grow.

[0003] In SLAM technology, relocation is a crucial concept. It refers to the ability of a positioning device to reposition itself in the environment by comparing its location with a previously established map after losing tracking. However, large SLAM maps can lead to problems such as prolonged relocation time, increased memory usage, and higher computational resource requirements. This is because performing feature matching and localization calculations on massive map data consumes significant computational resources. Summary of the Invention

[0004] In view of the above, the present invention provides a method, a positioning device, and a computer-readable storage medium for performing repositioning, which can be used to solve the above-mentioned technical problems.

[0005] This invention provides a method for performing relocation, executed by a positioning device applying Simultaneous Localization and Mapping (SLAM), comprising: the positioning device performing SLAM to acquire a plurality of first keyframes, and sensing first environmental sensing information corresponding to each first keyframe when acquiring each first keyframe; when the positioning device performs a relocation mechanism, the positioning device sensing current environmental sensing information, and identifying at least one second keyframe among the plurality of first keyframes by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first keyframe; and the positioning device performing the relocation mechanism based on the at least one second keyframe.

[0006] This invention provides a positioning device applying Simultaneous Localization and Mapping (SLAM), including a storage circuit and a processor. The storage circuit stores program code. The processor is connected to the storage circuit and accesses the program code to execute: performing SLAM to acquire a plurality of first keyframes, and sensing first environmental sensing information corresponding to each first keyframe when acquiring each first keyframe; when executing a repositioning mechanism, the positioning device senses the current environmental sensing information, and at least finds at least one second keyframe among the plurality of first keyframes by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first keyframe; and performing the repositioning mechanism based on the at least one second keyframe.

[0007] This invention provides a computer-readable storage medium recording an executable computer program. The executable computer program is loaded by a localization device applying Simultaneous Localization and Mapping (SLAM) to perform the following steps: performing SLAM to acquire a plurality of first keyframes, and sensing first environmental sensing information corresponding to each first keyframe upon acquisition of each first keyframe; when performing a relocalization mechanism, the localization device senses current environmental sensing information and identifies at least one second keyframe among the plurality of first keyframes by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first keyframe; and performing the relocalization mechanism based on the at least one second keyframe. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of a positioning device according to an embodiment of the present invention.

[0009] Figure 2 This is a flowchart illustrating a method for performing relocation according to an embodiment of the present invention. Detailed Implementation

[0010] Please refer to Figure 1 This is a schematic diagram of a positioning device according to an embodiment of the present invention. In different embodiments, the positioning device 100 may be implemented as various devices applicable to SLAM, such as service robots, industrial robots, drones, autonomous vehicles, head-mounted displays (HMDs) for providing reality services (e.g., augmented reality, virtual reality, mixed reality, etc.), mobile phones, etc., but is not limited thereto.

[0011] exist Figure 1 In the positioning device 100, there are storage circuits 102 and processors 104.

[0012] The storage circuit 102 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record multiple program codes or modules.

[0013] The processor 104 is coupled to the storage circuit 102 and may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, an ARM-based processor, and the like.

[0014] In an embodiment of the present invention, the processor 104 may access the modules and program code recorded in the storage circuit 102 to implement the execution relocation method proposed in the present invention, the details of which are described below.

[0015] Please refer to Figure 2 This is a flowchart illustrating a method for performing relocation according to an embodiment of the present invention. The method of this embodiment can be performed by... Figure 1 The positioning device 100 is executed, and the following is the matching Figure 1 Component description shown Figure 2 Details of each step.

[0016] In step S210, the processor 104 performs SLAM to acquire multiple first keyframes and senses first environmental sensing information corresponding to each first keyframe when acquiring each first keyframe.

[0017] Generally, keyframes in SLAM technology are a selection of representative image frames selected and stored by the SLAM system. These image frames contain rich environmental feature information and can be used to record the observations of a camera (e.g., tracking) or localization device at a specific location. Each keyframe is associated with the camera's pose (which can be represented as a six-degree-of-freedom system) and extracted map points. Keyframes are not necessarily image frames at every moment, but rather image frames selected according to a certain strategy. For example, when the camera pose and / or the pose of the localization device 100 changes significantly, or when environmental features change dramatically, the corresponding image frame may be recorded as a keyframe. These keyframes constitute the map in the SLAM system and are used in subsequent localization processes to help the system re-localize and recover the pose of the localization device 100.

[0018] Therefore, each first keyframe in step S210 can be understood as a keyframe collected during the SLAM process performed by the positioning device 100, but it is not limited to this.

[0019] In one embodiment, assuming that the positioning device 100 performs SLAM in a specific field, the positioning device 100 may capture image frames with its camera (e.g., a tracking camera) as it moves around in this specific field, and a portion of these image frames may be recorded as first keyframes, but is not limited thereto.

[0020] In an embodiment of the present invention, the processor 104 may sense first environmental sensing information corresponding to each first keyframe when acquiring each first keyframe.

[0021] In an embodiment of the present invention, the first environmental sensing information corresponding to each first key frame includes at least one of the long-range force information and ambient light information sensed by the positioning device 100 when acquiring each first key frame.

[0022] In one embodiment, the long-range force information may include at least one of the magnetic direction, magnetic intensity, gravitational direction, and long-range force angle, wherein the long-range force angle includes the angle between the magnetic direction and the gravitational direction.

[0023] In one embodiment, the ambient light information may include ambient brightness and exposure parameters.

[0024] In one embodiment, when the processor 104 acquires an image frame captured by the camera of the aforementioned specific field, it controls the relevant environmental sensors to sense the corresponding environmental sensing information. Subsequently, if the captured image frame is determined to be one of the first keyframes (hereinafter referred to as the first frame), the processor 104 may further determine the environmental sensing information corresponding to the first frame as the corresponding first environmental sensing information, but this is not limited to this.

[0025] In different embodiments, the aforementioned environmental sensor may be coupled to the processor 104, and may be implemented as at least one of, for example, an electronic compass, an inertial measurement unit (IMU), or an optical sensor, but is not limited thereto.

[0026] In embodiments where the environmental sensor includes an electronic compass, the environmental sensing information corresponding to the first frame includes, for example, magnetic direction and / or magnetic intensity, wherein the magnetic direction is, for example, the direction of the magnetic field measured by the electronic compass, and the magnetic intensity is, for example, the magnetic field intensity measured by the electronic compass.

[0027] In embodiments where the environmental sensor includes an electronic compass and an inertial measurement unit, the environmental sensing information corresponding to the first frame includes, for example, at least one of the magnetic direction, magnetic intensity, gravitational direction, and distance force angle. In one embodiment, the distance force angle is, for example, an angle between one of the magnetic direction and the gravitational direction and the other of the magnetic direction and the gravitational direction.

[0028] In embodiments where the environmental sensor includes a light sensor, the environmental sensing information corresponding to the first frame includes, for example, ambient brightness and exposure parameters.

[0029] In embodiments where the environmental sensor includes an electronic compass, an inertial measurement unit, and an optical sensor, the environmental sensing information corresponding to the first frame includes, for example, at least one of the following: magnetic direction, magnetic intensity, gravity direction, distance force angle, ambient brightness, and exposure parameters.

[0030] In other embodiments, the environmental sensor may also include sensors with other environmental sensing functions, and is not limited to the elements mentioned above.

[0031] In step S220, when the relocation mechanism is executed, the processor 104 senses the current environment sensing information and finds at least one second keyframe among the multiple first keyframes by comparing the current environment sensing information with the first environment sensing information corresponding to each first keyframe.

[0032] In some embodiments, when a device using a SLAM system loses tracking for some reason and is unable to determine its current pose, the device can perform a relocalization mechanism. Generally, when performing a relocalization mechanism, the device matches the currently observed image frame (hereinafter referred to as the current frame) with recorded keyframes. Specifically, the device extracts feature points from the current frame and uses feature matching techniques to find keyframes with similar feature points. Then, the device can calculate the pose transformation of the current camera relative to the matched keyframes based on the geometric relationships between these matched feature points. In this way, the device can re-determine its current accurate position based on the location of the keyframes, thereby restoring localization and continuing normal operation.

[0033] However, as the SLAM running time of the positioning device 100 increases, the number of recorded first keyframes will increase accordingly. In this case, when the positioning device 100 executes the repositioning mechanism, it needs to compare the feature points of the current frame with a large number of first keyframes, which may reduce the efficiency of SLAM.

[0034] Therefore, instead of directly comparing the current frame with the plurality of first keyframes, the processor 104 can first filter out at least one second keyframe from the first keyframes through step S220, and perform a relocation mechanism based on the smaller number of the at least one second keyframe. This improves the efficiency of SLAM.

[0035] Specifically, when the positioning device 100 needs to perform a repositioning mechanism, the processor 104 may, for example, control the environmental sensor to sense the current environmental sensing information as the current environmental sensing information considered in step S220.

[0036] In different embodiments, the current environmental sensing information may include at least one of the following, depending on the state of the environmental sensor: magnetic direction, magnetic intensity, gravity direction, distance force angle, ambient brightness, and exposure parameters, but is not limited to these.

[0037] After obtaining the current environment sensing information, the processor 104 can find at least one second keyframe among the multiple first keyframes by comparing the current environment sensing information with the first environment sensing information corresponding to each first keyframe.

[0038] In one embodiment, in response to determining that the first environmental sensing information corresponding to the first frame in the first keyframe matches the current environmental sensing information, the processor 104 may determine that the first frame belongs to the at least one second keyframe. On the other hand, in response to determining that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information, the processor 104 may determine that the first frame does not belong to the at least one second keyframe.

[0039] From another perspective, determining that the first frame belongs to the at least one second keyframe can be understood as determining that the first frame is one of the at least one second keyframe, that is, the first frame can be used to execute the subsequent relocation mechanism. Conversely, determining that the first frame does not belong to the at least one second keyframe can be understood as determining that the first frame is not one of the at least one second keyframe, that is, the first frame will not be used to execute the subsequent relocation mechanism, but it is not limited to this.

[0040] In one embodiment, in response to determining that the difference between the first environmental sensing information corresponding to the first frame and the current environmental sensing information is within the error range, the processor 104 may determine that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. On the other hand, in response to determining that the difference between the first environmental sensing information corresponding to the first frame and the current environmental sensing information is not within the error range, the processor 104 may determine that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

[0041] For example, suppose the first environmental sensing information corresponding to the first frame is the magnetic direction (hereinafter referred to as A1) at the time the first frame was acquired, and the current environmental sensing information is the magnetic direction (hereinafter referred to as B1) currently sensed by the positioning device 100. In this case, if the difference between the magnetic direction A1 and the magnetic direction B1 (e.g., the angle between the magnetic direction A1 and the magnetic direction B1) is within the error range (e.g., a preset angle range), then the processor 104 can, for example, determine that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. Conversely, if the difference between the magnetic direction A1 and the magnetic direction B1 is not within the error range, then the processor 104 can, for example, determine that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

[0042] For another example, suppose the first environmental sensing information corresponding to the first frame is the magnetic field strength at the time the first frame was acquired (hereinafter referred to as A2), and the current environmental sensing information is the magnetic field strength currently sensed by the positioning device 100 (hereinafter referred to as B2). In this case, if the difference between magnetic field strength A2 and magnetic field strength B2 (e.g., the intensity difference between magnetic field strength A2 and magnetic field strength B2) is within the error range (e.g., a preset intensity range), then the processor 104 may determine, for example, that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. Conversely, if the difference between magnetic field strength A2 and magnetic field strength B2 is not within the error range, then the processor 104 may determine, for example, that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

[0043] For another example, suppose the first environmental sensing information corresponding to the first frame is the range force angle (hereinafter referred to as A3) at the time the first frame was acquired, and the current environmental sensing information is the range force angle (hereinafter referred to as B3) currently sensed by the positioning device 100. In this case, if the difference between the range force angle A3 and the range force angle B3 (e.g., the angular difference between the range force angle A3 and the range force angle B3) is within the error range (e.g., a certain preset angular range), then the processor 104 may determine, for example, that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. Conversely, if the difference between the range force angle A3 and the range force angle B3 is not within the error range, then the processor 104 may determine, for example, that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

[0044] For other types of environmental sensing information (such as ambient brightness, exposure parameters, etc.), those skilled in the art should be able to deduce the corresponding matching determination methods based on the above description, which will not be elaborated here.

[0045] In some embodiments, assuming that the environmental sensing information under consideration includes multiple states, the processor 104 may first perform weighting operations based on these states to obtain corresponding values ​​(which can be understood as scores), and then make the matching determination based on these values.

[0046] For example, suppose the first environmental sensing information corresponding to the first frame is the magnetic direction A1, magnetic intensity A2, and distance force angle A3 at the time the first frame was acquired, and the current environmental sensing information is the magnetic direction B1, magnetic intensity B2, and distance force angle B3 sensed by the positioning device 100 at the moment. In this case, the processor 104 may, for example, perform a weighted calculation based on the magnetic direction A1, magnetic intensity A2, and distance force angle A3 to determine the corresponding value (hereinafter referred to as value N1), and then perform the same weighted calculation based on the magnetic direction B1, magnetic intensity B2, and distance force angle B3 to determine the corresponding value (hereinafter referred to as value N2). Then, if the difference between value N1 and value N2 is within the error range (e.g., a preset value range), the processor 104 may, for example, determine that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. Conversely, if the difference between value N1 and value N2 is not within the error range, the processor 104 may, for example, determine that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

[0047] When the environmental sensing information under consideration includes other combinations of states, those skilled in the art should be able to deduce the corresponding matching determination method based on the above description, which will not be elaborated here.

[0048] In step S230, the processor 104 performs a repositioning mechanism based on the at least one second keyframe.

[0049] In one embodiment, after capturing a current frame corresponding to current environment sensing information, the processor 104 may determine whether one of the at least one second keyframe (hereinafter referred to as the second frame) matches the current frame.

[0050] In one embodiment, the processor 104 may determine the similarity between a plurality of first feature points in the second frame and a plurality of reference feature points in the current frame. In response to determining that the similarity is higher than a similarity threshold, the processor 104 may determine that the second frame matches the current frame. Conversely, in response to determining that the similarity is not higher than the similarity threshold, the processor 104 may determine that the second frame does not match the current frame.

[0051] In different embodiments, the processor 104 may determine the similarity based on, for example, Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Random Sample Consensus (RANSAC), or other similar feature matching algorithms, but is not limited thereto.

[0052] In one embodiment, in response to determining that the second frame matches the current frame, the processor 104 can re-determine the current accurate position of the positioning device 100 based on the position of the second frame, thereby restoring positioning and continuing normal operation.

[0053] As can be seen from the above, since the number of second keyframes obtained after screening is less than that of the first keyframes, the processor 104 can find the second frame that matches the current frame with better efficiency. This improves the efficiency of the positioning device 100 in executing the repositioning mechanism.

[0054] In some implementations, the method proposed in this invention can provide greater efficiency improvement when the area of ​​a specific field is large (e.g., not less than 900 square meters) and / or the number of first keyframes is large (e.g., not less than 300).

[0055] On the other hand, in response to determining that the second frame does not match the current frame, the processor 104 may continue to search for other frames that match the current frame in other second key frames, but may not be limited to this.

[0056] The present invention further provides a computer-readable storage medium. This computer-readable storage medium comprises a plurality of program instructions (e.g., setup program instructions and deployment program instructions) embodied therein. These program instructions can be loaded into and executed by the positioning device 100 to perform the methods described above and the functions of the positioning device 100.

[0057] In summary, the technical solution proposed in this invention can filter out a smaller number of second keyframes from the first keyframes by comparing environmental sensing information, and then perform a repositioning mechanism based on the second keyframes. This reduces the memory and computational load required to execute the repositioning mechanism, thereby improving the efficiency of repositioning.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for performing relocalization, performed by a localization device applying Simultaneous Localization and Mapping (SLAM), characterized in that, include: The positioning device performs SLAM to acquire multiple first keyframes, and senses first environmental sensing information corresponding to each first keyframe when acquiring each first keyframe. When the positioning device performs the repositioning mechanism, the positioning device senses the current environmental sensing information and at least finds at least one second key frame among the plurality of first key frames by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first key frame. as well as The repositioning mechanism is performed by the positioning device based on the at least one second keyframe.

2. The method according to claim 1, wherein the first environmental sensing information corresponding to each of the first keyframes includes at least one of the long-range force information and ambient light information sensed by the positioning device when acquiring each of the first keyframes.

3. The method according to claim 2, wherein the telescopic force information includes at least one of magnetic direction, magnetic intensity, gravitational direction and telescopic force angle, wherein the telescopic force angle includes the angle between the magnetic direction and the gravitational direction.

4. The method according to claim 2, wherein the ambient light information includes ambient brightness and exposure parameters.

5. The method according to claim 1, wherein the plurality of first keyframes are associated with a specific field where the positioning device is located, and the area of ​​the specific field is not less than 900 square meters.

6. The method according to claim 1, wherein the number of the plurality of first keyframes is not less than 300.

7. The method of claim 1, wherein sensing the current environmental sensing information comprises: The current environmental information is sensed at least through the environmental sensors of the positioning device.

8. The method of claim 1, wherein finding at least one second keyframe among the plurality of first keyframes by comparing the current environment sensing information with the first environment sensing information corresponding to each of the first keyframes comprises: In response to determining that the first environmental sensing information corresponding to the first frame among the plurality of first keyframes matches the current environmental sensing information, it is determined that the first frame belongs to the at least one second keyframe. as well as In response to determining that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information, it is determined that the first frame does not belong to the at least one second key frame.

9. The method of claim 8, comprising: In response to determining that the difference between the first environmental sensing information corresponding to the first frame and the current environmental sensing information is within the error range, it is determined that the first environmental sensing information corresponding to the first frame matches the current environmental sensing information. as well as In response to determining that the difference between the first environmental sensing information corresponding to the first frame and the current environmental sensing information is not within the error range, it is determined that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information.

10. A positioning device applying Simultaneous Localization and Mapping (SLAM), characterized in that, include: Storage circuitry that stores program code; The processor, which is coupled to the memory circuitry and accesses the program code to execute it, SLAM is performed to acquire multiple first keyframes, and first environmental sensing information corresponding to each first keyframe is sensed when each first keyframe is acquired. When the repositioning mechanism is executed, the positioning device senses the current environmental sensing information and at least finds at least one second keyframe among the plurality of first keyframes by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first keyframe. as well as The repositioning mechanism is based on at least one second keyframe.

11. The positioning device according to claim 10, wherein the first environmental sensing information corresponding to each of the first keyframes includes at least one of the long-range force information and ambient light information sensed by the positioning device when acquiring each of the first keyframes.

12. The positioning device according to claim 11, wherein the distance force information includes at least one of magnetic direction, magnetic intensity, gravity direction and distance force angle, wherein the distance force angle includes the angle between the magnetic direction and the gravity direction.

13. The positioning device according to claim 11, wherein the ambient light information includes ambient brightness and exposure parameters.

14. The positioning device according to claim 10, wherein the plurality of first keyframes are associated with a specific field where the positioning device is located, and the area of ​​the specific field is not less than 900 square meters, or the number of the plurality of first keyframes is not less than 300.

15. The positioning device of claim 10, further comprising an environmental sensor coupled to the processor, wherein the processor is configured to perform: The current environmental information is sensed at least by the environmental sensor of the positioning device.

16. The positioning device of claim 10, wherein the processor is configured to perform: In response to determining that the first environmental sensing information corresponding to the first frame among the plurality of first keyframes matches the current environmental sensing information, it is determined that the first frame belongs to the at least one second keyframe; and In response to determining that the first environmental sensing information corresponding to the first frame does not match the current environmental sensing information, it is determined that the first frame does not belong to the at least one second key frame.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium records an executable computer program, which is loaded by a positioning device applying Simultaneous Localization and Mapping (SLAM) to perform the following steps: SLAM is performed to acquire multiple first keyframes, and first environmental sensing information corresponding to each first keyframe is sensed when each first keyframe is acquired. When the repositioning mechanism is executed, the positioning device senses the current environmental sensing information and at least finds at least one second keyframe among the plurality of first keyframes by comparing the current environmental sensing information with the first environmental sensing information corresponding to each first keyframe. as well as The repositioning mechanism is based on at least one second keyframe.

Images