Map updating method, electronic device and non-transient computer readable storage medium
By performing image recognition and map point classification transformation on keyframes, inferred map points are generated to update the map, which solves the problems of low efficiency and insufficient accuracy in the extraction of distant object features in existing technologies, and improves the efficiency and accuracy of map construction.
Patent Information
- Application Number
- CN202411171166.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-07
- Filing Date
- 2024-08-26
- Publication Date
- 2025-11-07
AI Technical Summary
Existing simultaneous localization and mapping (SMR) systems are inefficient and lack accuracy in extracting features from distant objects, resulting in low efficiency and accuracy in map building.
By performing image recognition on keyframes to determine whether they contain repetitive graphics, multiple corresponding map points are obtained. These map points are then classified and transformed to generate inferred map points to update the map.
It improves the efficiency and accuracy of map building and reduces the number of times and time required to sense the environment.
Smart Images
Figure CN120910168A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to a method, and particularly, to a map updating method for an electronic device. BACKGROUND
[0002] In existing systems using simultaneous localization and mapping (SLAM) technique, image quality is quite crucial for map construction. However, image quality is practically limited by the accuracy of the camera itself. Due to the limitation, these systems are more difficult to accurately obtain features of distant objects when applied to a wide environment, which results in low efficiency and low accuracy of map construction. Therefore, it is important to propose a new map construction method with high efficiency and high accuracy. SUMMARY
[0003] One aspect of the present disclosure is a map updating method. The map updating method is for an electronic device. The electronic device is used to establish a map of an environment. The map updating method comprises: performing image recognition on a key frame to determine whether the key frame contains a repetitive pattern; obtaining a plurality of map points corresponding to the repetitive pattern; generating a plurality of estimated map points according to the plurality of map points; and updating the map according to the plurality of estimated map points.
[0004] In some embodiments, obtaining the plurality of map points corresponding to the repetitive pattern comprises: mapping a plurality of existing map points of the map to a plurality of feature points extracted from the key frame according to a pose data corresponding to the key frame; and using the plurality of existing map points matching the plurality of feature points extracted from the repetitive pattern as the plurality of map points.
[0005] In some embodiments, generating the plurality of estimated map points according to the plurality of map points comprises: classifying the plurality of map points according to a basic repeating unit of the repetitive pattern to obtain a transformation data; and transforming at least one of the plurality of map points corresponding to the basic repeating unit by the transformation data to generate at least one of the plurality of estimated map points.
[0006] In some embodiments, classifying the plurality of map points according to the basic repeating unit of the repetitive pattern to obtain the transformation data comprises: classifying the plurality of map points into a plurality of map point groups according to a number of the basic repeating unit; and calculating the transformation data by performing a transformation operation between two adjacent map point groups in the plurality of map point groups.
[0007] In some embodiments, the number of map points in one of the two adjacent map point groups is greater than the number of map points in the other of the two adjacent map point groups.
[0008] In some embodiments, transforming, by the transformation data, the at least one of the plurality of map points corresponding to the basic repeating unit to generate the at least one of the plurality of putative map points comprises multiplying map points in a first group of map points in the plurality of map points by the transformation data to generate putative map points in a second group of map points in the plurality of map points, wherein the first group of map points is directly adjacent to the second group of map points.
[0009] In some embodiments, the number of putative map points in the second group of map points is greater than or equal to the number of map points in the second group of map points.
[0010] In some embodiments, the sum of the number of putative map points in the second group of map points and the number of map points in the second group of map points is equal to the number of map points in a reference group of map points in the plurality of map points.
[0011] In some embodiments, updating the map in accordance with the plurality of putative map points comprises adding one of the plurality of putative map points to the map when none of the plurality of map points is close to a spatial coordinate of the one of the plurality of putative map points.
[0012] In some embodiments, the map updating method further comprises adding a new feature point to a planar coordinate on the key frame, wherein the one of the plurality of putative map points is projected to the planar coordinate on the key frame.
[0013] In some embodiments, updating the map in accordance with the plurality of putative map points comprises adjusting one of the plurality of map points to a spatial coordinate of one of the plurality of putative map points when the one of the plurality of map points is close to the spatial coordinate.
[0014] In some embodiments, the map updating method further comprises adjusting a feature point to a planar coordinate on the key frame, wherein the one of the plurality of map points at the spatial coordinate is mapped to the feature point at the planar coordinate.
[0015] In some embodiments, when the key frame does not contain the repeating pattern, the map updating method further comprises updating the map using the key frame; and obtaining another key frame.
[0016] Another aspect of the disclosure is an electronic device. The electronic device is configured to build a map of an environment and includes a camera and a processor. The camera is configured to capture at least one image of the environment. The processor is coupled to the camera and configured to: obtain a keyframe from the at least one image; perform image recognition on the keyframe to determine whether the keyframe includes a repetitive pattern; obtain a plurality of map points corresponding to the repetitive pattern; generate a plurality of tentative map points based on the plurality of map points; and update the map based on the plurality of tentative map points.
[0017] Another aspect of the disclosure is a non-transitory computer-readable storage medium having a computer program configured to perform a map updating method, wherein the map updating method is applicable to an electronic device configured to build a map of an environment, and the map updating method includes: performing image recognition on a keyframe to determine whether the keyframe includes a repetitive pattern; obtaining a plurality of map points corresponding to the repetitive pattern; generating a plurality of tentative map points based on the plurality of map points; and updating the map based on the plurality of tentative map points.
[0018] In summary, by obtaining map points corresponding to a repetitive pattern in a keyframe, the electronic device of the disclosure can generate tentative map points to patch a set of map points that are inaccurate and / or incomplete due to limitations of the camera itself. In addition, the tentative map points help the electronic device to reduce the number and time of sensing the environment for map building. Therefore, the electronic device and the map updating method have advantages of improving the efficiency and accuracy of map building. BRIEF DESCRIPTION OF DRAWINGS
[0019] Figure 1 A block diagram of an electronic device according to some embodiments of the disclosure.
[0020] Figure 2 A flowchart of a map updating method according to some embodiments of the disclosure.
[0021] Figure 3 A schematic diagram of a keyframe having a repetitive pattern according to some embodiments of the disclosure.
[0022] Figure 4 A flowchart of an operation of a map updating method according to some embodiments of the disclosure.
[0023] Figure 5 A schematic diagram of mapping existing map points to a keyframe according to some embodiments of the disclosure.
[0024] Figure 6Flowchart of an operation of a map updating method according to some embodiments of the present disclosure.
[0025] Figure 7 Diagram of classifying map points corresponding to repetitive patterns according to some embodiments of the present disclosure.
[0026] Figure 8 Diagram of generating a tentative map point from a map point corresponding to a repetitive pattern according to some embodiments of the present disclosure.
[0027] Figure 9 Flowchart of an operation of a map updating method according to some embodiments of the present disclosure.
[0028] Figure 10 Diagram of generating a tentative map point from a map point corresponding to a repetitive pattern according to some embodiments of the present disclosure.
[0029] Legend:
[0030] 100: electronic device
[0031] 101: processor
[0032] 103: camera
[0033] 105: storage device
[0034] 200: map updating method
[0035] IKF: key frame
[0036] IMG: image
[0037] M1: map
[0038] MP: map point
[0039] MPE: existing map point
[0040] MPI: tentative map point
[0041] RP: repetitive pattern
[0042] RU: basic repetitive unit
[0043] S201-S204: operation
[0044] S401-S402, S601-S603, S901-S903: sub-operation
[0045] TD: transformation data DETAILED DESCRIPTION
[0046] The following detailed description is made with reference to the accompanying drawings, of which the embodiments described are only to explain the present disclosure and are not intended to limit the present disclosure, and the description of the structure and operation is not intended to limit the order of execution, and any structure recombined by elements, the device generated has equal efficiency, is covered by the scope of the present disclosure.
[0047] As used herein, "coupled" or "connected" can mean two or more elements are in direct physical or electrical contact with each other, or can mean that two or more elements are not in direct contact with each other, but can still cooperate or interact with each other.
[0048] Referring to Figure 1 , Figure 1 is a schematic diagram of an electronic device 100 according to some embodiments of the present disclosure. In some embodiments, the electronic device 100 is capable of sensing an environment. Specifically, the environment can be a public place (e.g., a station, a game place, a restaurant, etc.) or a private place (e.g., a workplace, a residence, etc.). Also, the electronic device 100 can be implemented by a wearable device (e.g., a head-mounted device) worn by a user, a vehicle (e.g., a car) operated by the user, or an autonomous mobile robot (e.g., a cleaning robot, a sweeping robot, a food delivery robot, etc.), which means that the electronic device 100 can move to different spaces in the environment. In this way, the electronic device 100 can sense different spaces in the environment, thereby building a map M1 of the environment and positioning itself in the map M1 of the environment.
[0049] In some embodiments, as shown in Figure 1 , the electronic device 100 includes a processor 101, a camera 103, and a storage device 105. The processor 101 is communicatively and electrically coupled to the camera 103 and the storage device 105, thereby cooperating with the camera 103 and the storage device 105. For example, the camera 103 is used to capture at least one image IMG of the environment. By receiving the image IMG of the environment from the camera 103, the processor 101 can use at least one vision-based positioning technology (e.g., a simultaneous localization and mapping technology, etc.) to perform some operations, such as calculating a pose (i.e., an orientation and a position) of the electronic device 100, building a map M1 of the environment, positioning the electronic device 100 in the map M1 of the environment, etc. In addition, the storage device 105 can store signals, data, and / or information (e.g., the image IMG, the pose of the electronic device 100, the map M1, etc.) required by the processor 101 to perform the some operations.
[0050] According to the above embodiments, by using the vision-based positioning technique, the processor 101 can obtain a key frame IKF from the images IMG captured by the camera 103. Specifically, when an image IMG is separated from a previous / latest key frame (not shown in the figure) by a predetermined number of frames or more, the processor 101 can determine that the image IMG is a candidate for a key frame IKF. Further, the processor 101 can perform feature extraction on the image IMG to determine whether the number of common feature points between the image IMG and the previous key frame is equal to or less than a predetermined threshold value. When the number of common feature points between the image IMG and the previous key frame is equal to or less than the predetermined threshold value, the processor 101 sets the image IMG as the key frame IKF and stores the key frame IKF in the storage device 105 as shown in Figure 1 The processor 101 can use the key frame IKF to update the map Ml and calculate the pose of the electronic device 100 corresponding to the key frame IKF. By sequentially obtaining a plurality of key frames IKF, the map Ml can be continuously updated, thereby ensuring the integrity of the map Ml.
[0051] In the above embodiments, for those images IMG that are not set as key frames IKF, the processor 101 simply estimates the pose of the electronic device 100 using only the relative relationship between those images IMG and the previous key frame. After that, those images IMG will be discarded, i.e., those images IMG will not be stored in the storage device 105.
[0052] In the above embodiments, the processor 101 can be implemented by a central processing unit, an application specific integrated circuit, a microprocessor, a system on a chip, or other suitable processing circuit. The camera 103 can be implemented by an image capturing structure including, for example, a lens, an image sensor, an image processor, and the like. In addition, the storage device 105 can be implemented by a volatile memory, a non-volatile memory, or both.
[0053] It should be understood that the configuration of the electronic device 100 is not limited to that shown in Figure 1 For example, in some embodiments, the storage device 105 is omitted from Figure 1 and the signals, data, and / or information required by the processor 101 to perform some of the operations can be stored in a remote unit (e.g., a server) and can be accessed by the processor 101 through a wireless communication (e.g., a network).
[0054] Detailed operations of the electronic device 100 will be described in the following paragraphs in conjunction with a map updating method 200. Please refer to Figure 2 , Figure 2 is a flowchart of the map updating method 200 according to some embodiments of the present disclosure. In some embodiments, the map updating method 200 is applicable to the electronic device 100 and includes operations S201-S204.
[0055] Generally, there are some repetitive structures in an environment. For example, multiple columns of the same structure are arranged at equal intervals on a station platform. In another example, multiple windows of the same structure are arranged in an array on a wall of a high-rise building. Accordingly, a key frame IKF obtained from the image IMG captured by the camera 103 may inadvertently include a repetitive pattern RP corresponding to one repetitive structure in the environment, which will be described in conjunction with Figure 3 . Figure 3 is a schematic diagram of a key frame IKF having a repetitive pattern RP according to some embodiments of the present disclosure.
[0056] In Figure 3 , the repetitive pattern RP corresponds to multiple columns on a station platform, and three basic repetitive units RU[A]-RU[C] in the repetitive pattern RP are labeled. Each of the basic repetitive units RU[A]-RU[C] corresponds to one of the multiple columns on the platform. Based on the field of view of the camera corresponding to the key frame IKF in Figure 3 , the basic repetitive unit RU[A] having the largest size among the basic repetitive units RU[A]-RU[C] corresponds to the column closest to the electronic device 100, and the two basic repetitive units RU[B]-RU[C] smaller than the basic repetitive unit RU[A] can correspond to the two columns immediately following the column closest to the electronic device 100. In addition, the basic repetitive unit RU[C] is smaller than the basic repetitive unit RU[B], which means that the column corresponding to the basic repetitive unit RU[C] is farther from the column corresponding to the basic repetitive unit RU[A] than the column corresponding to the basic repetitive unit RU[B].
[0057] In the following embodiments, if a reference sign of an element is used without indicating its letter or number index, it means that the reference sign of the element refers to any one of the element group to which the element belongs. For example, the basic repetitive unit RU refers to any one of the basic repetitive units RU[A]-RU[C].
[0058] In the above embodiments, it is understood that the repetitive pattern RP can include at least two base repeating units RU. That is, if there are at least two base repeating units RU in the key frame IKF, it can be indicated that the key frame IKF includes the repetitive pattern RP.
[0059] In some embodiments, operation S201 is performed when the processor 101 obtains the key frame IKF from the image IMG captured by the camera 103.
[0060] At operation S201, the electronic device 100 performs image recognition on the key frame IKF by the processor 101 to determine whether the key frame IKF includes the repetitive pattern RP. Specifically, the image recognition can be implemented by any algorithm or well-trained neural network model capable of detecting whether there are at least two base repeating units RU in the image data (i.e., the key frame IKF). In some embodiments, as shown in FIG. 2, when the processor 101 determines that the key frame IKF does not include the repetitive pattern RP, operation S201 can be performed again to perform image recognition on another key frame IKF newly obtained. Figure 2
[0061] In the embodiments of FIG. 2, when the processor 101 determines that the key frame IKF includes the repetitive pattern RP, operation S202 is performed. At operation S202, the electronic device 100 obtains, by the processor 101, a plurality of map points MP (shown in FIG. 3) corresponding to the repetitive pattern RP, which will be described in conjunction with Figure 3 Figure 7 Figure 4 Figure 4 A flowchart of operation S202 of the map updating method 200 according to some embodiments of the present disclosure is shown in FIG. 4. In some embodiments, as shown in FIG. 4, operation S202 includes a plurality of sub-operations S401-S402. Figure 4
[0062] At sub-operation S401, the processor 101 maps a plurality of existing map points MPE of the map M1 to a plurality of feature points (not shown in the figure) extracted from the key frame IKF according to pose data corresponding to the key frame IKF, which will be described in conjunction with Figure 5 Figure 5 FIG. 1 illustrates a schematic diagram of mapping existing map points MPE to a key frame IKF according to some embodiments of the present disclosure. In some embodiments, the processor 101 calculates a pose of the electronic device 100 when the key frame IKF (or the image IMG corresponding to the key frame IKF) is captured as the pose data corresponding to the key frame IKF. Also, each of the existing map points MPE can include a corresponding descriptor capable of indicating spatial coordinates and features (e.g., color, shape, texture, etc.) in the map M1. Accordingly, the processor 101 can find the existing map points MPE that can be projected onto the key frame IKF according to the descriptors of the existing map points MPE and the pose data corresponding to the key frame IKF.
[0063] According to the above description, each of the feature points extracted from the key frame IKF can include a corresponding descriptor capable of indicating planar coordinates and features (e.g., color, shape, texture, etc.) in the key frame IKF. The processor 101 can further match the existing map points MPE projected onto the key frame IKF with the feature points extracted from the key frame IKF according to the descriptors of the existing map points MPE and the descriptors of the feature points. Accordingly, the processor 101 can obtain the existing map points MPE matched or mapped with the feature points extracted from the repetitive pattern RP in the key frame IKF, and the sub-operation S402 will be performed.
[0064] At the sub-operation S402, the processor 101 uses the existing map points MPE matched with the feature points extracted from the repetitive pattern RP as the map points MP corresponding to the repetitive pattern RP. Then, the operation S203 will be performed.
[0065] Please refer again to Figure 2 , at the operation S203, the electronic device 100 generates, by the processor 101, a plurality of inferred map points MPI (shown in Figure 8 ) according to the map points MP corresponding to the repetitive pattern RP, which will be described in Figure 6 . Figure 6 FIG. 2 illustrates a flowchart of the operation S203 of the map updating method 200 according to some embodiments of the present disclosure. In some embodiments, as shown in Figure 6 , the operation S203 includes a plurality of sub-operations S601-S603.
[0066] At the sub-operation S601, the processor 101 classifies the map points MP into a plurality of map point groups according to the number of the basic repeating units RU. Please refer to Figure 7 , Figure 7 FIG. 3 illustrates a schematic diagram of classifying the map points MP corresponding to the repetitive pattern RP according to some embodiments of the present disclosure. As shown inFigure 4-5 and Figure 7 As can be seen from the explanation, Figure 7 The multiple map points MP shown can be Figure 5 The MPE is a set of existing map points that match multiple feature points extracted from repetitive graphs (RP).
[0067] At Figure 7 In this embodiment, because there are three basic repeating units RU[A] to RU[C], multiple map points MP are classified into three map point groups. Specifically, multiple map points MP[A] that match multiple feature points extracted from the basic repeating unit RU[A] can be referred to as a first map point group (which is in Figure 7 (represented by a hollow square in the middle), multiple map points MP[B] that match multiple feature points extracted from the basic repeating unit RU[B] can be referred to as a second map point group (which is in the hollow square in the middle). Figure 7 (represented by a hollow triangle), and multiple map points MP[C] that match multiple feature points extracted from the basic repeating unit RU[C] can be referred to as a third map point group (which is in Figure 7 (Represented by hollow rhombuses in the middle). Also, the first map point group is directly adjacent to (or immediately adjacent to) the second map point group in map M1, and the second map point group is directly adjacent to (or immediately adjacent to) the third map point group in map M1.
[0068] in accordance with Figure 7 As explained, since the beams and columns corresponding to the basic repeating units RU[A] to RU[C] are structurally similar, theoretically, the spatial distribution of map point MP in these three map point groups should be similar. However, due to the limitations of the camera 103 itself, the second and third map point groups corresponding to distant beams and columns may have the risk of inaccurate and / or incomplete map points MP[B] and MP[C]. Because the basic repeating unit RU[A] corresponds to the nearest beam and column, in practice, the spatial distribution of map point MP[A] corresponding to the basic repeating unit RU[A] in map M1 should be the most accurate and complete. Furthermore, as... Figure 7 As shown, the number of map points MP[A] in the first map point group corresponding to the basic repeating unit RU[A] is likely the largest among the three map point groups. That is, map points MP[A] in the first map point group can be used as the most reliable reference to repair map points MP[B] and MP[C] corresponding to the basic repeating units RU[B] and RU[C] (i.e., the first map point group can be referred to as a reference map point group). Furthermore, when the beams and columns corresponding to the basic repeating units RU[A] to RU[C] are equally spaced on the platform, the relationship between directly adjacent first and second map point groups can also be used to repair map points MP[C] in the third map point group.
[0069] Accordingly, in sub-operation S602, processor 101 calculates transformation data TD by performing a transformation operation between two adjacent map point groups. In some embodiments, processor 101 performs a transformation operation between a first map point group and a second map point group to calculate transformation data TD. For example, such as Figure 8 As shown, processor 101 transforms map point MP[A][1] in the first map point group into map point MP[B][1] in the second map point group, thereby obtaining data that allows map point MP[A][1] to have the spatial coordinates of map point MP[B][1] as transformation data TD. It should be noted that the spatial relationship between map point MP[A][1] and other map points MP[A] in the first map point group is similar to the spatial relationship between map point MP[B][1] and other map points MP[B] in the second map point group. This means that map point MP[A][1] corresponds to map point MP[B][1]. Therefore, other map points MP[A] that can correspond to other map points MP[B] can be transformed to obtain their corresponding transformation data TD.
[0070] As can be seen from the description of sub-operations S601 to S602, in some embodiments, the electronic device 100 classifies multiple map points MP according to the basic repeating unit RU of the repeating pattern RP to obtain transformation data TD.
[0071] In sub-operation S603, processor 101 transforms at least one of a plurality of map points MP corresponding to the basic repeating unit RU by transforming data TD, to generate at least one of a plurality of estimated map points MPI. (See also...) Figure 8 , Figure 8 This is a schematic diagram illustrating the generation of a presumed map point MPI based on a map point MP corresponding to a repeating graphic RP, according to some embodiments of the present disclosure.
[0072] In some embodiments of sub-operation S603, such as Figure 8 As shown, processor 101 multiplies map point MP[B][1] in the second map point group by transformation data TD to generate an estimated map point MPI[C][1] in the third map point group. Because the transformation data TD used to generate the estimated map point MPI[C][1] is obtained by transforming map point MP[A][1] in the first map point group into map point MP[B][1] in the second map point group, the spatial relationship between the estimated map point MPI[C][1] and multiple map points MP[C] in the third map point group is similar to the spatial relationship between map point MP[A][1] and other map points MP[A] in the first map point group (or the spatial relationship between map point MP[B][1] and other map points MP[B] in the second map point group).
[0073] For example Figure 8As shown, processor 101 multiplies map point MP[B][2] in the second map point group by transformation data TD to generate an estimated map point MPI[C][2] in the third map point group. It should be understood that the transformation data TD used to generate the estimated map point MPI[C][2] can be obtained by transforming map point MP[A][2] in the first map point group into map point MP[B][2] in the second map point group.
[0074] Also, at Figure 8 In one embodiment, other map points MP[B] in the second map point group can be multiplied by their corresponding transformation data TD to generate other estimated map points MPI[C] in the third map point group (in Figure 8 (Represented by a rhombus filled with dots). Therefore, because Figure 2 The number of map points MP[B] in the second map point group is 4, and the number of presumed map points MPI[C] generated in the third map point group can also be 4. Accordingly, in some embodiments, the number of presumed map points MPI[C] generated in the third map point group is greater than the number of map points MP[C] in the third map point group (i.e., 2 in Figures 7 and 8). However, this disclosure is not limited thereto, and the map points MP[C] in the third map point group in Figures 7 and 8 are shown for illustrative purposes. In some embodiments, the number of presumed map points MPI[C] generated in the third map point group can be equal to the number of map points MP[C] in the third map point group.
[0075] Please refer to it again. Figure 9 In operation S204, electronic device 100 updates map M1 via processor 101 based on multiple estimated map points MPI, which will be used in conjunction with... Figure 9 Please provide an explanation. Figure 9 This is a flowchart of operation S204 of a map update method 200 according to some embodiments of the present disclosure. In some embodiments, such as Figure 8 As shown, operation S204 includes multiple sub-operations S901 to S903.
[0076] At sub-operation S901, the processor 101 determines whether a map point MP is close to the spatial coordinate of a presumed map point MPI. Specifically, when a distance difference between the spatial coordinate of the map point MP and the spatial coordinate of the presumed map point MPI is greater than or equal to a preset minimum distance, the processor 101 determines that the map point MP is not close to the spatial coordinate of the presumed map point MPI. When the distance difference between the spatial coordinate of the map point MP and the spatial coordinate of the presumed map point MPI is less than the preset minimum distance, the processor 101 determines that the map point MP is close to the spatial coordinate of the presumed map point MPI. In some embodiments of sub-operation S901, the processor 101 determines that none of the map points MP is close to the spatial coordinate of the presumed map point MPI[C][1], so that sub-operation S902 is performed. At sub-operation S902, the processor 101 adds the presumed map point MPI[C][1] to the map M1.
[0077] In some embodiments of sub-operation S901, the processor 101 determines that the map point MP[C][2] is close to the spatial coordinate of the presumed map point MPI[C][2], so that sub-operation S903 is performed. At sub-operation S903, the processor 101 adjusts the map point MP[C][2] to the spatial coordinate of the presumed map point MPI[C][2]. It should be understood that, before the map point MP[C][2] is adjusted, the spatial coordinate of the map point MP[C][2] is not the same as the spatial coordinate of the presumed map point MPI[C][2]. In the embodiment in which the number of the presumed map points MPI[C] generated from the third map point group is 4, by adding the presumed map point MPI[C] to the map M1 and / or adjusting the map point MP[C], the total number of the presumed map points MPI[C] and the map points MP[C] in the third map point group can be equal to the number of the map points MP[B] in the second map point group.
[0078] In some further embodiments, when map point MP[C][2] is adjusted to the spatial coordinates of estimated map point MPI[C][2], processor 101 also adjusts a feature point that matched map point MP[C][2] before adjusting map point MP[C][2]. Specifically, based on the pose data corresponding to keyframe IKF, estimated map point MPI[C][2] can be projected onto a planar coordinate on keyframe IKF. The feature point is adjusted from another planar coordinate indicated by its descriptor to the planar coordinate on keyframe IKF. In this way, map point MP[C][2] adjusted to the spatial coordinates of estimated map point MPI[C][2] can be mapped to the feature point adjusted to the planar coordinate. By also adjusting the feature point extracted from keyframe IKF, it is beneficial to match this keyframe IKF with another newly acquired keyframe IKF. Furthermore, when the estimated map point MPI[C][1] is added to map M1 (i.e., sub-operation S902), a new feature point can be added to a planar coordinate on keyframe IKF, wherein the estimated map point MPI[C][1] is also projected onto this planar coordinate on keyframe IKF. The operation of adding the new feature point to keyframe IKF is equivalent to adding the estimated map point MPI[C][1] to keyframe IKF.
[0079] At Figure 10 In one embodiment, processor 101 multiplies map point MP[B] in the second map point group by transform data TD to generate an estimated map point MPI[C] in the third map point group. However, this disclosure is not limited thereto. For example, please refer to... Figure 10 , Figure 10 This is a schematic diagram illustrating the generation of a presumed map point MPI based on a map point MP corresponding to a repeating graphic RP, according to some embodiments of the present disclosure.
[0080] In some embodiments, such as Figure 10 As shown, processor 101 multiplies map point MP[B][1] in the second map point group by transformation data TD to generate an estimated map point MPI[B][3] in the second map point group. It should be understood that the transformation data TD used to generate the estimated map point MPI[B][3] can be obtained by transforming map point MP[A][1] in the first map point group to map point MP[A][3] in the first map point group. Similarly, processor 101 can generate other estimated map points MPI[B] in the second map point group (in... Figure 10(represented by triangles filled with dots) such that the number of map points MP[A] in the first map point group (i.e., the reference map point group) is equal to the sum of the number of estimated map points MPI[B] generated in the second map point group and the number of map points MP[B] in the second map point group.
[0081] At Figure 2 In some further embodiments, the processor 101 multiplies the estimated map point MPI[B][3] generated from the second map point group by transformation data TD to generate the estimated map point MPI[C][3] from the third map point group. It should be understood that the transformation data TD used to generate the estimated map point MPI[C][3] can be obtained by transforming map point MP[A][3] in the first map point group to the estimated map point MPI[B][3] generated from the second map point group. However, this disclosure is not limited thereto. For example, the processor 101 can multiply a map point MP[C] in the third map point group by the corresponding transformation data TD to generate the estimated map point MPI[C][3] from the third map point group. Specifically, this corresponding transformation data TD can be obtained by performing a transformation operation between two corresponding map points MP[A] in the first map point group. Similarly, the processor 101 can generate other estimated map points MPI[C] in the third map point group, such that the number of map points MP[A] in the first map point group (i.e., the reference map point group) is equal to the sum of the number of estimated map points MPI[C] generated in the third map point group and the number of map points MP[C] in the third map point group.
[0082] It should be understood that map update method 200 is not limited to... Figure 8 The flowchart is shown below. For example, in some embodiments, when processor 101 determines that keyframe IKF does not contain repeating graph RP, processor 101 will update map M1 using keyframe IKF and obtain another keyframe IKF. Generally, when updating map M1 using keyframe IKF, processor 101 can create new map points in map M1 and / or adjust the descriptor of at least one existing map point MPE in map M1 based on feature points extracted from keyframe IKF. It should be noted that these new map points created in map M1 based on feature points extracted from keyframe IKF are different from the aforementioned presumed map points MPI added to map M1, because presumed map points MPI are generated from existing map points MPE in map M1, not from feature points.
[0083] In some embodiments, the presumed map point MPI is stored in map M1 as an inference record. For example, The putative map point MPI[C][1] in the above equation can be recorded as an inference equation: MPI[C][1] = MP[B][1] * TD(MP[A][1]:MP[B][1]), where TD(MP[A][1]:MP[B][1] in the inference equation represents the transformation data TD used to transform map point MP[A][1] to map point MP[B][1]. When the spatial coordinates of map point MP[A][1] (and / or map point MP[B][1]) are updated according to the feature points extracted from the key frame IKF, the putative map point MPI[C][1] will be updated at the same time due to the change in the inference equation. Further, since the repetitive pattern RP involves the orderly arrangement of similar objects (e.g., beam columns on a platform), the intensity of the corresponding feature points of the plurality of map point groups will also be similar, which causes the closer feature points to have higher accuracy. Under this characteristic, the above inference will occur in the map point groups corresponding to distant objects (e.g., the second or third map point groups), and the phenomenon of circular inference will not occur.
[0084] From the above description of the inference method, it can be seen that, by using the above inference method, the second and third map point groups have the opportunity to obtain the same number of map points MP as the first map point group. In practice, in some embodiments, these putative map points MPI are first stored in an inference collection database (storage or independent of storage device 105) and are not immediately added to the map M1. Further, before the putative map points MPI are added to the map M1 and the corresponding key frame IKF, the existing map points MPE will be checked. Once the putative map point MPI is surrounded by no other map points MP when projected onto the key frame IKF, the putative map point MPI will be added to the key frame IKF and the map M1 (i.e., sub-operation S902). In some embodiments, all map points (including map points MP and putative map points MPI) on the key frame IKF maintain a predetermined minimum distance from each other to ensure that all map points are uniformly distributed on the key frame IKF and effectively control the number and quality of all map points on the key frame IKF.
[0085] There are many ways to keep all the map points on the key frame IKF apart from each other. The disclosure provides a simple example that divides the key frame IKF into N equal parts and only allows one map point (i.e., a map point MP or an inferred map point MPI) in each part. When the above further embodiments of the sub-operation S902 are implemented, the processor 101 can further determine whether there is only a new feature point in one of the N equal parts of the key frame IKF when a new feature point is added to the key frame IKF. When there is only a new feature point in one of the N equal parts of the key frame IKF, the new feature point will remain on the key frame IKF. However, the disclosure is not limited thereto. There are still many ways to keep all the map points evenly distributed, and this example is only used to illustrate one of the cases of the sub-operation S901.
[0086] As can be seen from the above embodiments of the disclosure, by obtaining the map points MP corresponding to the repetitive pattern RP in the key frame IKF, the electronic device 100 of the disclosure can generate the inferred map points MPI to repair those map point groups (i.e., the second map point group or the third map point group) that are inaccurate and / or incomplete due to the limitations of the camera 103 itself. In addition, the inferred map points MPI help the electronic device 100 to reduce the number of times and the time required to sense the environment for map construction. Therefore, the electronic device 100 and the map updating method 200 have the advantages of improving the efficiency and accuracy of map construction.
[0087] The method of the disclosure can exist in the form of program code. The program code can be included in a physical medium, such as a floppy disk, an optical disk, a hard disk, or any other transitory or non-transitory computer-readable storage medium, wherein the computer becomes a device for implementing the method when the program code is loaded and executed by the computer. The program code can also be transmitted through some transmission medium, such as a wire or cable, through an optical fiber, or through any other transmission form, wherein the computer becomes a device for implementing the method when the program code is received, loaded and executed by the computer. When implemented on a general-purpose processor, the program code combines the processor to provide a unique device whose operation is similar to that of an application-specific logic circuit.
[0088] Although the disclosure has been disclosed as above with embodiments, it is not intended to limit the disclosure, and those skilled in the art can make various modifications and improvements without departing from the spirit and scope of the disclosure, and therefore the protection scope of the disclosure should be defined by the appended claims.
Claims
1. A map update method characterized by comprising: A method for updating a map of an environment, wherein the map is built by an electronic device, the method comprising: performing image recognition on a key frame to determine whether the key frame contains a repetitive pattern; obtaining a plurality of map points corresponding to the repetitive pattern; generating a plurality of tentative map points based on the plurality of map points; and updating the map based on the plurality of tentative map points. The obtaining of the plurality of map points corresponding to the repetitive pattern comprises:
2. The map update method according to claim 1, characterized by, mapping a plurality of existing map points of the map to a plurality of feature points extracted from the key frame based on a pose data corresponding to the key frame; and using the plurality of existing map points matching the plurality of feature points extracted from the repetitive pattern as the plurality of map points. The generating of the plurality of tentative map points based on the plurality of map points comprises:
3. The map update method according to claim 1, characterized by, classifying the plurality of map points based on a basic repeating unit of the repetitive pattern to obtain a transformation data; and transforming at least one of the plurality of map points corresponding to the basic repeating unit based on the transformation data to generate at least one of the plurality of tentative map points. The classifying of the plurality of map points based on the basic repeating unit of the repetitive pattern to obtain the transformation data comprises: classifying the plurality of map points into a plurality of map point groups based on a number of the basic repeating unit; and 4. The map update method according to claim 3, characterized by, calculating the transformation data by performing a transformation operation between two adjacent map point groups of the plurality of map point groups. The number of map points in one of the two adjacent map point groups is greater than the number of map points in the other of the two adjacent map point groups. The transforming of the at least one of the plurality of map points corresponding to the basic repeating unit based on the transformation data to generate the at least one of the plurality of tentative map points comprises:
5. The map update method according to claim 4, characterized by, multiplying map points in a first map point group of the plurality of map points by the transformation data to generate tentative map points in a second map point group of the plurality of map points, wherein the first map point group and the second map point group are directly adjacent.
6. The map update method according to claim 3, wherein The number of tentative map points in the second map point group is greater than or equal to the number of map points in the second map point group. The sum of the number of tentative map points in the second map point group and the number of map points in the second map point group is equal to the number of map points in a reference map point group of the plurality of map points.
7. The map update method according to claim 6, wherein The updating of the map based on the plurality of tentative map points comprises:
8. The map update method according to claim 6, characterized by, adding one of the plurality of tentative map points to the map when none of the plurality of map points is close to a spatial coordinate of the one of the plurality of tentative map points.
9. The map update method according to claim 1, wherein The method further comprises: adding a new feature point to a plane coordinate on the key frame, wherein the one of the plurality of tentative map points is projected to the plane coordinate on the key frame.
10. The map update method according to claim 9, characterized by, The updating of the map based on the plurality of tentative map points comprises: adjusting one of the plurality of map points to a spatial coordinate of one of the plurality of tentative map points when the one of the plurality of map points is close to the spatial coordinate of the one of the plurality of tentative map points.
11. The map update method according to claim 1, wherein The method further comprises: 12. The map update method according to claim 11, wherein adjusting a feature point to a planar coordinate on the key frame, wherein the one of the plurality of map points in the spatial coordinate is mapped to the feature point in the planar coordinate.
13. The map update method according to claim 1, characterized by, when the key frame does not contain the repetitive pattern, the map updating method further comprises: updating the map using the key frame; and obtaining another key frame.
14. An electronic device, comprising: a method for building a map of an environment, comprising: a camera for capturing at least one image of the environment; and a processor coupled to the camera and configured to: obtain a key frame from the at least one image; perform image recognition on the key frame to determine whether the key frame contains a repetitive pattern; obtain a plurality of map points corresponding to the repetitive pattern; generate a plurality of tentative map points based on the plurality of map points; and update the map based on the plurality of tentative map points. a computer program for executing a map updating method, wherein the map updating method is applicable to an electronic device configured to build a map of an environment, and the map updating method comprises:
15. A non-transitory computer-readable storage medium, comprising: performing image recognition on a key frame to determine whether the key frame contains a repetitive pattern; obtaining a plurality of map points corresponding to the repetitive pattern; generating a plurality of tentative map points based on the plurality of map points; and updating the map based on the plurality of tentative map points.