Method and system for simultaneous localization and alignment for a mobility vehicle

Abstract

A simultaneous localization and alignment (SLAM) system for a mobility vehicle includes a camera configured to capture a front image of the mobility vehicle and a motion data sensor configured to detect motion data of the mobility vehicle. The system also includes a controller configured to receive the front image from the camera and detect a feature point from the front image. The controller is also configured to receive the motion data from the motion data sensor, track the detected feature point, and store a state of the feature point tracking.The state is selected from a first state indicating that a previously tracked first feature point is being tracked normally, a second state indicating that the first feature point has been lost and a second feature point is being tracked, and a third state indicating that the first and second feature points have been lost.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0193479, filed on December 27, 2023, the entire contents of which are incorporated herein by reference. BACKGROUND(a) Technical field

[0002] The present disclosure relates to a method and system for simultaneous localization and alignment (SLAM) for a mobility vehicle. (b) Description of the prior art

[0003] Simultaneous localization and ranging (SLAM) is a technology that enables a vehicle to detect its current position and simultaneously create a map of its surroundings while navigating through an unknown environment. SLAM is primarily implemented using Lidar (Light Detection and Ranging). However, Lidar-based SLAM is expensive because a Lidar sensor must be installed on the vehicle. Accordingly, Lidar-based SLAM is rarely applied to vehicles that do not transport passengers and are used only indoors. Instead, SLAM is primarily implemented using camera images.

[0004] However, camera-based indoor environment SLAM may encounter difficulties in accurately extracting feature points from images detected by the camera due to noise such as white walls, pointless planes, and strong light, and SLAM may fail if the feature points are lost for a long time.

[0005] The matters described in this prior art section are provided to enhance understanding of the background of the present disclosure and may include matters not yet known to those of ordinary skill in the art to which the present technology pertains. SUMMARY

[0006] Embodiments of the present disclosure provide a method and system for simultaneous localization and mapping (SLAM) based on camera images for a mobility vehicle that continues SLAM with motion data sensor-based localization despite losing feature points during SLAM with object tracking.

[0007] According to one embodiment of the present disclosure, a simultaneous localization and alignment (SLAM) system for a mobility vehicle is provided. The system includes a camera configured to obtain an image of the mobility vehicle. The system also includes a motion data sensor configured to detect motion data of the mobility vehicle. The system further includes a controller configured to receive the image of the mobility vehicle from the camera and detect a feature point from the front image. The controller is also configured to receive the motion data of the mobility vehicle from the motion data sensor. The controller is further configured to track the detected feature point and store a state of the tracking of the feature point.The state may be selected from a first state indicating that a previously tracked first feature point is being tracked normally, a second state indicating that the first feature point has been lost and a second feature point is being tracked, and a third state indicating that the first feature point and the second feature point have been lost.

[0008] The controller may be further configured to update a location of the mobility vehicle based on the movement data of the mobility vehicle in the third state.

[0009] The motion data sensor may comprise at least one of the following elements: i) an encoder configured to detect information about the rotation of a drive motor or a wheel provided on the mobility vehicle and transmit the information to the controller, or ii) an inertial sensor configured to detect information about a motion situation of the mobility vehicle, including a speed and a direction of the mobility vehicle, gravity and acceleration, and transmit the information to the controller.

[0010] The controller may be further configured to track the first feature point in the first state and store a frame including the first feature point as a first key frame. The controller may also be configured to temporarily track the second feature point in the second state after the first state and store a frame including the second feature point as a second key frame. The controller may be further configured to determine whether a distance between the first key frame and the second key frame is less than a set distance. The controller may additionally be configured, in response to determining that the distance between the first key frame and the second key frame is less than the set distance, to input the first state as the state and store a current frame as the first key frame.

[0011] The controller may be further configured to determine, in response to determining that the distance between the first key frame and the second key frame is equal to or greater than the set distance, whether a time for which the state is maintained in the second state is greater than a set time.

[0012] The controller may be further configured to input the second state as the state and temporarily track the second feature point when it is determined that the time during which the state is held in the second state is less than the set time.

[0013] The controller may be further configured to, in response to the loss of the first feature point that was tracked and the detection of the second feature point, input the second state as the state, store a frame including the second feature point as a second key frame, and update a location of the mobility vehicle based on the second key frame.

[0014] The controller may be further configured to input the state as the third state when a number of detected feature points is equal to or less than a set number.

[0015] According to another embodiment of the present disclosure, a simultaneous localization and alignment (SLAM) method for a mobility vehicle is provided. The method includes receiving, by a controller, a front image of the mobility vehicle from a camera in a state corresponding to a first state indicating that a previously tracked first feature point is being tracked normally, a second state indicating that the first feature point has been lost and a second feature point is being tracked, and a third state indicating that the first and second feature points have been lost. The method also includes detecting a feature point from the front image.The method also includes, in response to determining that a number of feature points greater than a set number has been detected, storing, by the controller, a frame including a currently detected first feature point as a first key frame. The method further includes determining a location of the mobility vehicle based on the first feature point. The method additionally includes, by the controller, inputting the third state as a state in response to determining that a number of feature points equal to or less than the set number has been detected. The method further includes determining, by the controller, a location of the mobility vehicle based on motion data of the mobility vehicle.

[0016] The method may further comprise receiving, by the controller, the front image of the mobility vehicle from the camera in the third state and detecting the feature point from the front image. The method may also comprise inputting, by the controller, the second state as the state in response to determining that a number of feature points greater than the set number has been detected. The method may additionally comprise determining, by the controller, the location of the mobility vehicle based on the currently detected second feature point.

[0017] The method may further comprise receiving, by the controller, the front image of the mobility vehicle from the camera in the second state and detecting the feature point from the front image. The method may also comprise inputting, by the controller, the third state as the state in response to determining that the feature points equal to or less than the set number have been detected. The method may additionally comprise determining, by the controller, the location of the mobility vehicle based on the data of the movement of the mobility vehicle.

[0018] The method may further comprise receiving, by the controller, the front image of the mobility vehicle from the camera in the second state and detecting the feature point from the front image. The method may also comprise, in response to determining that a number of feature points greater than the set number has been detected, storing, by the controller, a frame containing the currently detected second feature point as a second key frame and temporarily determining the location of the mobility vehicle based on the second feature point. The method may additionally comprise determining, by the controller, whether a distance between the first and second key frames is less than a set distance.The method may further comprise, in response to determining that the distance between the first key frame and the second key frame is less than the set distance, inputting by the controller the first state as the state and storing a current frame as the first key frame.

[0019] The method may further include, in response to determining that the distance between the first key frame and the second key frame is equal to or greater than the set distance, determining, by the controller, whether a time for which the state is maintained in the second state is greater than a set time. The method may also include, in response to determining that the time for which the state is maintained in the second state is greater than a set time, inputting, by the controller, the first state as the state and considering the second feature point as a new first feature point and tracking the second feature point.

[0020] The method may further comprise, in response to determining that the time for which the state is maintained in the second state is less than the set time, inputting by the controller the second state as the state and temporarily determining the location of the mobility vehicle based on the second feature point.

[0021] The mobility vehicle can be detected by a motion data sensor. The motion data sensor can comprise at least one of the following elements: i) an encoder configured to detect information about the rotation of a drive motor or a drive wheel on the mobility vehicle and transmit the information to the controller, or ii) an inertial sensor configured to detect information about the movement of the mobility vehicle, including a speed and direction of the mobility vehicle, gravity, and acceleration, and transmit the information to the controller.

[0022] According to the present embodiments of the present disclosure, even if feature points are lost during simultaneous localization and mapping (SLAM) with object tracking, SLAM can continue with motion data sensor-based localization. This can ensure the robustness of SLAM to disturbances such as light.

[0023] Additionally, if the lost feature point is rediscovered during SLAM using the motion data sensor-based localization, the location of the mobility vehicle can be corrected based on a previous location and a current location of the feature point. For this purpose, SLAM can be accurately performed.

[0024] Other effects that can be obtained or are expected to be achieved by embodiments of the present disclosure are directly or implicitly disclosed in the detailed description of the present disclosure. In other words, various effects that can be obtained or expected according to the present disclosure are directly or implicitly disclosed in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Embodiments of the present disclosure should be better understood by reference to the following description taken in conjunction with the accompanying drawings, in which the same or functionally similar elements are designated by identical or similar reference numerals. Fig. 1 is a block diagram of a simultaneous localization and alignment (SLAM) system for a mobility vehicle according to a present embodiment of the present disclosure. Fig. 2 and Fig. 3 are flow diagrams of a method for SLAM for a mobility vehicle according to a present embodiment of the present disclosure. Fig. 4 is a diagram illustrating an example scenario for a method for SLAM for a mobility vehicle according to a present embodiment of the present disclosure.

[0026] It should be understood that the above-referenced drawings are not necessarily drawn to scale. The drawings present a rather brief expression of various characteristics that illustrate the basic principles of the present disclosure. For example, the specific characteristics of the present disclosure, including, but not limited to, a particular dimension, direction, location, and shape, may be determined in part by a particular intended application and environment. DETAILED DESCRIPTION

[0027] The terms used herein are for the purpose of describing specific embodiments. The terms are not intended to be limiting of the present disclosure. As used herein, singular terms also include plural terms unless the context clearly indicates otherwise. The terms "comprise," "including," "having," "having," or the like, when used in the present description, determine the presence of the recited characteristics, integers, operations, constituent elements, and / or components. It should be understood, however, that such terms do not preclude the possibility of the presence or addition of one or more other characteristics, integers, operations, operations, constituent elements, components, and / or groups thereof. As used herein, the term "and / or" includes any or all combinations of the connected and recited elements.

[0028] As used herein, "mobility vehicle" or "mobility vehicle" or other similar terms as used herein include motor vehicles in general. Such motor vehicles include passenger vehicles, sport utility vehicles (SUVs), buses, trucks, and various commercial vehicles. Such vehicles also include marine mobility vehicles, such as various boats and ships, and aerial mobility vehicles, such as aircraft and drones. Such vehicles generally include any object capable of moving with the aid of a motorized source. Further, the terms "mobility vehicle" or "mobility vehicle" or similar terms as used herein include hybrid mobility vehicles, electric mobility vehicles, plug-in hybrid mobility vehicles, hydrogen-powered mobility vehicles, and other alternative fuel mobility vehicles (e.g., fuel derived from resources other than petroleum).Hybrid mobility vehicles include mobility vehicles that have two or more power sources, such as gasoline-powered and electric-powered mobility vehicles. Mobility vehicles according to the present embodiments of the present disclosure include both manually powered mobility vehicles and autonomously and / or automatically powered mobility vehicles.

[0029] Additionally, it should be understood that one or more of the methods according to embodiments of the present disclosure, or aspects thereof, may be performed by at least one or more controllers. The term "controller" may refer to a hardware device including a memory and a processor. The memory is configured to store program instructions, and the processor is specifically programmed to execute program instructions to perform one or more processes described in more detail below. The controller may control operations of units, modules, components, devices, or the like, as described herein.Further, it will be understood that methods according to the present embodiments of the present disclosure may be performed by an apparatus including a controller along with one or more other components as would be recognized by those skilled in the art.

[0030] Furthermore, the controller of the present disclosure may be implemented as a non-transitory, computer-readable writing medium comprising processor-executable program instructions. Examples of the computer-readable writing medium include read-only memory (ROM), random access memory (RAM), compact disc (CD) ROM, magnetic tapes, floppy disks, flash drives, smart cards, and / or optical data storage devices. However, the computer-readable writing medium is not limited thereto. The computer-readable writing medium may also be distributed over a computer network to store and execute program instructions through a distributed method, such as a telematics server or a controller area network (CAN).

[0031] When a component, device, element, or the like of the present disclosure has a purpose or performs an operation, function, or the like, the component, device, or element should be considered herein to be “configured” to fulfill that purpose or perform that operation or function.

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0033] Fig. 1 is a block diagram of a simultaneous localization and alignment (SLAM) system for a mobility vehicle according to a present embodiment of the present disclosure.

[0034] As in Fig. 1, a SLAM system for a mobility vehicle according to a present embodiment of the disclosure includes an encoder 20, an inertial sensor 30, a camera 40, a controller 50, and a mobility vehicle 60.

[0035] The encoder 20 can detect information about a rotation of a drive motor or a wheel of the mobility vehicle 60. The encoder 20 can be connected to the controller 50 and transmit information about the detected rotation of the drive motor or wheel to the controller 50. The controller 50 can calculate movement data of the mobility vehicle 60, such as a movement speed and / or a distance traveled by the mobility vehicle 60, based on the information about the rotation of the drive motor or wheel.

[0036] The inertial sensor 30 can detect information about a movement situation of the mobility vehicle 60, including the speed and direction of the mobility vehicle 60, gravity, and acceleration. The inertial sensor 30 can be connected to the controller 50 and can transmit information about the detected movement situation of the mobility vehicle 60 to the controller 50. The controller 50 can detect or supplement the movement data of the mobility vehicle 60 based on the information about the movement situation of the mobility vehicle 60.

[0037] While it is described above that both the encoder 20 and the inertial sensor 30 are used as motion data sensors to detect the motion data of the mobility vehicle 60, in some embodiments, only one of the encoder 20 and the inertial sensor 30 may be used as the motion data sensor. Furthermore, the motion data sensor is not limited to the encoder 20 and the inertial sensor 30. Rather, the motion data sensor may additionally or alternatively include a variety of other sensors that detect motion data of the mobility vehicle 60.

[0038] The camera 40 can be mounted on the mobility vehicle 60. The camera 50 can obtain a front image of the mobility vehicle 60 within the detection range of the camera 40. The camera 40 can be connected to the controller 50 and can transmit the captured images to the controller 50.

[0039] The controller 50 can receive i) the information about the rotation of the drive motor or the wheel from the encoder 20, ii) the information about the movement situation of the mobility vehicle 60 from the inertial sensor 30, and iii) the front image of the mobility vehicle 60 from the camera 40.

[0040] The controller 50 is configured to detect objects (e.g., feature points) in the image using an object detection algorithm, such as an artificial neural network, based on the received front image. The controller 50 is configured to track the detected feature points and determine the location of the mobility vehicle 60 based on the tracked feature points.

[0041] The controller 50 can detect the data about the movement of the mobility vehicle 60 based on the information about the rotation of the drive motor or the wheel received from the encoder 20 or the information about the movement situation of the mobility vehicle 60 received from the inertial sensor 30. The controller 50 can estimate the location of the mobility vehicle 60 based on the movement data of the mobility vehicle 60 when the feature points that were tracked have been lost. The controller 50 can also determine an absolute position of the mobility vehicle 60 using the publicly known positioning method. The controller 50 is configured to perform SLAM based on the absolute location of the mobility vehicle 60 when the feature points that were tracked have been lost.

[0042] The controller 50 may be equipped with one or more microprocessors. The one or more microprocessors may be programmed to perform SLAM steps or operations according to the present embodiment of the present disclosure.

[0043] The controller 50 is connected to the mobility vehicle 60. The controller 50 can generate a route for the mobility vehicle 60 and / or control the movement of the mobility vehicle 60 using a map generated by the SLAM method according to the present embodiments. For example, the controller 50 can control the mobility vehicle 60 to track the object or control the mobility vehicle 60 to avoid the object.

[0044] Fig. 2 and Fig. 3 are flowcharts of a method for SLAM for a mobility vehicle according to a present embodiment of the present disclosure.

[0045] As in Fig. 2, the method for SLAM according to one embodiment of the present disclosure begins when the mobility vehicle 60 is turned on. For example, a user may press a start button on the mobility vehicle 60 or turn on the mobility vehicle 60 via a user interface.

[0046] Once the mobility vehicle 60 is turned on, in a step or operation S100, the user presses a SLAM button provided on the mobility vehicle 60 or triggers SLAM via the user interface. When SLAM is triggered, the controller 50 controls a first state as a state in a step or operation S110. The camera 40 detects the front image of the mobility vehicle 60, and the controller 50 receives the front image of the mobility vehicle 60 from the camera 40. In a step or operation S120, the controller 50 detects feature points from the front image. Here, the state refers to a state that tracks a feature point. The state may include a first state, a second state, and a third state.The first state refers to the state in which a previously tracked first feature point is tracked normally, the second state refers to the state in which a new second feature point is tracked after the first feature point is lost, and the third state refers to the state in which the first feature point or the second feature point is lost. In one embodiment, in step or operation S120, the first state is input as the default state when SLAM is triggered.

[0047] When the feature point is detected in step or operation S120, the controller 50 determines whether the number of feature points is greater than a set number N1 and the state is the first state in step or operation S130. Here, the set number N1 may be 20, but is not limited thereto.

[0048] If the processor 150 determines that the number of feature points is greater than the set number N1 and the state is the first state, the controller 50 performs feature point tracking based on a previous frame in a step or operation S140. In one example, the controller 50 tracks the first feature point detected in the previous frame, which was detected in a previous object detection cycle immediately before a current object detection cycle. In other words, the controller 50 tracks the first feature point detected in the previous frame and matches the feature point detected in the current frame with the first feature point detected in the previous frame. If there is no previous frame, for example, if SLAM has just begun, the controller 50 does not perform step or operation S140, and the method continues with a step or operation S150.

[0049] When the feature point has been tracked based on the previous frame in step or operation S140, the controller 50 inputs the first state as the state and stores the current frame as a first key frame in step or operation S150. Here, the first key frame refers to the frame that is the reference for tracking the first feature point in a next object detection cycle in the first state. In other words, the first key frame includes all the first feature points tracked in the first state. For example, the first key frame stored in step or operation S150 becomes the previous frame in step or operation S140 in the next object detection cycle.

[0050] In a step or operation S160, the controller 50 determines the location of the mobility vehicle 60 based on the tracked first feature point. The method for determining the location of the mobility vehicle 60 based on the first feature point is well known to those of ordinary skill in the art, and a detailed description thereof has been omitted.

[0051] Once the location of the mobility vehicle 60 has been determined, the controller 50 determines whether SLAM is complete in a step or operation S170. If the processor 50 determines that SLAM is complete, the controller 50 terminates the method according to the present embodiment of the disclosure. Conversely, if the processor 50 determines that SLAM is not complete, the method proceeds to step or operation S120, and when the next object detection cycle arrives, the controller 50 detects the feature point from the image captured by the camera 40.

[0052] Referring again to step or operation S130, if the number of feature points is equal to or less than the set number N1 or the state is not the first state, the method proceeds to step or operation S200. As in Fig. 3, the controller 50 determines in step or operation S200 whether the number of feature points is greater than the set number N1.

[0053] If it is determined in step or operation S200 that the number of feature points is greater than the set number N1, the controller 50 determines whether the state is the second state in a step or operation S210. In other words, since the number of feature points is greater than the set number N1, the controller 50 determines whether the state is the second state or the third state.

[0054] If the state is determined to be the second state in step or operation S210, the controller 50 performs tracking of the feature points based on the previous frame in a step or operation S220. If the number of feature points is greater than the set number N1 and the state is the second state, some of the first feature points that were tracked have been lost, but new second feature points have been detected, and the new second feature points can be tracked. Therefore, if the state determined in step or operation S210 is the second state and indicates that the new second feature point is being tracked, the controller 50 tracks the second feature point detected in the previous frame, which was detected in the previous object detection cycle immediately before the current object detection cycle.In other words, the controller 50 tracks the second feature point detected in the previous frame and matches the feature point detected in the current frame with the second feature point detected in the previous frame.

[0055] When the feature point has been tracked based on the previous frame in step or operation S220, the controller 50 stores the current frame as a second key frame in step or operation S230. Here, the second key frame refers to the frame that represents the reference for tracking the second feature point in the next object detection cycle in the second state. In other words, the second key frame includes all the second feature points tracked in the second state. For example, the second key frame stored in step or operation S230 becomes the previous frame in step or operation S220 in the next object detection cycle.

[0056] When the second key frame is stored in step or operation S230, the controller 50 determines in a step or operation S240 whether a distance between the first and second key frames is smaller than a set distance D1. If the first key frame with the previously tracked first feature point and the second key frame with the currently tracked second feature point are close to each other, it can be determined that the previously tracked and lost first feature point has been recovered. Thus, the processor 50 determines in step or operation S240 whether the currently tracked second feature point is identical to the previously lost first feature point. The set distance D1 can be set to any distance determined to be appropriate by those skilled in the art.

[0057] If the distance between the first and second key frames is smaller than the set distance D1 in step or operation S240, the process proceeds to step or operation S150, and the controller 50 inputs the first state as the state and stores the current frame as the first key frame. If the distance between the first and second key frames is smaller than the set distance D1, the lost first feature point is recovered. Accordingly, the controller 50 returns the state to the first state and stores the current frame as the first key frame.

[0058] If the distance between the first and second key frames is equal to or greater than the set distance D1 in step or operation S240, the controller 50 determines whether the time for which the state is maintained in the second state is greater than a set time T1 in step or operation S250. If the time for which the state is maintained in the second state in step or operation S250 is greater than the set time T1, it is determined that the lost first feature point has not been recovered, but the new second feature point is stably tracked. Accordingly, in step S260, the controller 50 inputs the first state as the state, and the process proceeds to step or operation S120. Thus, when the next object detection cycle arrives, the controller 50 detects the feature point from the image captured by the camera 40 in step or operation S120.

[0059] If the controller 50 determines that the time for which the state is maintained in the second state is equal to or less than the set time T1 in step or operation S250, the controller 50 attempts to recover the lost first feature point and maintains the second state by tracking the new second feature point in a step or operation S270. Thereafter, the method proceeds to step or operation S120, and when the next object detection cycle arrives, the controller 50 detects the feature point from the image captured by the camera 40.

[0060] Referring again to step or operation S210, if the controller 50 determines that the state is not the second state, the controller 50 has lost the first feature point that was tracked but found the second feature point sufficient to perform feature point tracking. Accordingly, in step or operation S290, the controller 50 inputs the second state as the state and begins tracking the new second feature point. To this end, in step or operation S300, the controller 50 stores the current frame as a second key frame to store a frame that serves as a reference point for tracking the feature point in the next object detection cycle in the second state. Thereafter, the controller 50 updates the location of the mobility vehicle 60 based on the new second feature point in step or operation S310.The method then proceeds to step or operation S160 in which the controller 50 determines the location of the mobility vehicle 60.

[0061] Referring again to step or operation S250, when the number of feature points is equal to or less than the set number N1, the controller 50 determines that the first feature point or the second feature point being tracked has been lost and the feature point-based SLAM cannot be performed, and inputs the third state as the state in a step or operation S280. Thereafter, the controller 50 updates the location of the mobility vehicle 60 based on the movement data of the mobility vehicle 60 in a step or operation S320.For example, the controller 50 may recognize the movement data of the mobility vehicle 60 based on the information about the rotation of the drive motor or the wheel received from the encoder 20 or the information about the movement situation of the mobility vehicle 60 received from the inertial sensor 30, and update the current location of the mobility vehicle 60 by adding the movement data of the mobility vehicle 60 to the previous location of the mobility vehicle 60.

[0062] Once the location of the mobility vehicle 60 is updated, the method proceeds to step or operation S120, and when the next object detection cycle arrives, the controller 50 detects the feature point from the image captured by the camera 40.

[0063] Fig. 4 is a diagram illustrating an example scenario of a method for SLAM for a mobility vehicle according to a present embodiment of the present disclosure.

[0064] Referring to Fig. 4, the controller 50 begins tracking the first feature point 62 at a time T and continues tracking the first feature point 62 until a time (T+2). If this is the case, steps or operations S110-S170 are repeated.

[0065] At a time (T+3), the controller 50 loses the first feature point that was tracked. In this case, if the number of feature points is equal to or less than the set number N1 in step or operation S130, the process proceeds to step S200. Further, in step S200, since the number of feature points is equal to or less than the set number N1, the controller 50 inputs the third state as the state in step S280 and updates the location of the mobility vehicle 60 based on the movement data of the mobility vehicle 60.

[0066] At a time (T+4), the controller 50 detects a new second feature point 64 in step or operation S120, and since the number of feature points 64 is greater than the set number N1, but the state is the third state, the method proceeds through steps or operations S130, S200, and S210 to step or operation S290. The controller 50 begins tracking the new second feature point 64 by inputting the second state as a state in step or operation S290. Accordingly, the controller 50 stores the current frame as the second key frame in step or operation S300, updates the location of the mobility vehicle 60 based on the new second feature point 64 in step or operation S310, and determines the location of the mobility vehicle 60 in step or operation S160.

[0067] At a time (T+5), the controller 50 detects and tracks the second feature point 64 and simultaneously detects another first feature point 62 in step or operation S120. Since the number of feature points 62 and 64 is greater than the set number N1, but the state is the second state, the process proceeds through steps S130, S200, and S210 to step or operation S220. The controller 50 tracks the second feature point 64 based on the previous frame (i.e., the second key frame stored at time (T+4)) in step or operation S220 and stores the current frame as the second key frame in step or operation S230. The second key frame includes the second feature point 64 currently being tracked and another first feature point 62.

[0068] Thereafter, in step or operation S240, the controller 50 compares the distance between the first key frame (the previously tracked but lost first feature point 62) and the second key frame (the currently tracked second feature point 64 and another first feature point 62). Since the distance between the first and second key frames is smaller than the set distance D1 (i.e., the lost first feature point 62 has been recovered), the process proceeds to step or operation S150, and the controller 50 inputs the first state as the state and stores the current frame as the first key frame. Further, by comparing the locator of the mobility vehicle 60 based on the second feature point 64 with the locator of the mobility vehicle 60 based on the first feature point 62, the locator of the mobility vehicle 60 can be corrected and optimized.

[0069] Although the present disclosure has been described in connection with the embodiments presently contemplated as practical, the disclosure is not limited to the described embodiments. On the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] KR 10-2023-0193479

[0001]

Claims

[1] Simultaneous Localization and Alignment (SLAM) system for a mobility vehicle, the system comprising: a camera configured to obtain a front image of the mobility vehicle; a motion data sensor configured to detect motion data of the mobility vehicle; and a controller that is configured to receive the front image of the mobility vehicle from the camera, to recognize a feature point from the front image, to receive the movement data of the mobility vehicle from the movement data sensor, to track the detected feature point and to save a state of tracking the feature point, where the state is selected from a first state indicating that a previously tracked first feature point is tracked normally, a second state indicating that the first feature point has been lost and a second feature point is being tracked, and a third state indicating that the first feature point and the second feature point have been lost. [2] The system of claim 1, wherein the controller is further configured to update a location of the mobility vehicle based on the data of movement of the mobility vehicle in the third state. [3] The system of claim 1, wherein the motion data sensor comprises at least one of the following: an encoder configured to detect information about a rotation of a drive motor or a drive wheel provided on the mobility vehicle and transmit the information to the controller, or an inertial sensor configured to detect information about a movement situation of the mobility vehicle, including a speed and a direction of the mobility vehicle, gravity and acceleration, and to transmit the information to the controller. [4] The system of claim 1, wherein the controller is further configured: track the first feature point in the first state and store a frame including the first feature point as a first key frame; temporarily track the second feature point in the second state after the first state and store a frame including the second feature point as a second key frame; to determine whether a distance between the first key frame and the second key frame is smaller than a set distance; and in response to determining that the distance between the first key frame and the second key frame is smaller than the set distance, input the first state as the state and store a current frame as the first key frame. [5] The system of claim 4, wherein the controller is further configured: in response to determining that the distance between the first key frame and the second key frame is equal to or greater than the set distance, determining whether a time for which the state is held in the second state is greater than a set time; and in response to determining that the time for which the state is maintained in the second state is greater than the set time, inputting the first state as the state and considering the second feature point as a new first feature point and tracking the second feature point. [6] The system of claim 5, wherein the controller is further configured to input the second state as the state and temporarily track the second feature point when it is determined that the time for which the state is held in the second state is less than the set time. [7] The system of claim 1, wherein the controller is further configured, in response to the loss of the first feature point that was tracked and the detection of the second feature point input the second state as a state; store a frame including the second feature point as a second key frame; and update a location of the mobility vehicle based on the second key frame. [8] The system of claim 1, wherein the controller is further configured to input the state as the third state when a number of detected feature points is equal to or less than a predetermined number. [9] A method for simultaneous localization and alignment (SLAM) for a mobility vehicle, the method comprising: Receiving, by a controller, a front image of the mobility vehicle from a camera in a state that is a first state, from the first state indicating that a previously tracked first feature point is being tracked normally, a second state indicating that the first feature point has been lost and a second feature point is being tracked, and a third state indicating that the first and second feature points have been lost; Detecting a feature point from the front image; in response to determining that a number of feature points greater than a set number has been detected, storing, by the controller, a frame including a currently detected first feature point as a first key frame; Determining a location of the mobility vehicle based on the first feature point; inputting, by the controller, the third state as the state in response to determining that a number of feature points equal to or less than the set number has been detected; and Determining, by the controller, the location of the mobility vehicle based on movement data of the mobility vehicle. [10] The method of claim 9 further comprising: Receiving, by the controller, the front image of the mobility vehicle from the camera in the third state and detecting the feature point from the front image; inputting, by the controller, the second state as the state in response to determining that a number of feature points greater than the set number has been detected; and Determining, by the controller, the location of the mobility vehicle based on the currently detected second feature point. [11] The method of claim 10 further comprising: Receiving, by the controller, the front image of the mobility vehicle from the camera in the second state and detecting the feature point from the front image; inputting, by the controller, the third state as the state in response to determining that the feature points equal to or less than the set number were detected; and Determining, by the controller, the location of the mobility vehicle based on the data of the movement of the mobility vehicle. [12] The method of claim 10 further comprising: Receiving, by the controller, the front image of the mobility vehicle from the camera in the second state and detecting the feature point from the front image; in response to determining that a number of feature points greater than the set number has been detected, storing, by the controller, a frame containing the currently detected second feature point as a second key frame and temporarily determining the location of the mobility vehicle based on the second feature point; Determining, by the controller, whether a distance between the first key frame and the second key frame is less than a set distance; and in response to determining that the distance between the first key frame and the second key frame is less than the set distance, inputting, by the controller, the first state as the state and Save a current frame as the first keyframe. [13] The method of claim 12 further comprising: in response to determining that the distance between the first key frame and the second key frame is equal to or greater than the set distance, determining, by the controller, whether a time for which the state is held in the second state is greater than a set time; and in response to determining that the time for which the state is maintained in the second state is greater than the set time, inputting, by the controller, the first state as the state and considering the second feature point as a new first feature point and tracking the second feature point. [14] The method of claim 13, further comprising, in response to determining that the time the state is maintained in the second state is less than the set time, inputting, by the controller, the second state as the state and temporarily determining the location of the mobility vehicle based on the second feature point. [15] The method of claim 9, wherein the movement data of the mobility vehicle is detected by a movement data sensor, the movement data sensor comprising at least one of the following: an encoder configured to detect information about the rotation of a drive motor or a wheel provided on the mobility vehicle and to transmit the information to the controller, or an inertial sensor configured to detect information about the movement of the mobility vehicle, including a speed and a direction of the mobility vehicle, gravity, and acceleration, and to transmit the information to the controller.

Images