Method and device for making high-precision indoor map optimized through path closing

Abstract

The present invention is a method for operating a computing device for performing a method for making a precise indoor map using a closed path, the method including: using at least one sensor module including a lidar sensor, an inertial measurement unit sensor, and one or more visual sensors; a step of moving the position of the robot for generating the image of the precision map and the sensor data to a first position; a step of constructing a first closed path for pose map optimization according to the odometer estimation from the first position in order to generate a first map using the movement of the robot; updating a second map generation path corresponding to the current path of the robot and at least a part of the sensor data by using at least one of path information of the first closed path and the sensor data; and outputting a second map updated on the basis of the updated second map generation path and the sensor data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to a high-precision indoor map making method and apparatus using path closure optimization, and more particularly, to a map making method and apparatus for performing indoor map precision using path closure in the process of generating an indoor map using images and laser radars, etc. BACKGROUND

[0002] A mobile robot not only needs to grasp its own position within a given environment, but also needs to generate a map related to its surrounding picture on its own in the case where it is placed in a completely new environment that has not been experienced before.

[0003] The map generation of a mobile robot refers to an operation of memorizing in an appropriate method after grasping the placement positions of obstacles or objects of the surroundings and open spaces that are freely movable.

[0004] As an example of a map generation technique of a mobile robot, Korean Patent Laid-Open Publication No. 10-2010-0070922 (published on June 28, 2010) discloses a technique that can generate a final grid map for position recognition of a mobile robot in linkage with position information of landmarks after generating a grid map using distance information from surrounding objects.

[0005] In particular, in the relative movement path of a mobile robot, i.e., ODOMETRY estimation and path optimization process as described above, a one-way open path method of generating an indoor map while moving randomly within a prescribed space or moving in a manner of avoiding repeated entry into an already traveled area is generally used.

[0006] However, for a site such as a disaster site where a state change can occur, it can be difficult to optimize and update a partial map, and errors can continue to accumulate, resulting in a problem of generating an error in a precise map.

[0007] PRIOR ART DOCUMENT

[0008] PATENT DOCUMENT

[0009] (Patent Document 1) Korean Patent Laid-Open No. 1020180109410 (2018.10.08) [Applicant: Naver Corp.] SUMMARY

[0010] TECHNICAL PROBLEM TO BE SOLVED

[0011] The present invention is intended to solve the problems as described above, and aims to provide a high-precision indoor map making method and apparatus using a closed path formed by a path that has already been traveled, which improves a dead reckoning estimation of a current path and a path optimization process using the closed path, thereby minimizing an error and generating a precise map.

[0012] The problems to be solved by the present invention are not limited to the above, but can be extended to various matters that can be derived from the embodiments of the invention described below.

[0013] Means for solving the problems

[0014] In an embodiment of the present specification, there can be provided an indoor map refinement method using a closed path, as an indoor map refinement method using a closed path executed by a computing device, including: a step of moving a position of a robot for generating an image and sensor data for a refined map to a first position using at least one of a sensor module including a laser radar sensor, an inertial measurement unit (IMU) sensor, and one or more vision sensors; a step of constructing a first closed path optimized in a pose graph from a dead reckoning estimation from the first position in order to generate a first map using movement of the robot; a step of updating at least a part of a second map generation path and sensor data corresponding to a current path of the robot using at least one of path information of the first closed path and the sensor data; and a step of outputting a second map updated based on the second map generation path and the sensor data, further including: a step of setting condition information of the first closed path corresponding to the first position; the condition information including a condition of selecting a point from which the robot can return as the first position from a surrounding position, the returnable point being determined based on at least one of state information of a ground surface obtained from sensor information of the robot, information of whether there are two or more passages available for movement of the mobile robot in the surrounding, and inclination information, the condition information including condition information of determining a second position adjacent within a certain distance with relative position information of the first position as an alternative end point of the first closed path in a case where the mobile robot cannot return to the first position due to an obstacle or a structural change, determining the second map generation path consisting of a second closed path moving between the first position and the second position in a case where the second position is determined as the alternative end point, the updating step including a step of performing update of an indoor map based on an image and sensor data collected on the second closed path using original collected information of at least a part of an interval from the first position of the first closed path to the first position even in a case where the robot moves on the second closed path.

[0015] In an embodiment of the present specification, the indoor map refinement method using a closed path can include a process of further including, in the updating step, performing pose graph optimization on the second map generation path using at least one of path information of the first closed path and the sensor data.

[0016] In an embodiment of the present disclosure, the updating step can further include a process of updating the local map information corresponding to the current path of the robot and the odometry estimation by optimizing the acquired local map information based on the pose graph generated by the second map generation path. Thereby, the accuracy of the local map can be improved and the cumulative error can be reduced.

[0017] In an embodiment of the present disclosure, the pose graph optimization of the second map generation path can include two types of optimization processes: one is a loop closure optimization that optimizes the overall path of the robot based on the first closed path; and the other is a smoothing process that performs re-optimization on at least a part of the path moved through the first closed path using the information collected so far to improve the consistency and continuity of the overall map.

[0018] In an embodiment of the present disclosure, the method can further include a step of setting condition information of the first closed path corresponding to the first position. The condition information can include selecting a point that the robot can return from among the surrounding positions as the first position, which is determined based on floor state information acquired from sensor information of the robot, information on whether there are two or more passages available for the robot to move, and inclination information.

[0019] In an embodiment of the present disclosure, the condition information of the first closed path can further include a condition of setting the first position as a specific area within a certain adjacent range based on position information of a distinctive object that can be easily recognized in an image among the surrounding positions, thereby improving the accuracy of path recognition and closed path setting.

[0020] In an embodiment of the present disclosure, the condition information can further include a condition of determining a second position adjacent within a certain distance from the first position as a substitute end point of the first closed path based on relative position information with the first position in the case where the first position cannot be returned due to an obstacle or a structural change, and in the case where the second position is set as the substitute end point, constructing a second closed path formed by a movement path between the first position and the second position and performing a map generation and data update process on the path, so that the map can be merged and optimally updated using original data collected in the first closed path even if the robot moves only on the second closed path.

[0021] Effects of Invention

[0022] According to the present embodiment, a first closed path optimized based on a pose graph optimization from a dead reckoning estimation of a first position can be configured for generating a first map, at least one of path information of the first closed path and sensor data can be used to update at least a portion of a second map generation path and sensor data corresponding to a current path of the robot, and an updated second map can be constructed based on the updated second map generation path and sensor data.

[0023] Accordingly, the present application can provide a high-precision indoor map making method and apparatus using path closure optimization that can minimize errors and generate a precise map by improving dead reckoning estimation and path optimization processes using a closed path formed by a path that has already been traveled.

[0024] It is to be understood that the effects of the present specification are not limited to what has been described in the foregoing content. That is, the effects include all the effects that can be deduced from the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a schematic diagram illustrating an example of an operating environment of a system according to an embodiment of the present specification.

[0026] Figure 2 is a block diagram for explaining an internal configuration of the computing device 200 in an embodiment of the present specification.

[0027] Figure 3 is a schematic diagram for explaining a hardware configuration of an investigation robot connected to the computing device according to an embodiment of the present application.

[0028] Figure 4 is an example diagram of a sensor module according to an embodiment of the present application.

[0029] Figure 5 is an example diagram of a damage information processing section according to an embodiment of the present application.

[0030] Figure 6 is a block diagram specifically illustrating a crack analysis section according to an embodiment of the present application.

[0031] Figure 7 is a block diagram specifically illustrating a service processing section according to an embodiment of the present application.

[0032] Figure 8 is a flowchart for explaining an operation of the computing device according to an embodiment of the present application.

[0033] Figure 9is a diagram illustrating accuracy test results of an embodiment using the present application DETAILED DESCRIPTION

[0034] In describing embodiments of the present specification, when it is determined that a detailed description of related known configurations or functions can make the gist of the embodiments of the present specification unclear, a detailed description thereof will be omitted. Also, portions irrelevant to the description of the embodiments of the present specification are omitted in the drawings, and similar portions are assigned similar reference numerals.

[0035] In the embodiments of the present specification, when described as a certain constituent element is "connected", "coupled", or "in contact" with other constituent element, it can include not only a direct connection relationship but also an indirect connection relationship with other constituent element present therebetween. Also, when described as a certain constituent element "includes" or "has" other constituent element, unless explicitly stated otherwise, it does not mean that other constituent element is excluded, but it can also include other constituent element.

[0036] In the embodiments of the present specification, terms such as first and second are used only to distinguish one constituent element from other constituent elements, and unless explicitly stated otherwise, it does not mean a limitation on the order or importance of the constituent elements. Therefore, within the scope of the embodiments of the present specification, a first constituent element in an embodiment can also be referred to as a second constituent element in other embodiments, and similarly, a second constituent element in an embodiment can also be referred to as a first constituent element in other embodiments.

[0037] In the embodiments of the present specification, mutually distinct constituent elements are only for clear description of each feature, and it does not mean that the constituent elements must be separated. That is, a plurality of constituent elements can be integrated into one hardware or software unit, and one constituent element can be distributed into a plurality of hardware or software units. Therefore, even if not separately mentioned, embodiments of integration or distribution as described above are included within the scope of the embodiments of the present specification.

[0038] In the present specification, a network can be a concept including both wired and wireless networks. At this time, the network can mean a communication network through which data exchange is possible between devices and systems and between devices, and is not limited to a specific network.

[0039] The embodiments described in this specification can all be hardware, or one part can be hardware and another part can be software, or all can be software. In this specification, "unit", "apparatus", or "system" and the like refer to hardware, a combination of hardware and software, or a computer-related entity such as software. For example, in this specification, a unit, module, apparatus, or system and the like can be a process running on a computer, a process, an object, an executable, a thread of execution, a program, and / or a computer, but are not limited thereto. For example, an application running on a computer and both sides of the computer can correspond to a unit, module, apparatus, or system and the like in this specification.

[0040] In addition, in this specification, the device can not only be a mobile device such as a smartphone, a tablet PC, a wearable device, and a head-mounted display (HMD), but also a stationary device such as a personal computer (PC) or a home appliance equipped with a display function. In addition, as an example, the device can be a car navigation device or an Internet of Things (IoT) device. That is, in this specification, the device can refer to an instrument that can run an application function, and is not limited to a specific type. Next, for the convenience of explanation, the instrument that runs the application will be referred to as a device.

[0041] In the present specification, the communication manner of the network is not particularly limited, and the connection between the respective constituent elements can not employ the same network manner. The network can include not only a communication manner using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, and the like) but also near distance wireless communication between devices. For example, the network can include all communication methods that can establish interconnection between objects and objects, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods.For example, the wired and / or network can refer to a communication network based on a communication method selected from one or more of a group consisting of a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Global System for Mobile Network (GSM), an Enhanced Data GSM Environment (EDGE), a High Speed Downlink Packet Access (HSDPA), a Wideband Code Division Multiple Access (W-CDMA), a Code Division Multiple Access (CDMA), a Time Division Multiple Access (TDMA), Bluetooth, Zigbee, a Wireless LAN (Wi-Fi), a Voice over Internet Protocol (VoIP), LTE Advanced, IEEE 802.16m, Wireless MAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), Ultra Mobile Broadband (UMB, formerly EV-DO Rev. C), Flash-OFDM, High Speed Wireless Broadband and Mobile Broadband Wireless Access systems (iBurst and MBWA (IEEE 802.20) systems), HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and an ultrasonic application communication, but is not limited thereto.

[0042] The components described in the various embodiments are not essential components, and some of them can be optional components. Therefore, embodiments constituted by a part of the components described in the embodiments are also included in the scope of the embodiments of the present specification. Furthermore, embodiments including other components in addition to the components described in the various embodiments are also included in the scope of the embodiments of the present specification.

[0043] Next, the embodiments of the present specification will be described in detail with reference to the accompanying drawings.

[0044] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 An example of a working environment of a system according to an embodiment of the present disclosure is shown. Referring to Figure 1 , the user device 110 is connected to one or more servers 120, 130, 140 through a network 1. Figure 1 The number of user devices or servers is not limited to this example.

[0045] The user device 110 can be a fixed or mobile terminal implemented as a computer system. The user device 110 can include, for example, a smartphone, a mobile phone, a navigation device, a computer, a notebook, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet, a game console, a wearable device, an Internet of Things (IoT) device, a virtual reality (VR) device, and an augmented reality (AR) device. For example, in the present embodiment, the user device 110 can substantially refer to one of various physical computer systems capable of communicating with the servers 120-140 through the network 1 by wireless or wired communication.

[0046] Each server can be implemented as a computer device or a plurality of computer devices that provide instructions, codes, files, contents, and services by communicating with the user device 110 through the network 1. For example, the server can be a system that provides various services to the user device 110 through the network 1. More specifically, the server can provide services required by a corresponding application (e.g., information provision, etc.) to the user device 110 through a computer program (application) installed and executed on the user device 110. Another example is that the server can distribute files for installing and executing the above-described application to the user device 110, receive user input information, and provide a corresponding service.

[0047] Figure 2 is a block diagram of the internal configuration of a computing device 200 according to an embodiment of the present disclosure. The computing device 200 can be applied to Figure 1 The user device 110 or the servers 120-140 described above can each have the same or similar internal configuration by adding or subtracting some components.

[0048] Referring to Figure 2 The computing device 200 can include a memory 210, a processor 220, a communication module 230, and a transceiver 240. The memory 210 is a non-volatile computer-readable recording medium and can include a random access memory (RAM), a read-only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, and the like permanent mass storage device. Among them, the ROM, the SSD, the flash memory, and the disk drive, and the like permanent mass storage device are independent permanent storage devices different from the memory 210, and can be included in the above-mentioned device or server. In addition, the memory 210 can store an operating system and at least one program code (for example, a code of a browser installed and running on the user device 110, or an application installed on the user device 110 to provide a specific service). These software components can be loaded from a separate computer-readable recording medium different from the memory 210. The separate computer-readable recording medium can include a floppy disk drive, a magnetic disk, a magnetic tape, a DVD / CD-ROM drive, and a memory card, and the like.

[0049] In another embodiment, the software components can be loaded into the memory 210 through the communication module 230, rather than through the computer-readable recording medium. For example, at least one program can be loaded into the memory 210 based on a computer program (for example, the above-mentioned application) installed by a file provided by a developer or a file distribution system (for example, the above-mentioned server) responsible for distributing application installation files through the network 1.

[0050] The processor 220 can process the instructions of the computer program by performing arithmetic, logic, and input / output, and the like basic operations. The instructions can be provided to the processor 220 by the memory 210 or the communication module 230. For example, the processor 220 can execute the received instructions according to the program code stored in the recording device such as the memory 210.

[0051] The communication module 230 can provide a function of enabling the user device 110 and the servers 120-140 to communicate with each other through the network 1, and a function of enabling the device 110 and / or the servers 120-140 to communicate with other electronic devices.

[0052] The transceiver 240 can be a means of interfacing with external input / output devices (not shown). For example, the external input device can include a keyboard, a mouse, a microphone, and a camera, and the external output device can include a display, a speaker, and a haptic feedback device. Another example is that the transceiver 240 can be a means of interfacing with a device having an input and output integration function such as a touch screen.

[0053] In addition, in other embodiments, the computing device 200 can contain more or less components than those shown in FIG. 1, depending on its application, and the components can be combined or omitted.Figure 2 The computing device 200 can be implemented to include at least a part of the above-described input / output device, or further include other components such as a transceiver, a Global Navigation Satellite System (GNSS) module, a camera, various sensors, and a database, for example, when the computing device 200 is applied to the user device 110. More specifically, when the user device is a smartphone, the computing device 200 can be implemented to further include various types of components commonly found in a smartphone, such as an acceleration sensor or a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input / output port, and a vibrator for vibration.

[0054] The above-described computing device 200 can be implemented by a device including a processor and a memory. The memory can store instructions, and the processor can execute the operations described below according to the instructions stored in the memory. According to the present disclosure, the device can be implemented by at least a part of the configuration shown in Figure 1 or Figure 2 The above-described computing device 200 can be implemented by a device including a processor and a memory. The memory can store instructions, and the processor can execute the operations described below according to the instructions stored in the memory. According to the present disclosure, the device can be implemented by at least a part of the configuration shown in

[0055] Next, the description will be made based on a computing device that is operated based on Figure 1 and Figure 2 The description will be made based on a computing device that is operated based on Figure 2 The robot 300 can include the components included in the computing device 200 in

[0056] Figure 3 is a diagram for explaining a hardware configuration of a survey robot connected to a computing device according to an embodiment of the present disclosure.

[0057] Referring to Figure 3 The robot 300 is combined with a multi-sensor 320 including at least one or more sensors by means of a head 330 that can achieve fine adjustment and light source formation, and the multi-sensor 320 is connected to a robot arm 310 through a separate connection component, so that it can perform indoor structure analysis.

[0058] The robot arm 310 is designed to be rotatable by 360 degrees, and can be equipped in a folding or extending form.

[0059] Further, a control unit 340 that controls hardware, power supply, and communication of the robot 300, and a movement unit 350 that controls movement of the movement assembly can be equipped at the lower end of the robot 300, so that movement for structural damage analysis can be performed in an indoor environment such as a disaster. As an example of the movement unit 350, the movement unit can be a wheel or other configuration for performing a function of moving in an indoor or outdoor space, and is not limited to a specific form.

[0060] As described above, the robot 300 is configured to be movable in an indoor or outdoor space, and can be equipped with a communication function that integrates and transmits various kinds of information to other devices or servers. Further, the robot 300 can acquire sensing information by being combined with a plurality of sensors 320. Among them, the plurality of sensors 320 can include a laser radar sensor, an inertial measurement unit (IMU) sensor, a vision camera, a depth camera, a gyro sensor, and other sensors for sensing surrounding information, and is not limited to a specific sensor.

[0061] The laser radar (LiDAR) sensor can be a sensor that emits a laser pulse and measures the time of reflection, thereby measuring the position coordinates of the reflecting object. Further, the inertial measurement unit (IMU) sensor, as a motion sensor, can be a sensor that measures velocity, direction, and magnetism by a speedometer, a gyroscope, and a magnetometer, thereby measuring direction, acceleration, and position. Further, the vision camera can sense image information of surrounding images, and the depth camera can be a sensor that measures or senses a 3D image (i.e., a depth image) of surrounding objects based on a stereo method or a time of flight (ToF) method. Other sensors can include sensors for sensing temperature or air pressure and other surrounding information, and are not limited to a specific embodiment.

[0062] As an example, the robot 300 can include a laser radar sensor (or module) for indoor positioning and map construction. Among them, the laser radar sensor can be made in a small size and combined with the robot 300. In particular, the laser radar sensor can operate based on a simultaneous localization and mapping (SLAM) technique and generate an indoor map in real time, and thereby identify an indoor structure.

[0063] Further, as an example, in order to perform indoor and outdoor positioning at a disaster site, a Global Navigation Satellite System (GNSS) can be used. However, when communication and radio connection with the outside are not smooth due to obstruction by a building and damage in an indoor space at a disaster site, it can be necessary to identify a position and construct a map in the indoor and outdoor spaces on one's own, and for this purpose, the robot 300 can include a camera and a laser radar as the multiple sensors 320. At this time, as an example, since the laser radar sensor measures time based on a time-of-flight (ToF) method, it can be sensitive to vibration caused by external factors. In consideration of the above-described problem, the multiple sensors 320 can include an inertial measurement unit (IMU) sensor to sense a value related to movement and vibration of the robot 300 and perform position movement in consideration of vibration and analysis accuracy, etc., thereby stably performing crack detection and damage information analysis.

[0064] As described above, the multiple sensors 320 can include sensors for enabling the robot 300 to smoothly perform sensing in an indoor structure collapsed due to an earthquake, and sensing information is not limited to a specific embodiment. Next, for the convenience of explanation, a case in which the robot 300 incorporates the multiple sensors 320 and the robot 300 performs sensing through the multiple sensors 320 in order to analyze an indoor structure will be described as a basis. That is, the robot 300 described in the following content can be a robot 300 equipped with the multiple sensors 320, and is not limited to a specific embodiment.

[0065] Figure 4 FIG. 1 is an example of a sensor module mounted on the robot 300 according to an embodiment of the present application, and is a schematic diagram illustrating hardware of the multiple sensors 320.

[0066] Referring to Figure 4 The multiple sensors 320 according to an embodiment of the present application can include a camera sensor 321 at an upper end, an inertial measurement unit (IMU) sensor 322 axially coupled to a lower end thereof, and a laser radar sensor 323 axially coupled to a lower end thereof.

[0067] Here, the respective sensor modules of the multiple sensors 320 are coupled in hardware, and can synchronize three-dimensional space sensing axes by being coupled with a main shaft 320a penetrating the respective modules in a vertical direction as a center.

[0068] By means of the coordinate axis synchronization of the multiple sensors 320 as described above, the positioning and rotation information of the robot 300 movement and into the tilt interval, etc. can be processed in real time, which is embodied when acquiring the sensing information of the multiple sensors 320. For example, the output of each sensor module acquired by the coordinate axis synchronization can be mapped with the position information centered on the synchronized main coordinate axis information, and the image information, depth sensing information, rotation information, angle information, acceleration information, and vibration information acquired synchronously from each sensor module can be mapped with the position information, thereby being used for the position tracking of the robot 300 and the damage detection processing of the indoor and outdoor space facilities.

[0069] In addition, in order to facilitate the processing of the sensing information of the multiple sensors 320 as described above, it is appropriate to use the well-known data processing based on the Simultaneous Localization And Mapping (SLAM) technology. The Simultaneous Localization And Mapping (SLAM) is a technology for estimating the position using various sensors and generating a three-dimensional map of the corresponding environment in an unknown environment, and is a technology that has been applied in various fields with the improvement of computer processing speed and the development of sensor technologies such as cameras and lidar, and in particular, as a technology for indoor positioning and mapping in the absence of a ready-made map in a disaster site or the like, spatial information is constructed by scanning, data structuring, and three-dimensional indoor map visualization based on lidar and cameras.

[0070] By this means, the robot 300 can dataize the information sensed from the multiple sensors 320 to construct a three-dimensional indoor map, and can share the damage degree of each facility after estimating the facilities from the constructed indoor map and detecting the damage degree of each facility based on the crack information. For example, the robot 300 can be put into an indoor structure collapsed due to an earthquake and construct spatial information based on the robot position in the indoor, and then analyze the damage degree of each facility and output.

[0071] At this time, as an example, the indoor structure collapsed due to an earthquake can belong to a dangerous environment, and the accuracy of sensing can be improved by using data set in advance in a dangerous condition. That is, the robot 300 can be put into the indoor by considering the environment of the indoor structure collapsed due to an earthquake and perform sensing by the sensors. Among them, the dangerous condition can be exemplified by classification conditions such as a flat ground, a deceleration lane, and gravel detected according to average acceleration and maximum acceleration sensing.

[0072] The robot 300 and the plurality of sensors 320 configured in the above-described manner can be controlled by the computing device 200 according to the embodiment of the present application. As an example, the robot 300 can perform indoor positioning and map construction and damage analysis using simultaneous localization and mapping (SLAM) based on the plurality of sensors 320 described above, and the control of the robot 300 can be performed by the connected computing device 200. As an example, the robot 300 and the computing device 200 can exchange information through a communication connection between each other. At this time, the robot 300 can perform sensing in indoor and outdoor spaces based on the control action of the computing device 200, and the computing device 200 can control the position of the plurality of sensors 320 or the position of the moving part 350 in real time in order to improve the accuracy of crack detection.

[0073] For example, vibrations that affect the sensitivity of a laser radar sensor or light source environments that affect the photographing of a camera, etc. can vary depending on the situation of the disaster site. Therefore, distortion can occur in the sensing information, and in order to minimize the distortion as described above, various computing processes can be processed in the computing device 200 and control signals for the robot 300 can be constructed.

[0074] As an example, the internal position and rotation of the plurality of sensors 320 equipped in the robot 300 can be controlled, and this can be achieved by the computing device 200 controlling the driving of the robot arm 310.

[0075] In addition, as an example, the computing device 200 can perform driving control of the head 330 in conjunction with the plurality of sensors 320, so that it can serve to achieve more accurate sensing actions by performing fine position adjustment of the sensor modules equipped in the plurality of sensors 320 and light source control of light emitting diode (LED) lighting mounted to the head 330, etc.

[0076] Further, the computing device 200 can control the moving position and moving speed of the robot 300, and the moving position and moving speed can be optimized in a direction to prevent distortion or error, taking into account the recognition rate of the vibration sensor and the vision sensor based on changes in the indoor environment, etc.

[0077] Figure 5 is a block diagram that specifically illustrates a service processing part 250 of a computing device 200 for driving a robot 300 according to an embodiment of the present application and outputting facility analysis information corresponding to three-dimensional analysis of indoor and outdoor spaces.

[0078] Referring to Figure 5The computing device 200 according to the embodiment of the present application can further include a service processing section 250. The service processing section 250 can be configured by a processor module for various service processes for data processing for the robot 300 control according to the embodiment of the present application, which is executed in the processor 220 processing and input / output operations. The computing device 200 can be equipped with the service processing section 250 inside the processor 220, or perform service processing by the service processing section 250 using data processed in a device equipped in the form of an external processor, thereby providing the generated service processing result to the user device 110 or one or more servers 120, 130, 140.

[0079] Through the processing as described above, the user device 110 or one or more servers 120, 130, 140 can receive the analysis information constructed by the action of the service processing section 250 according to the embodiment of the present application and display output by a display device, or for transmitting and receiving data through an external network.

[0080] In detail, first, referring to Figure 5 The service processing section 250 includes a robot driving section 251, an odometry estimation section 253, a pose graph optimization section 255, and a precise map construction section 257.

[0081] The robot driving section 251 generates a control signal for performing hardware driving control and transmits it to the robot 300 in order to control the position movement and sensing action of the robot 300 as illustrated in Figure 3 and Figure 4 and receives a corresponding response to perform driving control.

[0082] Among them, the robot driving section 251 can perform driving control on the multiple sensors 320, the head 330, the robot arm 310, the control section 340, and the movement section 350 of the robot 300 as described above, respectively, and thereby perform movement, internal position, and sensing control of the multiple sensors 320 of the robot 300.

[0083] In addition, the robot driving section 251 can perform data processing of acquiring sensor signals of each sensor equipped in the multiple sensors 320 and constructing sensing information synchronized according to each position based on simultaneous localization and mapping (SLAM), and further transmitting it to the odometry estimation section 253.

[0084] The sensor signals can include sensor signals of the laser radar sensor 323, sensor signals of the inertial measurement unit (IMU) sensor 322, and visual sensing signals of the camera sensor 321. In addition, the sensor module driving unit 253 can construct simultaneous localization and mapping (SLAM) data by mutually mapping and converting the signals acquired from the respective sensor modules of the multi-sensor 320, and deliver the constructed simultaneous localization and mapping (SLAM) data to the odometry estimation unit 253.

[0085] The simultaneous localization and mapping (SLAM) data can include not only the sensor signals of the basic laser radar sensor 323, the inertial measurement unit (IMU) sensor 322, and the camera sensor 321, but also information converted for constructing three-dimensional space data.

[0086] For example, the simultaneous localization and mapping (SLAM) data can further include indoor position coordinate information calculated from the inertial measurement unit (IMU) sensor 322, a raw (RAW) image mapping image acquired from the camera sensor 321, pose (POSE) information of the camera and the laser radar calculated from the inertial measurement unit (IMU) sensor 322, and feature point cloud (Point Cloud) information acquired from the raw image data.

[0087] In addition, the robot driving unit 251 can request the robot 300 to move the position of the robot 300 or precisely control the position of the multi-sensor 320 in order to improve the accuracy of the respective sensor information, in the case where the accuracy of the respective sensor information is below a critical value as described above.

[0088] In addition, the odometry estimation unit 253 can perform odometry estimation for estimating the relative movement distance and direction of the robot 300 from the initial position, with respect to the sensor data delivered from the robot driving unit 251.

[0089] In particular, according to an embodiment of the present application, the odometry estimation unit 253 can acquire movement state information of the robot 300 based on one or more pieces of closed path information that have been tracked, and further deliver the acquired movement state information to the pose graph optimization unit 255 after mapping the acquired movement state information together with key frame scan information such as the simultaneous localization and mapping (SLAM) or inertial measurement unit (IMU) data.

[0090] Further, the pose graph optimization unit 255 can perform first optimization using one or more of the closed path information that has been tracked, and perform second optimization using the key frame scan information and / or inertial measurement unit (IMU) data that has been collected, with respect to a portion of the overall movement path of the robot 300 that constitutes a closed path, when movement state information is acquired, so that pose graph optimization (PGO) that more accurately completes localization while smoothing the path can be performed.

[0091] Next, the odometry estimation unit 253 acquires the state information received from the pose graph optimization unit 255 and the local map information update information, and performs update processing of odometry information based thereon, and further continuously performs collection of movement state information of the robot 300 based thereon.

[0092] Further, the fine map construction unit 257 constructs fine map information based on the key frame scan information and the state information of the robot 300 with respect to the path that has been optimized by the pose graph optimization unit 255, and outputs the constructed fine map information through a separate output device, or to one or more devices 110.

[0093] Figure 6 is a flowchart for describing a closed path-based path optimization process for indoor map refinement according to an embodiment of the present application.

[0094] As Figure 6 shown, when generating a map based on a robot, odometry estimation using an open loop filter is performed in the related art, and at this time, the map generation path is maximized to avoid repeated visits to the same place, and is constituted in such a way that it will only move on an open type fixed or randomly optimized path.

[0095] However, for a site such as a local map that can dynamically change such as a disaster site, this is an inefficient process. In particular, when local errors are accumulated and continuously added, it can cause problems such as exploration of an inefficient path or errors in the generation of a fine map.

[0096] Therefore, in step S101, the odometry estimation unit 253 according to an embodiment of the present application can form a closed path according to initial odometry estimation, and next, using the information collected in the closed path, perform odometry estimation based on a closed path filter with respect to at least a portion of the section.

[0097] Next, in step S103, the pose graph optimization section 255 can perform pose graph optimization using the state information acquired from the closed path filter-based odometry estimation and the key frame scan information.

[0098] For example, the pose graph optimization section 255 can perform loop closure optimization that optimizes the entire path of the robot when a closed curve is drawn in the path of the robot 300 after the initial odometry estimation is completed (i.e., when the robot visits a location that has been visited before), and can perform smoothing optimization that re-optimizes at least a portion of the path that has been traveled by using the information that has been collected so far, and can update the optimization information even in the case of a local dynamic change in the map in order to more accurately perform positioning.

[0099] By doing so, it is possible to solve the problem of continuous accumulation of errors due to the inability to update the local map as a pose graph-optimized map in the open path method of the prior art that employs one-way odometry and optimization processes without feedback, and to provide a pipeline that can improve the precision error by re-passing a portion of the map that has been optimized in the original closed path and the current state to the odometry estimation section 253, thereby enabling path optimization for generating a more precise map.

[0100] Figure 7 and Figure 8 is an explanatory diagram for describing path optimization according to an embodiment of the present application.

[0101] Referring to Figure 7 and Figure 8 , in Figure 7 , a first closed path 10 (indicated by an arrow) formed from initial odometry is indicated by a solid line, and a second closed path 20 updated by path optimization based on the first closed path 10 is indicated by a dashed line.

[0102] As described above, the pose graph optimization section 255 according to an embodiment of the present application can be configured to acquire closed path information that constructs the first closed path 10 from a portion of the movement path of the robot 300 from the odometry estimation section 253, and to update the map information collected along the first closed path 10 on the subsequent movement path based on the map generated at the start point and the end point of the first closed path 10 when generating the second closed path 20.

[0103] Generally speaking, because the absolute coordinates of the robot 300 cannot be known in indoor space, the odometer estimation unit 253 generates a map of the indoor space by merging the surrounding images continuously captured based on relative movement. However, due to the characteristics of disaster sites, errors may occur due to factors such as the movement and posture changes of the robot 300, and the measurement environment may also change dynamically.

[0104] Therefore, according to the pose graph optimization unit 255 of an embodiment of the present invention, for example, a starting point A for constituting the first closed path 10 can be set, and the robot 300 stores and manages the surrounding images and sensor data collected in the first closed interval 10 on the premise that the surrounding images and sensor data acquired at point A are the same as the surrounding images and sensor data acquired when returning to point A, and then, when performing the optimization of the second closed interval 20, performs sensor data correction and optimization using the information of the first closed interval 10.

[0105] The pose graph optimization unit 255 may predefine conditional information for setting the starting point A. The starting point A may be determined based on surrounding images and sensor data captured during the movement of the robot 300. For example, a location with a high probability of return and / or a location that facilitates movement to the next location may be selected as the starting point and return point A. The location with a high probability of return may be determined based on the state of the ground (material, space, and smoothness), whether there are two or more paths available for the mobile robot to move around, and the inclination. The location that facilitates movement to the next location may be, for example, a step for moving to an upper or lower level.

[0106] In particular, in one example, if Figure 7 For the first floor Figure 8 To ascend one level from the first floor to the second floor, step 701-1 can be a path connecting the first and second floors, while step 711 can be a path connecting to the third floor. When capturing images for creating indoor maps within a multi-story building, environmental changes that occur when moving between floors can cause problems such as difficulty matching images captured at various points or errors. To address this issue, robot 71 can identify the location of the steps and capture images at shorter intervals than typical scanning locations when ascending or descending the steps. Furthermore, the locations of the steps can be identified as absolute coordinates within the building, so the steps can be used as reference points to determine the starting or ending point of a closed path.

[0107] Re-read Figure 8, are performed while moving to the 7th to 15th scan positions, respectively, and in selecting the scan positions, one or more of the stability and height of the floor (preferably selecting a higher position compared to a lower position) and the brightness is decided, or is additionally or solely decided based on the size and type of the object in the image taken at the past scan position. For example, when an object in the 14th direction (front) is recognized from the scan image taken at the 13th position, and there is no newly added object when moving in the 14th direction, the next scan position can be decided after moving a distance (8 m in the present embodiment) farther than in the general case. In deciding the scan position, a position where an object not recognized at the past scan position can be decided. Further, there can be a problem in that the accuracy of object recognition in the image acquired in real time can be lowered due to the influence of the illumination in the room, additional collapse, or a beacon light entering from the outside. Figure 8

[0108] In the above-described case, the robot according to one embodiment of the present application can set a critical value for a newly occurring environmental change (temperature, change in the object in the image, and change in brightness), and immediately stop moving when one of the elements in the set critical value exceeds the critical value, and decide the position where the movement is stopped as a new scan position.

[0109] Further, in deciding the scan position, it can be necessary to exclude elements irrelevant to the making of the indoor map from the external environmental factors. For this, based on the type or position of the recognized object, it can be possible not to stop moving even in the case where the critical value is exceeded. For example, for other objects captured in a window object or a change in brightness captured in a window object, etc., it can be possible to decide whether to stop moving after recognizing the additional state. This is because in the case where there is a large window and the movement of a construction device is captured in the window or the light of a repair device is radiated into the inside, etc., it is necessary to ignore them.

[0110] Further, referring to Figure 8 , a case where the originally planned moving direction of the robot (scan positions 7 to 15) is changed is illustrated. Referring to Figure 8 , the robot coming up the stairs 701-1 can decide one of the 15th scan position side and the 8th scan position side by scanning the surroundings in the 7th position. For example, in the case where there are many cracks or the collapse risk is large in the 15th scan position side, in order to preferentially move in the 15th scan position direction, it can be possible to move in the direction of the 7th to 15th to 14th position. Further, the position 77 is a position where the path closure is completed in the moving path, and in the case where the floor at the corresponding position 77 is not flat or it is difficult to acquire the surrounding image, it can be possible to perform the path closure and the image synthesis based on the image acquired at the position B in the above-described manner. In the above-described case, the robot according to one embodiment of the present application can set a critical value for a newly occurring environmental change (temperature, change in the object in the image, and change in brightness), and immediately stop moving when one of the elements in the set critical value exceeds the critical value, and decide the position where the movement is stopped as a new scan position.​Figure 8 In the figure, scan positions 7 to 15 do not refer to the absolute movement direction of the robot.

[0111] Furthermore, the pose graph optimization unit 255 can select the starting point A based on a specific area within a certain range adjacent to a unique object (e.g., an emergency exit sign) that is easily identifiable in the surrounding image. The position information of the determined starting point A can then be transmitted to the robot 300 via the robot drive unit 251.

[0112] In addition, if Figure 8 As shown, when setting the starting point A, the end point of the first closed section 10 must also be set to A. However, during the movement of the robot 300, it may be difficult to return to the originally set position and form a closed section (path closure). For example, a wall may suddenly collapse or an object may fall, creating an obstacle.

[0113] In such situations, the robot driver 251 according to an embodiment of the present invention can control the robot 300 to dynamically change its starting and ending points based on the surrounding environment. For example, if the robot driver 251 determines point A as the starting and ending point of the first closed path 10 and controls the path, and then cannot return to point A due to obstacles or structural changes, the robot driver 251 can determine a specific point B, whose relative position to point A is clearly known, as the alternative end point of the closed path, thereby optimizing the pose graph optimization unit 255 to form the second closed path 20.

[0114] Among them, because the absolute coordinates of point A and point B are known, the relative displacement between the two can be calculated as a fixed value. Even if the robot 300 does not return to point A, the pose graph optimization unit 255 can map part of the images and sensor data collected in the first closed path 10 to the second closed path 20 that optimizes the image information and sensor data captured at point B using the displacement value between A and B. The precision map construction unit 257 can use it to update the overall indoor map, thereby maintaining the precision of the indoor map.

[0115] For example, see Figure 7 as well as Figure 9 At both points A and B, there are objects, namely emergency exits, whose absolute coordinates are known. This allows the precise map construction unit 257 to utilize the relative positional difference between the two points. Even when the robot 300 moves to the second closed path 20, it can use the existing information of a portion of the section assuming it departed from and returned to A on the first closed path 10 to perform additional updates to the indoor map based on the images and sensor data collected on the current second closed path 20.

[0116] Further, as the B point which is the alternative end point, a position displacement of at least a position close to the A point within a certain distance or a position which can include the environment of the A point in the peripheral image is selected. This is because when the A position is included in the peripheral image photographed at the B point, the position of the line of sight and the map operation can be corrected according to the relative difference.

[0117] The setting and the restriction of each point as described above can be processed according to the condition setting of the map generation path. For this, the robot driving part 251 can set each path condition and command the robot 300 to perform the position movement and the sensor data collection based thereon.

[0118] Specifically, the robot driving part 251 sets the condition information of the first closed path corresponding to the first position such as the A, the condition information including a condition of selecting a point which the robot can return from among the peripheral positions as the first position, the returnable point being determined based on at least one of the state information of the ground obtained from the sensor information of the robot, information of whether there are two or more passages available for the mobile robot to move in the periphery, and the inclination information.

[0119] Further, the condition information can include condition information of determining the first position as a specific area within a certain adjacent range based on the position information of an inherent object which can be easily recognized in the image in the periphery, and the condition information can include condition information of determining a second position (B point) adjacent to the relative position information of the first position within a certain distance as an alternative end point of the first closed path in the case where the mobile robot cannot return to the first position due to an obstacle or a structural change.

[0120] Further, in the case where the second position is determined as the alternative end point, the second map generation path composed of a second closed path 20 which moves between the first position and the second position can be determined, and the precise map construction part 257 can perform the update of the indoor map based on the image and the sensor data collected on the second closed path using the original collected information of at least a part of the section from the first position of the first closed path to the first position even in the case where the robot moves on the second closed path.

[0121] Further, as described above, it is preferable that the second position be determined as a point which includes the environment of the first position in the peripheral image collected by the robot in order to generate the map, so that the relative difference can be corrected.

[0122] Figure 9 is a diagram illustrating the results of the accuracy test experiment using the embodiment of the present application.

[0123] Referring to Figure 9 , the performance of the existing open path pipeline is distinguished using blue, and the performance of the closed path pipeline according to the embodiment of the present application is distinguished using yellow. Among them, the map generated by the BLK360 high-precision sensor and the three-dimensional map generated by the simultaneous localization and mapping (SLAM) algorithm based on each pipeline are compared, and the accuracy thereof is evaluated and tested. As Figure 9 indicated, it can be confirmed that the current estimate value can be appropriately corrected and the error accumulated in the odometer can be removed through the optimized local map, so that the overall performance can be improved while improving the odometer performance.

[0124] The above-described embodiments can be at least partially implemented through a computer program and recorded in a computer-readable storage medium. The computer-readable storage medium recorded with the program for implementing the embodiments includes any storage device that can store computer-readable data. Examples of the computer-readable storage medium include read-only memory (ROM), random access memory (RAM), compact disc (CD-ROM), magnetic tape, and optical data storage device. In addition, the computer-readable storage medium can be distributed over network-coupled computer systems so that the computer-readable code is stored and executed in a distributed manner.

[0125] In addition, the functional programs, codes, and code segments for implementing the embodiments will be easily understood and implemented by those skilled in the art. Although the present disclosure has been described in conjunction with the embodiments shown in the drawings, this description is only for illustrative purposes, and those skilled in the art can understand that various modifications and changes can be made without departing from the spirit and scope of the present disclosure. However, it should be noted that these modifications should be considered to fall within the scope of the technical protection of the present disclosure. Therefore, the true technical protection scope of the present disclosure should be interpreted according to the appended claims and their technical spirits, including other implementations, other embodiments, and the equivalents of the claims.

Claims

1. A method for refining an indoor map using a closed path, As a method for refining an indoor map using a closed path, which is performed by a computing device, comprising: a step of moving a position of a robot for generating an image and sensor data for a refined map to a first position using at least one of a sensor module including a laser radar sensor, an inertial measurement unit (IMU) sensor, and one or more vision sensors; a step of constructing a first closed path optimized in a pose graph from a dead reckoning estimation from the first position in order to generate a first map using movement of the robot; a step of updating at least a portion of a second map generation path and sensor data corresponding to a current path of the robot using at least one of path information of the first closed path and the sensor data; and a step of outputting a second map updated based on the second map generation path and the sensor data updated; further comprising a step of setting condition information of the first closed path corresponding to the first position; the condition information, including a condition of selecting a point from which the robot can return as the first position from among surrounding positions, the returnable point being determined based on at least one of state information of a ground obtained from sensor information of the robot, information of whether there are two or more passages available for movement of the robot in the surroundings, and inclination information, the condition information, including condition information of determining a second position adjacent within a certain distance with relative position information of the first position as an alternative end point of the first closed path in a case where the robot cannot return to the first position due to an obstacle or a structural change, in a case where the second position is determined as the alternative end point, determining the second map generation path consisting of a second closed path moving between the first position and the second position, the updating step including a step of performing update of an indoor map based on an image and sensor data collected on the second closed path using original collected information of at least a portion of an interval from the first position of the first closed path to the first position and back even in a case where the robot moves on the second closed path. 2.The method for refining an indoor map using a closed path according to claim 1, the updating step including: a step of performing pose graph optimization of the second map generation path using at least one of path information of the first closed path and the sensor data. 3.The method for refining an indoor map using a closed path according to claim 2, a step of updating local map information corresponding to a current path of the robot and a dead reckoning estimation using local map information updated according to the pose graph optimization of the second map generation path. 4.The method for refining an indoor map using a closed path according to claim 2, the pose graph optimization of the second map generation path, including loop closure optimization that optimizes the overall path of the robot (300) according to the first closed path, and smoothing optimization that re-optimizes at least a part of the path that has been passed through by the first closed path using information that has been collected so far.

5. The indoor map refinement method using a closed path according to claim 1, Also included are: a step of setting condition information of the first closed path corresponding to the first position; the condition information, including a condition of selecting a point that the robot can return from among surrounding positions as the first position, the returnable point is determined based on all information of state information of the ground obtained from sensor information of the robot, information of whether there are two or more passages available for the mobile robot to move in the surrounding, and inclination information.

6. The indoor map refinement method using a closed path according to claim 5, the condition information, including condition information of determining the first position as a specific area within a certain adjacent range based on position information of an inherent object that can be easily recognized within an image in the surrounding position.

7. The indoor map refinement method using a closed path according to claim 5, the condition information, including condition information of determining a second position that is adjacent within a certain distance and has relative position information with the first position as an alternative end point of the first closed path in a case where the mobile robot cannot return to the first position due to an obstacle and a structural change.

Images