Visual positioning method, control server using the same, and building

Abstract

The present invention relates to a visual positioning method, and a control server and a building using the same. The visual positioning method according to one embodiment of the present invention may include a step of receiving from a robot a surrounding image of a space in which the robot is located, a step of selecting a positioning module according to spatial characteristics set in the space, and a step of generating positioning information for the robot from the surrounding image using the selected positioning module.

Description

Technical Field

[0001] The present invention relates to a visual positioning method for positioning the position and pose (pose: posture) of a robot in order to control the running of the robot, a control server using the same, and a building.

Background Art

[0002] VL (Visual Localization: visual positioning, visual position determination) is a technology that can estimate the global position of a robot based on an image captured by the robot or the like. The robot transmits the captured image to a VL server, and the VL server can estimate the global position of the robot based on the received image. However, such VL requires a preliminary operation of scanning the entire space to generate an overall map of the space. At this time, expensive equipment such as a lidar is required. Moreover, when recognizing the position of a robot using only VL without a given initial position, a false positive (false positive) is likely to occur because an overall search is performed using a query image. In addition, in an open space such as the outdoors, there is a problem that existing VL is difficult to utilize due to changes in scale and environment.

[0003] As another example, there is a technology for estimating the global position of a robot by utilizing a two-dimensional image marker. For example, Korean Patent Publication No. 10-2006-0129960 relates to a mobile robot and a method for calculating its position and pose. The map data memory stores map data of the moving area, position data of markers at predetermined positions in the moving area, identification data of the markers, and position data of boundary lines adjacent to the markers in the moving area. The marker detection unit detects a marker from an image based on the position data and identification data of the marker, the boundary line detection unit detects a boundary line adjacent to the marker from the image, and the parameter calculation unit calculates the parameters of the boundary line from the image. Then, the position and pose calculation unit calculates the position and pose of the mobile robot in the moving area based on the parameters of the boundary line and the position data of the boundary line. [Overview of the project] [Problems that the invention aims to solve]

[0004] The present invention aims to provide a visual positioning method that can perform more accurate visual positioning by utilizing not only general visual positioning methods but also multiple visual positioning methods specialized according to the spatial characteristics of each space, as well as a control server and building that use this method.

[0005] The present invention aims to provide a visual positioning method that has robust properties to spatial characteristics by integrating multiple positioning modules that perform visual positioning, as well as a control server and a building that use this method.

[0006] The present invention aims to provide a building that includes a robot that provides services while moving around inside the building. [Means for solving the problem]

[0007] A visual localization method according to one embodiment of the present invention relates to a visual localization method for a control server that determines the position of a robot within a target area based on surrounding images acquired from a camera mounted on the robot, and may include the steps of: receiving surrounding images from the robot for the space in which the robot is located; applying a positioning module based on spatial characteristics set in the space; and generating positioning information for the robot from the surrounding images using the positioning module.

[0008] A control server according to one embodiment of the present invention performs visual localization to determine the position of the robot within a target area based on surrounding images acquired from a camera mounted on the robot, and may include a video receiving unit that receives surrounding images of the space in which the robot is located from the robot, a positioning module determination unit that applies a positioning module based on the spatial characteristics set in the space, and a positioning unit that generates positioning information for the robot from the surrounding images using the positioning module.

[0009] A building according to one embodiment of the present invention is equipped with at least one robot that provides services while traveling within the building, the robot being controlled by communication with a control server, the control server performing visual localization to determine the robot's position within the building based on surrounding images acquired from a camera mounted on the robot, and includes at least one processor embodied to execute computer-readable instructions, the at least one processor receiving surrounding images from the robot regarding the space in which the robot is located, applying a positioning module based on the spatial characteristics set in the space, and generating positioning information for the robot from the surrounding images using the positioning module.

[0010] It should be noted that the means for solving the above-mentioned problems do not fully describe all the features of the present invention. The various features of the present invention and the advantages and effects thereof can be understood in more detail by referring to the specific embodiments described below. [Effects of the Invention]

[0011] According to one embodiment of the present invention, the visual positioning method, the control server and building using the same, in addition to general visual positioning methods, multiple visual positioning methods specialized for the spatial characteristics of each space are also utilized, making it possible to perform more accurate visual positioning.

[0012] A visual positioning method according to one embodiment of the present invention, a control server using the same, and a building can perform visual positioning for a robot by integrating multiple positioning modules specialized for spatial characteristics, thereby providing robust visual positioning for various spatial characteristics.

[0013] However, the effects that can be achieved by the visual positioning method according to the embodiment of the present invention, the control server using the same, and the building are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those with ordinary skill in the art to which the present invention pertains from the following description. [Brief explanation of the drawing]

[0014] [Figure 1] This is a schematic diagram showing a control system for controlling the operation of a robot according to one embodiment of the present invention. [Figure 2] This is a block diagram showing a control server according to one embodiment of the present invention. [Figure 3] This is an illustrative diagram showing visual positioning using a main positioning module according to one embodiment of the present invention. [Figure 4] This is an illustrative diagram showing visual positioning using a marker positioning module according to one embodiment of the present invention. [Figure 5] This is an illustrative diagram showing visual positioning using a signage positioning module according to one embodiment of the present invention. [Figure 6] This is an illustrative diagram showing visual positioning using an elevator positioning module according to one embodiment of the present invention. [Figure 7] This is a schematic diagram showing a control server according to one embodiment of the present invention. [Figure 8] This is a schematic diagram showing a building in which a control system using visual positioning according to one embodiment of the present invention is constructed. [Figure 9] This flowchart shows a visual positioning method according to one embodiment of the present invention. [Modes for carrying out the invention]

[0015] The embodiments disclosed herein will be described in detail below with reference to the accompanying drawings. However, in the drawings, identical or similar components will be given the same reference numeral, and redundant descriptions thereof will be omitted. The suffixes “module” and “unit” used in the following description for components are considered or used interchangeably for the sake of facilitating the writing of the specification and do not have a distinct meaning or role in themselves. That is, the term “unit” as used in the present invention means a software, FPGA or ASIC or other hardware component, and a “unit” can play any role. However, “unit” is not limited to software or hardware. A “unit” may be configured to reside in an addressable storage medium and may be configured to regenerate one or more processors. Thus, for example, a “unit” includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and the functions provided by the "units" may be combined into a smaller number of components and "units," or they may be further separated into additional components and "units."

[0016] Furthermore, when describing the embodiments disclosed herein, detailed explanations of relevant prior art are omitted if it is determined that such explanations would obscure the essence of the embodiments disclosed herein. In addition, the accompanying drawings are merely for the purpose of facilitating the understanding of the embodiments disclosed herein, and the accompanying drawings do not limit the technical idea disclosed herein. All modifications, equivalents, or substitutions within the concept and technical scope of the invention should be understood as being included in the invention.

[0017] Figure 1 is a schematic diagram showing a control system for controlling the operation of a robot according to one embodiment of the present invention.

[0018] Referring to FIG. 1, a control system according to an embodiment of the present invention may include a robot 1 and a control server 100.

[0019] Hereinafter, referring to FIG. 1, a control system according to an embodiment of the present invention will be described.

[0020] The robot 1 can perform various services while moving within the target area A. Here, the target area A may be an indoor space such as a building. The robot 1 can move within the target area A in an autonomous driving mode, but the positioning of the robot 1 or the control of its movement may be performed by the control server 100. That is, the robot 1 can perform operations such as movement through communication with the control server 100.

[0021] The robot 1 may include a camera and can use the camera to capture peripheral images corresponding to the movement of the robot 1. The robot 1 can transmit the peripheral images to the control server 100, and the control server 100 can perform visual localization on the robot 1 based on the received peripheral images.

[0022] According to some embodiments, in addition to the camera, the robot 1 may further include various sensors such as an IMU (Inertial Measurement Unit) sensor, a wheel encoder sensor, and a lidar sensor. In addition to visual localization, the control server 100 can also perform further localization on the robot 1 in other ways.

[0023] The control server 100 can communicate with the robot 1 located within the target area A via the network N, and can control various actions of the robot 1, including its movement. Here, the communication method between the control server 100 and the robot 1 is not limited and can include communication methods that utilize communication networks that the network N may include (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks, satellite networks, etc.), as well as short-range wireless communication between devices. For example, the network N may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the internet. Furthermore, the network N may include, but is not limited to, one or more arbitrary network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks.

[0024] The control server 100 communicates with the robot 1 via the network N and can be implemented as a computer device or multiple computer devices that provide commands, codes, files, content, services, etc. Depending on the embodiment, the control server 100 can also be implemented as a cloud server.

[0025] In order for robot 1 to move within target area A, it is necessary to perform positioning to generate positioning information including robot 1's current position and pose. Here, specific map information for target area A may be stored in advance, and the control server 100 can match robot 1's current position and pose with the map information and map robot 1's current position and pose within the map information. Depending on the embodiment, the position and pose can be represented by 3D coordinate values ​​and 3-axis rotation values ​​(pitch, roll, yaw), respectively.

[0026] The control server 100 can receive surrounding video from the robot 1 and perform visual positioning of the robot 1 based on the received surrounding video. In other words, the control server 100 can confirm the precise position and pose of the robot 1 in the map information from the results of the visual positioning of the robot 1, and based on this, can determine the direction of travel, direction of rotation, speed of travel, etc. of the robot 1 and control the movement of the robot 1.

[0027] Specifically, the control server 100 uses the received surrounding video as a query image and can search for a reference image corresponding to the query image from the map information. Here, the position and pose of the measuring equipment corresponding to the reference image may be set, so by comparing the reference image with the surrounding video, it is possible to determine the current position and pose of robot 1. In other words, the position and pose of robot 1 corresponding to the query image can be determined by local feature matching.

[0028] However, if the space in which the robot 1 is moving is repetitive or lacks spatial features, problems may arise such as a decrease in the performance of visual positioning using surrounding images. On the other hand, the control server 100 according to one embodiment of the present invention utilizes not only general visual positioning methods but also multiple visual positioning methods specialized according to the spatial characteristics of each space, making it possible to perform more accurate visual positioning. The control server according to one embodiment of the present invention will be described below.

[0029] Figure 2 is a block diagram showing a control server 100 according to one embodiment of the present invention.

[0030] Referring to Figure 2, the control server 100 according to one embodiment of the present invention may include a video receiving unit 110, a positioning module determination unit 120, a positioning unit 130, and a map database 140.

[0031] The video receiving unit 110 can receive surrounding video of the space in which the robot 1 is located from the robot 1. The robot 1 can generate surrounding video by taking pictures of its surroundings while moving using a camera installed on the robot itself, and can transmit the captured surrounding video to the video receiving unit 110 via the network. In some embodiments, the camera can be positioned to point vertically upward in order to minimize the obscuration of the surrounding video by objects or people close to the robot 1 and to reduce changes in scale within the video. That is, the surrounding video may be a picture taken above the robot 1, capturing the ceiling of the target area A.

[0032] The positioning module determination unit 120 can apply a positioning module based on the spatial characteristics set for the space. That is, the control server 100 may contain multiple positioning modules, and the positioning module determination unit 120 can apply the positioning module that is most suitable for the spatial characteristics of the space where the robot 1 is currently located from among the multiple positioning modules.

[0033] Specifically, the target area A may be divided into multiple spaces, and the map information for the target area A stored in the map database 140 may contain spatial characteristics specific to each space. For example, if the target area A is a building, the building may contain spaces such as a separate charging station to assist in charging the robot 1, and since precise position control is required for charging the robot 1 within the charging station, markers or the like may be included to assist in this. In addition, in the office space within the building, signage such as signboards may be installed according to a set standard defined by certain rules so that workers can recognize their position within the space, and in the case of an elevator within the building, specific spatial characteristics within the elevator space may be included.

[0034] In other words, since the map information contains spatial characteristics for each space, the optimal positioning module can be pre-set based on these spatial characteristics. Therefore, the positioning module determination unit 120 can apply each pre-set positioning module to each space based on its spatial characteristics. In this case, since the optimal positioning module can be selected according to the spatial characteristics of each space, it becomes possible to perform visual positioning within that space more accurately and quickly.

[0035] Here, the positioning module determination unit 120 first needs to distinguish which space each robot 1 is currently located in. To this end, the positioning module determination unit 120 can utilize map information corresponding to the target area A and the robot 1's previous positioning information. The map information may be precisely generated and stored in advance so that the robot 1 can travel within the target area A, and the map information may contain pre-stored information on each space included within the target area A and the spatial characteristics set for that space. Therefore, based on the robot 1's previous positioning information, the positioning module determination unit 120 can determine which space the robot 1 is currently located in on the map information and extract the spatial characteristics set for that space. In some embodiments, the positioning module determination unit 120 can also obtain the odometry of the robot 1 by utilizing another IMU sensor or wheel encoder sensor included in the robot 1, and apply this odometry to the previous positioning information to identify the space where the robot 1 is currently located.

[0036] Furthermore, it is also possible to implement a configuration in which the positioning module determination unit 120 distinguishes between different spaces from the surrounding video. For example, if the surrounding video includes markers or signage, the positioning module determination unit 120 can determine the characteristics of each space, such as the marker space or the signage space, and apply the corresponding positioning module.

[0037] Here, the positioning module may include a main positioning module V1, a marker positioning module V2, a signage positioning module V3, an elevator positioning module V4, etc. However, it is not limited to this, and depending on the embodiment, it may further include various positioning modules, etc.

[0038] The main positioning module V1 can extract feature points from surrounding video footage, compare these feature points with feature points in map information, and generate positioning information for robot 1. Here, the map database included in the control server 100 may contain a feature point map corresponding to the target area A. Therefore, after setting the surrounding video footage as the query image, the main positioning module V1 can extract global feature points from the query image using a deep learning model or the like. Then, as shown in Figure 3, the extracted feature points can be used to search the feature point map for reference image i2 corresponding to query image i1. Here, each reference image may be tagged with information such as the position and pose of the measurement equipment used when generating the feature point map. Therefore, the main positioning module V1 can compare the reference image and the query image to identify the current position and pose of robot 1 and generate positioning information.

[0039] On the other hand, since the main positioning module V1 is applicable to the entire space of the target area A where the feature point map is generated, the positioning module determination unit 120 can apply the main positioning module V1 to perform positioning in general spaces, and thereafter, when entering a specific space, it can apply a positioning module corresponding to the characteristics of that space. At this time, the positioning module determination unit 120 can switch from the main positioning module to other positioning modules, but depending on the embodiment, it is also possible to have other positioning modules operate together with the main positioning module V1. That is, the positioning module determination unit 120 may select multiple positioning modules instead of just one, and perform visual positioning of the robot 1's position.

[0040] The marker positioning module V2 can extract markers from surrounding video footage and generate positioning information for robot 1 based on the coordinate information recognized from the markers.

[0041] Markers may be manufactured in the form of printed materials that can be attached to surfaces, such as posters, wallpaper, or stickers. In some embodiments, markers M can be generated in the form of April tags, as shown in Figure 4(a). Each marker M may contain a different identification code, which can be used to distinguish between the markers. The marker positioning module V2 can use the identification codes of the markers M to extract pre-set 3D coordinate information for each marker M1, M2, as shown in Figure 4(b). The map database 140 may contain information about the marker space to which each marker in the target area A is attached, and a marker map stored by matching the identification tag of each marker with the 3D coordinates to which the marker is attached. Therefore, the marker positioning module V2 can use the surrounding video captured by the robot 1 as a query image, detect markers M from within the query image, and obtain positioning information, including position and pose relative to the query image, based on the coordinate information matched to the identification tag contained within the marker M. In other words, the marker positioning module V2 can generate positioning information for the position and pose of robot 1 from the position and angle of marker M included in the query image.

[0042] On the other hand, since the marker positioning module V2 can perform positioning only within the marker space, the positioning module determination unit 120 can apply the marker positioning module V2 as the positioning module if the spatial characteristics of the space where the robot 1 is located are those of the marker space. In other words, the marker space may be a space where precise control of the robot 1's position is necessary, or a space where accurate positioning is difficult with the existing main positioning module V1 alone, and the positioning module determination unit 120 can apply the marker positioning module V2 if it corresponds to a space set as a marker space in the map information.

[0043] The Signage Positioning Module V3 can extract signage from surrounding video footage, obtain the coordinate information of the signage, and then generate positioning information for Robot 1 based on the coordinate information of the signage.

[0044] The signage may be installed in a predetermined location within the space according to a set standard, and may be a signboard installed so that workers, robots, etc., within the target area A can recognize their location within the internal space. In some embodiments, as shown in Figure 5(a), signage S1 and S2 may be attached to the space. Here, since the signage is installed on a predetermined location according to a set standard, the signage positioning module V3 can extract the signage included in the surrounding video and obtain the coordinate information of the signage from the signage's set standard. In some embodiments, the signage positioning module V3 can also recognize characters on the signage using OCR (Optical Character Reader), distinguish the signage by the recognized characters, and extract the 3D coordinate information of the signage. The map database 140 may include a signage map that maps and stores information about the signage space to which each signage within the target area A is attached, and the 3D coordinates to which each signage is attached. Therefore, the signage positioning module V3 can use the surrounding video captured by robot 1 as a query image, detect the signage S from within the query image, and obtain positioning information, including the position and pose relative to the query image, based on the coordinate information obtained from the signage S.

[0045] On the other hand, since the signage positioning module V3 can perform positioning only within the signage space, the positioning module determination unit 120 can apply the signage positioning module V3 as the positioning module if the spatial characteristics of the space where the robot 1 is located are those of a signage space.

[0046] The elevator positioning module V4 can extract specific feature points of the elevator from surrounding video footage and generate positioning information for robot 1 based on these feature points. Since the interior of an elevator generally has a rectangular ceiling shape, the rectangular ceiling shape can be extracted as a feature of the elevator, and positioning information for the position and pose of robot 1 inside the elevator can be generated based on the elevator ceiling shape that appears in the surrounding video footage. Depending on the embodiment, the elevator may be a dedicated elevator that only robot 1 rides in.

[0047] Specifically, as shown in Figure 6(a), the elevator ceiling may appear as a rectangle R, and as shown in Figure 6(b), specific feature points F1, F2, and F3 of the elevator can be extracted from the rectangular ceiling shape that appears in the surrounding image. For example, positioning information for the position and pose of robot 1 inside the elevator can be generated based on the gradients F1 and F2 of each side forming the rectangle, coordinate information F3 where the center point of the surrounding image is located within the rectangle, etc. Furthermore, the map database 140 may include an elevator map that includes information about the elevator space in which elevators are installed within the target area A, and the types of specific feature points to be extracted from each elevator. Therefore, the elevator positioning module V4 can use the surrounding image captured by robot 1 as a query image, extract specific feature points of the elevator from the query image, and obtain positioning information including the position and pose relative to the query image based on the specific feature points.

[0048] On the other hand, since the elevator positioning module V4 can only perform positioning within the elevator space, the positioning module determination unit 120 can apply the elevator positioning module V3 as the positioning module if the spatial characteristics of the space where the robot 1 is located are those of the elevator space.

[0049] In this example, it is assumed that the positioning module determination unit 120 determines the positioning module based on the spatial characteristics of each space. However, in some embodiments, it is also possible to implement a system that uses another positioning module when positioning using one positioning module fails. For example, the positioning module determination unit 120 may be set to perform positioning using the main positioning module V1 as the base, and thereafter, if positioning fails due to a decrease in the accuracy of positioning by the main positioning module V1, the positioning module can be changed to apply the marker positioning module V2 or the signage positioning module V3.

[0050] Specifically, the main positioning module V1 can apply a RANSAC (Random Sample Consensus) algorithm-based PNP (Perspective-N-Point) after matching a reference image with a query image. That is, by comparing the matched feature points in the query image and the reference image using PNP, the pose of robot 1, such as its rotation angle, can be determined. However, there may be cases where the number of feature points classified as normal (inlier) among the matched feature points in the query image and the reference image is less than or equal to a set number. In this case, it can be determined that spatial positioning using the main positioning module V1 has failed. In other words, this corresponds to a situation where the matching between the query image and the reference image is not sufficient to allow comparison, resulting in an inaccurate positioning result from the main positioning module V1. Therefore, the positioning module determination unit 120 can apply a marker positioning module V2 or a signage positioning module V3 instead of the main positioning module V1, and in some embodiments, the marker positioning module V2 and the signage positioning module V3 can be applied simultaneously.

[0051] Within the space included in the target area A, there may be areas where spatial positioning by the main positioning module V1 is difficult due to its characteristics. Markers or signage can be pre-attached to such areas. Therefore, the positioning module determination unit 120 can be implemented to attempt positioning using the marker positioning module V1 or the signage positioning module V2 instead of the main positioning module V1 if positioning by the main positioning module V1 fails.

[0052] The positioning unit 130 can generate positioning information for the robot from surrounding video footage using positioning modules. Specifically, positioning information can be generated by performing positioning using the main positioning module V1, marker positioning module V2, signage positioning module V3, elevator positioning module V4, etc. However, the specific positioning methods used by each positioning module have been described above, so the details will be omitted here.

[0053] On the other hand, the positioning module determination unit 120 can apply multiple positioning modules simultaneously, in which case the positioning unit 130 can perform positioning using multiple positioning modules. Specifically, the positioning unit 130 can collect positioning information measured by each positioning module to generate final positioning information. At this time, it can set a weight value for each positioning module and combine the positioning information to which each weight value has been applied to generate the final positioning information. For example, when the main positioning module V1 and the marker positioning module V2 are selected simultaneously, the weight value of the main positioning module V1 can be set to 1 and the weight value of the marker positioning module V2 can be set to 9 to generate the final positioning information. In addition, positioning information measured by each positioning module can be combined by utilizing pose graph optimization or Kalman filters, which are widely used in the field of sensor fusion. Depending on the implementation, the dynamic covariance scaling method among the pose graph optimization techniques can be utilized. This allows for robust pose estimation by adjusting the covariance during the optimization process, even if some inaccurate poses are input.

[0054] Figure 7 is a block diagram showing a control server 100 according to one embodiment of the present invention.

[0055] Referring to Figure 7, the control server 100 may include memory 10, a processor 20, a communication interface 30, an input / output interface 40, etc. Memory 10 is a computer-readable recording medium and may include RAM (random access memory), ROM (read-only memory), and a permanent mass storage device such as a disk drive. Here, ROM and the permanent mass storage device such as a disk drive may be included in the control server 100 as separate permanent storage devices distinct from memory 10. Memory 10 may also store an operational system and at least one program code. Such software components may be loaded into memory 10 from a computer-readable recording medium separate from memory 10. Such a separate computer-readable recording medium may include a floppy drive, a disk, magnetic tape, a DVD / CD-ROM drive, a memory card, etc. In other embodiments, the software components may be loaded into memory 10 via the communication interface 30 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 10 of the control server 100 based on a computer program installed by a file received through the network N.

[0056] The processor 20 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 20 by memory 10 or a communication interface 30. For example, the processor 20 may be configured to execute instructions received by program code stored in a recording device such as memory 10.

[0057] The communication interface 30 can provide a function for the control server 100 to communicate with other devices (e.g., the aforementioned storage devices) via the network N. For example, requests, instructions, data, files, etc., generated by program code stored in a storage device such as memory 10 by the processor 20 of the control server 100 may be transmitted to other devices via the network N under the control of the communication interface 30. Conversely, signals, instructions, data, files, etc., from other devices may be received via the network N through the communication interface 30 of the control server 100. Signals, instructions, data, etc., received via the communication interface 30 may be transmitted to the processor 20 or memory 10, and files, etc., may be stored in a storage medium (the aforementioned permanent storage device) that the control server 100 may further include.

[0058] The input / output interface 40 may be a means for interface with the input / output device 50. For example, the input device may include a microphone, keyboard, or mouse, and the output device may include a display or speaker. As another example, the input / output interface 40 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 50 may be configured as a single device with the control server 100.

[0059] Figure 8 is a schematic diagram showing a building B on which a control system using visual positioning according to one embodiment of the present invention is constructed. As shown in Figure 8, building B can be set as a target area, and multiple robots 1 that provide services within the target area may be included within building B. Building B also includes an indoor area on which the robots travel, and the indoor area may be divided into multiple spaces according to its spatial characteristics. Here, among the multiple spaces, building B may include a marker space on which markers are attached, a signage space including signage, an elevator space including an elevator, etc. The robots 1 can communicate with a control server 100 via a network N, and can perform actions such as movement under the control of the control server 100. The control server 100 does not have to be constructed within building B, and may be implemented as a cloud server or the like, separate from building B.

[0060] The control server 100 can perform visual positioning to determine the location of each robot 1 within building B based on the surrounding video footage acquired from the camera installed on robot 1. The specific operation of the control server 100 has been described above, so a detailed explanation will be omitted here.

[0061] Figure 9 is a flowchart illustrating a visual positioning method according to one embodiment of the present invention. Here, each step in Figure 9 may be performed by a control server according to one embodiment of the present invention.

[0062] The control server can receive surrounding video footage of the space in which the robot is located from the robot (S10). The robot can generate surrounding video footage by taking pictures of its surroundings while moving using a camera installed on the robot itself, and can transmit the captured surrounding video footage to the control server via the network. In some embodiments, the camera can be positioned to point vertically upward in order to minimize the obstruction of the surrounding video footage by objects or people close to the robot. That is, the surrounding video footage may be footage of the ceiling of the target area taken above the robot.

[0063] The control server can select a positioning module based on the spatial characteristics set for the space (S20). That is, the control server may contain multiple positioning modules, and the control server can select the positioning module that is best suited to the spatial characteristics of the space where the robot is currently located.

[0064] Specifically, the target area may be divided into multiple spaces, and the map information stored in the map database may have pre-defined spatial characteristics for each space included within the target area. Therefore, after distinguishing each space in which the robot is located, the control server can select the optimal positioning module based on the spatial characteristics of each space, thereby enabling more accurate and rapid visual positioning within that space.

[0065] Here, the control server first needs to determine which space each robot is currently located in. To do this, the control server can utilize map information corresponding to the target area and the robot's previous positioning information. In other words, based on the robot's previous positioning information, the control server can determine which space the robot is currently in on the map information, and use this to extract the spatial characteristics set for that space. In some embodiments, the robot's odometry can be obtained using sensor values ​​received from other IMU sensors or wheel encoder sensors included in the robot, so the control server can apply this odometry to the previous positioning information to accurately identify the space in which the robot is currently located.

[0066] Here, the positioning module may include a main positioning module, a marker positioning module, a signage positioning module, an elevator positioning module, etc. The marker positioning module can extract markers from the surrounding video and generate robot positioning information based on the coordinate information recognized from the markers. The signage positioning module can extract signage from the surrounding video, obtain the coordinate information of the signage, and then generate robot positioning information based on the coordinate information of the signage. The elevator positioning module can extract specific feature points of the elevator from the surrounding video and generate robot positioning information based on the specific feature points. The main positioning module can extract feature points from the surrounding video, compare the feature points with feature points in map information, and generate robot positioning information.

[0067] In some embodiments, the control server first selects the main positioning module as the positioning module, and then, depending on the spatial characteristics, can switch to a marker positioning module, signage positioning module, elevator positioning module, etc. That is, since the main positioning module can perform visual positioning for the entire target area, the main positioning module is used as the basic module, but if there is a positioning module specialized for the spatial characteristics, it can be switched to that positioning module. In other embodiments, it is also possible to operate the main positioning module and the positioning modules added according to the respective spatial characteristics simultaneously.

[0068] The control server can generate positioning information for the robot from the surrounding video using the selected positioning module (S30). That is, positioning information can be generated by performing positioning using the main positioning module, marker positioning module, signage positioning module, elevator positioning module, etc. However, the specific positioning methods used by each positioning module have been described above, so the specific details will be omitted here.

[0069] On the other hand, the control server can select multiple positioning modules simultaneously and perform positioning using the selected multiple positioning modules at the same time. Specifically, the control server can collect positioning information measured by each positioning module to generate final positioning information. At this time, it can set weight values ​​for each positioning module and combine the positioning information to which each weight value has been applied to generate the final positioning information. In addition, depending on the embodiment, it is also possible to combine the positioning information measured by each positioning module by utilizing techniques such as pose graph optimization and Kalman filters, which are widely used in the field of sensor fusion.

[0070] The present invention, as described above, can be embodied as computer-readable code on a medium on which a program is recorded. The computer-readable medium may be used to continuously store computer-executable programs or to temporarily store them for execution or download. The medium may be various recording or storage means in the form of a combination of one or more hardware components, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of mediums include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical mediums such as floptical disks, and ROMs, RAMs, flash memories, etc., which may be configured to store program instructions. Other examples of mediums include recording media or storage media managed by app stores that distribute applications and other sites and servers that supply or distribute various software. Therefore, the above detailed description should not be interpreted restrictively in any way and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and any modifications within the equivalent scope of the present invention are included within the scope of the present invention.

[0071] The present invention is not limited by the embodiments described above and the accompanying drawings. It will be apparent to a person with ordinary skill in the art to which the present invention pertains that the components of the present invention can be substituted, modified, and changed without departing from the technical spirit of the invention.

Claims

1. A visual localization method for a control server that determines the position of the robot within a target area based on surrounding images acquired from a camera mounted on the robot, The steps include receiving a video of the surrounding area from the robot, and The steps include applying at least one of multiple positioning modules based on the spatial characteristics set in the aforementioned space, The step includes generating positioning information for the robot from the surrounding video using the positioning module, A visual positioning method for a control server, comprising the step of applying the positioning module, which includes distinguishing spatial characteristics that indicate features in the space where the robot is located using map information corresponding to the target area, and applying the positioning module according to the spatial characteristics.

2. The step of applying the positioning module is: A visual positioning method for a control server according to claim 1, characterized by distinguishing the space in which the robot is located based on map information corresponding to the target area and the robot's positioning information or the robot's odometry, and determining the spatial characteristics set in the space.

3. The positioning module is, A marker positioning module extracts markers from the surrounding video and generates positioning information for the robot based on the coordinate information recognized from the markers. A signage positioning module extracts signage from the surrounding video, obtains the coordinate information of the signage, and then generates positioning information for the robot based on the coordinate information of the signage. An elevator positioning module that extracts specific feature points of the elevator from the surrounding video and generates positioning information for the robot based on the specific feature points, A visual positioning method for a control server according to claim 2, characterized by including at least one of the following: a main positioning module that extracts feature points from the surrounding video, compares the feature points with feature points in the map information, and generates positioning information for the robot.

4. The step of applying the positioning module is: The visual positioning method for a control server according to claim 3, characterized in that, after applying the main positioning module as the positioning module, it is converted to one of the marker positioning module, signage positioning module, and elevator positioning module depending on the spatial characteristics, or is further applied to operate simultaneously with the positioning module.

5. The step of applying the positioning module is: The visual positioning method for a control server according to claim 3, characterized in that if the spatial characteristics indicate a space in which a marker containing coordinate information is attached, the marker positioning module is applied.

6. The step of applying the positioning module is: The visual positioning method for a control server according to claim 3, characterized in that if the spatial characteristics indicate a space including signage installed in a predetermined location inside according to a set standard, the signage positioning module is applied.

7. The step of applying the positioning module is: The visual positioning method for a control server according to claim 3, characterized in that the elevator positioning module is applied if the spatial characteristics indicate a space in which an elevator exists.

8. A computer program stored on a medium for use in conjunction with hardware to perform the visual positioning method of a control server according to any one of claims 1 to 7.

9. A control server that performs visual localization to determine the position of the robot within a target area based on surrounding images acquired from a camera mounted on the robot, The robot has a video receiving unit that receives video of the surrounding space in which the robot is located, A positioning module determination unit that applies at least one of a plurality of positioning modules based on the spatial characteristics set in the aforementioned space, The system includes a positioning unit that generates positioning information for the robot from the surrounding video using the positioning module, The positioning module determination unit is a control server that distinguishes spatial characteristics indicating features in the space where the robot is located using map information corresponding to the target area, and applies the positioning module according to the spatial characteristics.

10. A building, and the said building is The indoor area includes the area in which the robot travels, the indoor area is divided into multiple spaces according to its spatial characteristics, and at least one of the multiple spaces includes a marker or signage. At least one robot is positioned to provide services while moving around inside the building. The robot is controlled by communication with the control server described in claim 9, in a building.

Images