Robot

Abstract

A robot according to one aspect of the present disclosure comprises: a memory in which a map is stored; a processor for generating a traveling path to a destination set on the map; a plurality of obstacle detection sensors for detecting an obstacle during traveling based on the traveling path; and a camera sensor for acquiring an image during traveling, wherein the processor recognizes the attribute of an obstacle existing at the destination or on the traveling path on the basis of the image acquired by the camera sensor, and increases an obstacle size set on the map on the basis of the recognized attribute of the obstacle.

Description

robot

[0001] The present disclosure relates to an autonomous robot, and more particularly, to a robot capable of preventing collisions while driving.

[0002] Robots were developed for industrial use and have played a role in factory automation. Recently, the scope of robotics applications has expanded, with developments in medical and aerospace robots, as well as domestic robots for everyday use. Among these robots, those capable of autonomous navigation are called mobile robots.

[0003] As robot use increases, there is a growing demand for robots that can provide diverse information, entertainment, and services beyond the repetitive performance of simple functions.

[0004] Accordingly, various robots are being developed to provide convenience to people by being placed in homes, restaurants, stores, public places, etc.

[0005] Robots are equipped with multiple sensors to avoid obstacles while driving. For example, optical sensors determine the presence and distance of obstacles based on the amount of light reflected back from the obstacle or the time it takes for the light to be received.

[0006] Prior art document 1 (Korean Patent Publication No. 10-2020-0108631) relates to a safety device for mobile robots to prevent accidents, and prevents accidents of mobile robots by combining a primary safety device using multiple RGB-D cameras and a secondary safety device using a bumper.

[0007] Furthermore, prior art document 2 (US Patent No. US9535421) discloses a serving robot. The serving robot disclosed in prior art document 2 includes a touch-based display that provides an interface for user requests and a container for serving items, which is positioned inside the robot. The user places the serving items into the container of the serving robot, and the serving robot begins delivery upon receiving input from the display regarding the transport location of the serving items.

[0008] Depending on the service location and type, the robot may be required to approach the service recipient very closely. Furthermore, the robot may need to navigate narrow passages created by dynamic and fixed obstacles. Therefore, technology is needed to prevent collisions when the robot approaches a person or navigates through narrow passages.

[0009]

[0010] The purpose of this disclosure is to provide a robot that can safely move to a location where a service recipient can conveniently receive service.

[0011] The purpose of the present disclosure is to provide optimal obstacle recognition and driving technology that takes into account the characteristics of sensors equipped with a robot.

[0012] The purpose of the present disclosure is to provide a robot that can effectively respond to and drive in narrow paths caused by obstacles.

[0013] The purpose of the present disclosure is to provide a robot with high efficiency and reliability in an environment with many changing situations.

[0014] The purpose of the present disclosure is to provide a robot that can effectively establish a path depending on the situation.

[0015] To achieve the above or other purposes, a robot according to one aspect of the present disclosure can prevent collisions by changing obstacle information stored in a map according to obstacles present at a destination or on a driving path.

[0016] A robot according to one aspect of the present disclosure can determine the properties of an obstacle based on an image, and safely move to the closest possible location to a customer at a destination by considering the type of sensor, sensing method, and properties of the obstacle.

[0017] A robot according to one aspect of the present disclosure can effectively provide a service by actively changing the destination point depending on whether the destination point is occupied by an obstacle.

[0018] In order to achieve the above or other purposes, a robot according to one aspect of the present disclosure includes a memory in which a map is stored, a processor for generating a driving path to a destination point set on the map, a plurality of obstacle detection sensors for detecting obstacles during driving based on the driving path, and a camera sensor for obtaining images during the driving, wherein the processor recognizes attributes of obstacles existing at the destination point or on the driving path based on images obtained from the camera sensors, and increases an obstacle size set on the map based on attributes of the recognized obstacles.

[0019] The processor may increase the size of the obstacle set on the map if the property of the recognized obstacle corresponds to a false detection risk obstacle set in accordance with the type and sensing method of the obstacle detection sensor.

[0020] The above map may be a cost map including a plurality of grids and having costs assigned to the plurality of grids.

[0021] The processor may increase the number of grids in which the recognized obstacle is located in the cost map. Here, the processor may also increase the cost values ​​of the grids in which the recognized obstacle is located in the cost map.

[0022] The above processor can increase the cost values ​​of grids in which the recognized obstacle is located in the cost map.

[0023] The above obstacle detection sensor may include a LiDAR sensor and a ToF sensor.

[0024] The above processor can lower the noise software filtering level of the LiDAR sensor.

[0025] The processor can increase the size of the obstacle in a situation where the vehicle must drive within a predetermined distance of the recognized obstacle, and maintain the size of the obstacle in other situations.

[0026] The processor may change the destination point from a first destination point to a second destination point if the obstacle occupies the destination point.

[0027] The processor may determine, based on the obstacle placement distribution adjacent to the first destination point, the point farthest from the first destination point among the movable candidate points as the second destination point.

[0028] The above movable candidate point may be a point to which the robot can move among the points around the table closest to the first destination point.

[0029] The processor determines the properties of an obstacle forming a narrow path on the driving path, and if the properties of the recognized obstacle correspond to a dangerous obstacle that is set to be detected in accordance with the type and sensing method of the obstacle detection sensor, the processor can increase the size of the obstacle set on the map and regenerate the driving path.

[0030] In order to achieve the above or other purposes, a robot according to one aspect of the present disclosure includes a memory in which a map is stored, a processor for generating a driving path to a first destination point set on the map, a plurality of obstacle detection sensors for detecting obstacles during driving based on the driving path, and a camera sensor for obtaining an image during the driving, wherein the processor, if there is an obstacle occupying the first destination point, recognizes an attribute of the obstacle occupying the first destination point based on the image obtained from the camera sensor, and if the attribute of the recognized obstacle corresponds to a falsely detected dangerous obstacle set in response to a type and a sensing method of the obstacle detection sensor, changes the destination point to a second destination point.

[0031] If the above processor cannot change the destination point to the second destination point,

[0032] The size of the obstacles set on the above map can be increased.

[0033] The above map is a cost map including a plurality of grids and having costs assigned to the plurality of grids, and the processor can increase the number of grids in the cost map where the recognized obstacle is located. Here, the processor can also increase the cost values ​​of the grids in the cost map where the recognized obstacle is located.

[0034] The above map is a cost map including a plurality of grids to which costs are assigned, and the processor can increase the cost values ​​of grids in which the recognized obstacle is located in the cost map.

[0035] The processor may determine, based on the obstacle placement distribution adjacent to the first destination point, the point farthest from the first destination point among the movable candidate points as the second destination point.

[0036] The above obstacle detection sensor includes a LiDAR sensor, and the processor can lower a noise software filtering level of the LiDAR sensor.

[0037] In this embodiment, the processor determines the properties of an obstacle forming a narrow path on the driving path, and if the properties of the recognized obstacle correspond to a falsely detected dangerous obstacle set in accordance with the type and sensing method of the obstacle detection sensor, the processor can increase the size of the obstacle set on the map and regenerate the driving path.

[0038]

[0039] According to at least one of the embodiments of the present disclosure, the robot can safely move to a location where the service recipient can comfortably receive the service, thereby improving usability and reliability.

[0040] According to at least one of the embodiments of the present disclosure, it is possible to implement optimal obstacle recognition and driving technology that takes into account the characteristics of the sensors equipped with the robot.

[0041] According to at least one of the embodiments of the present disclosure, it is possible to effectively respond to and drive on narrow paths caused by obstacles.

[0042] According to at least one of the embodiments of the present disclosure, a robot with high efficiency and reliability can be provided even in an environment with many changing situations by effectively establishing a path according to the situation and driving while avoiding obstacles.

[0043] Figure 1 illustrates an AI device according to one embodiment of the present disclosure.

[0044] FIG. 2 illustrates an AI server according to one embodiment of the present disclosure.

[0045] FIG. 3 illustrates an AI system according to one embodiment of the present disclosure.

[0046] Figure 4 illustrates a robot according to an embodiment of the present disclosure.

[0047] FIG. 5 is a block diagram illustrating a control relationship between major components of a robot according to an embodiment of the present disclosure.

[0048] FIG. 6 is a flowchart illustrating a method of operating a robot according to an embodiment of the present disclosure.

[0049] FIGS. 7 to 16c are drawings for reference in the description of the method of operating a robot according to an embodiment of the present disclosure.

[0050] FIG. 17 is a flowchart illustrating a method of operating a robot according to an embodiment of the present disclosure.

[0051] FIGS. 18 to 20 are drawings for reference in the description of the method of operating a robot according to an embodiment of the present disclosure.

[0052] FIG. 21 is a drawing for reference in a description of sensing software processing according to an embodiment of the present disclosure.

[0053] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. However, the present disclosure is not limited to these embodiments and can be modified in various forms.

[0054] In the drawings, parts that are not related to the description are omitted in order to clearly and concisely explain the present disclosure, and the same drawing reference numerals are used for the same or extremely similar parts throughout the specification.

[0055] Meanwhile, the suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification, and do not impart any particularly significant meaning or role to them. Therefore, the terms "module" and "part" may be used interchangeably.

[0056] Additionally, while terms such as "first" and "second" may be used in this specification to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another.

[0057] Artificial Intelligence (AI) refers to the field of study that studies artificial intelligence or the methodologies for creating it, while Machine Learning (ML) refers to the field that defines various problems in the field of AI and studies the methodologies for solving them. Machine learning is also defined as an algorithm that improves performance on a task through consistent experience.

[0058] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0059] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0060] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0061] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0062] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0063] Supervised learning refers to a method for training an artificial neural network when given labels for the training data. The labels can refer to the correct answer (or output value) that the artificial neural network must infer when the training data is input to the artificial neural network. Unsupervised learning can refer to a method for training an artificial neural network when the training data is not given labels. Reinforcement learning can refer to a learning method in which an agent defined within a given environment is trained to select actions or action sequences that maximize the cumulative reward in each state.

[0064] Machine learning implemented with a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, the term "machine learning" is used to encompass deep learning.

[0065] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolution Neural Networks) model.

[0066] The YOLO (You Only Look Once) model is a model that can predict objects and their locations within an image by looking at the image only once.

[0067] The YOLO (You Only Look Once) model divides the original image into grids of equal size. For each grid, it predicts the number of bounding boxes in a predefined shape centered around the grid center, and calculates confidence based on this.

[0068] Afterwards, whether the image contains an object or is just a background is included, and locations with high object confidence are selected so that the object category can be identified.

[0069] The Faster R-CNN (Regions with Convolution Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0070] This article specifically explains the Faster R-CNN (Regions with Convolution Neural Networks) model.

[0071] First, feature maps are extracted from the image using a Convolution Neural Network (CNN) model. Based on the extracted feature maps, multiple regions of interest (RoIs) are extracted. RoI pooling is performed for each region of interest.

[0072] RoI pooling is a process of setting a grid to a predetermined size of H x W for the feature map onto which the region of interest is projected, extracting the largest value for each cell included in each grid, and extracting a feature map with a size of H x W.

[0073] A feature vector is extracted from a feature map having a size of H x W, and object identification information can be obtained from the feature vector.

[0074] A robot can be defined as a machine that automatically performs or operates a given task based on its own capabilities. Specifically, a robot capable of perceiving its environment, making independent judgments, and performing actions can be called an intelligent robot.

[0075] Robots can be classified into industrial, medical, household, and military types depending on their purpose or field of use.

[0076] Robots are equipped with actuators or motors, enabling them to perform various physical actions, such as moving robot joints. Furthermore, mobile robots include wheels, brakes, propellers, and other actuators within their actuators, enabling them to move on the ground or fly in the air.

[0077] Self-driving refers to the technology of driving on one's own, and an autonomous vehicle refers to a vehicle that drives without user intervention or with minimal user intervention.

[0078] For example, autonomous driving can include technologies that maintain the driving lane, technologies that automatically adjust speed such as adaptive cruise control, technologies that automatically drive along a set route, and technologies that automatically set a route and drive when a destination is set.

[0079] Vehicles include vehicles equipped only with internal combustion engines, hybrid vehicles equipped with both internal combustion engines and electric motors, and electric vehicles equipped only with electric motors, and may include not only automobiles but also trains, motorcycles, etc.

[0080] At this time, autonomous vehicles can be viewed as robots with autonomous driving functions.

[0081] FIG. 1 illustrates an AI device (100) according to one embodiment of the present disclosure.

[0082] The AI ​​device (100) can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a desktop computer, a laptop, a digital broadcasting terminal, a PDA (personal digital assistant), a PMP (portable multimedia player), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, digital signage, a robot, a vehicle, etc.

[0083] Referring to FIG. 1, the terminal (100) may include a communication interface (110), an input interface (120), a running processor (130), a sensing unit (140), an output interface (150), a memory (170), and a processor (180).

[0084] The communication interface (110) can transmit and receive data with external devices such as other AI devices (100a to 100e) or AI servers (200) using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0085] At this time, the communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0086] The input interface (120) can obtain various types of data.

[0087] At this time, the input interface (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input interface for receiving information from a user, etc. Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

[0088] The input interface (120) can obtain input data to be used when obtaining output using learning data and learning models for model learning. The input interface (120) can also obtain unprocessed input data, in which case the processor (180) or learning processor (130) can extract input features as preprocessing for the input data.

[0089] The learning processor (130) can train a model composed of an artificial neural network using learning data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer result values ​​for new input data other than the learning data, and the inferred values ​​can be used as a basis for making decisions regarding certain actions.

[0090] At this time, the running processor (130) can perform AI processing together with the running processor (240) of the AI ​​server (200).

[0091] At this time, the running processor (130) may include a memory integrated or implemented in the AI ​​device (100). Alternatively, the running processor (130) may be implemented using a memory (170), an external memory directly coupled to the AI ​​device (100), or a memory maintained in an external device.

[0092] The sensing unit (140) can obtain at least one of internal information of the AI ​​device (100), information about the surrounding environment of the AI ​​device (100), and user information using various sensors.

[0093] At this time, the sensors included in the sensing unit (140) include a proximity sensor, a light sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, a radar, etc.

[0094] The output interface (150) can generate output related to visual, auditory, or tactile sensations.

[0095] At this time, the output interface (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0096] The memory (170) can store data that supports various functions of the AI ​​device (100). For example, the memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input interface (120).

[0097] The processor (180) may determine at least one executable operation of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm. Then, the processor (180) may control components of the AI ​​device (100) to perform the determined operation.

[0098] To this end, the processor (180) can request, retrieve, receive or utilize data from the running processor (130) or memory (170), and control components of the AI ​​device (100) to execute at least one of the executable operations, a predicted operation or an operation determined to be desirable.

[0099] At this time, if the processor (180) requires connection of an external device to perform a determined operation, it can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0100] The processor (180) can obtain intent information for user input and determine the user's requirements based on the obtained intent information.

[0101] At this time, the processor (180) can obtain intent information corresponding to the user input by using at least one of an STT (Speech To Text) engine for converting voice input into a string or a natural language processing (NLP) engine for obtaining intent information of natural language.

[0102] At this time, at least one of the STT engine or the NLP engine may be configured with an artificial neural network, at least in part, trained according to a machine learning algorithm. Furthermore, at least one of the STT engine or the NLP engine may be trained by the learning processor (130), the learning processor (240) of the AI ​​server (200), or through distributed processing thereof.

[0103] The processor (180) can collect history information including the operation details of the AI ​​device (100) or the user's feedback on the operation, and store the information in the memory (170) or the learning processor (130), or transmit the information to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0104] The processor (180) can control at least some of the components of the AI ​​device (100) to drive an application program stored in the memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination to drive the application program.

[0105] FIG. 2 illustrates an AI server (200) according to one embodiment of the present disclosure.

[0106] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or utilizes a trained artificial neural network. Here, the AI ​​server (200) may be composed of multiple servers to perform distributed processing, and may be defined as a 5G network. In this case, the AI ​​server (200) may be included as part of the AI ​​device (100) and may perform at least a portion of the AI ​​processing.

[0107] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0108] The communication interface (210) can transmit and receive data with an external device such as an AI device (100).

[0109] The memory (230) may include a model storage (231). The model storage (231) may store a model (or artificial neural network, 231a) being learned or learned through a learning processor (240).

[0110] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as an AI device (100).

[0111] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0112] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0113] Figure 3 illustrates an AI system (1) according to one embodiment of the present disclosure.

[0114] Referring to FIG. 3, an AI system (1) is connected to a cloud network (10) by at least one of an AI server (200), a robot (100a), an autonomous vehicle (100b), an XR device (100c), a smartphone (100d), or an appliance (100e). Here, a robot (100a), an autonomous vehicle (100b), an XR device (100c), a smartphone (100d), or an appliance (100e) to which AI technology is applied may be referred to as an AI device (100a to 100e).

[0115] A cloud network (10) may refer to a network that constitutes part of a cloud computing infrastructure or exists within a cloud computing infrastructure. Here, the cloud network (10) may be configured using a 3G network, a 4G or LTE (Long Term Evolution) network, a 5G network, etc.

[0116] That is, each device (100a to 100e, 200) constituting the AI ​​system (1) can be connected to each other via a cloud network (10). In particular, each device (100a to 100e, 200) can communicate with each other via a base station, but can also communicate with each other directly without going through a base station.

[0117] The AI ​​server (200) may include a server that performs AI processing and a server that performs operations on big data.

[0118] The AI ​​server (200) is connected to at least one of the AI ​​devices constituting the AI ​​system (1), such as a robot (100a), an autonomous vehicle (100b), an XR device (100c), a smartphone (100d), or a home appliance (100e), through a cloud network (10), and can assist at least part of the AI ​​processing of the connected AI devices (100a to 100e).

[0119] At this time, the AI ​​server (200) can train an artificial neural network according to a machine learning algorithm on behalf of the AI ​​devices (100a to 100e), and can directly store the learning model or transmit it to the AI ​​devices (100a to 100e).

[0120] At this time, the AI ​​server (200) can receive input data from the AI ​​devices (100a to 100e), infer a result value for the received input data using a learning model, and generate a response or control command based on the inferred result value and transmit it to the AI ​​devices (100a to 100e).

[0121] Alternatively, the AI ​​device (100a to 100e) may infer a result value for input data using a direct learning model and generate a response or control command based on the inferred result value.

[0122] Below, various embodiments of AI devices (100a to 100e) to which the above-described technology is applied are described. Here, the AI ​​devices (100a to 100e) illustrated in FIG. 3 can be viewed as specific embodiments of the AI ​​device (100) illustrated in FIG. 1.

[0123] The robot (100a) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, etc. by applying AI technology.

[0124] The robot (100a) may include a robot control module for controlling movement, and the robot control module may mean a software module or a chip that implements the same in hardware.

[0125] The robot (100a) can obtain status information of the robot (100a), detect (recognize) the surrounding environment and objects, generate map data, determine a movement path and driving plan, determine a response to user interaction, or determine an action using sensor information obtained from various types of sensors.

[0126] Here, the robot (100a) can use sensor information acquired from at least one sensor among lidar, radar, and camera to determine a movement path and driving plan.

[0127] The robot (100a) can perform the above-described operations using a learning model comprised of at least one artificial neural network. For example, the robot (100a) can recognize its surroundings and objects using the learning model, and determine operations using the recognized surrounding environment information or object information. Here, the learning model may be learned directly by the robot (100a) or by an external device such as an AI server (200).

[0128] At this time, the robot (100a) may perform an action by generating a result using a direct learning model, but may also perform an action by transmitting sensor information to an external device such as an AI server (200) and receiving the result generated accordingly.

[0129] The robot (100a) can determine a movement path and a driving plan using at least one of map data, object information detected from sensor information, or object information acquired from an external device, and control a driving unit to drive the robot (100a) according to the determined movement path and driving plan.

[0130] Map data may include object identification information for various objects positioned in the space where the robot (100a) moves. For example, map data may include object identification information for fixed objects such as walls and doors, as well as movable objects such as flower pots and desks. Furthermore, object identification information may include name, type, distance, location, etc.

[0131] Additionally, the robot (100a) can perform actions or drive by controlling the driving unit based on the user's control / interaction. At this time, the robot (100a) can acquire intention information regarding the interaction based on the user's actions or voice utterances, and determine a response based on the acquired intention information to perform the action.

[0132] An autonomous vehicle (100b) can be implemented as a mobile robot, vehicle, or unmanned aerial vehicle by applying AI technology.

[0133] The autonomous vehicle (100b) may include an autonomous driving control module for controlling autonomous driving functions. The autonomous driving control module may refer to a software module or a chip implementing the same as hardware. The autonomous driving control module may be included internally as a component of the autonomous vehicle (100b), but may also be configured as separate hardware and connected to the exterior of the autonomous vehicle (100b).

[0134] An autonomous vehicle (100b) can obtain status information of the autonomous vehicle (100b), detect (recognize) the surrounding environment and objects, generate map data, determine a movement path and driving plan, or determine an action using sensor information obtained from various types of sensors.

[0135] Here, the autonomous vehicle (100b) can use sensor information acquired from at least one sensor among lidar, radar, and camera, similar to the robot (100a), to determine a movement path and driving plan.

[0136] In particular, the autonomous vehicle (100b) can recognize the environment or objects in an area where the field of view is obstructed or an area beyond a certain distance by receiving sensor information from external devices, or can receive information recognized directly from external devices.

[0137] The autonomous vehicle (100b) can perform the above-described operations using a learning model comprised of at least one artificial neural network. For example, the autonomous vehicle (100b) can recognize its surroundings and objects using the learning model, and determine a driving route using the recognized surrounding environment information or object information. Here, the learning model may be learned directly by the autonomous vehicle (100b) or by an external device such as an AI server (200).

[0138] At this time, the autonomous vehicle (100b) may perform an action by generating a result using a direct learning model, but may also perform an action by transmitting sensor information to an external device such as an AI server (200) and receiving the result generated accordingly.

[0139] An autonomous vehicle (100b) can determine a movement path and a driving plan using at least one of map data, object information detected from sensor information, or object information acquired from an external device, and control a driving unit to drive the autonomous vehicle (100b) according to the determined movement path and driving plan.

[0140] Map data may include object identification information for various objects located in the space (e.g., a road) where the autonomous vehicle (100b) travels. For example, map data may include object identification information for fixed objects such as streetlights, rocks, and buildings, as well as movable objects such as vehicles and pedestrians. Furthermore, object identification information may include name, type, distance, location, and the like.

[0141] Additionally, the autonomous vehicle (100b) can perform actions or drive by controlling the driving unit based on the user's control / interaction. At this time, the autonomous vehicle (100b) can acquire intention information regarding the interaction based on the user's actions or voice utterances, and determine a response based on the acquired intention information to perform the action.

[0142] The robot (100a) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, etc. by applying AI technology and autonomous driving technology.

[0143] A robot (100a) to which AI technology and autonomous driving technology are applied may refer to a robot itself with autonomous driving function, or a robot (100a) that interacts with an autonomous vehicle (100b).

[0144] A robot (100a) with an autonomous driving function can be a general term for devices that move on their own along a given path without user control or move by determining the path on their own.

[0145] A robot (100a) with autonomous driving capabilities and an autonomous vehicle (100b) may use a common sensing method to determine one or more of a movement path or a driving plan. For example, a robot (100a) with autonomous driving capabilities and an autonomous vehicle (100b) may use information sensed via lidar, radar, and cameras to determine one or more of a movement path or a driving plan.

[0146] A robot (100a) interacting with an autonomous vehicle (100b) may exist separately from the autonomous vehicle (100b), and may be linked to autonomous driving functions within the autonomous vehicle (100b) or perform actions linked to a user riding in the autonomous vehicle (100b).

[0147] At this time, the robot (100a) interacting with the autonomous vehicle (100b) can control or assist the autonomous driving function of the autonomous vehicle (100b) by acquiring sensor information on behalf of the autonomous vehicle (100b) and providing it to the autonomous vehicle (100b), or by acquiring sensor information and generating surrounding environment information or object information and providing it to the autonomous vehicle (100b).

[0148] Alternatively, a robot (100a) interacting with an autonomous vehicle (100b) may monitor a user riding in the autonomous vehicle (100b) or control functions of the autonomous vehicle (100b) through interaction with the user. For example, if the robot (100a) determines that the driver is drowsy, it may activate the autonomous driving function of the autonomous vehicle (100b) or assist in controlling the driving unit of the autonomous vehicle (100b). Here, the functions of the autonomous vehicle (100b) controlled by the robot (100a) may include not only the autonomous driving function, but also functions provided by a navigation system or audio system installed inside the autonomous vehicle (100b).

[0149] Alternatively, a robot (100a) interacting with an autonomous vehicle (100b) may provide information to the autonomous vehicle (100b) or assist functions from outside the autonomous vehicle (100b). For example, the robot (100a) may provide traffic information, including signal information, to the autonomous vehicle (100b), such as a smart traffic light, or may interact with the autonomous vehicle (100b) to automatically connect an electric charger to a charging port, such as an automatic electric charger for an electric vehicle.

[0150] FIG. 4 illustrates an AI device (100) according to an embodiment of the present disclosure. More specifically, FIG. 4 illustrates a robot (100a) according to an embodiment of the present disclosure.

[0151] Fig. 4 illustrates serving robots (100a1, 100a2) that primarily serve items such as food. Referring to Fig. 4, the robot (100a) may include a main body (400) and a service module (490).

[0152] The main body (400) may form the base of the robot (100a). The main body (400) may be equipped with a driving wheel (402) that drives the robot (100a). The driving wheel (402) may protrude from the lower side of the main body (400).

[0153] The main body (400) can accommodate various components. The main body (400) can be equipped with a plurality of sensors.

[0154] For example, the main body (400) may be equipped with a lidar sensor (420). The lidar sensor (420) may be placed in a recessed portion formed in the main body (400).

[0155] Additionally, the main body (400) may be equipped with a plurality of ToF (time of flight) sensors (430). The plurality of ToF sensors (430) may be spaced apart from each other in the circumferential direction of the main body (400). More specifically, the ToF sensors (430) are arranged on the outer circumference of the main body (400) to detect objects around the robot (100a).

[0156] The ToF sensor (430) may be installed at a lower position than the lidar sensor (420). More specifically, the height of the ToF sensor (430) may be lower than the height of the lidar sensor (420) based on the lower end of the main body (400).

[0157] The ToF sensor (430) may be an optical ToF sensor (430) that calculates distance by measuring the time (time of flight) that light takes to hit an object and reflect. Alternatively, the ToF sensor (430) may use an ultrasonic sensor that measures ToF (time of flight) using ultrasonic waves.

[0158] A camera module (410) may be placed in the main body (400). The camera module (410) may include one or more cameras. The camera module (410) may include a camera sensor (412) for acquiring images and a depth camera (411) capable of measuring the distance to an obstacle.

[0159] The depth camera (411) can acquire depth images and measure the distance to obstacles. The camera sensor (412) can be an RGB 2D camera and can acquire images of the surroundings of the robot (100a).

[0160] Unlike the example of FIG. 4, the depth camera (411) and the camera sensor (412) may be individually positioned at different locations. The camera module (410) may be an RGB-D camera in which the depth camera (411) and the camera sensor (412) are integrated.

[0161] The service module (490) can cover the main body (400) from the upper side. The service module (490) includes a plurality of shelves and can accommodate items. The robot (100a) can autonomously drive while items are accommodated in the service module (490).

[0162] The robot (100a) according to the embodiment of the present disclosure is not limited to the example of FIG. 4. The serving robot (100a1) illustrated in FIG. 4 (a) and the serving robot (100a2) illustrated in FIG. 4 (b) differ in the shapes of the main body (400) and the service module (490), the sensors (410, 420, 430), and the positions of the driving wheels (402). In this way, the robot (100a) according to the embodiment of the present disclosure can be configured in various ways in terms of the appearance, the positions of each component, and the combination thereof.

[0163] Fig. 5 is a block diagram illustrating the control relationship between the main components of a robot (100a) according to one embodiment of the present disclosure. Descriptions overlapping with Fig. 1 are omitted.

[0164] Referring to FIG. 5, a robot (100a) according to an embodiment of the present disclosure includes a processor (180) that controls the overall operation of the robot (100a), a memory (170) that stores various data, a driving unit (190) that moves a main body (400) including a driving wheel (402), a sensing unit (140) that includes a plurality of sensors, and an input interface (120). In addition, the robot (100a) further includes the above-described learning processor (130) and can utilize AI for obstacle recognition, etc.

[0165] The processor (180) can control the overall operation of the robot (100a) by controlling the memory (170), driving unit (190), sensing unit (140), etc. within the robot (100a).

[0166] The memory (170) records various information necessary for controlling the robot (100a) and may include a volatile or non-volatile recording medium.

[0167] A map of the driving area may be stored in the memory (170). The map may be input by an external terminal, server, etc. that can exchange information with the robot (100a) via wired or wireless communication, or may be generated by the robot (100a) through self-learning.

[0168] The map for the driving area stored in the memory (170) may be a navigation map used for driving, a SLAM (Simultaneous localization and mapping) map used for location recognition, a global location map used for global location recognition, a local map used for local location recognition, an obstacle recognition map in which information about recognized obstacles is recorded, etc.

[0169] A global path refers to the path from the robot's starting or current location to the target point. A local path refers to a newly created path that the robot creates while driving along the global path or a newly created local path. This path is created by the robot by detecting surrounding obstacles at regular intervals.

[0170] Meanwhile, as described above, maps can be stored and managed in the memory (170) by purpose. However, the maps may not be clearly divided by purpose. Multiple pieces of information can be stored in a single map so that it can be used for at least two purposes. For example, a local map may include information about obstacles.

[0171] The driving unit (190) can move the main body (400) under the control of the processor (180). The driving unit (190) may include at least one driving wheel (402) that moves the main body of the robot (100a). The driving unit (190) may include a driving motor (not shown) that is connected to the driving wheel (402) and rotates the driving wheel (402).

[0172] Drive wheels (402) may be provided on the left and right sides of the main body (400), respectively, and are hereinafter referred to as left and right wheels, respectively. The left and right wheels may be driven by a single drive motor, but, if necessary, a left-wheel drive motor for driving the left wheel and a right-wheel drive motor for driving the right wheel may be provided, respectively. By differentiating the rotational speeds of the left and right wheels, the driving direction of the main body (400) can be changed to the left or right.

[0173] The robot (100a) may include a sensing unit (140) that includes sensors that sense various data related to the operation and status of the robot (100a).

[0174] The robot (100a) may include a motion detection sensor (141) that detects the motion of the robot (100a) and outputs motion information. For example, a gyro sensor, a wheel sensor, an acceleration sensor, etc. may be used as the motion detection sensor.

[0175] The gyro sensor detects the direction of rotation and the angle of rotation when the robot (100a) moves according to the driving mode. The gyro sensor detects the angular velocity of the robot (100a) and outputs a voltage value proportional to the angular velocity. The processor (180) calculates the direction of rotation and the angle of rotation using the voltage value output from the gyro sensor.

[0176] The wheel sensor is connected to the left and right wheels and detects the rotational speed of the wheels. Here, the wheel sensor may be a rotary encoder. The rotary encoder detects and outputs the rotational speed of the left and right wheels.

[0177] The processor (180) can calculate the rotational speed of the left and right wheels using the rotational speed. In addition, the processor (180) can calculate the rotational angle using the difference in the rotational speed of the left and right wheels.

[0178] The acceleration sensor detects changes in the robot's (100a) speed, such as changes in the robot's (100a) speed due to starting, stopping, changing direction, or colliding with an object. The acceleration sensor is attached to a location adjacent to the main wheel or auxiliary wheel, and can detect wheel slippage or idling.

[0179] Additionally, the acceleration sensor can detect changes in the speed of the robot (100a). That is, the acceleration sensor detects the amount of impact resulting from the change in speed and outputs a corresponding voltage value. Therefore, the acceleration sensor can function as an electronic bumper.

[0180] The processor (180) can calculate changes in the position of the robot (100a) based on motion information output from the motion detection sensor. This position corresponds to an absolute position using image information, and thus becomes a relative position. Through this relative position recognition, the robot (100a) can improve the performance of position recognition using image information and obstacle information.

[0181] The motion detection sensor (141) may include an IMU (Inertial Measurement Unit) sensor. The IMU sensor can recognize the movement status of the robot (100a) using an accelerometer, a gyrometer, a magnetometer, and an altimeter.

[0182] The robot (100a) may include a camera sensor (412) capable of capturing a predetermined range. The camera sensor (412) captures the surroundings of the robot (100a), the external environment, etc., and multiple camera sensors (412) may be installed in each area to improve capturing efficiency.

[0183] The processor (180) can judge external situations or recognize users, obstacles, etc. based on images captured by the camera sensor (412).

[0184] Additionally, the processor (180) can control the robot (100a) to drive based on an image captured by the camera sensor (412).

[0185] Meanwhile, the image captured by the camera sensor (412) can be stored in the memory (170).

[0186] The robot (100a) may include an obstacle detection sensor (143) that detects obstacles. The obstacle detection sensor (143) may include an infrared sensor, an ultrasonic sensor, a ToF sensor (430), an RF sensor, a geomagnetic sensor, a PSD (Position Sensitive Device) sensor, a cliff detection sensor that detects the presence of a cliff on the ground within the driving area, a Lidar (light detection and ranging) sensor (420), etc.

[0187] The obstacle detection sensor (143) can measure not only the presence of an obstacle but also the distance between the obstacle and the robot (100a). Therefore, the obstacle detection sensor (143) may also be referred to as a distance sensor.

[0188] Meanwhile, the obstacle detection sensor (143) detects objects, particularly obstacles, present in the driving (moving) direction of the robot (100a) and transmits obstacle information to the processor (180). At this time, the processor (180) can control the movement of the robot (100a) according to the location of the detected obstacle.

[0189] Meanwhile, the lidar sensor (420) outputs a laser and provides information such as the distance, position, direction, and material of an object that reflects the laser, and can acquire topographic information of the driving area. The robot (100a) can acquire 360-degree topographic (geometry) information using the lidar sensor (420).

[0190] The robot (100a) can generate a map by identifying the distance, location, direction, etc. of objects sensed by the lidar sensor (420). The robot (100a) can obtain topographic information of the driving area by analyzing the laser reception pattern, such as the time difference or signal intensity of the laser reflected from the outside and received. In addition, the robot (100a) can generate a map using the topographic information acquired through the lidar sensor (420).

[0191] For example, the robot (100a) can perform a lidar slam to recognize the current location by comparing the surrounding terrain information acquired at the current location through the lidar sensor (420) with previously stored terrain information based on the lidar sensor or by comparing the acquired terrain information.

[0192] The processor (180) can receive a destination through the user input interface of the input interface (120). Alternatively, the processor (180) can receive the destination through the communication interface (110). The processor (180) can generate a driving path of the robot (100a). When the processor (180) receives a destination, it can load a map stored in the memory (170). For example, the user can select a predetermined location, such as in front of a table, in front of a chair, or a specific location, as a destination within the map. Accordingly, the processor (180) can convert the destination selected by the user into coordinates corresponding to the map and generate a driving path from the starting point, which is the current location, to the arrival point corresponding to the destination.

[0193] Meanwhile, the map stored in the memory (170) may be a grid map including a plurality of grids for the driving area. The grid map includes a plurality of grid cells formed by a plurality of vertical grid lines and a plurality of horizontal grid lines. A grid cell is the smallest unit formed by four grid lines and may also be referred to as a grid.

[0194] Additionally, a map containing obstacle information may be a cost map in which costs are assigned to multiple grids. For example, a global map used for global path planning and / or a local map used for local path planning may include a cost map.

[0195] The processor (180) can generate a map of the driving area based on information detected by the sensing unit (140). The processor (180) can generate a two-dimensional or three-dimensional map by fusing data acquired from various sensors of the sensing unit (140).

[0196] The processor (180) can assign a cost to each grid included in the cost map based on obstacle information detected by the sensing unit (140) and generate a driving route with the lowest cost from the starting point to the destination point.

[0197] Additionally, the processor (180) can update the cost map to reflect dynamic obstacles detected during driving and newly detected fixed obstacles. Furthermore, the processor (180) can regenerate the route to the destination based on the updated cost map.

[0198] Meanwhile, when a robot (100a) provides a serving service, such as delivering food or goods to a restaurant or store, it is required that the robot (100a) approach the table or person as closely as possible. For example, upon arriving at a customer's table, the robot (100a) must approach the customer within arm's reach. Furthermore, the robot (100a) must be able to recognize narrow paths / widths and easily pass through dynamic and non-fixed obstacles, and situations may arise where the robot must navigate even in close proximity to obstacles.

[0199] However, depending on the specifications, combination, and placement of obstacle detection sensors (143) such as the lidar sensor (420), ToF sensor (430), and depth camera (411), there may be objects or ranges that are not detected or are difficult to detect.

[0200] For example, the optical obstacle detection sensor (143) may be misdetected by reflective materials such as metal, and the ultrasonic sensor may be misdetected by materials that can absorb sound waves. In addition, the depth camera (411) may be misdetected by repetitive patterns, and the lidar sensor (420) may be misdetected by thin objects.

[0201] Meanwhile, the depth camera (411) is capable of detecting a predetermined height range, but the sensing range varies depending on the placement location. The ToF sensor (430) has a relatively narrower sensing range than the depth camera (411), and the lidar sensor (420) can only detect objects of a specific height.

[0202] In this way, the obstacle detection sensor (143) has objects that may be misdetected, and its sensing range is limited. Therefore, when the robot (100a) drives close to an object that is not well detected or is misdetected, it may collide with the object or fail to pass through it without being detected. For example, the robot (100a) may collide with a high chair with thin legs, crash while driving on a narrow passage made of glass walls, or fail to pass through a narrow passage implemented with a vertical repeating pattern.

[0203] According to the present disclosure, considering the characteristics of the sensor equipped with the robot (100a), an object that may be misdetected is detected, and the purpose of the robot (100a) is to safely move to a location where a service recipient can comfortably receive service or to effectively respond to a narrow path.

[0204] The processor (180) can prevent collisions by modifying obstacle information stored in the map based on the presence of obstacles at the destination or along the driving route. Furthermore, the processor (180) can actively modify the destination based on whether the destination is occupied by obstacles, thereby appropriately responding to the current situation in the driving area.

[0205] The robot (100a) can determine the properties of an obstacle based on an image acquired from a camera sensor (142), and can safely move to the closest possible location to the customer at the destination by considering the type, sensing method, and properties of the obstacle detection sensor (143) provided.

[0206] The robot (100a) constructs a local map (coast map) based on sensing data from an obstacle detection sensor (143), plans a path to avoid obstacles displayed on the map, and drives in an avoidance manner.

[0207] While RGB camera-based methods for estimating object distances are available, their low accuracy makes them unsuitable for proximity avoidance maneuvers. Therefore, the local map (coast map) used by robots for evasive maneuvers should ideally be composed of 3D data based on distance sensors capable of measuring distances to objects. However, due to the characteristics of these distance sensors, objects that are not detected or whose distance sensing accuracy is low often exist as features of the store environment and interior design.

[0208] The robot (100a) includes a memory (170) in which a map is stored, a processor (180) that generates a driving route to a destination point set on the map, a plurality of obstacle detection sensors (143) that detect obstacles while driving based on the driving route, and a camera sensor (142) that acquires images while driving.

[0209] The memory (170) stores a map based on sensing data from an obstacle detection sensor (143) capable of measuring the distance to an obstacle.

[0210] The processor (180) recognizes an obstacle present at the destination or on the driving route based on an image acquired from the camera sensor (142). The processor (180) recognizes an obstacle included in the image acquired from the camera sensor (142). In addition, the processor (180) can determine whether the recognized obstacle is an object likely to be detected by the obstacle detection sensor (143).

[0211] According to an embodiment, the learning processor (130) may receive an image and learn to recognize an obstacle. The learning processor (130) may then send information about the recognized obstacle to the processor (180).

[0212] The camera sensor (142) may be an RGB 2D camera. The camera sensor (142) may be the camera sensor (412) of the camera module (410) illustrated in FIG. 4. Alternatively, the camera sensor (142) may be an RGB 2D camera separately provided in the depth camera (411).

[0213] It is preferable that the obstacle detection sensor (143) be equipped with multiple sensors of different types. By including sensors of different types, the obstacle detection sensor (143) can improve sensing accuracy and configure a diverse sensing range. For example, the obstacle detection sensor (143) may include a lidar sensor (420) and a ToF sensor (430). In addition, the obstacle detection sensor (143) may further include a depth camera (411).

[0214] The processor (180) can determine whether the obstacle has properties that may pose a risk of misdetection (whether it has thin / thin legs, whether it is made of a material that is difficult to detect such as glass, whether it has a vertical repeating pattern, etc.).

[0215] The processor (180) can increase the size of an obstacle set on the map based on the properties of the recognized obstacle. That is, the processor (180) can update the map by increasing the size of an obstacle set on the map.

[0216] The processor (180) can increase the size of the obstacle on the map in situations where it is necessary to drive within a predetermined distance of a recognized obstacle, and maintain the size of the obstacle in other situations. For example, situations where driving within close proximity may be required may include cases where an object occupies a set destination point or where an object forms a narrow path.

[0217] Additionally, the processor (180) can regenerate a path based on the updated map. Since the size of obstacles on the map has increased, the regenerated path is determined to avoid obstacles at a greater distance. Accordingly, the robot (100a) can approach obstacles with a risk of false detection without collision, up to the maximum performance of the equipped sensors.

[0218] When a situation arises where the robot (100a) must approach an object that is at risk of being detected by the obstacle detection sensor (143), the processor (180) determines what the object is based on an image acquired from the camera sensor (142).

[0219] When driving close to an object, the processor (180) uses a camera sensor (142) to determine whether the object is at risk of collision or difficult to pass through due to inaccurate distance. The processor (180) can determine whether the properties of the recognized obstacle correspond to a dangerous obstacle that is not detected.

[0220] The detected hazard obstacles can be set according to the type and sensing method of the obstacle detection sensor (143). The detected hazard obstacles can be set by type and can be set based on whether they contain specific properties, such as whether they are narrow, have a vertical repeating pattern, or are made of a material.

[0221] If the recognized obstacle's attributes correspond to a false detection risk obstacle, the processor (180) increases the obstacle size set on the map. If the object is a false detection risk obstacle with a high probability of false detection, the processor (180) can expand the size of the object on the local map, thereby limiting the accessible distance.

[0222] For example, if a dynamic obstacle such as a child's chair is placed at the arrival location of a store table, the chair on which the child is sitting is reflected larger on the map to prevent accidents by ensuring that the robot (100a) and serving items do not come as close to the child as possible.

[0223] Meanwhile, changing the destination may be more desirable from a safety perspective than arriving as close as possible to an object or person occupying the destination. Therefore, according to one embodiment of the present disclosure, the processor (180) may prioritize changing the destination based on the obstacle occupancy status of the destination. The processor (180) may prioritize changing the destination over changing the size of obstacles on the map based on the serving space, serving items, or user settings.

[0224] The processor (180) generates a driving path to a first destination point corresponding to the received destination, and the robot (100a) can drive along the generated driving path to the first destination point.

[0225] Meanwhile, if there is an obstacle occupying the first destination point, the processor (180) recognizes the properties of the obstacle occupying the first destination point based on the image acquired from the camera sensor (142).

[0226] If the property of the recognized obstacle corresponds to a false detection risk obstacle, the processor (180) can prevent false detection and collision in advance by changing the arrival point from the first arrival point to the second arrival point.

[0227] If an obstacle occupies the destination point, the processor (180) may change the destination point from the first destination point to the second destination point. In this case, the processor (180) may determine the farthest point from the first destination point among the movable candidate points as the second destination point based on the obstacle placement distribution adjacent to the first destination point.

[0228] In the case where there is a table near the destination point, the movable candidate point may be a point to which the robot (100a) can move among the points around the table closest to the first destination point.

[0229] If it is impossible to change the destination point to the second destination point, the processor (180) may increase the size of the obstacle set on the map. For example, there may be fixed obstacles such as walls, dynamic obstacles such as people and animals, or other obstacles near the first destination point, making it impossible to set a new destination point. In this case, the processor (180) updates the map by increasing the size of the obstacle set on the map. Based on the updated map, the processor (180) may bring the robot (100a) as close as possible to the destination point without colliding with the obstacle.

[0230] The processor (180) can change the size of the obstacle in a cost map if the recognized obstacle's attributes correspond to a falsely detected dangerous obstacle. The cost map can be a local map and can be used for obstacle avoidance driving.

[0231] More specifically, the processor (180) may increase the number of grids in which recognized obstacles are located in the cost map. Additionally, the processor (180) may also increase the cost values ​​of the grids in which recognized obstacles are located in the cost map.

[0232] Alternatively, the processor (180) may increase only the cost values ​​of the grids where the recognized obstacles are located in the cost map.

[0233] According to one embodiment of the present disclosure, if the obstacle occupies the destination point, the processor (180) changes the destination point to prevent a collision or a situation in which the vehicle cannot pass through.

[0234] Not only at the destination but also during driving, the processor (180) can change the size of an obstacle on the map according to the properties of the recognized obstacle. The processor (180) determines the properties of an obstacle forming a narrow path on the driving route. If the properties of the recognized obstacle correspond to a falsely detected dangerous obstacle set in accordance with the type and sensing method of the obstacle detection sensor (143), the processor (180) can increase the size of the obstacle set on the map and regenerate the driving route.

[0235] FIG. 6 is a flowchart illustrating a method of operating a robot according to an embodiment of the present disclosure. FIGS. 7 to 16c are drawings referenced in the description of a method of operating a robot according to an embodiment of the present disclosure.

[0236] Referring to Fig. 6, first, the robot (100a) moves to a set destination point (S610).

[0237] Referring to (a) and (b) of FIG. 7, there are one or more tables (710) within the driving area. Chairs (720) are arranged on the long side of the table (710), and a destination point (730) corresponding to the table (710) can be set on either of the short sides.

[0238] A user can input a destination into the robot (100a) by operating a predetermined terminal. The user's operation screen may display only the table (710), or the table (710) and the destination point (730) together.

[0239] When a user inputs a table (710) or a destination (730) as a destination, the processor (180) generates a driving route to the destination (730), and the robot (100a) moves to the destination (730) along the set driving route (S610).

[0240] If the destination point (730) is empty (S620), the robot (100a) moves to the destination point (730) and then ends driving (S610). An adult (740) sitting on a chair (720) can receive the serving items from the robot (100a) that has moved to the destination point (730).

[0241] Referring to (a) and (b) of FIG. 8, a child chair (810) for a child (820) who cannot sit on a chair (720) for adults may be placed at the destination (730).

[0242] Referring to (a) and (b) of FIG. 9, when there is a child chair (810) and a child (820) at the destination point (730) where the original robot (100a) was to arrive, the existing method distributes food near the child chair (810) and the child (820) at a predetermined distance (d1) according to the destination proximity arrival algorithm. In this case, the child is at risk due to exposure to the served food, and the adult (740) is inconvenient to take the food out right away because the distance (d2) for taking the food down from the robot (100a) is far.

[0243] Meanwhile, in FIGS. 8 and 9, a child's chair (810) with thin and long legs is exemplified as a sensory hazard obstacle, but the sensory hazard obstacle may vary depending on the material, shape, and sensor equipped with the robot (100a).

[0244] Figures 10a to 10f illustrate various sensory hazard obstacles.

[0245] Referring to FIG. 10a, a sign having a thin / thin metal rod (1010) similar to a child's chair (810) may also be considered a sensory hazard obstacle.

[0246] Referring to FIG. 10b, the glass pane (1020) may also be a potential hazard for the sensory hazard. The glass pane (1020) is a fixed obstacle in itself, but a narrow path with the glass pane (1020) on one side may be created due to another dynamic obstacle.

[0247] Referring to FIGS. 10c, 10e and 10f, vertical decorations (1030), blinds with repeating patterns (1050), and pet panels (1060) also have the risk of false detection.

[0248] Referring to FIG. 10d, the obstacle detection sensor (143) only detects objects and distances, and therefore cannot determine the risk status and level of risk. For example, the risk levels are very different between a paper box (1041) placed on a table (1040) and a burner and hot food (1042) placed on the table (1040).

[0249] Likewise, glass, thin / thin-legged objects, metal materials, vertical repeating pattern decorations, tabletop burners, etc. may be considered as potential hazards for false positive detection.

[0250] If the obstacle detection sensor (143) detects an obstacle occupying the destination point (730) (S620), the processor (180) determines the obstacle based on the image acquired from the camera sensor (142) (S630). That is, when there is an object at the destination point (730) and a situation occurs in which the robot (100a) must approach the object, the processor (180) can distinguish what the object is from the image acquired from the camera sensor (142). The processor (180) determines whether the object is an object that poses a risk of collision or is difficult to pass through because the sensing distance accuracy of the obstacle detection sensor (143) is not achieved when the object is driven in close proximity.

[0251] Referring to FIG. 11, an obstacle occupying a destination point (730) can be detected by an obstacle detection sensor (143) of a robot (100a) traveling along a first travel path (1150) (S620). The processor (180) can recognize a child chair (810) and / or a child (820) at the destination point (730) based on an image acquired from a camera sensor (142).

[0252] According to an embodiment, if the destination point (730) is occupied by an object whose distance is difficult to detect (S640), the processor (180) can change the location of the destination point (730) to another side and generate a path to go to the changed destination point (S650).

[0253] The processor (180) can control the robot (100a) to drive according to the changed destination point and route (S660).

[0254] Referring to FIG. 12, the processor (180) can change the destination point from the originally set first destination point (730) to the second destination point (1110). In addition, the processor (180) can generate a new second driving route (1250) to the second destination point (1110).

[0255] Referring to FIGS. 11 and 12, there are adults (740, 1120) on the long side of the table (710), and the side opposite the first arrival point (730) based on the table (710) is empty. Therefore, a second arrival point (1110) can be set on the empty side of the table (710).

[0256] Meanwhile, since the location of the long side of the table (710) where there are no adults (740, 1120) must pass the first arrival point (730), the second arrival point (1110) is better in terms of efficiency and safety.

[0257] If the determined obstacle or obstacle attribute corresponds to a falsified risk obstacle (S640), the processor (180) increases the obstacle size on the cost map and changes the cost map (S670).

[0258] Meanwhile, if the destination point (730) is not occupied by an object that is difficult to detect, and there is no problem with the distance measurement accuracy of the obstacle detection sensor (143), the robot (100a) can end driving after moving to a position as close as possible to the object according to the destination proximity arrival algorithm (S680). The destination proximity arrival algorithm is a driving algorithm that, if there is an obstacle at the destination, causes the robot (100a) to approach the original destination coordinates to an allowable proximity distance corresponding to the driving and sensing performance of the robot (100a), and the allowable proximity distance can be set in consideration of a margin for preventing collision.

[0259] Meanwhile, if the destination point is occupied by an object whose distance is difficult to detect (S640) and it is difficult to change the destination point location (S650), the processor (180) can increase the size of the cost map occupied by the corresponding obstacle (S670). Based on the changed cost map, the processor (180) can control the movement of the robot (100a) using a destination proximity arrival algorithm (S680).

[0260] That is, the processor (180) sets the distance limit for approaching a dangerous obstacle detected by the obstacle detection sensor (143) further away than other obstacles. By increasing the cost map occupancy size of the corresponding obstacle, the robot (100a) approaches less obstacles detected by the obstacle detection sensor (143) in proportion to the increased occupancy size. Therefore, collision between the robot (100a) and the obstacle at the destination can be prevented.

[0261] Referring to Fig. 13, the originally set first destination point (1310) is occupied by an object (1320) whose distance is difficult to detect. In addition, there is a wall (1340) and adults (1350, 1360) around the first table (1330), and the remaining locations require passing the first destination point (1310), making it difficult to change the destination point location.

[0262] At this time, the processor (180) can increase the cost map occupancy size of the obstacle (1320) to expand the access restriction range around the obstacle (1320).

[0263] Figure 14 conceptually illustrates an inflated obstacle (1400) with an increased size on the cost map. The inflated obstacle (1400) is the inflated size of an infant recognized by the robot (100a).

[0264] Even if a proximity arrival algorithm that arrives near the destination is applied when there is an obstacle (1320) at the destination, the child is drawn large as an obstacle (1400) on the cost map, so the robot (100a) arrives farther away from the infant (1320) and closer to the adult (1350).

[0265] Referring to FIG. 15A, the processor (180) can generate a cost map by assigning a cost to the environment of the driving area. For example, the processor (180) can set cost information in the grids of the cost map based on environmental information of the driving area.

[0266] Referring to FIGS. 15a and 15b, the hatched grid (1510) may represent the boundary of the driving area (wall, obstacle, etc.), and the white grid may represent the driving area in which the robot (100a) can drive.

[0267] Additionally, grids can be assigned a cost based on environmental information, such as the presence of obstacles, the type of obstacle, the size of the obstacle, and the proximity of the obstacles. Grids in a general area without obstacles can have a cost set to 0. Alternatively, grids in a general area without obstacles can be assigned a relatively low cost based on the distance to the obstacles.

[0268] For example, a first value is set for a risk area (1520) corresponding to a fixed object or boundary. Additionally, costs may be assigned to areas (1530, 1540) corresponding to obstacles such as tables and people. The cost value "10" exemplified herein is provided for ease of understanding, and the present disclosure is not limited thereto.

[0269] Meanwhile, when the obstacle detection sensor (143) detects an obstacle at the destination, the processor (180) can assign a cost to the corresponding grid (1550). In addition, the processor (180) can determine the obstacle based on an image acquired from the camera sensor (142).

[0270] If it is determined to be a false detection risk obstacle, the processor (180) can apply a cost set to the obstacle to an area (1560) that extends beyond the destination point (1550). In this way, the processor (180) can increase the size of the false detection risk obstacle on the cost map.

[0271] Meanwhile, each grid can be assigned various costs based on environmental information such as the presence of obstacles, the type of obstacle, the size of the obstacle, and whether obstacles are adjacent.

[0272] Referring to FIGS. 16A to 16C, a first value is set for a risk area (1620) corresponding to a fixed object or boundary line (1610). Additionally, a cost may be assigned to each area (1630, 1640) corresponding to obstacles such as tables and people.

[0273] For example, a first value may be set for a risk area (1620) adjacent to a fixed object or boundary (1610), and a second value lower than the first value may be set for obstacle detection areas (1630, 1630, 1650) other than the risk area (1620). Depending on the embodiment, the cost assigned to each grid may also be set differently depending on the type and size of the obstacle.

[0274] Cost information for each grid may indicate the likelihood that the robot (100a) will avoid or collide with an obstacle when moving through the grid, or the likelihood that the robot (100a) will move adjacent to an obstacle. In grids with a cost of 0 or a low cost, this may include cases where the robot (100a) is far from obstacles such as fixed or moving objects.

[0275] Meanwhile, cost maps can display grids in different display states based on environmental information, costs, and other factors. For example, each grid can display different colors, saturation, and other factors. Accordingly, cost maps can intuitively display costs in the form of contour lines.

[0276] A predetermined cost (e.g., 8) may be assigned to an area corresponding to a table (1630), and a different cost (e.g., 7) may be assigned to an area corresponding to an obstacle such as a person (1640).

[0277] Meanwhile, when the obstacle detection sensor (143) detects an obstacle at the destination, the processor (180) can assign a cost to the corresponding grid (1650). In addition, the processor (180) can determine the obstacle based on an image acquired from the camera sensor (142).

[0278] Referring to FIG. 16b, in the case where a dangerous obstacle is determined to be a false detection hazard, the processor (180) can apply a cost set to the obstacle to an area (1660) extended beyond the destination point (1650).

[0279] Referring to FIG. 16b, the processor (180) can increase the size of the detected risk obstacle on the cost map by giving the extended area (1660) a cost that is the same as or lower than the destination point (1650).

[0280] Referring to FIG. 16c, when a dangerous obstacle is determined to be a false detection hazard, the processor (180) can increase the cost in the destination point (1670) and the areas (1632, 1642, 1680) adjacent to the destination point (1670).

[0281] The processor (180) can increase the cost of the destination point (1670) where the five-sensing risk obstacle is located.

[0282] Additionally, the processor (180) can assign a set cost to an adjacent area (1680) where the cost was 0.

[0283] Additionally, the processor (180) may also increase the costs of adjacent areas (1632, 1642). Accordingly, the costs of the area (1630) corresponding to the table may differ between an area (1632) close to the hazard detection obstacle and an area (1631) far from the hazard detection obstacle. Additionally, the costs of the area (1640) corresponding to the person may differ between an area (1642) close to the hazard detection obstacle and an area (1641) far from the hazard detection obstacle.

[0284] FIG. 17 is a flowchart illustrating an operation method of a robot according to an embodiment of the present disclosure, and FIGS. 18 to 20 are drawings referenced in an explanation of an operation method of a robot according to an embodiment of the present disclosure.

[0285] Referring to FIGS. 17 and 18, first, the robot (100a) moves along the set driving path (1800) to the first destination point (1840) (S1710).

[0286] Referring to FIGS. 18 to 20, walls (1811, 1812) are positioned on both sides of the driving area, and a first table (1821) and a second table (1822) may be positioned between the walls (1811, 1812) on both sides.

[0287] When an input with the first table (1821) or the first destination (1840) as the destination is received, the processor (180) can generate the shortest driving path (1800) to the first destination (1840).

[0288] If the first destination point (1840) is empty and there are no special circumstances during the drive to the first destination point (1840) (S1720), the robot (100a) ends the drive after moving to the first destination point (1840) (S1710). An adult (1831, 1832) sitting on a chair near the first table (1821) can receive the serving items from the robot (100a) that has moved to the first destination point (1840).

[0289] Meanwhile, a narrow path (d3) may be set in the middle of the driving path (1800) along which the robot (100a) moves to the first table (1821) due to an obstacle (1860) such as a child's chair (S1720).

[0290] For example, a number of obstacles, such as a second table (1822), a toddler (1860), and adults (1833, 1834), may be located between the robot (100a) and the first destination point (1840). Alternatively, after the robot (100a) starts moving, new obstacles, such as a toddler (1860) and adults (1833, 1834), may appear.

[0291] When the robot (100a) moves along the driving path (1800) and the obstacle detection sensor (143) detects a narrow path due to an obstacle (S1720), the processor (180) determines the obstacle based on the image acquired from the camera sensor (142) (S1730). That is, when a situation occurs in which the robot (100a) approaches an object and must pass through a narrow path, the processor (180) can distinguish what the object is based on the image acquired from the camera sensor (142). The processor (180) determines whether the object is a dangerous obstacle that is difficult to detect or has a risk of collision due to an inaccurate sensing distance of the obstacle detection sensor (143) when driving in close proximity.

[0292] If the obstacle (1860) is not a dangerous obstacle that can be sensed (S1740), the processor (180) controls the robot (100a) to travel the shortest distance through the narrow path to the first destination point (1840) while measuring the distance at which a collision with the obstacle (1860) does not occur (S1710).

[0293] If the obstacle (1860) is a dangerous obstacle that is detected (S1740), the processor (180) changes the cost map by increasing the size of the obstacle on the cost map (S1750).

[0294] Referring to Figures 18 and 19, the occupied size on the cost map increases from the first size (1850) to the second size (1900). If there is an object whose distance is difficult to detect on the way to the destination (S1740), the processor (180) can prevent the robot (100a) from approaching the obstacle (1860) excessively by increasing the occupied size of the cost map of the obstacle.

[0295] Meanwhile, the processor (180) regenerates the driving route based on the changed cost map (S1760). If the obstacle forming the narrow path is an object that could pose a safety hazard if approached closely, such as a child or a glass wall, the processor (180) increases the cost map occupancy size of the object (1860) to recognize it. Referring to FIG. 19, the route (1800) that requires approaching the object (1860) is not generated.

[0296] Referring to FIG. 21, a regenerated driving path (2000) is generated away from the inflated infant (1990) (S1760), and the robot (100a) can move along the regenerated driving path (2000) to the destination point (1840) (S1710).

[0297] According to an embodiment of the present disclosure, when an object whose distance is difficult to detect is identified along the way to a destination, approach can be prevented by increasing the size of the obstacle's cost map. Furthermore, according to an embodiment of the present disclosure, if the obstacle constitutes a narrow path, increasing the size of the obstacle's cost map can set a further distance limit relative to other obstacles.

[0298] Meanwhile, the processor (180) can improve the sensing accuracy of a dangerous obstacle by lowering the noise software filtering level of the LiDAR sensor (430).

[0299] FIG. 21 is a drawing for reference in a description of sensing software processing according to an embodiment of the present disclosure.

[0300] Objects with thin legs, such as a toddler's chair (810), are difficult to reliably detect with the depth camera (411) and lidar sensor (430). In particular, since the legs of the toddler's chair (810) extend from the outside (in a ladder shape), there is a possibility of collision with the robot (100b) if the legs of the toddler's chair (810) are not detected.

[0301] LiDAR, by its very nature, has edge noise. This noise occurs at the edges of objects. Therefore, this noise must be removed using software filters. However, in the case of thin chair legs, this noise is often indistinguishable and thus deleted.

[0302] Figure 12 (a) illustrates a top view of raw data detected by a 2D lidar sensor, and Figure 12 (b) illustrates a top view of data detected by a 2D lidar sensor after software filtering has been performed.

[0303] Accordingly, the processor (180) relaxes the software (SW) noise removal filter of the lidar sensor (430) when approaching the infant chair (810) to better detect the thin chair legs.

[0304] In addition, the processor (180) increases the occupied size of the child's chair (810) on the cost map while lowering the noise software filtering level of the lidar sensor (430) to improve the detection performance. In this case, since a new path is not created, there is no problem in driving even if lidar edge noise occurs. Therefore, in the case where it is necessary to approach near the child's chair (810), in addition to detecting that it is the child's chair (810) and expanding and reflecting it in the cost map, the detection performance of moving objects is improved by weakening the SW filter strength of the lidar sensor (430).

[0305] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. Memory where the map is stored; A processor that generates a driving route to a destination point set on the above map; A plurality of obstacle detection sensors for detecting obstacles while driving based on the driving path; and A camera sensor for acquiring images during the driving; The above processor, Based on the image acquired from the camera sensor, the properties of obstacles existing at the destination point or on the driving path are recognized, A robot that increases the size of an obstacle set on the map based on the properties of the recognized obstacle.

2. In paragraph 1, The above processor, A robot that increases the size of the obstacle set on the map when the property of the above-mentioned recognized obstacle corresponds to a falsely detected dangerous obstacle set in accordance with the type and sensing method of the above-mentioned obstacle detection sensor.

3. In paragraph 1, The above map is a cost map that includes a plurality of grids and has costs assigned to the plurality of grids. The above processor, A robot that increases the number of grids in which the recognized obstacles are located in the above cost map.

4. In paragraph 3, The above processor, A robot that increases the cost value of grids where the recognized obstacles are located in the above cost map.

5. In paragraph 1, The above map is a cost map that includes multiple grids to which costs are assigned. The above processor, A robot that increases the cost value of grids where the recognized obstacles are located in the above cost map.

6. In paragraph 1, The above obstacle detection sensor, A robot that includes a LiDAR sensor and a ToF sensor.

7. In paragraph 6, The above processor, A robot that lowers the noise software filtering level of the above LiDAR sensor.

8. In paragraph 1, The above processor, In a situation where the vehicle must drive within a predetermined distance of the recognized obstacle, the size of the obstacle is increased, In other cases, a robot that maintains the above obstacle size.

9. In paragraph 1, The above processor, A robot that changes the destination point from the first destination point to the second destination point if the above obstacle occupies the destination point.

10. In paragraph 9, The above processor, A robot that determines the point farthest from the first destination point among the movable candidate points as the second destination point based on the distribution of obstacle placement adjacent to the first destination point.

11. In paragraph 10, The above movable candidate point is a robot that is a point around the table closest to the first destination point to which the robot can move.

12. In paragraph 1, The above processor, Determine the properties of obstacles forming a narrow path on the above driving route, If the property of the above-mentioned recognized obstacle corresponds to a false detection risk obstacle set in response to the type and sensing method of the obstacle detection sensor, the size of the obstacle set on the map is increased, A robot that regenerates the above driving path.

13. Memory where the map is stored; A processor that generates a driving route to a first destination point set on the above map; A plurality of obstacle detection sensors for detecting obstacles while driving based on the driving path; and A camera sensor for acquiring images during the driving; The above processor, If there is an obstacle occupying the above first destination point, Based on the image acquired from the camera sensor, the properties of the obstacle occupying the first destination point are recognized, A robot that changes the destination point to a second destination point when the property of the above-mentioned recognized obstacle corresponds to a false detection risk obstacle set in response to the type and sensing method of the above-mentioned obstacle detection sensor.

14. In paragraph 13, The above processor, If it is not possible to change the destination to the above second destination, A robot that increases the size of obstacles set on the above map.

15. In paragraph 14, The above map is a cost map that includes a plurality of grids and has costs assigned to the plurality of grids. The above processor, A robot that increases the number of grids in which the recognized obstacles are located in the above cost map.

16. In paragraph 15, The above processor, A robot that increases the cost value of grids where the recognized obstacles are located in the above cost map.

17. In paragraph 14, The above map is a cost map that includes multiple grids to which costs are assigned. The above processor, A robot that increases the cost value of grids where the recognized obstacles are located in the above cost map.

18. In paragraph 13, The above processor, A robot that determines the point farthest from the first destination point among the movable candidate points as the second destination point based on the distribution of obstacle placement adjacent to the first destination point.

19. In paragraph 13, The above obstacle detection sensor includes a LiDAR sensor, The above processor is a robot that lowers the noise software filtering level of the LiDAR sensor.

20. In paragraph 13, The above processor, Determine the properties of obstacles forming a narrow path on the above driving route, If the property of the above-mentioned recognized obstacle corresponds to a false detection risk obstacle set in response to the type and sensing method of the obstacle detection sensor, the size of the obstacle set on the map is increased, A robot that regenerates the above driving path.

Images