Different typed robot system having edge server and cloud server and control method thereof

Abstract

The present invention provides a cloud-based robot system comprising: a plurality of robots disposed in a plurality of arbitrarily divided spaces; a cloud server that generates and distributes a control model applicable to the plurality of robots; and a plurality of edge servers allocated to each space, transmitting and receiving with the cloud server, receiving the control model, and controlling the plurality of robots in the space, wherein the cloud server receives error values ​​in real time from the plurality of robots, generates an upgraded control model based on the error values, and transmits it to the edge servers. Accordingly, the present invention can perform efficient intelligence augmentation by sharing a deep learning model for controlling heterogeneous robots among edge servers. Furthermore, by updating the deep learning model of the local environment in which each robot is deployed, services optimized for each local environment can be augmented.

Description

Technology Field

[0001] The present invention provides a robot system comprising a plurality of edge servers that control heterogeneous robots and a cloud server that performs transmission and reception. Specifically, the present invention relates to a control method for a heterogeneous robot system that utilizes communication between a cloud server providing a base model and a robot. Background Technology

[0002] Robots were developed for industrial use and have played a role in factory automation. Recently, the fields of robot application have expanded further, with the development of medical and aerospace robots, as well as household robots for general use. Among these robots, those capable of driving under their own power are called artificial intelligence robots.

[0003] With the increasing use of robots, there is a growing demand for robots capable of providing diverse information, entertainment, and services beyond the repetitive performance of simple functions.

[0004] Accordingly, various robots are being developed to provide convenience to people by being deployed in homes, restaurants, stores, public places, etc.

[0005] In addition, heterogeneous robot systems are being developed in which different types of robots are deployed in a single defined space to perform their respective tasks.

[0006] Such a heterogeneous robot system can be defined as the convergence of Information and Communication Technologies (ICT) and existing service industries.

[0007] For example, a heterogeneous robot system is defined as a system for processing and communicating regarding various events in the space, based on technologies such as the Internet of Things, Big Data, Cloud Computing, and Cyber-Physical Systems (CPS).

[0008] Cloud computing utilizes internet technology to provide virtualized information and communication technology resources as a service.

[0009] Users can use information and communication technology resources (e.g., servers, storage, networks, and software, etc.) as needed through cloud computing.

[0010] In addition, users can receive real-time service scalability support based on the load of services provided through cloud computing, and pay for the provision of services.

[0011] When a cloud server performs control of numerous heterogeneous robots through cloud computing, an edge server may be additionally included for each space for the convenience of control or communication.

[0012] In this way, a robot system has been developed that includes an edge server controlling multiple robots at close range and a cloud server transmitting and receiving with multiple edge servers.

[0013] Such robot systems generate a very large amount of data traffic between the cloud server, the edge server, and the robot.

[0014] In the case of Korean published patent No. 2019-0119225, a technology is disclosed in which a robot trains its own first artificial intelligence and retrains the first artificial intelligence based on second analysis result data analyzed using a second artificial intelligence of a main management server utilizing a cloud network. However, in the case of machine learning between a main management server and a robot utilizing a cloud network, there is a disadvantage in that knowledge cannot be shared and trained between different robots.

[0015] In addition, in the case of Korean registered patent No. 10-2044665, it is disclosed that the robot system generates instructions for the first robot system to perform the task based on a procedure previously used by the second robot system to perform the task requested by the first robot system.

[0016] However, such a system merely shares information collected and utilized by different robots via the cloud, making it difficult to selectively receive and process information required by specific local environments from heterogeneous local environments.

[0017] Meanwhile, Korean registered patent No. 10-1692602 discloses that a cloud server receives a measurement value corresponding to an information request received from a first robot from a second robot and transmits it to the first robot, or transmits a control signal to the first robot to control the first robot based on the received measurement value.

[0018] However, in such cases, there is a disadvantage in that while information collected / used by different robots is shared via the cloud, it is not possible to selectively receive and augment or train information required by specific local environments from heterogeneous local environments. Prior art literature

[0019] Korean Published Patent No. 2019-0119225, Date of Publication: October 22, 2019 Korean Registered Patent No. 10-2044665, Date of Publication: February 12, 2019 Korean Published Patent No. 10-1692602, Date of Publication: July 21, 2016 The problem to be solved

[0020] The first objective of the present invention is to provide a service for efficient intelligence augmentation by sharing a deep learning model for controlling a plurality of discrete heterogeneous robots between edge servers.

[0021] The second objective of the present invention is to provide a robot system that can augment services optimized for each local environment by upgrading the deep learning model of the local environment in which each robot is deployed.

[0022] In addition, the third objective of the present invention is to provide a robot system capable of monitoring each local environment through each robot on a cloud server to identify local environments requiring new functions, and adding a learning model regarding the new functions to an existing base model. means of solving the problem

[0024] The present invention provides a cloud-based robot system comprising: a plurality of robots disposed in a plurality of arbitrarily divided spaces; a cloud server that generates and distributes a control model applicable to the plurality of robots; and a plurality of edge servers allocated to each space, transmitting and receiving with the cloud server, and receiving the control model to control the plurality of robots in the space, wherein the cloud server receives error values ​​in real time from the plurality of robots and generates an upgraded control model based on the error values ​​and transmits it to the edge servers.

[0025] The above plurality of robots may include different types of robots.

[0026] The above control model may be packaged with respective control models for multiple functions of different types of the robot.

[0027] The edge server receives the control model and can execute the control model by tuning it according to the type of robot controlled by the edge server.

[0028] The cloud server can obtain environment information from the edge server and perform deep learning error learning on the control model to generate the upgraded control model.

[0029] The cloud server can select other edge servers to which the upgraded control model is applied and transmit the upgraded control model to the selected edge servers.

[0030] The cloud server can select the edge server containing the robot to which the upgraded control model is applied and transmit the upgraded control model.

[0031] While executing the control model, the cloud server can perform deep learning error learning to generate the upgraded control model when the number of error values ​​exceeding a threshold from the robot is greater than a predetermined number.

[0032] The cloud server can generate a deep learning model for a new function when the number of error values ​​exceeding a threshold from the robot is greater than a predetermined number and a pattern is detected in the user's response to the error values.

[0033] The cloud server above can classify multiple edge servers into multiple groups and manage them.

[0034] Meanwhile, a cloud-based robot control method for controlling multiple robots placed in multiple arbitrarily divided spaces comprises: a step of generating a control base model applicable to the multiple robots in a cloud server; a step of distributing the control model to an edge server assigned to each space; a step of receiving error values ​​in real time from the multiple robots while controlling the multiple robots with the control model in the edge server; a step of upgrading the control model to a control model for the specific space in accordance with the error value from a specific robot in the specific space in the cloud server; and a step of distributing the upgraded control model to the edge server assigned to the specific space and controlling the specific robot with the upgraded control model.

[0035] The above plurality of robots may include different types of robots.

[0036] The above control model generation step can generate and package respective control models for multiple functions of different types of robots.

[0037] The method may further include the step of receiving the control model from the edge server and tuning the control base model according to the type of robot controlled by the edge server.

[0038] The above-mentioned upgrading step can generate an upgraded control model by obtaining environment information from the edge server on the cloud server and performing deep learning training on the space allocated to the edge server.

[0039] It may include the step of selecting another edge server to which the upgraded control model is applied from the cloud server, and the step of transmitting the upgraded control model to the selected other edge server.

[0040] The step of selecting the other edge server mentioned above may select the edge server that includes the robot to which the cloud server applies the upgraded control model.

[0041] The above upgrade step can generate the upgraded control model by performing the deep learning error learning when the error value exceeding a threshold while running the control model is greater than a predetermined number.

[0042] The cloud server may further include the step of generating a deep learning model for a new function when the number of error values ​​exceeding a threshold from the robot is greater than a predetermined number and a pattern is detected in the user's response to the error values.

[0043] The method may further include the step of classifying and grouping a plurality of the edge servers from the cloud server. Effects of the invention

[0044] Through the above solution, the present invention can perform efficient intelligence augmentation by sharing a deep learning model for controlling discrete heterogeneous robots between edge servers.

[0045] In addition, by updating the deep learning model of the local environment where each robot is deployed, services optimized for each local environment can be provided.

[0046] Specifically, by monitoring each local environment through each robot on a cloud server, local environments requiring new features can be identified, and a learning model for the new features can be added to the existing base model to provide them. Brief explanation of the drawing

[0048] FIG. 1 is a schematic diagram illustrating a cloud and edge-based environment according to one embodiment of the present invention. Figure 2 is a conceptual diagram showing the zone classification of the cloud server of the present invention. FIG. 3 is a detailed diagram illustrating a heterogeneous robot system based on one edge server of the present invention. Figure 4 is a configuration diagram according to an application example of the cloud server of Figure 3. FIG. 5 is a conceptual diagram illustrating the modeling of a cloud, an edge server, and a robot according to an embodiment of the present invention. Figure 6 is an overall flowchart of the robot cloud system of the present invention. Figure 7 is a flowchart showing the cloud server of Figure 6 upgrading the base model for one edge server. FIG. 8 is a flowchart illustrating a method for sharing an upgrade model to another edge server according to an embodiment of the present invention. Specific details for implementing the invention

[0049] The use of terms such as 'first, second,' etc., preceding the components mentioned below is intended solely to avoid confusion regarding the components being referred to, and is unrelated to the order, importance, or master-subordinate relationship between the components. For example, an invention including only the second component without the first component can be implemented.

[0050] In the drawings, the thickness or size of each component is exaggerated, omitted, or depicted schematically for the convenience and clarity of explanation. Additionally, the size and area of ​​each component do not fully reflect their actual size or area.

[0051] In addition, angles and directions mentioned in the process of describing the structure of the present invention are based on those described in the drawings. In the description of the structure in the specification, if the reference point and positional relationship for an angle are not clearly mentioned, the relevant drawings should be referenced.

[0052] FIG. 1 is a schematic diagram for explaining a cloud and edge-based environment according to an embodiment of the present invention, FIG. 2 is a conceptual diagram showing the zone classification of a cloud server of the present invention, and FIG. 3 is a detailed diagram for explaining a heterogeneous robot system based on one edge server of the present invention.

[0053] Referring to FIG. 1, a cloud-based robot system according to one embodiment of the present invention may include a cloud server (10), a plurality of edge servers (20), and a plurality of robots (30).

[0054] The cloud-based robot (30) system manages and controls multiple robots (30) distributed in various spaces spaced apart from each other.

[0055] At this time, each space is functionally or geographically separated, and each edge server (20) is set up to control a plurality of robots (30) placed within each space.

[0056] In each space, robots of the same type (30) may be placed, but, conversely, robots of different types (30) may be placed.

[0057] For example, in space 1 (A1), an edge server 1 (21) may be set up, and one order robot 1 (313) and two guide robots (311, 312) may be placed therein. Meanwhile, in space 2, an edge server 2 (22) may be set up, and two delivery robots (321, 323) and one guide robot (322) may be placed therein. Meanwhile, in space 3, an edge server n (23) may be set up, and an entertaining robot (331) and a guide robot (332) may be placed therein.

[0058] The heterogeneous or homogeneous robots (30) placed in each space transmit and receive with the edge server (20) and cloud server (10) set up in the space, and can provide assigned services by interacting with the user in a home or workplace, etc.

[0059] Preferably, a cloud robot system according to one embodiment of the present invention includes a plurality of heterogeneous robots (30) and each edge server (20) set up in each space, and a cloud server (10) that transmits and receives with each edge server (20) and heterogeneous robot (30) and provides integrated management.

[0060] The edge server (20) can monitor and control the status of multiple robots (30) remotely, so the cloud robot system can provide more effective services by using multiple robots (30).

[0061] A plurality of heterogeneous robots (30), edge servers (20), and cloud servers (10) may communicate with each other by having communication means (not shown) that support one or more communication standards.

[0062] For example, multiple heterogeneous robots (30), edge servers (20), and cloud servers (10) may be implemented to communicate wirelessly using wireless communication technologies such as IEEE 802.11 WLAN, IEEE 802.15 WPAN, UWB, Wi-Fi, Zigbee, Z-wave, Blue-Tooth, etc. The robot (30) may vary depending on the communication method of other devices or cloud servers (10) and edge servers (20) that wish to communicate.

[0063] In particular, multiple robots (30) can implement wireless communication with other robots (30) and / or edge servers (20) and cloud servers (10) via a 5G network. When a robot (30) communicates wirelessly via a 5G network, it can perform ultra-low latency / ultra-high capacity data transmission networks.

[0064] More specifically, 5G networks are communication technologies that provide transmission speeds of tens of Gbps in the wireless section. They enable the transmission of ultra-low latency, ultra-high capacity, and ultra-realistic data at speeds exceeding Gbps and ultra-low latency in the millisecond range across multiple modules to meet service-specific quality requirements. Such 5G networks can provide network quality equivalent to high-speed wired networks while simultaneously offering the advantages of being wireless.

[0065] This 5G network can be controlled to provide various services optimized for each heterogeneous robot (30) and each space between the robot (30) and the edge server (20), between the edge server (20) and the cloud server (10), and between the robot (30) and the cloud server (10) in a cloud machine learning / deep learning-based robot system.

[0066] Additionally, multiple robots (30) and edge servers (20) can communicate using MQTT (Message Queueing Telemetry Transport) and HTTP (HyperText Transfer Protocol), but are not limited thereto.

[0067] In some cases, the multiple robots (30), edge server (20), and cloud server (10) support two or more communication standards, and the optimal communication standard can be used depending on the type of communication data and the type of device participating in the communication.

[0068] The user can check information about the robots (30) within the robot (30) system while transmitting and receiving with the edge server (20).

[0069] In this specification, 'user' refers to a person who uses a service through a plurality of robots (30), and may include individual customers who purchase or rent the robots (30) and use them at a workplace, etc., as well as managers and employees of a company that provides services to employees or customers using the robots (30), and customers who use the services provided by such company. Accordingly, 'user' may include individual customers (Business to Consumer: B2C) and corporate customers (Business to Business: B2B).

[0070] The cloud server (10) designs / creates a model for controlling multiple physically distributed robots (30) and distributes the model to edge servers (20) set up in each space.

[0071] Specifically, the cloud server (10) generates a base model or engine. The base model or engine is a general-purpose model or general-purpose engine applied to the heterogeneous robot (30) in the environment of the heterogeneous robot system, and is not designed as a model specific to a specific situation.

[0072] The cloud server (10) can form and package multiple base models or engines that are functionally or structurally distinct for multiple heterogeneous robots (30), and can perform complex machine learning / deep learning that is difficult to implement in the edge server (20).

[0073] Machine learning is a field of artificial intelligence that refers to the technology of creating models capable of generalizing, classifying, and evaluating vast amounts of data, based on algorithms and techniques that enable computers to learn autonomously.

[0074] By creating such a base model and distributing it to multiple edge servers (20), control can be performed on multiple heterogeneous robots (30) that are physically separated.

[0075] Additionally, for each local environment, that is, each space where a plurality of heterogeneous robots (30) are deployed, the cloud server (10) can receive information about the environment from the edge server (20), receive error information in real time from the robot (30), and perform deep learning to upgrade the base model to be optimized for the environment by comprehensively judging the results.

[0076] The cloud server (10) can control each heterogeneous robot (30) with an optimized model for the space by transmitting the upgraded model to the edge server (20) of the space and executing the upgraded model.

[0077] In addition, the cloud server (10) can group multiple edge servers (20) by zone, and the edge servers (20) grouped in this way can be managed by group.

[0078] At this time, the grouped zones can be configured based on the physical distance or response time from the cloud server (10) to the edge server (20).

[0079] As shown in FIG. 2, a plurality of edge servers (21, 22) existing within a first distance (d1) or a first response time from a cloud server (10) can be classified into a first zone (z1), a plurality of edge servers (23, 24, 25) existing within a second distance (d2) greater than the first distance (d1) or a second response time longer than the first response time can be classified into a second zone (z2), and other edge servers (26, 27) can be classified into a third zone (z3) by a third distance (d3) or a third response time.

[0080] At this time, the distances between the first distance (d1), the second distance (d2), and the third distance (d3) may be the same, but are not limited thereto.

[0081] In addition, the difference between the first and third response times may be the same, but is not limited thereto. In this case, the response time may be defined as the number of network nodes until the packet arrives or the response time obtained via ping, but is not limited thereto.

[0082] In this way, when multiple edge servers (20) are classified by zone, they can be usefully applied to the modeling, distribution, scheduling, etc. of each edge server (20) in the cloud server (10).

[0083] The edge server (20) is set up in each space as shown in FIG. 3, and controls each robot (30) by tuning the base model into an adaptive model suitable for a plurality of heterogeneous or homogeneous robots (30) placed in the space, and tuning the newly received upgraded base model.

[0084] A general edge server (20) is a technology that processes data in the edge area where data is created, deviating from the method of processing data in the cloud server (10). Data requiring real-time processing is processed at the edge server (20), and if necessary, it is implemented in a way that communicates with the central cloud as a secondary task.

[0085] However, in the present invention, a cloud server (10) performs a base deep learning modeling, which is a general-purpose model or engine for heterogeneous robots (30) and services, and an edge server (20) receives the base model, tunes it to suit the space, and executes it, and the cloud server (10) receives error information from the robot (30) in the space, performs an upgrade on the base model, and provides it back to the edge server (20).

[0086] Specifically, the cloud server (10) performs deep learning / machine learning by utilizing learning data of the space previously collected from various multiple robots (30) to customize and upgrade the base deep learning model to suit each robot (30) and service.

[0087] At this time, the cloud server (10) can take into account various environmental information about the space and can be upgraded to an optimal customized deep learning model for each space.

[0088] Additionally, multiple heterogeneous robots (30) transmit feedback (errors, performance information, etc.) to a cloud server (10), and the cloud server (10) can share the upgraded model with other edge servers (20). Accordingly, other edge servers (20) that receive the shared model can remove the existing model, update it with the shared model, and tune and execute it to suit the robots (30) in the space.

[0089] Each edge server (20) is placed in a separate virtual space or a physically separated space (A1) as shown in FIG. 3, and controls a plurality of robots (30) that are placed in the space (A1) and provide services, and the services provided by the robots (30).

[0090] For example, as shown in FIG. 3, an edge server (21) can control a plurality of robots (30: 31, 32, 33, 34, 35) that provide services within a smart restaurant, and the plurality of robots (30) may include a guide robot, an ordering robot, a delivery robot, an entertaining robot, a cleaning robot or a cooking robot, a barista robot, etc.

[0091] In this way, a plurality of heterogeneous robots (30) placed in the space (A1) are dependent on the edge server (21) and provide services by transmitting and receiving data with the edge server (21) and the cloud server (10).

[0092] These cloud servers (10) and edge servers (20) can generally have similar configurations.

[0093] FIG. 4 is a configuration diagram according to one application example of the cloud server (10) of FIG. 3.

[0094] Referring to FIG. 4, the cloud server (10) may include memory, a computing data collection database (110), a control unit (100), and a communication unit (120).

[0095] Specifically, the cloud server (10) or edge server (20) may be implemented as at least one computing device composed of a memory and a processor that reads a program stored in the memory and performs a specific function.

[0096] First, the memory (110) can store an algorithm capable of performing processing for base deep learning in the case of a cloud server (10), and can store and manage data obtained based on IoT technology from multiple homogeneous or heterogeneous robots (30) within each space.

[0097] In the case of the edge server (20), it can store and manage data acquired based on IoT technology from multiple homogeneous or heterogeneous robots (30) in the space, and can store base deep learning models received from the cloud server (10).

[0098] Additionally, the memory (110) can store one or more of the application programs, data, and instructions required for functional operation according to an embodiment of the present invention.

[0099] This memory (110) may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and may also be web storage that performs the storage function of the memory (110) on the internet.

[0100] Additionally, in some embodiments, software components stored in memory (110) may include an operating system, a communication module (or set of instructions), a contact / motion module (or set of instructions), a graphics module (or set of instructions), a text input module (or set of instructions), a satellite positioning system (GPS) module (or set of instructions), and applications (or sets of instructions).

[0101] The processor (100) of the cloud server (10) may be equipped with a more advanced processor capable of performing machine learning / deep learning.

[0102] In the case of the edge server (20), it can control and drive the overall operation of multiple robots (30).

[0103] Such a processor (100) can be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0104] The processor (100) of the cloud server (10) can analyze various data obtained from multiple robots (30) and perform appropriate processing, and can perform machine learning / deep learning on its own to perform learning for upgrading the base deep learning model.

[0105] The cloud server (10) can transmit the upgraded model to the edge server (20) after machine learning / deep learning based on data received from the robot (30), data input by the user, etc.

[0106] The processor (100) of the edge server (20) can analyze the data obtained from a plurality of robots (30) respectively and apply each of the analyzed data to the corresponding robot.

[0107] Meanwhile, the cloud server (10) or edge server (20) may include a communication unit (120), and as previously described, various communication methods are possible depending on the communication method required by the environment. In particular, communication between the robot (30) and the edge server (20), between the edge server (20) and the cloud server (10), between the edge server (20) and the edge server (20), and between the robot (30) and the cloud server (10) is possible using a 5G network, and the algorithm of a large-capacity deep learning modeling itself can be transmitted and received.

[0108] Meanwhile, various heterogeneous or homogeneous robots (30) placed in each space can transmit space, object, and usage-related data to the edge server (20).

[0109] Here, the data may be data related to the recognition of space and object recognized by the robot (30), or image data regarding space and object acquired by the image acquisition unit.

[0110] According to an embodiment, the robot (30) and the cloud server (10) may include an artificial neural network (ANN) in the form of software or hardware that is trained to recognize at least one of the attributes of an object, such as a user, voice, attributes of space, and obstacles.

[0111] According to one embodiment of the present invention, a cloud server (10) may include a deep neural network (DNN), such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Deep Belief Network (DBN), which is trained by deep learning. For example, a deep neural network structure (DNN), such as a Convolutional Neural Network (CNN), may be installed in the processor of the cloud server (10).

[0112] Additionally, usage-related data is data obtained through the use of the robot (30), and may include usage history data, detection signals obtained from the sensor unit (110), etc.

[0113] The robot (30) can receive recognition input data and recognize the attributes of people, objects, and spaces included in the input data, and output the result.

[0114] In addition, the above-mentioned learned deep neural network structure (DNN) receives input data for recognition, analyzes and learns data related to the use of the robot (30), and can recognize usage patterns, usage environments, etc.

[0115] Meanwhile, data related to space, object, and usage can be transmitted to an edge server (20) and / or a cloud server (10) through the communication unit of the robot (30).

[0116] The cloud server (10) can train a deep neural network (DNN) based on the received data, and then transmit the updated deep neural network (DNN) structure data to the corresponding edge server (20) to update it.

[0117] A robot (30) according to one embodiment of the present invention may be a mobile robot (30) that travels through each space and performs a set service or mission.

[0118] Such a robot (30) may include a control unit that controls overall operation, a storage unit that stores various data, and a communication unit that transmits and receives data with other devices such as a cloud server (10) and an edge server (20).

[0119] The control unit can control the communication unit, various sensors, etc. within the robot (30) to control the overall operation of the robot (30).

[0120] The communication unit includes at least one communication module to enable the artificial intelligence robot (30) to be connected to the internet or a predetermined network and to communicate with other devices.

[0121] Additionally, the communication unit can handle data transmission and reception between the robot (30) and the edge server (20) and / or the cloud server (10) by connecting to a communication module provided in the edge server (20) and the cloud server (10).

[0122] A robot (30) according to one embodiment of the present invention may further include a voice input unit that receives voice input from a user through a microphone.

[0123] The voice input unit may include a processing unit that converts analog sound into digital data or is connected to the processing unit, and may convert the user input voice signal into data so that it can be recognized by the control unit or the edge server (20).

[0124] Meanwhile, the control unit can control the robot (30) to perform a predetermined operation based on the voice recognition result.

[0125] Meanwhile, the robot (30) may include various modules that display predetermined information as an image or output it as sound, depending on the type.

[0126] The robot (30) may include a display that displays information corresponding to a command input, a processing result corresponding to a user's command input, an operation mode, an operation state, an error state, etc., as an image.

[0127] According to an embodiment, at least a portion of the display may be configured as a touchscreen by forming a layered structure with a touchpad. In this case, the display configured as a touchscreen may also be used as an input device capable of inputting information by user touch, in addition to being an output device.

[0128] In addition, the sound output unit can output sound, notification messages such as warning sounds, operating modes, operating statuses, and error statuses, information corresponding to user command inputs, and processing results corresponding to user command inputs, according to the control of the control unit. The sound output unit can convert electrical signals from the control unit into audio signals and output them. To this end, a speaker or the like may be provided.

[0129] According to an embodiment, the robot (30) may further include an image acquisition unit capable of capturing a predetermined range.

[0130] The image acquisition unit captures the surroundings of the artificial intelligence robot (30), the external environment, etc., and may include a camera module. Multiple such cameras may be installed in each part to improve shooting efficiency.

[0131] The image acquisition unit can capture an image for user recognition. The control unit can determine the external situation or recognize the user (target of guidance) based on the image captured and acquired by the image acquisition unit.

[0132] In addition, if the robot (30) is an artificial intelligence robot (30), the control unit can control the robot (30) to drive based on the image acquired by the image acquisition unit.

[0133] The robot (30) may further include a driving unit for movement, and the driving unit may move the main body according to the control of the control unit.

[0134] The drive unit may be positioned within the driving unit of the robot (30) and may include at least one drive wheel (not shown) that moves the main body. The drive unit may include a drive motor (not shown) connected to the drive wheel to rotate the drive wheel. The drive wheels may be provided on the left and right sides of the main body, respectively, and are referred to as the left wheel and the right wheel, respectively.

[0135] The left and right wheels may be driven by a single drive motor, but if necessary, a left-wheel drive motor to drive the left wheel and a right-wheel drive motor to drive the right wheel may be provided separately. The driving direction of the main body can be changed to the left or right by creating a difference in the rotational speeds of the left and right wheels.

[0136] Meanwhile, the robot (30) may include a sensor unit that includes sensors for sensing various data related to the operation and state of the robot (30).

[0137] The sensor unit may further include a motion detection sensor that detects the motion of the robot (30) and outputs motion information. For example, a gyro sensor, a wheel sensor, an acceleration sensor, etc., may be used as the motion detection sensor.

[0138] The sensor unit may include an obstacle detection sensor that detects obstacles, and the obstacle detection sensor may include an infrared sensor, an ultrasonic sensor, an RF sensor, a geomagnetic sensor, a PSD (Position Sensitive Device) sensor, a cliff detection sensor that detects whether there is a cliff on the floor within the driving area, a Lidar (light detection and ranging: Lidar), etc.

[0139] Meanwhile, the obstacle detection sensor detects objects, particularly obstacles, that exist in the driving (movement) direction of the artificial intelligence robot (30) and transmits obstacle information to the control unit. At this time, the control unit can control the movement of the robot (30) according to the location of the detected obstacle.

[0140] Meanwhile, the control unit can control the transmission of the operating status of the artificial intelligence robot (30) or user input, etc., to the edge server (20) and cloud server (10) through the communication unit.

[0141] As shown in FIG. 3, such a robot (30) may have a structure in which specific tasks, such as a delivery robot, an order robot, and a guide robot, are assigned to a single space, and a work section may be separately included to suit the task.

[0142] Each robot (30) can be controlled from each edge server (20) according to the assigned mission.

[0143] Hereinafter, the control of a robot (30) system including a heterogeneous robot (30) will be described in detail with reference to FIGS. 5 and FIGS. 6.

[0144] FIG. 5 is a conceptual diagram illustrating the modeling of a cloud and edge server (20) according to one embodiment of the present invention, and FIG. 6 is an overall flowchart of the robot (30) cloud system of the present invention.

[0145] Referring to FIG. 6, a robot (30) cloud system according to one embodiment of the present invention performs model creation / distribution / sharing between a cloud server (10) and a plurality of edge servers (20) that transmit and receive data.

[0146] Specifically, the cloud server (10) creates a base model that can be universally applied to all robots (30) controlled by the cloud server (10), that is, robots (30) that are controlled by the cloud server (10) or connected to store information, or robots (30) connected to the edge server (20) (S10).

[0147] That is, the cloud server (10) performs machine learning / deep learning on the processor (100) to generate a model that can be universally used for controlling multiple heterogeneous robots (30).

[0148] When a heterogeneous robot (30) controlled through an edge server (20) that transmits and receives to a cloud server (10) is a delivery robot, barista robot, ordering robot, cooking robot, guidance robot, etc. as shown in FIG. 5, an object recognition model (101), a navigation model (102), a voice recognition model (103), a location recognition model (104), an emotion recognition model (105), a control intelligence model (106), etc., can be generated as models that can be selectively utilized for each of the robots (30).

[0149] Such models (101-106) can be selectively applied to a specific robot (30), and in the case of a cloud server (10), all required models (101-106) are generated respectively in the concept of a union.

[0150] Each of these deep learning models provides a general-purpose packaged base model to the edge server (20), and optimizes the environment by tuning each edge server (20) according to each space and robot (30).

[0151] Specifically, the cloud server (10) creates a packaged base model and distributes the packaged base model to each registered edge server (20) (S10).

[0152] Meanwhile, the cloud server (10) performs grouping by registering multiple edge servers (20) and classifying them into specific zones (S20).

[0153] A zone can be established based on the distance or response time of each registered edge server (20), and information about the zone assigned to multiple edge servers (20) in the same zone and information about other edge servers (20) registered in the same zone can be transmitted.

[0154] At this time, when a service registration request is received from the new edge server 2 (22) (S21), the cloud server (10) performs zone allocation for the new edge server 2 (22) (S30).

[0155] That is, the physical location of Edge Server 2 (22) is determined through the IP address of the new Edge Server 2 (22), and the response time is measured through Ping or traceroute.

[0156] The cloud server (10) reads the distance from the cloud server (10) based on the response time or physical location and assigns the new edge server (20) to a specific zone (S30).

[0157] The cloud server (10) transmits information about the zone allocation, such as the range of the zone and other edge servers (21, 23, etc.) registered within the zone, to the new edge server 2 (22), and transmits to the other edge servers (21, 23, etc.) of the zone that the new edge server 2 (22) has been registered.

[0158] Next, the cloud server (10) deploys the corresponding packaged base model to the registered edge server 2 (22) (S31).

[0159] At this time, the distributed base model is a package containing a model or module related to various functions capable of managing or controlling various heterogeneous robots (30), and an application for tuning the corresponding base model can also be distributed together when the base model is transmitted.

[0160] When the edge server 2 (22) receives the base model, it tunes the base model to be suitable for a plurality of heterogeneous robots (30) managed by the edge server 2 (22) (S40).

[0161] In this tuning, as shown in FIG. 5, when the edge server 2 (22) controls only the delivery robot (30), the order robot (30), and the guide robot (30), the control intelligence model among the received base models is not activated, and only other models such as the object recognition model (101), the navigation model (102), the voice recognition model (103), the location recognition model (104), and the emotion recognition model (105) are activated.

[0162] In other words, tuning is defined as the process of activating only the models necessary in each area and saving the model that is not needed in an inactive state.

[0163] The edge server (20) can reduce costs and time by reducing computational programs by selecting and activating only the models required for the heterogeneous robots (30) in the zones (A1, A2, A3*?*).

[0164] For example, in the case of edge server 1 (21) of FIG. 5, if a delivery robot, a barista robot, an ordering robot, a cooking robot, and a guide robot are placed within the zone (A1), all models packaged as base models can be activated.

[0165] In cases where only the barista robot, order robot, and cooking robot exist, such as with edge server 3 (23), the robots do not move, so control can be performed without the navigation model (102), location recognition model (104), etc. being activated (S41).

[0166] Each heterogeneous robot (30) is controlled according to a tuned base model and performs assigned tasks while providing services to customers (S50).

[0167] At this time, the heterogeneous robot (30) transmits the input, output, and error values ​​for each operation to the cloud server (10) to perform a report (S51).

[0168] Next, the cloud server (10) can drive the base model tuned through the edge server (20) to control each heterogeneous robot (30), and when certain data is obtained, each base model can be upgraded to suit each robot environment (S60).

[0169] The upgrade of such a base model will be explained in detail later with reference to FIGS. 7 and FIGS. 8.

[0170] When an upgrade to the base model of the edge server (20) is performed on the cloud server (10), the cloud server (10) transmits the upgraded model to the edge server (10) (S61).

[0171] At this time, the edge server (20) that receives the upgraded base model can delete the previous base model, convert and update it to the upgraded base model, and control the robot (30) within the allocated area (S62).

[0172] Thus, the control system of heterogeneous robots (30) using the cloud server (10) of the present invention includes an edge server (20) for controlling robots (30) at a distance, and while each edge server (20) appropriately tunes and uses the base model provided by the cloud server (10), it receives an error report for each robot from the cloud server (10) and individually upgrades the base model for each space to perform machine learning / deep learning.

[0173] In this way, the base model upgraded to be specialized for each environment in the cloud server (10) can be transmitted to the edge server (20) in the corresponding space, and can also be transmitted to and received by other edge servers (20) that require it.

[0174] In this way, by upgrading the base model through the cloud server (10), the machine learning / deep learning algorithm computations used for upgrading the base model are intensively performed on the cloud server (10), thereby reducing the computational burden on individual edge servers.

[0175] Accordingly, individual edge servers (20) perform only the function of a simple processor or controller, thereby reducing the computation time for commands and reception for each robot (30), and reducing the size of the module and computation size of individual edge servers (20), so that the economic burden on each edge server (20) can be reduced.

[0176] In addition, since the base model upgraded directly from the cloud server (10) to another edge server (20) that needs to be upgraded is transmitted and received, the selection and transmission of other edge servers (20) that need to be upgraded can be carried out in a single module, and unnecessary traffic can be reduced.

[0177] Below, the upgrade of the base model in the cloud server (10) and the sharing of the upgraded base model will be explained with reference to FIGS. 7 and FIGS. 8.

[0178] FIG. 7 is a flowchart showing the cloud server of FIG. 6 upgrading the base model, and FIG. 8 is a flowchart showing a method for sharing an upgrade model to another edge server according to an embodiment of the present invention.

[0179] First, referring to FIG. 7, the cloud server (10) sets a threshold value and a minimum number of data for deep learning error learning from each of the plurality of robots (30) for the base model tuned at the edge server (20) (S100).

[0180] Specifically, a threshold value (Eth) and a minimum number of data points (N) can be set.

[0181] The threshold value (Eth) can be defined as a criterion value for selecting values ​​classified as error values ​​among the values ​​received for each base model, and the minimum number of data (N) can be defined as the minimum number of data that allows deep learning model training.

[0182] At this time, the threshold value (Eth) and the minimum number of data points (N) can be set differently for each model.

[0183] Next, while the edge server (20) controls a plurality of heterogeneous robots (30) using each model, the cloud server (10) receives input values ​​transmitted by the edge server (20) from each robot (30), output values ​​and error values ​​(E) output from each robot (30) respectively (S101).

[0184] Such input values, output values, and error values ​​(E) can be received for each service or each function's progress and for each event's occurrence.

[0185] The cloud server (10) stores the received data along with the deep learning model and training data in correspondence with each local environment, i.e., the edge server (20).

[0186] Next, the cloud server (10) monitors the characteristics of each local environment to determine whether the deep learning base model needs to be upgraded.

[0187] The cloud server (10) analyzes the learning completeness of the deep learning model in the local environment for each edge server (20), user reactions, and patterns of the robot (30) and user to determine whether to upgrade the current deep learning model or create a model with new functions.

[0188] This judgment can be determined by analyzing the pattern of the user's reaction to the output of the robot (30) (S102).

[0189] That is, if no pattern is formed in the user's response, it can be determined as a general error situation, and if the error value (E) is greater than or equal to the threshold value (Eth) and the number of data points (N) is greater than or equal to the minimum number of data points, it can be determined as a situation to upgrade the current deep learning model (S103).

[0190] That is, if the error value (E) is greater than or equal to the threshold (Eth), it is determined to be true, stored as training data, and the number of data points (n=n+1) is counted.

[0191] In this way, while determining whether the error value (E) for the data is greater than the threshold (Eth) in succession, if the counted number of data does not satisfy the minimum number of data (N), the next data is read and the true / false of the error value (E) is determined again, and if the number of data with an error value greater than the threshold (Eth) satisfies the minimum number of data (N), it is determined that the deep learning model can be trained and an upgrade is performed through deep learning training of the model (S104).

[0192] At this time, the cloud server (10) can be upgraded by changing the number of nodes, the number of layers, and the activation function values ​​along with the added error values ​​(E), and at this time, the activation function can utilize multiple candidate sets stored in advance.

[0193] For example, when applying the speech recognition model, if the error value (E) exceeds the minimum data value, the program can be programmed to minimize the error by applying a different activation function accordingly.

[0194] The upgraded model is transmitted to the edge server (20) as version 2 (V2), and the edge server (20) can use it as a replacement for the previous version, version 1 (V1) (S105).

[0195] Meanwhile, the cloud server (10) receives information about the local environment of each space from the edge server (20) and combines this with data from the robot to determine whether a new function is needed for the local environment (S102).

[0196] The cloud server (10) analyzes the learning completeness of the deep learning model in the local environment for each edge server (20), user reactions, and patterns of the robot (30) and user to determine whether to create a new model with different functions from the current deep learning model.

[0197] This judgment can be determined by analyzing the pattern of the user's reaction to the output of the robot (30).

[0198] In other words, if a pattern is detected in the user's response and it is determined that such a pattern of response is intentional, it is determined that there is a need for a new feature.

[0199] Accordingly, the cloud server can perform deep learning / machine learning to generate a deep learning base model capable of controlling new functions for new response patterns.

[0200] This new deep learning model is packaged together with other models being upgraded and transmitted to the edge server (20).

[0201] The new model transmitted in this way can be stored in the memory (101) of the edge server (20) as version 2 (V2), and can be used to replace the previous version, version 1 (V1).

[0202] When a change is made to the upgraded model, the edge server (20) transmits the details of such change to the cloud server (10) and controls the heterogeneous robot (30) with the upgraded model.

[0203] In this way, by performing modeling suitable for each environment using environmental information from the edge server (20) and real-time data from the robot on the cloud server, the most optimized version of the modeling can be produced.

[0204] In addition, by receiving environmental information from edge servers as well as robots and analyzing user responses to perform modeling, it is possible to upgrade the model by reflecting pattern analysis of user responses.

[0205] In addition, since each activation function and change option is already stored, upgrading is possible by modifying them, allowing for a relatively simple upgrade.

[0206] In this way, the upgraded base model for a specific environment in which a specific edge server (20) is located can be shared with other edge servers (20) that control similar robots (30).

[0207] Specifically, referring to FIG. 8, the cloud server (10) creates and distributes a base model that can be universally applied to all robots (30) controlled by the cloud server (10), that is, robots (30) that are controlled by the cloud server (10) or connected to store information, or robots (30) connected to the edge server (20) (S200).

[0208] That is, the cloud server (10) performs machine learning / deep learning on the processor (100) to generate a model that can be universally used for controlling multiple heterogeneous robots (30).

[0209] Each of these deep learning models provides a general-purpose packaged base model to the edge server (20), and each edge server (20) optimizes the environment by tuning according to each space and robot (30) (S201).

[0210] When Edge Server A (21) receives the base model, it tunes the base model to be suitable for multiple heterogeneous robots 1 (30) managed by Edge Server A (21).

[0211] Each edge server (20) controls the heterogeneous robot (20) in the corresponding space, that is, the allocated space where multiple heterogeneous robots are placed, using a tuned base model, and each robot 1 (30) provides an output to the user to perform a task set according to the input from the edge server (S202).

[0212] Each robot (30) transmits the error value that occurs while performing the set task to the cloud server in real time along with the input and output (S203).

[0213] The cloud server (10) obtains data from error reporting from each of the robots (30) and environmental information from the edge server (20), and combines them to perform an upgrade of the deep learning model for the corresponding edge server A (21).

[0214] Therefore, an upgraded deep learning model is created for the environment, that is, the space where the edge server A (21) is allocated (S204).

[0215] When a model upgrade occurs for edge server A (21), the cloud server (10) sends the deep learning model of the upgraded version to edge server A (21) (S205).

[0216] Edge server A (21) changes to the upgraded model to control the heterogeneous robot (S206), and the heterogeneous robot (30) performs the task according to the control by the new model and sends the error report to the cloud server (10) in the same way.

[0217] Meanwhile, when an upgrade model for edge server A (21) is created, the cloud server (10) performs a selection of other edge servers (22) that require a model upgrade to prevent such errors, as many errors are expected to occur in the model and similar errors are expected to occur in other edge servers (22) (S211).

[0218] That is, the cloud server (10) selects other edge servers (22) that have a similar environment based on the upgraded model, the upgraded changes and the related robot (30) model name, etc.

[0219] That is, the cloud server (10) can select other edge servers (22) that control the same robot (30), and can also select edge servers (22) that mainly utilize the upgraded model.

[0220] At this time, the cloud server (10) can select an edge server (20) that has a space similar to the current edge server (20), that is, an edge server (20) that has similar visual characteristics, but is not limited thereto.

[0221] Additionally, the cloud server (10) may select only the edge servers (22) within the zone to which the upgraded edge server A (21) belongs by referring to the zone-specific relevance of each edge server (20), but is not limited thereto.

[0222] In this way, the cloud server (10) selects the edge server (20) that requires it and sends an update preparation request to the corresponding edge server (22) (S211).

[0223] Each selected edge server (22) is notified that an upgrade of the corresponding base model has occurred, and each selected edge server (22) prepares to receive the upgraded base model by means of such notification from the cloud server (10).

[0224] At this time, the cloud server (10) may notify the selected edge servers (22) of the deep learning model change and the related robot (30) model name, and request them to prepare for an update to the upgraded base model.

[0225] Selected edge servers (22) move the currently stored deep learning base model to temp and store it temporarily (S212).

[0226] Next, the selected edge servers (22) reply to the cloud server (10) that they are ready and send an update request (S213).

[0227] The cloud server (10) distributes the upgraded deep learning model to each selected edge server (22) (S214).

[0228] When a selected edge server (22) receives an upgraded deep learning model, it stores the model and performs a trial run.

[0229] If there are no errors in the execution of the upgraded deep learning model, the temp is discarded to delete the previous deep learning base model (S215).

[0230] With this operation, the version of the updated model is notified to the cloud server (10) as version 2 (V2) (S216), and the robot (30) is controlled by performing an update of the deep learning base model (S217).

[0231] The edge server (22) selected in this way can each execute the base model received from the cloud server (10) by tuning it, and when upgrade information and selection information are received from the cloud server (10), it can receive the upgraded model, update the tuned base model to the next version, and control the robots (30) in the area with the upgraded version.

[0232] In addition, the cloud server (0) can upgrade the model by deep learning to the next version based on counting a predetermined number of data and determining an error, as shown in Fig. 7, even when controlling the upgraded version, and this can also be shared with the edge server (20).

[0233] An embodiment of the present invention can perform an upgrade to a model optimized for a specific environment while controlling an edge server (20) and a robot (30) from a cloud server (10).

[0234] In addition, by sharing the upgraded model between two or more edge servers (20), the evolved model can be shared and continuously upgraded as it evolves in each other's environments.

[0235] The robot system according to the present invention is not limited to the configuration and method of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to be made.

[0236] Meanwhile, the control method of a robot system according to an embodiment of the present invention can be implemented as processor-readable code on a processor-readable recording medium. A processor-readable recording medium includes all types of recording devices in which data that can be read by a processor is stored. Examples of processor-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include implementation in the form of a carrier wave, such as transmission via the Internet. Furthermore, the processor-readable recording medium may be distributed across networked computer systems, so that processor-readable code can be stored and executed in a distributed manner.

[0237] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0238] 10: Cloud Server 20: Edge Server 30: Robot

Claims

Claim 1 A cloud-based robot system comprising: a plurality of robots disposed in a plurality of arbitrarily divided spaces; a cloud server that generates and distributes a base control model applicable to the plurality of robots; and a plurality of edge servers assigned to each space, transmitting and receiving with the cloud server, receiving the base control model, and controlling the plurality of robots in the space, wherein the plurality of robots include robots of different types, and the edge servers receive the base control model, tune and execute the base control model according to the type of robot being controlled, and receive error values ​​in real time from the plurality of robots, and the cloud server obtains the error values ​​and environmental information from the edge servers, performs deep learning error learning for a specific space assigned to the edge servers, generates an upgraded control model based on the error values ​​and transmits it to the edge servers, and selects other edge servers to which the upgraded control model is applied and transmits the upgraded control model to the selected other edge servers. Claim 2 delete Claim 3 A cloud-based robot system according to claim 1, characterized in that the base control model has a respective control model for a plurality of functions of different types of robots packaged therein. Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 A cloud-based robot system according to claim 1, characterized in that the cloud server selects the edge server containing the robot to which the upgraded control model is applied and transmits the upgraded control model. Claim 8 A cloud-based robot system according to claim 7, characterized in that the cloud server, while executing the base control model, performs deep learning error learning to generate the upgraded control model when the error value exceeding a threshold from the robot is greater than or equal to a predetermined number. Claim 9 A cloud-based robot system according to claim 7, characterized in that the cloud server generates a deep learning model for a new function when the number of error values ​​exceeding a threshold from the robot is greater than a predetermined number and a pattern is detected in the user's response to the error values. Claim 10 delete Claim 11 A cloud-based robot control method for controlling multiple robots placed in multiple arbitrarily divided spaces, comprising: generating a base control model applicable to the multiple robots on a cloud server; distributing the base control model to an edge server allocated to each space; receiving error values ​​in real time from the multiple robots while controlling the multiple robots using the base control model on the edge server; and upgrading the control model to a control model for the specific space on the cloud server according to an error value from a specific robot in the specific space. A cloud-based robot control method comprising the steps of: distributing the upgraded control model to the edge server allocated to the specific space; controlling the specific robot with the upgraded control model; wherein the plurality of robots includes robots of different types; and the upgrading step further comprises the steps of: obtaining environment information from the edge server at the cloud server and generating an upgraded control model by performing deep learning error learning on the space allocated to the edge server; receiving the base control model at the edge server and tuning and executing the base control model according to the type of robot being controlled; selecting another edge server to which the upgraded control model is applied at the cloud server; and transmitting the upgraded control model to the selected other edge server. Claim 12 delete Claim 13 A cloud-based robot control method according to claim 11, wherein the step of generating the base control model comprises generating and packaging respective control models for multiple functions of different types of robots. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 A cloud-based robot control method according to claim 11, wherein the step of selecting the other edge server is characterized by selecting the edge server to which the cloud server includes a robot to which the upgraded control model is applied. Claim 18 A cloud-based robot control method according to claim 17, wherein the upgrading step is characterized by performing deep learning error learning to generate the upgraded control model when the error value exceeding a threshold while executing the base control model is greater than or equal to a predetermined number. Claim 19 A cloud-based robot control method according to claim 17, characterized in that the cloud server further includes the step of generating a deep learning model for a new function when the error value exceeding a threshold from the robot is greater than or equal to a predetermined number and a pattern is detected in the user's response to the error value. Claim 20 delete

Images